Choosing an AI-enabled research tool is rarely a decision made by one person alone. A researcher selecting a platform for a review, a librarian recommending a platform, a methodologist defending a protocol, a research lead preparing a procurement request, or a research office evaluating institutional investment will all eventually face the same underlying question: Can you justify this choice?
That question takes different forms. A procurement team asks whether the tool meets security and governance requirements. A peer reviewer asks whether its use can be reported and reproduced. A research office or faculty leader asks whether the platform supports institutional research capability and represents a worthwhile investment. Different audiences, but the same underlying concern: is the decision transparent, defensible and supported by evidence?
Here are the four questions you’re likely to be asked, and the evidence you can use to answer them.
Questions every researcher should ask
Is it methodologically defensible?
A trustworthy AI tool should explain why each automated task is appropriate given the consequences of getting it wrong. Look for a framework that considers task risk, the level of human oversight required and the model performance needed for each use case, rather than relying on headline accuracy figures alone. The tool should also be transparent about what it does not automate, the limitations of its models, and the contexts in which performance may differ (for example, when a model has only been evaluated on English-language studies).
Covidence uses a decision matrix to evaluate every AI-assisted feature before release. Each feature is assessed according to the potential impact of errors, the model performance required and the level of human oversight appropriate for the task. Features that do not meet the required standard are not released. Each feature is also accompanied by documentation describing its intended use, evaluation methodology, known limitations and appropriate contexts for use, helping reviewers determine whether the feature is suitable for their review. Reviewers retain responsibility for all review decisions.
Example tool assessment language: “We selected Covidence because its AI-assisted features are governed by a decision matrix that assesses task risk, required model performance and the appropriate level of human oversight before features are released. The platform supports the review workflow while preserving reviewer responsibility for all review decisions.”
Researchers should be able to clearly see what an AI tool did during the review, and readers should be able to understand and evaluate that use from the reported methods. Look for automation that is visible, reversible and well documented, with clear guidance on how to report its use in your protocol and manuscript.
Is it transparent and reportable?
In Covidence, automated actions remain visible and reversible throughout the review. PRISMA flow diagrams update automatically, and each AI-assisted feature is supported by documentation describing its intended use, evaluation methodology, performance, known limitations and recommended reporting language, aligned with the RAISE guidelines.
Example reporting language: “We used Covidence’s ‘Remove references reporting on non-RCTs before screening’ feature before title and abstract screening. The feature uses the Cochrane RCT Classifier, which has demonstrated greater than 99.5% recall for identifying health-related RCTs (Thomas et al., 2021). All automatically removed references remained available for reviewer inspection and could be restored to manual screening at any time. Use of the feature was documented in the PRISMA flow diagram and reported in accordance with the RAISE guidelines.”
Questions institutional decision-makers will ask
Is it safe to use on our data?
Institutions increasingly expect clear answers about how their data is handled. Ask directly:
- Is our data used to train AI models?
- Where is our data stored?
- What independent security certifications does the platform hold?
These answers should be easy to find and supported by independent evidence. SOC 2 Type II and ISO 27001 certification are widely recognised indicators that security controls have been independently assessed.
Example procurement or ethics submission language: “Covidence is currently SOC 2 Type I certified. External audits for SOC 2 Type II and ISO 27001 are underway, with certification expected in late 2026. Documents uploaded to the platform are not stored or shared by LLM providers and are not used to train AI models. Data handling is governed by Covidence’s privacy policy and customer data agreements.”
Is the organisation mission-aligned?
Choosing an AI-enabled research tool also means choosing the organisation behind it. Alongside evaluating the technology itself, consider whether the organisation’s mission, governance and incentives are aligned with supporting rigorous, reproducible research. These influence how AI features are prioritised, how risks are managed and how transparently limitations are communicated.
Ask questions such as:
- What is the organisation’s mission?
- Does it publish evidence of limitations as well as performance?
- Are its incentives aligned with supporting rigorous, reproducible research?
- Is its governance transparent?
Covidence is developed by the not-for-profit Future Evidence Foundation, whose mission is to strengthen evidence synthesis. That mission shapes our approach to automation. We introduce AI where there is evidence it improves review quality or efficiency, while preserving reviewer oversight for consequential scientific decisions.
Example procurement language: “In evaluating AI-enabled research tools, we considered not only technical capability but also organisational governance, transparency and alignment with responsible research practices.”
Is it worth it?
Efficiency claims are only meaningful when you understand how they were measured. “50% faster” or “95% accurate” says little without knowing the task, the evaluation conditions or whether those results apply to your research context.
The strongest business case comes from your own institution’s evidence rather than vendor benchmarks. Look for reporting that helps you understand research activity, researcher engagement, adoption of AI-assisted features and, where appropriate, workflow efficiency. This allows institutional decision-makers to justify investment using their own data, connecting platform adoption to research productivity, researcher capacity and responsible AI governance. Because every automation feature is optional and reversible, institutions can adopt AI incrementally and choose the level of automation appropriate for different review types and risk profiles.
Example business case language: “Based on 12 months of institutional usage, Covidence supported 144 active reviews across our institution. The Research Insights Dashboard shows review activity, researcher engagement and workflow progress across teams. Together with workflow automation that reduces manual effort, including automatic open access full-text retrieval and optional AI-assisted features, this provides evidence of institutional value to support procurement, renewal and governance decisions.”
Justifying AI means demonstrating good governance
AI tools should not be judged by how impressive their demonstrations are, but by how well they withstand scrutiny.
If you can explain what the tool does, why each automation is appropriate, how its performance has been evaluated, how its outputs can be audited, how institutional data is protected, and why you trust the organisation behind it, you are no longer relying on marketing claims. You are making an evidence-based case.
That is exactly what evidence synthesis is built on. Ultimately, the standard for AI-enabled research tools should be the same as the standard for research itself: transparent, well documented and defensible.
This is the sixth article in Covidence’s six-part series on AI:
- Part 1: Covidence’s approach to responsible automation (AI)
- Part 2: How Covidence decides what AI to release (and what to hold back)
- Part 3: Beyond evaluation: deciding when AI is appropriate in evidence synthesis
- Part 4: How we evaluate AI models for evidence synthesis
- Part 5: AI won’t replace methods skills, it depends on them
- Part 6: How to justify AI tools to institutions
How do you feel about Covidence’s approach to automation (AI)? Share your perspective
