Can AI produce reliable and consistent data analysis?

The Stockholm Environment Institute (SEI) conducted a pilot study to evaluate the SEI AI Reader, an LLM-based tool designed to extract data from policy documents. By testing the tool against human analysis of four documents using an analytical framework of 12 variables and 46 questions, researchers found the AI achieved approximately 85% accuracy and consistency between 69% and 90%. The study highlights that iterative prompt refinement—specifically combining broad structured queries with specific analytical questions and using 'context variables'—is essential to avoid generic responses. The authors conclude that AI-generated data is sufficiently high-quality for academic scrutiny, offering a scalable way to synthesize grey literature on climate policy implementation.

• The SEI AI Reader achieved an average accuracy of approximately 85% when compared to manual human assessment across four test documents.
• Consistency of the tool's outputs across the four analyzed documents ranged from 69% to 90%, although occasional discrepancies occurred where one run would differ from others.
• Initial broad exploratory questions were ineffective, often resulting in generic or omitted responses; however, a combination of a broad but structured query with a specific analytical question proved most effective.
• The researchers did not observe the fabrication of facts or 'hallucinations', attributing this to prompt safeguards such as requiring direct quotes and page references.

APA

Kappe, K., & Dzebo, A. (2025). Can AI produce reliable and consistent data analysis? Stockholm Environment Institute. https://www.sei.org/wp-content/uploads/pdfs/201162/can-ai-produce-reliable-consistent-data-analysis.pdf

Chicago

Kappe, Kira, and Adis Dzebo. Can AI produce reliable and consistent data analysis? Stockholm Environment Institute, 2025. https://www.sei.org/wp-content/uploads/pdfs/201162/can-ai-produce-reliable-consistent-data-analysis.pdf.

Wikipedia

{{cite report |last1=Kappe |first1=Kira |last2=Dzebo |first2=Adis |title=Can AI produce reliable and consistent data analysis? |publisher=Stockholm Environment Institute |date=August 2025 |url=https://www.sei.org/wp-content/uploads/pdfs/201162/can-ai-produce-reliable-consistent-data-analysis.pdf |access-date=17 August 2026 |via=Climate Insights Directory}}

BibTeX

@techreport{kappe2025can, author = {Kappe, Kira and Dzebo, Adis}, title = {{Can AI produce reliable and consistent data analysis?}}, institution = {Stockholm Environment Institute}, year = {2025}, month = aug, url = {https://www.sei.org/wp-content/uploads/pdfs/201162/can-ai-produce-reliable-consistent-data-analysis.pdf}, urldate = {2026-08-17}, note = {Indexed by Climate Insights Directory} }