can-ai-produce-reliable-consistent-data-analysis-c33bcc687a0ba70b.pdf

This briefing by the Stockholm Environment Institute (SEI) details a pilot project evaluating the use of the SEI AI Reader, a tool based on large language models, to conduct systematic policy analysis. The project tested the tool's ability to extract data from policy evaluations and audits using a framework of 12 independent variables and 46 questions, focusing on climate policy implementation. The results indicate that with iterative prompt calibration, the AI can achieve high levels of accuracy and consistency, making it a viable method for processing large volumes of grey literature to identify drivers of successful policy outcomes.

• The SEI AI Reader achieved an average accuracy of approximately 85% when compared to manual human analysis across four test documents. The researchers noted that the tool did not exhibit hallucination of facts or wrongful answers, which they attribute to prompt design safeguards such as requiring direct quotes and page references.
• Consistency of the AI's outputs across the four documents ranged from 69% to 90%. While most runs produced similar results, occasional discrepancies occurred, leading the researchers to suggest running each document twice in the full dataset analysis to mitigate outliers.
• Initial attempts using broad exploratory questions were ineffective, often resulting in generic or omitted responses. The researchers found that the most effective results were achieved by combining a broad but structured query with a specific, analytical question, and by using a 'context variable' to provide guidelines and correct previous inconsistencies.
• The pilot suggests that AI-generated data is sufficiently reliable and consistent to meet academic scrutiny, offering a scalable solution for synthesizing vast amounts of climate policy evaluations and grey literature to identify patterns of effective implementation.

APA

Stockholm Environment Institute (n.d.). can-ai-produce-reliable-consistent-data-analysis-c33bcc687a0ba70b.pdf. https://www.sei.org/wp-content/uploads/pdfs/201162/can-ai-produce-reliable-consistent-data-analysis.pdf

Chicago

Stockholm Environment Institute. can-ai-produce-reliable-consistent-data-analysis-c33bcc687a0ba70b.pdf. n.d. https://www.sei.org/wp-content/uploads/pdfs/201162/can-ai-produce-reliable-consistent-data-analysis.pdf.

Wikipedia

{{cite report |author=Stockholm Environment Institute |title=can-ai-produce-reliable-consistent-data-analysis-c33bcc687a0ba70b.pdf |url=https://www.sei.org/wp-content/uploads/pdfs/201162/can-ai-produce-reliable-consistent-data-analysis.pdf |access-date=17 August 2026 |via=Climate Insights Directory}}

BibTeX

@techreport{stockholmenvironmentinstitutendcanaiproducereliableconsistentdataanalysisc33bcc687a0ba70bpdf, author = {{Stockholm Environment Institute}}, title = {{can-ai-produce-reliable-consistent-data-analysis-c33bcc687a0ba70b.pdf}}, institution = {Stockholm Environment Institute}, url = {https://www.sei.org/wp-content/uploads/pdfs/201162/can-ai-produce-reliable-consistent-data-analysis.pdf}, urldate = {2026-08-17}, note = {Indexed by Climate Insights Directory} }