Global Approaches to Auditing Artificial Intelligence

The report analyzes the fragmented ecosystem of AI audits, distinguishing between organizational conformity assessments and academic performance evaluations. It details the roles of internal, external, and community auditors, the methodologies used—such as input variation and benchmarking—and the current state of enforceable regulations in the US, Canada, EU, UK, and China. The document argues for a more diverse, trustworthy audit ecosystem with better data access and a focus on global impacts, including environmental and labor concerns, to move beyond 'null compliance'.

• AI audits are categorized into two primary orientations: organizational/institutional audits, which focus on risk and conformity to policies, and academic audits, which typically test for direct or indirect societal harms such as bias and discrimination.
• The report identifies a limited number of jurisdictions with enforceable AI audit regulations: the United States, Canada, the European Union, the United Kingdom, and China.
• The EU AI Act introduces conformity assessments and post-market monitoring for high-risk AI systems, while New York City's Local Law 144 mandates bias audits for automated employment decision tools (AEDTs) using a specific 'impact ratio'.
• Auditors are classified by their relationship to the audited entity: internal auditors (company employees), second-party auditors (contracted by developers), external/independent auditors (academics or journalists), and community/user auditors.
• Audit methodologies vary by access level (black-box vs. white-box) and include input variation, benchmarking, automated comparison using other ML models, and the use of audit artifacts like model cards and data sheets.
• The report warns of 'null compliance' and 'audit-washing', where internal or second-party audits are used to demonstrate adherence to norms without verifiable results or independent scrutiny, specifically citing the lack of public disclosure for NYC Local Law 144 audits.
• Current AI audit regimes are criticized for being concentrated in the 'global north' and failing to account for global supply chain impacts, such as the working conditions of data annotators and the massive water and energy consumption of AI systems.
• Comprehensive auditability is hindered by a lack of access to data and model components; the report argues that black-box access is insufficient for rigorous auditing.

APA

Sylvester, P., Chun, W. H. K., Hannák, A., Mendoza, M., Broussard, M., Enikolopov, R., Nelson, A., Sandvig, C., Sesan, G., Sporle, A., Srihari, R., Srinivasan, J., Wilson, C., Zou, M., Valenzuela, S., & Howard, P. (2024). Global Approaches to Auditing Artificial Intelligence. International Panel on the Information Environment. https://www.ipie.info/research/global-approaches-to-auditing-artificial-intelligence

Chicago

Sylvester, Prem, Wendy Hui Kyong Chun, Anikó Hannák, Marcelo Mendoza, Meredith Broussard, Ruben Enikolopov, Alondra Nelson, et al. Global Approaches to Auditing Artificial Intelligence. International Panel on the Information Environment, 2024. https://www.ipie.info/research/global-approaches-to-auditing-artificial-intelligence.

Wikipedia

{{cite report |last1=Sylvester |first1=Prem |last2=Chun |first2=Wendy Hui Kyong |last3=Hannák |first3=Anikó |last4=Mendoza |first4=Marcelo |last5=Broussard |first5=Meredith |last6=Enikolopov |first6=Ruben |last7=Nelson |first7=Alondra |last8=Sandvig |first8=Christian |last9=Sesan |first9=Gbenga |display-authors=etal |title=Global Approaches to Auditing Artificial Intelligence |publisher=International Panel on the Information Environment |date=2024 |url=https://www.ipie.info/research/global-approaches-to-auditing-artificial-intelligence |access-date=23 October 2026 |via=Climate Insights Directory}}

BibTeX

@techreport{sylvester2024global, author = {Sylvester, Prem and Chun, Wendy Hui Kyong and Hannák, Anikó and Mendoza, Marcelo and Broussard, Meredith and Enikolopov, Ruben and Nelson, Alondra and Sandvig, Christian and Sesan, Gbenga and Sporle, Andrew and Srihari, Rohini and Srinivasan, Janaki and Wilson, Christo and Zou, Mimi and Valenzuela, Sebastián and Howard, Philip}, title = {{Global Approaches to Auditing Artificial Intelligence}}, institution = {International Panel on the Information Environment}, year = {2024}, url = {https://www.ipie.info/research/global-approaches-to-auditing-artificial-intelligence}, urldate = {2026-10-23}, note = {Indexed by Climate Insights Directory} }