It is often hard to decide what which organizations and insights are “in and out” of a conversation about climate communication and science . This has historically been solved by “Are they a government? Do they have academic credentials? Is what is said consistent with the findings of the IPCC?”
While these kinds of appeal to authority are important for information integrity, as has been well documented over the last 5 years by the Climate Action Against Disinformation Coalition, having “trustworthy” science is not enough.
Instead, climate science based policy implementations happens because a network of different stakeholders, including business, civil society, media and governments of different sizes, communicate with each other and explain how the science goes from global evidence to local reality. Decision makers need answers to questions like: How does scientific evidence enter local policy debates? What is the overall “perspective” needed to see the value of that evidence? How do the nuances of our local context change the meaning of that data?
That is often where civil society and other grey literature comes in. Our editorial strategy focuses on finding evidence based civil society communication that bridges climate science and policy discussions in as many contexts as possible. And I wanted to walk you through the process of choosing which organizations and documents to index.
In order to go from “we think this might be a good publisher to include” to “reliable documents we think should be in the corpus” we have a three step process: picking organizations, picking documents, and making sure the documents are well described.
What civil society organizations are writing about climate? Should we include them?
The first step in our curation process has been identifying organizations that might be good candidates for including in the corpus.
For every 1 organization that is going into our collection (at time of writing we have identified ~400, and we estimate that in our current scope will include at least 1000), we review at least 2 other organizations publishing about topics parallel to climate. We identify candidates between a mix of reliable sources (i.e. citations on Wikipedia and Wikidata or in research publications), snowballing from our existing collection (i.e. links that are present in websites we have already collected from), referrals from subject matter experts in specific topics and general knowledge searches on multiple other digital platforms.
After we have identified a candidate, each website is manually reviewed for the following:
- Is the organization part of civil society? We are including publications from a range of different kinds of organizations, including non-profits, volunteer organizations, think tanks and others.
- Are they publishing evidence based documents?
- Some organizations and websites are mostly focused on campaigning, or single issue policy efforts. We only want organizations producing research.
- Do they publish significant collections on climate change?
- If an organization is explicitly mission-focused on climate, we include their whole collection of documents, even if some of the documents are on the edge of climate communications.
- For some organizations we only include part of their collection. We screen organizations focused on another sector, like nature/conservation or human and economic rights, for either a publication “Tag” on their website or Program line explicitly focused on climate or energy.
- Do we have any evidence of disinformation or funding from polluting industries?
- Our team has long paid attention to the research coming from the CAAD network, and uses research networks like, like DeSmog, to identify active misinformation organizations.
- We also check “About us” sections of a website, to make sure we understand what kind of funding applies to the organization. We have identified more than 10 websites that on the surface looked like they met the first two criteria, but on reviewing their funding and supporters indicated they didn’t qualify.
Other things we are trying to exclude:
- Collections of documents created by coalitions that don’t include new publications unique to that publisher. In developing the project, we found that most of these collections reproduced UN Agency and large Civil Society reports.
- Websites built for campaigning, news, or single policy initiatives. Our focus is on the harder to find “backfile” of reports and evidence created by research focused organizations that could contribute to a long-term research conversation.
- Any websites with access restrictions (i.e. pay or registration for access research, or bot access restrictions).
So far, the criteria seem to allow us to find a quite wide range of organizations. We are also trying to make the process as robust as possible while sharing how we make decisions here on the Updates. This is the first in a series of Updates that will explain different parts of our curatorial process.
We are excited to see how this approach expands our understanding of which insights matters for climate communication: the smallest organizations in our pipeline have as few as 4-5 reports on their website, while the snowballing process has helped us find a number of organizations that weren’t “obvious” from public knowledge platforms and generic research search tools. We also have found that the snowballing reveals a number of organizations and documents (especially off-site PDFs) that external web searches don’t seem to “see”.
How do you choose which documents to include?
Most organizations have a mix of both organizational communications and research based publications on their website. We only want to index those documents meant to provide analysis or insight into climate action.
This means that we have to make sure each document is one of the content types we want to include. Our goal is to find evergreen expertise and evidence based research and statements that show the conversation happening by civil society. To do this we decided to include the following kinds of documents:
- Reports, research papers and other kinds of articles written by the organization themselves, including complex analysis (for example this country focused report from the African Climate Foundation)
- Briefings on specific policies or topics that are shorter, and are not as extensive as reports (such as this briefing by Zero Carbon Analytics about the economic opportunity created by offshore wind)
- Evidence based statements typically compiled with coalitions of organizations (such as this statement by the African Youth Climate Movement found on the South African Institute of International Affairs)
- Guides that help decision makers or implementers evaluate a problem and act on it (such as this Guide by Climate Policy Initiative on evaluating Adaptation Finance)
- Case studies that are based on some kind of evidence based documentation (such as this case study by Green Cape about solar installations)
Examples of documents we are excluding include:
- Organizational content, such as updates on blogs, events, profiles of staff, or calls for proposals.
- News type content—i.e. reactions to events or responses or statements made in reaction to specific news articles
- Content without a lot of text that is “readable” (such as comics, presentations and podcast summaries)
In our test batch of the ~200,000 document pages from ~50 organizations, we identified 30-40k candidate documents from organizations that meet our inclusion criteria, which means we are excluding over 160,000 public pieces of web content. As of now, we were conservative in the records that we’ve made available, only publishing records for 20,000 documents so that we could improve quality control before processing the rest.
One of our biggest challenges has also been that unlike academic publishing, the vast majority of civil society websites can have 2-3 different documents that refer to the same “object”, and if they are publishing in multiple languages, can have a version of the same object in each of those languages. A good example of this in our corpus at the moment is Global Energy Monitor’s Annual Boom or Bust Coal Reports. These are all similar documents but in different languages.
So far, when reviewing our final collection of documents: we accurately include the correct document types 90% of the time when compared with human reviewers. This accuracy will likely increase in future collections and we will publish more about our technical strategy, once we have optimized the overall data pipeline.
From Document to Record: What is the most important insight to highlight?
While knowing a document exists is useful, creating a useful interface for finding the document requires us to add descriptive metadata to create a full record that is useful for different kinds of search. We want each document to be visible both in our internal search, and for providing another discovery point for external search tools.
Once we identify a document as “in scope” we take several steps to create the metadata:
- First we double check the accuracy of the metadata that we extracted from the document, including Author, Publisher and date of publication.
- Then we send the document to a series of small LLMs (a locally controlled version of the open weight model Gemma), with clear instructions on how to determine what climate topics and geographies the document is about.
- We then use another version of our small AI tool to write very short summaries about the document grounded entirely in the document itself. Important to how we are organizing the corpus are “insights”, which summarize significant pieces of evidence or argumentation. Each of these is a short summary of information found in the original document, that explores a significant piece of data or evidence.
We chose to take this approach, because we wanted to make sure that we described the most significant parts of a document in a way that was both easy for us to find, and didn’t substitute for the whole document (i.e. a 500 or 600 word abstract that might capture the entire argument of a paper). This also allows us to use the “insights” to map out different kinds of evidence used to create conversations between civil society orgs in our “In Focus” section (we will describe our methodology for this in an upcoming blog post). You can see the first few examples of the In Focus pages on South Africa, Nigeria and Uganda.
Help us find more climate civil society publications!
As you may have noticed our first batch had ~50 organizations in the first round of publications. This is just the beginning. We estimate that the full scope of the corpus is likely to include at least 1000 organizations publishing about climate. Our next couple batches of content will take us close to 200. We are just at the beginning, and we need your help. If you can recommend a civil society organisation publishing on climate, especially in languages outside of the UN languages, send us an email: [email protected]