AI uses for fact-checking in Brazil

Adriano Belisario

Overview

  • About me

  • Language models for fact-checking firearm violence reports

  • Claim-matching and cross-platform misinformation

  • Considerations on AI uses for fact-checking

Link for this presentation: belisards.github.io/slides-bert-fact-checking/

Hi!

Data journalist from Brazil, experienced in human rights investigations.

Since 2013, I have collaborated with projects from several Brazilian and international organizations, such as Witness, UNHCR, Forensic Architecture, Open Knowledge, and Bellingcat.

Here, I will present two papers based on my research at the Oxford Internet Institute.

Fogo Cruzado

Instituto Fogo Cruzado (Crossfire Institute) has monitored and collected data from firearm violence events in Rio de Janeiro since 2017. They currently operate in four cities.

Fogo Cruzado’s human rights analysts follow online and on-the-ground sources seven days a week.

Fogo Cruzado’s website: https://fogocruzado.org.br/

Fact-checking gun violence

Human rights analysts follow multiple online sources simultaneously.

If a gun violence report is identified on Twitter, they ask follow-up questions to the users.

Then, they need to fact-check all reports and, if they are true, record an event in their database and provide real-time alerts.

The needle in the haystack

To keep the message flow manageable, restrictive search filters were used (e.g. using geographic instead of linguistic filters).

However, less than half of Twitter data is geotagged, meaning many reports may have been missed.

The needle in the haystack

Even so, finding a gun violence report among ordinary text is difficult. Analysts must conciliate the demand for verifying the cases and keeping up with an unstoppable flow of new posts.

Goals

  • Train an open-source language model on analysts’ interactions to detect what a gun violence report is in colloquial Brazilian Portuguese.

  • Reduce signal-to-noise ratio for analysts monitoring firearm violence.

  • Design an experiment to assess the model’s performance in a real-world setting.

Methods

I fine-tuned a Transformer neural network (the T in ChatGPT) using an open-source base model for Portuguese (BERTimbau) and ~24k tweet messages.

In addition to standard evaluation using holdout datasets, I integrated the model in a prototype designed to provide real-time predictions and evaluated it in real-world conditions with an experiment.

https://huggingface.co/neuralmind/bert-large-portuguese-cased

Example

The prototype

  1. Fetch new tweets using keywords and/or location and language filters.

  2. The model predicts a binary classification for the message: gun violence report or ordinary text.

  3. The text message and all metadata available - including the predictions - are uploaded to Airtable.

The experiment

I conducted an experiment to evaluate the model performance in the wild: for one month, analysts from Rio de Janeiro adopted the AI-based classification to assist their online search, whereas other regions still used only fully human-made reviews.

The experiment was evaluated through surveys, interviews, and the number of Twitter interactions by region (diff-in-diff analysis controlled by external factors).

Classification metrics

Results

  • The model performed well in all classification metrics (80-90% of success) when tested with holdout datasets.

  • Analysts have approved and adopted the prototype as part of their standard workflow to monitor new cases.

  • The experiment showed that using AI increased analysts’ interactions with users reporting armed violence.

Cross-platform misinformation

Analyzing Misinformation Claims During the 2022 Brazilian General Election on WhatsApp, Twitter, and Kwai: arxiv.org/abs/2401.02395

We partnered with the Brazil’s election authority and three fact-checking organizations running WhatsApp tiplines.

We compared the content submitted by users with posts from Twitter and Kwai.

Results

We observed little overlap between the users of different fact-checking tiplines and a high correlation between the number of users and the amount of unique content, suggesting that WhatsApp tiplines are far from reaching a saturation point.

Similarly, we also found little overlap between contents across platforms, indicating the need for further research with cross-platform approaches to identify misinformation dynamics.

Final considerations

  • Language gap in AI

  • Cross-platform misinformation

  • Multimodal disinformation

References

Into the crossfire: evaluating the use of a language model to crowdsource gun violence reports: arxiv.org/abs/2401.12989

Access this presentation online: belisards.github.io/slides-bert-fact-checking/

Analyzing Misinformation Claims During the 2022 Brazilian General Election on WhatsApp, Twitter, and Kwai: arxiv.org/abs/2401.02395

Contact me

I’m glad to hear from you!

Email: adrianobf (at) gmail (dot) com

Twitter: @belisards

LinkedIn: Adriano Belisario