About me
Language models for fact-checking firearm violence reports
Claim-matching and cross-platform misinformation
Considerations on AI uses for fact-checking
Link for this presentation: belisards.github.io/slides-bert-fact-checking/
Data journalist from Brazil, experienced in human rights investigations.
Since 2013, I have collaborated with projects from several Brazilian and international organizations, such as Witness, UNHCR, Forensic Architecture, Open Knowledge, and Bellingcat.
Here, I will present two papers based on my research at the Oxford Internet Institute.
Instituto Fogo Cruzado (Crossfire Institute) has monitored and collected data from firearm violence events in Rio de Janeiro since 2017. They currently operate in four cities.
Fogo Cruzado’s human rights analysts follow online and on-the-ground sources seven days a week.
Fogo Cruzado’s website: https://fogocruzado.org.br/
Human rights analysts follow multiple online sources simultaneously.
If a gun violence report is identified on Twitter, they ask follow-up questions to the users.
Then, they need to fact-check all reports and, if they are true, record an event in their database and provide real-time alerts.
To keep the message flow manageable, restrictive search filters were used (e.g. using geographic instead of linguistic filters).
However, less than half of Twitter data is geotagged, meaning many reports may have been missed.
Even so, finding a gun violence report among ordinary text is difficult. Analysts must conciliate the demand for verifying the cases and keeping up with an unstoppable flow of new posts.
Train an open-source language model on analysts’ interactions to detect what a gun violence report is in colloquial Brazilian Portuguese.
Reduce signal-to-noise ratio for analysts monitoring firearm violence.
Design an experiment to assess the model’s performance in a real-world setting.
I fine-tuned a Transformer neural network (the T in ChatGPT) using an open-source base model for Portuguese (BERTimbau) and ~24k tweet messages.
In addition to standard evaluation using holdout datasets, I integrated the model in a prototype designed to provide real-time predictions and evaluated it in real-world conditions with an experiment.
Fetch new tweets using keywords and/or location and language filters.
The model predicts a binary classification for the message: gun violence report or ordinary text.
The text message and all metadata available - including the predictions - are uploaded to Airtable.
I conducted an experiment to evaluate the model performance in the wild: for one month, analysts from Rio de Janeiro adopted the AI-based classification to assist their online search, whereas other regions still used only fully human-made reviews.
The experiment was evaluated through surveys, interviews, and the number of Twitter interactions by region (diff-in-diff analysis controlled by external factors).
The model performed well in all classification metrics (80-90% of success) when tested with holdout datasets.
Analysts have approved and adopted the prototype as part of their standard workflow to monitor new cases.
The experiment showed that using AI increased analysts’ interactions with users reporting armed violence.
Analyzing Misinformation Claims During the 2022 Brazilian General Election on WhatsApp, Twitter, and Kwai: arxiv.org/abs/2401.02395
We partnered with the Brazil’s election authority and three fact-checking organizations running WhatsApp tiplines.
We compared the content submitted by users with posts from Twitter and Kwai.
We observed little overlap between the users of different fact-checking tiplines and a high correlation between the number of users and the amount of unique content, suggesting that WhatsApp tiplines are far from reaching a saturation point.
Similarly, we also found little overlap between contents across platforms, indicating the need for further research with cross-platform approaches to identify misinformation dynamics.
Language gap in AI
Cross-platform misinformation
Multimodal disinformation
Into the crossfire: evaluating the use of a language model to crowdsource gun violence reports: arxiv.org/abs/2401.12989
Access this presentation online: belisards.github.io/slides-bert-fact-checking/
Analyzing Misinformation Claims During the 2022 Brazilian General Election on WhatsApp, Twitter, and Kwai: arxiv.org/abs/2401.02395
I’m glad to hear from you!
Email: adrianobf (at) gmail (dot) com
Twitter: @belisards
LinkedIn: Adriano Belisario