Assessing Algorithmic Bias in Large Language Models' Predictions of Public Opinion Across Demographics
Khai Tran,Sev Geraskin,Doroteya Stoyanova,Jord Nguyen · Team Algorithm Avengers
Submitted to AI and Democracy Hackathon: Demonstrating the Risks. Sprint projects are early-stage work by participants, not Apart Research publications.
The rise of large language models (LLMs) has opened up new possibilities for gauging public opinion on societal issues through survey simulations. However, the potential for algorithmic bias in these models raises concerns about their ability to accurately represent diverse viewpoints, especially those of minority and marginalized groups. This project examines the threat posed by LLMs exhibiting demographic biases when predicting individuals' beliefs, emotions, and policy preferences on important issues. We focus specifically on how well state-of-the-art LLMs like GPT-3.5 and GPT-4 capture the nuances in public opinion across demographics in two distinct regions of Canada - British Columbia and Quebec.
Reviews
Data for GPT 3.5 looks strange
The problem analysis seems super on point. The automation of key institutional features requires significantly super-human implementation to avoid creating distrust and thereby fragmentation. I would like to see more attempts at solving the representativeness problem.
A really cool question to study empirically with lots of potential for relevant insight.
You’ve taken a very interesting approach of polling several LLMs as if they were humans and comparing that with real-world polling data. This is a very bold claim, and I think that the experimental design is lacking in a some aspects to back it up. For example, the was very little prompt engineering, for GPT the demographic data was presented with no preamble; there was little justification to look only at the “strong” response classification; every eval was done only once; different demographic slices were weighed the same (e.g. 86-95 non-binary MSc from Quebec has the same weight as everyone else).
Cite this project
@misc{tran2024assessing,
title = {{Assessing Algorithmic Bias in Large Language Models' Predictions of Public Opinion Across Demographics}},
author = {Khai Tran and Sev Geraskin and Doroteya Stoyanova and Jord Nguyen},
year = {2024},
month = may,
note = {Submitted to AI and Democracy Hackathon: Demonstrating the Risks, an Apart Research Sprint},
howpublished = {\url{https://apartresearch.com/sprints/projects/assessing-algorithmic-bias-in-large-language-models-predictions-of-public-opinion-across-demographics}},
url = {https://apartresearch.com/sprints/projects/assessing-algorithmic-bias-in-large-language-models-predictions-of-public-opinion-across-demographics}
}More from AI and Democracy Hackathon: Demonstrating the Risks
- View project: THE ROLE OF AI IN COMBATING POLITICAL DEEPFAKES IN AFRICAN DEMOCRACIES
THE ROLE OF AI IN COMBATING POLITICAL DEEPFAKES IN AFRICAN DEMOCRACIES
Team 1
The role of AI in combating political deepfakes in African democracies.
- View project: LEGISLaiTOR: A tool for jailbreaking the legislative process
LEGISLaiTOR: A tool for jailbreaking the legislative process
Team Managed Democracy
In this work, we consider the ramifications on generative artificial intelligence (AI) tools in the legislative process in democratic governments. While other research focuses on the micro-level details associated with …
- View project: Subtle and Simple Ways to Shift Political Bias in LLMs
Subtle and Simple Ways to Shift Political Bias in LLMs
Shifty
An informed user knows that an LLM sometimes has a political bias in their responses, but there’s an additional threat that this bias can drift over time, making it even harder to rely on LLMs for an objective …