Large language models can predict the results of social science experiments

This demo accompanies the paper Large language models can predict the results of social science experiments (Nature, 2026; OSF) and can be used for predicting experimental treatment effects on U.S. adults.

1. What treatment text would participants read?

2. What question would participants answer, after reading the treatment text?

Lower label

Upper label

3. Simulation model

Ensemble (GPT + Gemma + Qwen) GPT-5.6 Luna Gemma 3 27B Qwen 3.5 27B

Predict

🎛️ Advanced 🗃️ My history

Population All U.S. adults Women Men Black Americans White Americans Democrats Republicans

➕ Add another treatment to compare 📄 Upload a list of treatments

Compare up to 10 treatments on the same outcome, or upload one treatment per line.

FAQs

What does this tool do?

This tool uses Large Language Models (LLMs) to predict experimental treatment effects on survey outcomes for U.S. adult samples. Users can select a dependent variable and one or more text-based treatment messages. Once you click Submit, the tool uses an LLM to simulate American participant responses in an RCT experiment. It then displays the predicted treatment effect for each treatment.

Note that this is a technical demo, and not a substitute for conducting experiments with real human participants. It is only a way to predict experimental results, and should be used as a complement to, rather than a replacement for, research with humans participants.

How accurate are the predictions?

We have conducted a series of large-scale assessments of the accuracy of predictions generated using LLMs, details of which are provided in the paper. Briefly:

• For survey experiments (experiments with text-based treatments and measures), predicted effect sizes were strongly correlated with actual effect sizes (r = .85).
• For large, many-treatment experiments ("megastudies", including survey and field experiments), we found that the predictions' accuracy (r = .34) matched that of expert human forecasters (r = .26).

What can I use this for?

There may be several applications:

• Pilot testing of study materials. LLMs may help researchers pilot test study materials prior to launching experiments, thereby informing decisions about which materials to use.
• Intervention design. As LLMs can evaluate many treatment messages in very little time, they may help optimize the development of effective message-based interventions (e.g. to promote public health behaviors) by helping researchers narrow the field of messages to test in an RCT.
• Minimizing harm to human participants. For research that involves potential risk to human participants (such as exposing subjects to misinformation in order to subsequently test the impact of an intervention), LLMs may be used to conduct a simulated test of an intervention before exposing any human participants.

Note that this is a technical demo, based on our evaluation of LLMs' predictions for experiments conducted in the US. Studies are beginning to evaluate the strengths and limitations of using LLMs to simulate participants, including concerns about bias, risks of over-reliance, and misuse. For discussion, see [1][2].

Why are there no confidence intervals?

Because it is easy to generate extremely large samples of simulated participants, a confidence interval on the simulated treatment effects would be extremely narrow, yet this does not capture the error of the model in predicting human responses.

Can I make predictions on demographic subpopulations of the U.S.?

This is possible using the advanced (🎛️) menu.

However, we recommend extra caution in interpreting estimated subpopulation effects. Some studies simulating survey responses using LLMs have found simulated responses to often be biased against groups that are historically underrepresented or misrepresented in news or other media. (see e.g. [1][3]).

Can I compare multiple treatments at once?

Yes, this feature is accessible using the advanced (🎛️) menu. You can type in multiple treatments or upload a txt file with one treatment per line (up to 10 treatments).

What control group does the demo use?

All predicted treatment effects are relative to a “control group” that received no message. That is, we compare LLM-predicted outcome scores for simulated participants read a treatment message versus no message. If you wish to include another control group, simply add this as an additional treatment (see prior FAQ answer).

Are there usage restrictions?

This tool contains guardrails to prevent misuse. Our goal with these guardrails is to support scientific research uses, while minimizing the possibility that the tool can be used for socially harmful purposes (such as optimizing misinformation).

You can view the specific guardrails currently implemented here.

Can I use different dependent variable to the ones shown?

Yes, you may write any dependent variable in the box above. Note that the upper end of the scale should correspond to the intended direction of the treatment.