Research Findings
Six experiments that reveal what drives synthetic survey quality — from the levers that matter most to the biases hiding in default model behavior.
What Matters Most
Effect size of each optimization lever, sorted by maximum impact. Ensemble blending delivers +2.7–5.8 SPS points at zero additional cost.
Horizontal bar chart of effect sizes for each optimization lever (ensemble, temperature, conditioning, and more), sorted by maximum impact.
The Biggest Lever: Ensemble Blending
For each dataset, ensemble blending of 3 models outperforms the best single leaderboard entry at matched question count by +2.7 to +5.8 SPS points. The blend itself is arithmetic — no new model calls beyond the three constituent runs — but producing all three costs roughly 3× a single-model run.
Bar chart comparing the SPS of the best single model against a 3-model ensemble for each dataset. Ensemble gains range from +5.2 to +7.0 points.
Comparison set: best_single = highest recomputed-SPS non-ensemble, non-baseline leaderboard row (raw or product framework) evaluated on at least as many questions as the ensemble, after per-(model, framework, dataset) dedup; random_baseline = recomputed SPS of the random-baseline run for the same dataset (n in random_baseline_n).
Cost vs Performance Pareto
Each point is one configuration. Pareto-optimal configurations (no other run is cheaper AND higher-SPS) are highlighted; dominated runs are muted. Lower-left of the frontier is the cost/quality trade you cannot beat.
Scatter plot of LLM cost in USD against Survey Parity Score for each non-baseline configuration. Pareto-optimal points are highlighted; dominated points are muted.
Show data table
| Configuration | Cost (USD) | SPS | Pareto Status |
|---|---|---|---|
| SynthPanel Ensemble (3-model) (product) | $1.07 | 0.879 | Pareto-optimal |
| SynthPanel Ensemble (3-model) (product) | $6.12 | 0.877 | Dominated |
| Gemini 2.5 Flash (raw) | $1.61 | 0.829 | Dominated |
| SynthPanel (Sonnet 4) (product) | $8.86 | 0.829 | Dominated |
| SynthPanel (Haiku 4.5) (product) | $4.92 | 0.829 | Dominated |
| SynthPanel (GPT-4o-mini) (product) | $0.718 | 0.823 | Pareto-optimal |
| SynthPanel (Gemini Flash Lite) (product) | $0.084 | 0.821 | Pareto-optimal |
| Llama 3.3 70B (raw) | $0.468 | 0.819 | Dominated |
| SynthPanel (Gemini Flash Lite) (product) | $0.479 | 0.816 | Dominated |
| Claude Haiku 4.5 (raw) | $4.92 | 0.815 | Dominated |
| SynthPanel Ensemble (3-model) (product) | $0.537 | 0.813 | Dominated |
| GPT-4o (raw) | $3.50 | 0.813 | Dominated |
| GPT-4o-mini (raw) | $0.718 | 0.813 | Dominated |
| SynthPanel (Haiku 4.5) (product) | $0.864 | 0.809 | Dominated |
| Llama 3.3 70B (raw) | $0.082 | 0.796 | Pareto-optimal |
| SynthPanel (GPT-4o-mini) (product) | $0.126 | 0.787 | Dominated |
| SynthPanel (GPT-4o-mini) (product) | $0.063 | 0.786 | Pareto-optimal |
| Gemini 2.5 Flash (raw) | $0.071 | 0.783 | Dominated |
| Claude Sonnet 4 (raw) | $14.77 | 0.782 | Dominated |
| Gemini 2.5 Flash (raw) | $0.071 | 0.770 | Dominated |
| GPT-4o-mini (raw) | $0.032 | 0.770 | Pareto-optimal |
| Claude Haiku 4.5 (raw) | $0.864 | 0.768 | Dominated |
| Llama 3.3 70B (raw) | $0.021 | 0.762 | Pareto-optimal |
| SynthPanel (Gemini Flash Lite) (product) | $0.042 | 0.762 | Dominated |
| GPT-4o-mini (raw) | $0.032 | 0.749 | Dominated |
| Claude Sonnet 4.6 (raw) | $0.654 | 0.738 | Dominated |
| Gemini Flash Lite (raw) | $0.042 | 0.738 | Dominated |
| Claude Haiku 4.5 (raw) | $0.216 | 0.726 | Dominated |
| SynthPanel (Haiku 4.5) (product) | $0.432 | 0.725 | Dominated |
Cross-Provider JSD Matrix
Pairwise Jensen-Shannon divergence between each raw-LLM pair, averaged
across shared questions per dataset. Low values (cooler) mean providers
agree with each other — a red flag for HBR's
trendslop hypothesis (cross-model consensus reached on shared errors). High values (warmer) mean
providers genuinely diverge. Ground-truth concordance (model vs. human) stays
a separate axis — see mean_human_jsd below the chart.
Heatmap of pairwise Jensen-Shannon divergence between raw-LLM providers, per dataset. Lower off-diagonal values indicate cross-model agreement; higher values indicate divergence.
Show data table
How the Ensemble Is Built
The biggest lever in the benchmark is also the simplest. Six things worth knowing about how blending turns three good models into one better one.
What we blend
Per-question response distributions from three models — Claude Haiku 4.5, Gemini Flash Lite, and GPT-4o-mini — run through the SynthPanel conditioning framework on the same inputs.
How the blend works
Equal-weight arithmetic average of the three distributions per question. For each option on each question, we take the mean of the three models' probability mass. No learned weights, no router, no post-hoc calibration.
Why it works
The three models make uncorrelated errors on different questions. Blending cancels noise that is specific to any one model while preserving signal they agree on — 72–81% of questions improve under blending across datasets.
What we tested against it
Equal-weight, score-proportional, and inverse-JSD weighting — scores are indistinguishable to three decimals. The blending weights don't matter; the ensemble itself does.
What we didn't beat
Per-question oracle selection (the best single model picked retroactively) adds negligible headroom over equal-weight blending. There is little left to extract from this set of models without adding new ones.
What it costs
The blend itself is pure offline arithmetic — no new API calls beyond the three constituent runs. But it is not free: you have to run all three base models, so producing a 3-model blend costs roughly 3× a single-model run.
Temperature: It Depends on the Model
SPS across temperature settings for each model. Error bars show ±1 std across replicates. Gemini Flash Lite gains +4.5 SPS points from 0.3 to 2.0 (well outside its ±0.003 std), while Claude Haiku 4.5 moves under its own noise band.
Line chart of SPS across temperature settings for each model. Error bars show ±1 standard deviation across replicates.
Demographic Conditioning Reveals Model Bias
How much conditioning on a demographic group shifts model responses (p_cond). Republican conditioning is 2.2x stronger than Democrat — the model already leans progressive by default.
Chart showing how conditioning on different demographic groups shifts model responses (p_cond). Republican conditioning produces a larger shift than Democrat — evidence of a default progressive lean.
SPS by Topic
Best-per-model performance across the ten topic categories. Baselines and temperature variants hidden; scroll the legend to toggle models.
Grouped bar chart of best-per-model SPS across ten topic categories. Each group is a topic; each bar is a model.
SPS Convergence
How SPS stabilizes as replicate count increases. Showing the ensemble and the three blended single models.
Line chart showing how Survey Parity Score stabilizes as the number of replicates increases. One line per model plus the ensemble.
Temporal Drift Floor
Mean Jensen-Shannon divergence between the same Pew ATP question stems across different waves, grouped by year gap. A separate baseline from the Human Ceiling — drift measures how much real-world opinions shift, not sampling noise.
Temporal Drift Floor ships May 2026
The next publish cycle lifts the full cross-wave JSD table from
baselines.temporal_drift
in leaderboard.json —
mean drift, 95% CI, and per-year-gap pairs are already computed from repeated
Pew ATP stems on OpinionsQA. See the
Temporal Drift Floor definition
on the methodology page.