Imagine running a focus group with ten thousand “people” overnight, for roughly the price of a few coffees. No recruiting, no scheduling, no drop-outs. Just ask your questions and get answers by morning.<\/p>\n\n\n\n
That’s the promise of synthetic audiences<\/strong>: groups of AI agents, powered by the same large language models (LLMs) behind tools like ChatGPT, configured to stand in for human participants in surveys, experiments, and market research. The idea has exploded across marketing, psychology, economics, and design in just a couple of years.<\/p>\n\n\n\n In a recently published systematic review of over 100 related studies<\/a>, we looked at what these AI “participants” can actually do, where they fall apart, and how to use them in practice. This blog post present a summary of the main findings. We also list some of the key references at the end of the post.<\/p>\n\n\n\n The appeal becomes obvious the moment you look at the numbers. Studies that directly compared AI participants with human recruitment found cost differences of one to two orders of magnitude<\/strong> (that is, 10\u00d7 to 100\u00d7 cheaper per response). In one case, researchers generated tens of thousands of survey responses for roughly the API cost of a single query; in another, AI-generated open-ended answers for a user study cost a tiny fraction of what the equivalent human participants would have charged.<\/p>\n\n\n\n Beyond cost, there are practical superpowers:<\/p>\n\n\n\n The honest headline from the research: synthetic audiences are good at capturing broad, aggregate patterns<\/strong> (the overall direction of an effect, rankings, and average tendencies), even when they miss the fine detail.<\/p>\n\n\n\n The more capable the model, the better this gets. Newer models have reproduced classic results from economics and social psychology, closely matched human behavior in trust games, and broadly agreed with human raters on things like brand perceptions. In other words, if your question is “which option do people lean toward?” an AI panel can often point you in the right direction, fast.<\/p>\n\n\n\n The catch is that AI participants look<\/em> convincing, and that’s exactly the trap. Fluent, human-sounding answers make it easy to assume there’s human-like thinking behind them. There usually isn’t. The review found four recurring failure modes, each backed by hard evidence.<\/p>\n\n\n\n 1. They flatten the crowd.<\/strong> Real people disagree, surprise you, and sit at the extremes. AI participants tend to cluster around a bland “average.” One large replication study found an earlier model reproduced only 37.5%<\/strong> of a well-known set of psychology effects, versus a 50%<\/strong> rate for human samples. More tellingly, it kept giving near-identical answers to questions that normally produce lots of human variety. Other work documented a “hyper-accuracy” quirk, where AI participants give unrealistically precise, noise-free answers no real population would.<\/p>\n\n\n\n 2. They carry hidden biases.<\/strong> Because they learn from internet-scale text, AI participants tend to sound disproportionately Western, English-speaking, wealthy, and politically skewed. One study comparing AI-simulated public opinion against a global survey found it performed far better for Western, English-speaking, high-income countries while systematically misrepresenting attitudes across gender, age, education, and class. Worse, when asked to portray specific groups, models often produce caricatures<\/em>. Research on such groups found AI depictions that actual human participants judged not just inaccurate but actively harmful. The uncomfortable implication: synthetic audiences are least reliable precisely where fresh insight is most valuable, namely underrepresented and non-Western populations.<\/p>\n\n\n\n 3. They’re twitchy.<\/strong> Reword a question slightly, reorder the options, or tweak the persona, and the answers can swing dramatically. Studies showed AI survey responses with unstable estimates and strong sensitivity to wording, persona, and even timing<\/em>. In decision-making tasks, AI agents proved far more sensitive than humans to small “nudges”: a tiny change in framing produced outsized changes in behavior. That fragility makes results hard to reproduce.<\/p>\n\n\n\n 4. They can fake reasoning.<\/strong> An answer that looks thoughtful may just be pattern-matching on things the model absorbed during training. Researchers showed that changing one variable in a study (say, a product’s price) can quietly shift the model’s unstated assumptions<\/em> about everything else, contaminating any cause-and-effect conclusion. In another case, near-perfect economic “forecasts” turned out to be memorized history rather than genuine prediction. Add to this the finding that hallucinations are, mathematically, an unavoidable feature of these models, not a bug that will simply be patched away.<\/p>\n\n\n\n The field isn’t standing still. Three families of techniques are making synthetic audiences more trustworthy:<\/p>\n\n\n\n What makes this field genuinely exciting isn’t just today’s tools; it’s where the research is heading. The review highlights three directions that could turn synthetic audiences from a clever shortcut into a dependable scientific instrument.<\/p>\n\n\n\n 1. Open, transparent models built for human behavior.<\/strong> Most studies so far rely on closed, proprietary systems whose training data and updates are secret, which makes results hard to audit or reproduce (a “silent” update can change behavior between studies). Open models change that. The standout example is Centaur<\/em>, a model fine-tuned on over 10 million real human decisions across 160 psychology experiments<\/strong>. It outperformed traditional models at predicting how people actually behave and generalized to new tasks, offering a glimpse of purpose-built, inspectable “model populations” rather than borrowed chatbots.<\/p>\n\n\n\n 2. Standards, benchmarks, and reproducibility.<\/strong> Right now, every team prompts differently, making studies hard to compare. The field is moving toward shared benchmarks and tools, such as standardized environments for running AI-based surveys and experiments, so researchers can tell whether an improvement truly generalizes or just got lucky on one prompt. This is the unglamorous infrastructure that turned modern machine learning into a cumulative science, and synthetic audiences need it too.<\/p>\n\n\n\n 3. Looking inside the black box.<\/strong> The frontier is mechanistic interpretability<\/em>: tools that trace which internal “circuits” of a model produce a given behavior. The payoff would be enormous: imagine pinpointing the exact components that trigger a demographic stereotype and dialing them down, or verifying that a persona prompt genuinely activates the right knowledge rather than just changing the writing style. This work is early and mostly aspirational today, but it points toward synthetic audiences we can actually diagnose and correct, rather than merely observe.<\/p>\n\n\n\n If you’re tempted to use AI participants, the research suggests:<\/p>\n\n\n\n Synthetic audiences are one of the most exciting new tools in research, but they’re a tool, not a replacement for people. They’re best understood as a fast, cheap way to explore ideas and generate hypotheses, with their results treated as a starting point to be validated, not a final verdict.<\/p>\n\n\n\n Used carefully, they can democratize research and let small teams ask big questions. Used carelessly, they produce confident-sounding answers that quietly reflect an AI’s blind spots rather than reality. The difference comes down to knowing what they can and can’t do, and increasingly on the open models, shared standards, and interpretability tools that will determine how far this technology can really go.<\/p>\n\n\n\n This post summarizes the systematic literature review “Synthetic Audiences as Proxies for Human Users,” published in IEEE Access, which analyzed 100 studies on LLM-based synthetic audiences following established review protocols. It’s written for a general audience; see the full paper for methods, evidence, and the complete set of 100 citations.<\/em><\/p>\n\n\n\n Read the full paper:<\/strong> Lappas & Filippas, “Synthetic Audiences as Proxies for Human Users: A Systematic Literature Review,” IEEE Access<\/em>, 2026. https:\/\/ieeexplore.ieee.org\/document\/11563566\/<\/a><\/p>\n\n\n\n The specific findings mentioned above come from the studies below (a curated subset of the 100 reviewed). Links are provided where a stable one exists.<\/p>\n\n\n\n Applications, scale, and cost<\/strong><\/p>\n\n\n\n Limitations<\/strong><\/p>\n\n\n\n Improvement techniques and human-AI hybrids<\/strong><\/p>\n\n\n\n Future directions<\/strong><\/p>\n\n\n\n Imagine running a focus group with ten thousand “people” overnight, for roughly the price of a few coffees. No recruiting, no scheduling, no drop-outs. Just ask your questions and get answers by morning. That’s the promise of synthetic audiences: groups of AI agents, powered by the same large language models (LLMs) behind tools like ChatGPT, […]<\/p>\n","protected":false},"author":5,"featured_media":1756,"template":"","meta":{"_acf_changed":false},"tags":[],"content_types":[{"id":50,"name":"Blog Post","slug":"article"}],"ppma_author":[{"id":5,"display_name":"Ted Lappas","first_name":"Ted","last_name":"Lappas","nickname":"theodoros.lappas","user_nicename":"theodoros-lappas","user_email":"Theodoros.Lappas@wpp.com","biographical_info":"Ted co-leads WPP Research and serves as Head of Data Science at Satalia, co-founder of Conscium, and Assistant Professor in the Department of Marketing and Communication at the Athens University of Economics and Business. His research spans scalable algorithms for multimodal data, synthetic data generation, simulation-based verification for AI agents, and information diffusion and collective intelligence in expert networks. He publishes regularly in top-tier computer science and business venues.","avatar_url":"https:\/\/cms.research.wpp.com\/wp-content\/uploads\/2026\/04\/pic.png","job_title":"Head of Data Science","is_lead":false,"display_as_researcher":true,"order_priority":1}],"class_list":["post-1752","research_feed","type-research_feed","status-publish","has-post-thumbnail","hentry","content_type-article"],"acf":{"content":" <\/p>\n Imagine running a focus group with ten thousand “people” overnight, for roughly the price of a few coffees. No recruiting, no scheduling, no drop-outs. Just ask your questions and get answers by morning.<\/p>\n <\/p>\n <\/p>\n That’s the promise of synthetic audiences<\/strong>: groups of AI agents, powered by the same large language models (LLMs) behind tools like ChatGPT, configured to stand in for human participants in surveys, experiments, and market research. The idea has exploded across marketing, psychology, economics, and design in just a couple of years.<\/p>\n <\/p>\n <\/p>\n In a recently published systematic review of over 100 related studies<\/a>, we looked at what these AI “participants” can actually do, where they fall apart, and how to use them in practice. This blog post present a summary of the main findings. We also list some of the key references at the end of the post.<\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n The appeal becomes obvious the moment you look at the numbers. Studies that directly compared AI participants with human recruitment found cost differences of one to two orders of magnitude<\/strong> (that is, 10\u00d7 to 100\u00d7 cheaper per response). In one case, researchers generated tens of thousands of survey responses for roughly the API cost of a single query; in another, AI-generated open-ended answers for a user study cost a tiny fraction of what the equivalent human participants would have charged.<\/p>\n <\/p>\n <\/p>\n Beyond cost, there are practical superpowers:<\/p>\n <\/p>\n <\/p>\n \n<\/ul>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n The honest headline from the research: synthetic audiences are good at capturing broad, aggregate patterns<\/strong> (the overall direction of an effect, rankings, and average tendencies), even when they miss the fine detail.<\/p>\n <\/p>\n <\/p>\n The more capable the model, the better this gets. Newer models have reproduced classic results from economics and social psychology, closely matched human behavior in trust games, and broadly agreed with human raters on things like brand perceptions. In other words, if your question is “which option do people lean toward?” an AI panel can often point you in the right direction, fast.<\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n The catch is that AI participants look<\/em> convincing, and that’s exactly the trap. Fluent, human-sounding answers make it easy to assume there’s human-like thinking behind them. There usually isn’t. The review found four recurring failure modes, each backed by hard evidence.<\/p>\n <\/p>\n <\/p>\n 1. They flatten the crowd.<\/strong> Real people disagree, surprise you, and sit at the extremes. AI participants tend to cluster around a bland “average.” One large replication study found an earlier model reproduced only 37.5%<\/strong> of a well-known set of psychology effects, versus a 50%<\/strong> rate for human samples. More tellingly, it kept giving near-identical answers to questions that normally produce lots of human variety. Other work documented a “hyper-accuracy” quirk, where AI participants give unrealistically precise, noise-free answers no real population would.<\/p>\n <\/p>\n <\/p>\n 2. They carry hidden biases.<\/strong> Because they learn from internet-scale text, AI participants tend to sound disproportionately Western, English-speaking, wealthy, and politically skewed. One study comparing AI-simulated public opinion against a global survey found it performed far better for Western, English-speaking, high-income countries while systematically misrepresenting attitudes across gender, age, education, and class. Worse, when asked to portray specific groups, models often produce caricatures<\/em>. Research on such groups found AI depictions that actual human participants judged not just inaccurate but actively harmful. The uncomfortable implication: synthetic audiences are least reliable precisely where fresh insight is most valuable, namely underrepresented and non-Western populations.<\/p>\n <\/p>\n <\/p>\n 3. They’re twitchy.<\/strong> Reword a question slightly, reorder the options, or tweak the persona, and the answers can swing dramatically. Studies showed AI survey responses with unstable estimates and strong sensitivity to wording, persona, and even timing<\/em>. In decision-making tasks, AI agents proved far more sensitive than humans to small “nudges”: a tiny change in framing produced outsized changes in behavior. That fragility makes results hard to reproduce.<\/p>\n <\/p>\n <\/p>\n 4. They can fake reasoning.<\/strong> An answer that looks thoughtful may just be pattern-matching on things the model absorbed during training. Researchers showed that changing one variable in a study (say, a product’s price) can quietly shift the model’s unstated assumptions<\/em> about everything else, contaminating any cause-and-effect conclusion. In another case, near-perfect economic “forecasts” turned out to be memorized history rather than genuine prediction. Add to this the finding that hallucinations are, mathematically, an unavoidable feature of these models, not a bug that will simply be patched away.<\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n The field isn’t standing still. Three families of techniques are making synthetic audiences more trustworthy:<\/p>\n <\/p>\n <\/p>\n \n<\/ul>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n What makes this field genuinely exciting isn’t just today’s tools; it’s where the research is heading. The review highlights three directions that could turn synthetic audiences from a clever shortcut into a dependable scientific instrument.<\/p>\n <\/p>\n <\/p>\n 1. Open, transparent models built for human behavior.<\/strong> Most studies so far rely on closed, proprietary systems whose training data and updates are secret, which makes results hard to audit or reproduce (a “silent” update can change behavior between studies). Open models change that. The standout example is Centaur<\/em>, a model fine-tuned on over 10 million real human decisions across 160 psychology experiments<\/strong>. It outperformed traditional models at predicting how people actually behave and generalized to new tasks, offering a glimpse of purpose-built, inspectable “model populations” rather than borrowed chatbots.<\/p>\n <\/p>\n <\/p>\n 2. Standards, benchmarks, and reproducibility.<\/strong> Right now, every team prompts differently, making studies hard to compare. The field is moving toward shared benchmarks and tools, such as standardized environments for running AI-based surveys and experiments, so researchers can tell whether an improvement truly generalizes or just got lucky on one prompt. This is the unglamorous infrastructure that turned modern machine learning into a cumulative science, and synthetic audiences need it too.<\/p>\n <\/p>\n <\/p>\n 3. Looking inside the black box.<\/strong> The frontier is mechanistic interpretability<\/em>: tools that trace which internal “circuits” of a model produce a given behavior. The payoff would be enormous: imagine pinpointing the exact components that trigger a demographic stereotype and dialing them down, or verifying that a persona prompt genuinely activates the right knowledge rather than just changing the writing style. This work is early and mostly aspirational today, but it points toward synthetic audiences we can actually diagnose and correct, rather than merely observe.<\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n If you’re tempted to use AI participants, the research suggests:<\/p>\n <\/p>\n <\/p>\n \n<\/ul>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n Synthetic audiences are one of the most exciting new tools in research, but they’re a tool, not a replacement for people. They’re best understood as a fast, cheap way to explore ideas and generate hypotheses, with their results treated as a starting point to be validated, not a final verdict.<\/p>\n <\/p>\n <\/p>\n Used carefully, they can democratize research and let small teams ask big questions. Used carelessly, they produce confident-sounding answers that quietly reflect an AI’s blind spots rather than reality. The difference comes down to knowing what they can and can’t do, and increasingly on the open models, shared standards, and interpretability tools that will determine how far this technology can really go.<\/p>\n <\/p>\n <\/p>\n <\/p>\n This post summarizes the systematic literature review “Synthetic Audiences as Proxies for Human Users,” published in IEEE Access, which analyzed 100 studies on LLM-based synthetic audiences following established review protocols. It’s written for a general audience; see the full paper for methods, evidence, and the complete set of 100 citations.<\/em><\/p>\n <\/p>\n <\/p>\n Read the full paper:<\/strong> Lappas & Filippas, “Synthetic Audiences as Proxies for Human Users: A Systematic Literature Review,” IEEE Access<\/em>, 2026. https:\/\/ieeexplore.ieee.org\/document\/11563566\/<\/a><\/p>\n <\/p>\n <\/p>\n <\/p>\n <\/p>\n The specific findings mentioned above come from the studies below (a curated subset of the 100 reviewed). Links are provided where a stable one exists.<\/p>\n <\/p>\n <\/p>\n Applications, scale, and cost<\/strong><\/p>\n <\/p>\n <\/p>\n \n<\/ul>\n <\/p>\n <\/p>\n Limitations<\/strong><\/p>\n <\/p>\n <\/p>\n \n<\/ul>\n <\/p>\n <\/p>\n Improvement techniques and human-AI hybrids<\/strong><\/p>\n <\/p>\n <\/p>\nWhy everyone is excited<\/h2>\n\n\n\n
\n\n
Where they actually shine<\/h2>\n\n\n\n
Where they break<\/h2>\n\n\n\n
How researchers are fixing them<\/h2>\n\n\n\n
\n\n
The promising road ahead<\/h2>\n\n\n\n
Best Practice<\/h2>\n\n\n\n
\n\n
The bottom line<\/h2>\n\n\n\n
\n\n\n\nSources & further reading<\/h2>\n\n\n\n
\n\n
\n\n
\n\n
\n\n
Why everyone is excited<\/h2>\n
\n<\/p>\n
\n<\/p>\nWhere they actually shine<\/h2>\n
Where they break<\/h2>\n
How researchers are fixing them<\/h2>\n
\n<\/p>\n
\n<\/p>\n
\n<\/p>\nThe promising road ahead<\/h2>\n
Best Practice<\/h2>\n
\n<\/p>\n
\n<\/p>\nThe bottom line<\/h2>\n
\n<\/p>\nSources & further reading<\/h2>\n
\n<\/p>\n
\n<\/p>\n
\n<\/p>\n
\n<\/p>\n
\n<\/p>\n
\n<\/p>\n
\n<\/p>\n
\n<\/p>\n\n<\/p>\n
\n<\/p>\n
\n<\/p>\n
\n<\/p>\n
\n<\/p>\n
\n<\/p>\n
\n<\/p>\n
\n<\/p>\n
\n<\/p>\n