Author: Ted Lappas

Every year, brand-tracking programmes field hundreds of surveys: the same battery of questions, asked across audiences and categories, wave after wave. It is the backbone of how brands understand where they stand. It is also expensive, and a surprising amount of it tells you what you could already have worked out.

That last part is the interesting bit. In a new study using WPP’s Brand Asset® Valuator (BAV) data, we turned a familiar research question inside out. Instead of asking “how accurate can our tracking be?”, we asked the question a marketing or finance director would ask: “how few surveys can we run and still have data we are prepared to act on?” The answer, on the programme we tested, is about two-thirds of them. This post explains how we got there, what the trade-off looks like, and what it means for the way you plan fieldwork.

The expensive habit nobody questions

Brand tracking tends to treat every market the same. Point the same instrument at every audience and category, run it, then run it again next wave. The bill adds up fast. Published figures for professionally run surveys land around $17 per sampled person and $41 per completed questionnaire, rising to $110 per complete for some designs. These are US academic figures, but the economics travel, and they climb every year as response rates fall and each completed interview takes more effort to win.

There is a quality tax on top of the money. The more often and the longer you survey people, the more they rush, guess and drop out, so over-surveying can quietly erode the very accuracy it is meant to buy.

Meanwhile, markets are not equal in how much fresh measurement they need. Brand perceptions are sticky in some places and volatile in others. Some of your surveys move a lot between waves. Some barely move at all, and their results could have been predicted from the rest of the programme with useful precision. Treating both kinds the same means spending money where the answer was already in the archive.

The key insight: accuracy is a line you set, cost is what you cut

Here is the shift in thinking that makes the rest work.

Most conversations about predicting survey results get stuck on one question: “is the prediction accurate enough?” Asked that way, the answer is always “it could be better, at a price,” and the conversation goes nowhere.

We flipped it. The brand team decides in advance the lowest accuracy it is willing to act on. Call it the good-enough line. Then the job is to find the cheapest fieldwork programme that clears that line, and to predict everything else. Accuracy is not the thing you chase. It is a floor you refuse to go under. Cost is the thing you minimise.

Figure 1: Set the bar, find the cheapest plan, and predict the rest.

Put that way, three moves follow naturally.

1. Set the bar. For this study we set it at 0.75 on a standard accuracy measure (R², which runs from 0 for useless to 1 for perfect). That is a demonstration, not a recommendation. A tracker that feeds boardroom reviews can live with a lower bar than one that triggers spend market by market.
2. Find the cheapest plan. A “survival of the fittest” search (a genetic algorithm) tries thousands of combinations of surveys and hands back a named list of the ones to keep fielding. Named matters. A percentage tells you how much to cut; a list tells you what.
3. Predict the rest. A model trained on the surveys you did field fills in the ones you did not. Because every survey is described by the same handful of ingredients (audience, category, brand, attribute), the model learns how each ingredient shifts a brand’s scores and applies that to combinations it has never seen.

What each slice of budget buys you

The most useful thing the study produces is not a single number, it is a curve.

Figure 2: Prediction accuracy against the share of surveys fielded, showing the good-enough line at 0.75.

Read it left to right. Field 10% of your surveys and predictions are poor. Field 30% and they are already respectable. Field 60% and they are close to as good as they will ever get. After that, every extra survey buys less than the one before. On this programme, going from 63% of surveys to 100% of them lifts accuracy by under three points on a 100-point scale, and that final stretch is more than a third of the entire fieldwork budget.

This is the whole argument in one picture. The curve flattens, so the last few points of accuracy are the most expensive fieldwork you will ever buy. A programme that insists on the highest possible accuracy is paying its highest price for its smallest gain. Choosing to stop at the good-enough line is not a compromise on quality. It is the same logic you apply to every other budget you own.

The curve also lets you price any bar you like. Happy with 0.70? Field about 39% of the programme. Want 0.775? That will cost you about 91% of it. The conversation between research and finance can be had in one currency.

What we found

We tested this on the 2023 wave of BAV in the United States: 130 surveys (13 categories by 10 audiences), each scoring 10 brands on 46 attributes, nearly 60,000 data points in all. We held 26 surveys back as a hidden test set that the model never saw, so every accuracy figure below is on markets it had to predict cold.

• About a third of fieldwork is predictable. At the 0.75 bar, the cheapest plan fields roughly 63% of surveys and predicts the rest, a fieldwork reduction of about 37%. Fielding everything would reach only 0.78.
• The search returns a list, not just a number. On the training pool of 104 surveys, the genetic algorithm found a committable core of 81, a 22% reduction, with accuracy essentially unchanged (0.777 to 0.775). Being honest about this one: random selection can hit the bar with fewer surveys than the search found, because the search stops the moment it clears the line and does not keep pushing on cost. The search’s value is the named list you can act on, not a smaller number.
• The obvious shortcut does not work. We also tried filtering out the hardest-to-predict slices before compressing. On this data it made predictions worse at every threshold we tried. The model’s interaction features already capture what filtering would remove, so compression works at the level of whole surveys, which is also the level at which you actually buy fieldwork.

The trap: last year’s plan does not travel

Here is the finding that should change how you plan.

Take the surveys the search chose on 2023 and carry them, unchanged, into 2024 and 2025. Accuracy drops below the bar, to 0.73 and 0.69. Markets move, and a plan frozen on one wave goes stale on the next.

The fix is not to abandon the plan, it is to dilute it.

Figure 3: Accuracy rises as fresh surveys replace last year’s selections within a fixed 65-survey budget.

We held the budget fixed at 65 surveys and varied the mix. Spending all 65 on last year’s chosen surveys was the worst use of the money. Swapping just six of them for fresh surveys of the new wave lifted accuracy by five to six points. Swapping half was best, lifting it by eight points in both years at identical cost, and back above the bar. And the fresh surveys were picked at random. No clever selection needed; the model simply needs some measurement of the wave it is predicting.

So the deployment rule is simple. Under a fixed budget, reserve about half of it for fresh measurement of the wave you are planning, and let the model reconstruct the rest.

What this means if you run a tracker

Use it to plan, not to autopilot. Treat the predictions as a confident starting point that frees up budget, not a verdict that retires a market for good.

It is reallocation, not retreat. The point is not to survey less for its own sake. On these data, about a third of fieldwork spend could be redirected from predictable surveys towards the markets where fresh measurement demonstrably adds information, or simply saved.

Set the bar deliberately. The good-enough line is a management decision, not a modelling one. Decide it based on what the data is used for, and use the curve to see what it costs.

Refresh every wave. Do not carry last wave’s plan forward untouched. Mix in fresh surveys of the target wave, roughly half the budget on the evidence here.

Watch for shocks. A new entrant, a scandal, a viral moment can turn a stable market volatile overnight. The model cannot anticipate a structural break; only people watching the market can.

• It is reallocation, not retreat. The point is not to survey less for its own sake. It is to move a meaningful share of the budget — on these data, somewhere between a quarter and two-fifths of it — out of predictable slices and into volatile ones, where fresh measurement demonstrably adds information.
• Use savings strategically. Reduced fieldwork can mean lower cost, but it can also mean better coverage: expanding into new markets, adding new audiences, or going deeper where the evidence is most decision-relevant.
• Build in safeguards against drift. A slice that was predictable last year may not stay predictable forever. A new entrant, scandal, viral moment, campaign, or category disruption can change the market quickly. A deployed system should therefore include two safeguards. First, continue to field a small sample of otherwise “skippable” slices as a robustness check. Second, avoid using predicted values as if they were fresh observations in the next wave. In other words, do not let estimates compound into future estimates without being periodically re-anchored in real fieldwork.
• Use it to plan, not to autopilot. Treat the predictions as a confident starting point that frees up budget, not as a final verdict that retires a market for good.

The bottom line

Brand tracking does not have to be all-or-nothing, measure everything or fly blind. The data already in your archive can tell you, with useful precision, which surveys are worth re-fielding and which can be predicted from what you already know. And once you frame it as a budget decision rather than an accuracy contest, the answer to “why isn’t the prediction more accurate?” becomes obvious: because the last few points cost more than they are worth, and you chose not to buy them.

It is, in a sense, the mirror image of the synthetic-audiences debate. That conversation asks whether AI can replace the respondents. This one asks a more immediate and less risky question: of all the surveys you are about to run, which ones do you even need to? For roughly a third of them, the most honest answer is that you already have the data.

Sources and further reading

A curated subset of the work behind this post; full citations appear in the paper.

• Olson et al. (2024), “Examining variation in survey costs across surveys,” Sociological Methods & Research: the per-survey cost figures.
• Groves & Heeringa (2006), “Responsive design for household surveys,” Journal of the Royal Statistical Society A: survey cost as a constraint to be traded against accuracy.
• Bronnenberg, Dhar & Dubé (2009), “Brand history, geography, and the persistence of brand shares,” Journal of Political Economy: why some markets barely move.
• Prokhorenkova et al. (2018), “CatBoost: Unbiased boosting with categorical features,” NeurIPS: the recovery model.
• Yang et al. (2023), “Dataset pruning,” ICLR: why large fractions of training data are often redundant.
• Argyle et al. (2023), “Out of one, many: Using language models to simulate human samples,” Political Analysis: the synthetic-audiences counterpoint.

Disclaimer: This content was created with AI assistance. All research and conclusions are the work of WPP Research.

Imagine running a focus group with ten thousand “people” overnight, for roughly the price of a few coffees. No recruiting, no scheduling, no drop-outs. Just ask your questions and get answers by morning.

That’s the promise of synthetic audiences: groups of AI agents, powered by the same large language models (LLMs) behind tools like ChatGPT, configured to stand in for human participants in surveys, experiments, and market research. The idea has exploded across marketing, psychology, economics, and design in just a couple of years.

In a recently published systematic review of over 100 related studies, we looked at what these AI “participants” can actually do, where they fall apart, and how to use them in practice. This blog post present a summary of the main findings. We also list some of the key references at the end of the post.

Why everyone is excited

The appeal becomes obvious the moment you look at the numbers. Studies that directly compared AI participants with human recruitment found cost differences of one to two orders of magnitude (that is, 10× to 100× cheaper per response). In one case, researchers generated tens of thousands of survey responses for roughly the API cost of a single query; in another, AI-generated open-ended answers for a user study cost a tiny fraction of what the equivalent human participants would have charged.

Beyond cost, there are practical superpowers:

• • No logistics headaches. An AI agent always answers, never gets bored, and never abandons your survey halfway through.
• • Experiments you otherwise couldn’t run. One team simulated a bank run using tens of thousands of AI-generated depositors across hundreds of demographic groups, a scenario you obviously can’t stage with real customers and real money. Others have built entire simulated societies: sandbox towns of AI “residents” that develop their own social dynamics, and large-scale models with over 10,000 agents interacting in urban and economic environments.

Where they actually shine

The honest headline from the research: synthetic audiences are good at capturing broad, aggregate patterns (the overall direction of an effect, rankings, and average tendencies), even when they miss the fine detail.

The more capable the model, the better this gets. Newer models have reproduced classic results from economics and social psychology, closely matched human behavior in trust games, and broadly agreed with human raters on things like brand perceptions. In other words, if your question is “which option do people lean toward?” an AI panel can often point you in the right direction, fast.

Where they break

The catch is that AI participants look convincing, and that’s exactly the trap. Fluent, human-sounding answers make it easy to assume there’s human-like thinking behind them. There usually isn’t. The review found four recurring failure modes, each backed by hard evidence.

1. They flatten the crowd. Real people disagree, surprise you, and sit at the extremes. AI participants tend to cluster around a bland “average.” One large replication study found an earlier model reproduced only 37.5% of a well-known set of psychology effects, versus a 50% rate for human samples. More tellingly, it kept giving near-identical answers to questions that normally produce lots of human variety. Other work documented a “hyper-accuracy” quirk, where AI participants give unrealistically precise, noise-free answers no real population would.

2. They carry hidden biases. Because they learn from internet-scale text, AI participants tend to sound disproportionately Western, English-speaking, wealthy, and politically skewed. One study comparing AI-simulated public opinion against a global survey found it performed far better for Western, English-speaking, high-income countries while systematically misrepresenting attitudes across gender, age, education, and class. Worse, when asked to portray specific groups, models often produce caricatures. Research on such groups found AI depictions that actual human participants judged not just inaccurate but actively harmful. The uncomfortable implication: synthetic audiences are least reliable precisely where fresh insight is most valuable, namely underrepresented and non-Western populations.

3. They’re twitchy. Reword a question slightly, reorder the options, or tweak the persona, and the answers can swing dramatically. Studies showed AI survey responses with unstable estimates and strong sensitivity to wording, persona, and even timing. In decision-making tasks, AI agents proved far more sensitive than humans to small “nudges”: a tiny change in framing produced outsized changes in behavior. That fragility makes results hard to reproduce.

4. They can fake reasoning. An answer that looks thoughtful may just be pattern-matching on things the model absorbed during training. Researchers showed that changing one variable in a study (say, a product’s price) can quietly shift the model’s unstated assumptions about everything else, contaminating any cause-and-effect conclusion. In another case, near-perfect economic “forecasts” turned out to be memorized history rather than genuine prediction. Add to this the finding that hallucinations are, mathematically, an unavoidable feature of these models, not a bug that will simply be patched away.

How researchers are fixing them

The field isn’t standing still. Three families of techniques are making synthetic audiences more trustworthy:

• • Better prompting. Giving the AI richer personas (detailed demographics, life-story backstories, even real interview transcripts) measurably improves how well it matches a target group. The gains are real but fragile, since the same prompt-sensitivity problem lurks underneath.
• • Fine-tuning. Training models on real human responses for a specific population improves accuracy, though it tends to compress diversity unless done carefully.
• • Human-AI hybrids, the most reliable approach. Instead of replacing people, you collect a small human sample and use it to calibrate and correct the AI’s output. A statistical framework called prediction-powered inference makes this rigorous, and studies report cutting the required number of human participants by 20% to 30% in some settings, while another approach reduced the human data needed by up to 80%, all without sacrificing statistical validity. You keep much of the speed and cost savings while keeping the results honest.

The promising road ahead

What makes this field genuinely exciting isn’t just today’s tools; it’s where the research is heading. The review highlights three directions that could turn synthetic audiences from a clever shortcut into a dependable scientific instrument.

1. Open, transparent models built for human behavior. Most studies so far rely on closed, proprietary systems whose training data and updates are secret, which makes results hard to audit or reproduce (a “silent” update can change behavior between studies). Open models change that. The standout example is Centaur, a model fine-tuned on over 10 million real human decisions across 160 psychology experiments. It outperformed traditional models at predicting how people actually behave and generalized to new tasks, offering a glimpse of purpose-built, inspectable “model populations” rather than borrowed chatbots.

2. Standards, benchmarks, and reproducibility. Right now, every team prompts differently, making studies hard to compare. The field is moving toward shared benchmarks and tools, such as standardized environments for running AI-based surveys and experiments, so researchers can tell whether an improvement truly generalizes or just got lucky on one prompt. This is the unglamorous infrastructure that turned modern machine learning into a cumulative science, and synthetic audiences need it too.

3. Looking inside the black box. The frontier is mechanistic interpretability: tools that trace which internal “circuits” of a model produce a given behavior. The payoff would be enormous: imagine pinpointing the exact components that trigger a demographic stereotype and dialing them down, or verifying that a persona prompt genuinely activates the right knowledge rather than just changing the writing style. This work is early and mostly aspirational today, but it points toward synthetic audiences we can actually diagnose and correct, rather than merely observe.

Best Practice

If you’re tempted to use AI participants, the research suggests:

• • Evaluate the fit. Research shows promising evidence for early exploration, pilots, brainstorming, broad directional questions, and well-represented (Western, English-speaking) populations. Still a poor fit for fine-grained individual differences, underrepresented or non-Western groups, and high-stakes domains like policy or healthcare.
• • Explore with AI, validate with Humans. Even a modest dose of real-world data, used to check the AI’s answers, dramatically improves fidelity.

The bottom line

Synthetic audiences are one of the most exciting new tools in research, but they’re a tool, not a replacement for people. They’re best understood as a fast, cheap way to explore ideas and generate hypotheses, with their results treated as a starting point to be validated, not a final verdict.

Used carefully, they can democratize research and let small teams ask big questions. Used carelessly, they produce confident-sounding answers that quietly reflect an AI’s blind spots rather than reality. The difference comes down to knowing what they can and can’t do, and increasingly on the open models, shared standards, and interpretability tools that will determine how far this technology can really go.

---

This post summarizes the systematic literature review “Synthetic Audiences as Proxies for Human Users,” published in IEEE Access, which analyzed 100 studies on LLM-based synthetic audiences following established review protocols. It’s written for a general audience; see the full paper for methods, evidence, and the complete set of 100 citations.

Read the full paper: Lappas & Filippas, “Synthetic Audiences as Proxies for Human Users: A Systematic Literature Review,” IEEE Access, 2026. https://ieeexplore.ieee.org/document/11563566/

Sources & further reading

The specific findings mentioned above come from the studies below (a curated subset of the 100 reviewed). Links are provided where a stable one exists.

Applications, scale, and cost

• Argyle et al. (2023), “Out of one, many: Using language models to simulate human samples,” Political Analysis. https://doi.org/10.1017/pan.2023.2
• Hämäläinen et al. (2023), “Evaluating large language models in generating synthetic HCI research data,” CHI. https://doi.org/10.1145/3544548.3580688
• Kazinnik (2024), “Bank run, interrupted: Modeling deposit withdrawals with generative AI,” SSRN. https://doi.org/10.2139/ssrn.4656722
• Park et al. (2023), “Generative agents: Interactive simulacra of human behavior,” UIST (the “Smallville” study). https://doi.org/10.1145/3586183.3606763
• Piao et al. (2025), “AgentSociety: Large-scale simulation of LLM-driven generative agents,” https://arxiv.org/abs/2502.08691
• Aher et al. (2023), “Using large language models to simulate multiple humans and replicate human subject studies,” ICML. https://proceedings.mlr.press/v202/aher23a.html
• Xie et al. (2024), “Can large language model agents simulate human trust behavior?,” NeurIPS. https://arxiv.org/abs/2402.04559
• Li et al. (2024), “Frontiers: Determining the validity of large language models for automated perceptual analysis,” Marketing Science. https://doi.org/10.1287/mksc.2023.0454

Limitations

• Park et al. (2024), “Diminished diversity-of-thought in a standard large language model,” Behavior Research Methods (the 37.5% vs 50% replication finding). https://doi.org/10.3758/s13428-023-02307-x
• Qu & Wang (2024), “Performance and biases of large language models in public opinion simulation,” Humanities and Social Sciences Communications. https://doi.org/10.1057/s41599-024-03609-x
• Cheng et al. (2023), “CoMPosT: Characterizing and evaluating caricature in LLM simulations,” https://arxiv.org/abs/2310.11501
• Wang et al. (2025), “Large language models that replace human participants can harmfully misportray and flatten identity groups,” Nature Machine Intelligence. https://doi.org/10.1038/s42256-025-00986-z
• Gadiraju et al. (2023), “‘I wouldn’t say offensive but…’: Disability-centered perspectives on large language models,” FAccT. https://doi.org/10.1145/3593013.3593989
• Bisbee et al. (2024), “Synthetic replacements for human survey data? The perils of large language models,” Political Analysis. https://doi.org/10.1017/pan.2024.3
• Cherep et al. (2025), “LLM agents are hypersensitive to nudges,” https://arxiv.org/abs/2505.11584
• Gui & Toubia (2025), “The challenge of using LLMs to simulate human behavior: A causal inference perspective,” https://arxiv.org/abs/2312.15524
• Xu et al. (2025), “Hallucination is inevitable: An innate limitation of large language models,” https://arxiv.org/abs/2401.11817

Improvement techniques and human-AI hybrids

• Moon et al. (2024), “Virtual personas for language models via an anthology of backstories,” https://arxiv.org/abs/2407.06576
• Suh et al. (2025), “Language model fine-tuning on scaled survey data (SubPOP),” https://arxiv.org/abs/2502.16761
• Angelopoulos et al. (2023), “Prediction-powered inference,” Science. https://doi.org/10.1126/science.adi6000
• De Bartolomeis et al. (2025), “Efficient randomized experiments using foundation models,” https://arxiv.org/abs/2502.04262 (the 20% to 30% figure).
• Wang et al. (2024), “Large language models for market research: A data-augmentation approach,” https://arxiv.org/abs/2412.19363 (the up-to-80% figure).

Future directions

• Binz et al. (2024), “Centaur: A foundation model of human cognition,” https://arxiv.org/abs/2410.20268
• Shapira et al. (2024), “GLEE: A unified framework and benchmark for language-based economic environments,” https://arxiv.org/abs/2410.05254
• Sharkey et al. (2025), “Open problems in mechanistic interpretability,” https://arxiv.org/abs/2501.16496

What is Hermes ?

Hermes Agent is an open-source, self-hosted AI agent released by Nous Research, the lab behind the Hermes model family under an MIT license. Its main pitch is a built-in learning loop. Instead of resetting to zero every session, Hermes runs a post-execution review after each successful task, distills the steps that worked into a reusable, Markdown-defined “skill” and refines those skills the next time it hits a similar problem. It also keeps persistent memory across sessions, so it gradually builds a model of your projects and how you like things done, effectively learning through experience.

Unlike a copilot tethered to an IDE, Hermes is meant to live on a server and run unattended – a $5 VPS, a GPU box or serverless infra that costs almost nothing when idle. It talks to hundreds of LLMs through the OpenAI-compatible interface, can communicate via Telegram, Discord, Slack, WhatsApp, Signal, email and a CLI, supports natural-language cron for scheduled jobs, and can spin up subagents to parallelize work. It also ships with 40+ built-in skills out of the box.

What made it especially attractive to us is its ability to write and improve its own playbook. This ability made us think: could it learn how to do our job and fully automate the work we do at the Tech Pulse pod?

Our pod’s mission is to pick up state-of-the-art AI tools, experiment with them, and write an honest, evidence-backed assessment.

What makes our work tricky is the fact that every new tool we evaluate is different. We need to study it, figure out how to set it up, run it, and evaluate the results.

Our 6-step evaluation protocol

We begin by encoding our workflow into a strict, 6-step protocol that the agent can follow for every evaluation:

1. Workspace initialization – spin up a clean, isolated project environment so each evaluation starts from a known state.
2. Baseline replication – run the tool’s own “getting started” examples first, to confirm the headline claims reproduce before we push further.
3. Rigorous verification – design and run additional autonomous tests that probe the tool under conditions its authors didn’t pick, rather than extrapolating from the happy path.
4. Data synthesis & metrics control – measure the things that matter (recall, latency, accuracy) against ground truth data.
5. Adversarial peer review – hand the findings to a separate agent powered by a powerful LLM (Claude Opus 4.8), to receive feedback and iterate.
6. Finalization & delivery – format the assessment doc and ship it to the right channel (to our internal Notion knowledge base in our case).

The idea was simple: if these are the steps a human pod member walks through, can an agent walk through them unattended – and would the writeup at the end be any good?

The Skill system: how Hermes improves itself

The first time we ran Hermes on a real task, we walked it through the 6-step protocol by hand. Instead of just following along, Hermes wrote each step down as its own skill – a short Markdown file it can pull up at the start of any future run. So the protocol stopped being something we had to repeatedly provide as input; it became something the agent already knows.

Hermes used this knowledge to build a small set of skills covering everything we do: how to set up a clean workspace, how to test a new tool properly, and how to draft the write-up and run it past the reviewers. On top of those sits one master skill that holds the whole run together – it treats the 6 steps as a checklist and won’t let Hermes jump ahead before the previous step is actually done.

Impressively, when the LLM peer reviewer flagged something during an experiment (e.g. an unfair baseline, a missing caveat, a poorly designed experiment), Hermes would learn from the feedback and address the issue. It would also record its new learnings in the related skill files, so the next experiment started from a slightly stronger playbook.

That’s the self-improvement loop the Hermes pitch promises, and we actually watched it happen. The more experiments we run, the better the skills get, and the less we need to babysit Hermes for the next one. The underlying LLM that powers Hermes (Gemini 3.1 Pro) isn’t getting smarter – its playbook is, and Hermes is the one rewriting it.

Putting Hermes to the test

We gave Hermes a single, real assignment: take a brand-new open-source tool called Turbovec – which claims to store huge amounts of data in a tiny amount of memory and search it faster than the popular alternative – and find out whether those claims actually hold up.

We handed the agent the tool, its documentation, a bare cloud machine to work, and nothing else: no starter code, no template, no outline. Hermes had to decide what and how to test, run the experiments and, write the whole thing up on its own.

We reviewed Hermes’ output in the exact same way we would review a human colleague’s work, via three simple questions:

• Did it manage to set up the tool and run experiments? Yes! It wasn’t all smooth sailing. Hermes’s first pass used the wrong settings. One of the integrations that TurboVec advertised also didn’t work on the first try – a common challenge we face in the Tech Pulse Pod. However, rather than getting blocked by these stumbles, the agent noticed them, fixed them and left a clear trail of what went wrong and how it was corrected – exactly the kind of thing a rushed human reviewer might quietly skip over.
• Did it design a fair test? Yes! It first reproduced the tool authors’ own results, then set up an even-handed comparison against the leading alternative tool (FAISS), as a baseline. It was also careful enough to optimize the baseline tool’s configuration (rather than making it deliberately weak one), ensuring that the contest wasn’t rigged in Turbovec’s favor.
• Did it get the numbers right? Yes, with a bit of extra AI help. For inststance, its first attempt used a small data sample and then just assumed the results would scale up neatly. The Adversarial peer-review step (step 5 in our 6-step workflow) caught that this assumption was unsafe. Hermes accepted the criticism and re-ran the full-size test. The adversarial reviewer turned out to be right – using the small sample would have significantly skewed the results.
• Was the writeup appropriate? Yes, with a bit of extra AI help. Hermes’ original draft omitted some critical details and also inflated some of the findings. Thankfully, The LLM reviewer’s feedback in step 5 also ensured that claims got toned down to what the data actually supported.

So is it worth it?

Very promising. Left alone with a new GitHub repo and a blank machine, Hermes handled the mechanical, time-consuming work on its own: it downloaded the repo, installed everything, and ran some initial tests to make sure everything runs.

Even though it did stumble during the actual experimental design and assessment of the tool (TurboVec), the introduction of our second Reviewer agent was enough to address these issues and deliver, with no human in the loop.

Obviously this is just a single – albeit very promising – piece of evidence. We will keep pushing the limits of Hermes in the context of our pod’s work, with the intent to automate and scale-up our assessment efforts as much as possible.

Another angle we plan to explore is cost minimization. This first experiment showed the effectiveness of the iterative, dual-agent architecture:

• An affordable Gemini-powered Hermes to actually do the heavy lifting (open-ended, token-heavy)
• A more expensive Opus-powered Reviewer to review the report after each iteration and provide feedback (single-shot, token-lean)

A key question – and one that we keep facing in WPP Research – is: what is the cheapest LLM brain that we could use for each agent, while maintaining quality outcome?