Simulate the human response to any business decision.

Abstract— AI consumer simulation builds models of real consumers from behavioral data, then researches with those models as one would with human respondents. The field is new and largely unregulated in how it measures its own accuracy. Over one year we recorded the technical questions raised in 36 closed-door evaluations by consumer research teams across eight countries and six sectors. Setting aside commercial discussion, 150+ distinct concerns remained, resolving into four categories.

1 Method

Each evaluation was a working session in which a research team assessed whether synthetic consumers could take part in understanding their customers. Questions were recorded as asked, then coded and grouped by what the asker was trying to establish; the unit of analysis is a concern, not a meeting.

2 Data sourcing

Skepticism here is not about AI but about online data as evidence of behavior, and it arrives in three layers. The first is noise: teams describe trying to study their own category and finding a wall of sponsored content. The second is who is actually speaking — an influencer account is a business, the persona manufactured and the content scripted, so a model built on that material may be learning a performance.

The third is structural absence: the people generating the data are categorically not the people being studied. The users of adult incontinence products are in their seventies and absent from social media, while the accounts discussing the category belong to their children. B2B has the same gap.

3 Simulation fidelity

Half of all concerns land here, and the word conceals three questions. The first is individual reconstruction, where teams separate consistency from accuracy sharply: whether a model answers the same way twice is measurable, and beside the point next to whether what it infers matches what that person actually thinks. A related objection is distributional — real decision distributions are not normal.

The second is capability boundaries, where scenarios are refused on principle rather than on quality: sensory judgment of food and drink, reaction to an interface the model has never touched, products not yet launched. Each needs first-hand experiential cognition.

The third is knowledge reliability. Domain error is caught at once by experts, and tolerance for it is asymmetric: the same mistake is sampling noise from a human respondent and systemic risk from a machine. It is also the only category where teams shift from challenging to co-building.

4 Differentiation and access

“Why not do this with a general-purpose model” arrives after interest, not before it, and is a procurement question rather than a technical one. The answer that lands is about the standards of evidence a tool is held to, and whether they are visible. Access concerns are narrower: data flow, sovereignty, and whether input feeds external training.

5 Discussion

Traditional research earns trust through a disclosed method — how the sample was built, how respondents were recruited, where error came from. AI simulation has no equivalent section, and these four categories are what buyers assemble in its absence: the same four in Shanghai as in Zurich.

Abstract— Coding is the one task where AI agents have moved from demonstration to daily use, and it is where research effort has concentrated. Most businesses do not run on code; they run on research, and what people want from an agent in that work has not been mapped. We invited users of a commercial research platform to describe, in their own words, what they were trying to accomplish, and analyzed 1,338 of the resulting needs. Twelve business decisions recur, 67% of needs name two or more of them, and nearly all sit before a product exists.

1 Data and coding

Needs were volunteered as free text. Classifiers labeled each along three dimensions — the deliverable wanted, how the research should be done, and the business decision served; this paper concerns the third. Labels are multiple rather than exclusive, because one need frequently serves more than one decision.

2 Timing

These decisions share a position in time: they are asked before a product exists. The one exception, creative testing, is also the least common in the corpus. Commercial research spend has the opposite shape — in ESOMAR's accounting, satisfaction and CRM tracking are the largest category, and both happen after launch. That category appears nowhere in this corpus. The two measures are not equivalent, but the absence is total, not marginal.

3 Co-occurrence

Most needs do not stop at one decision. On average each touches just over two, and Table 2 gives the pairs that most often arrive together.

4 Implication for agent design

The most common pair is also the least compatible. Audience profiling calls for demographics, behaviors and pain points, usually through interviews; competitive analysis calls for positioning, benchmarking and share-of-voice tracking, usually through desk research. A team would staff the two differently and judge them by different standards.

The parts are also rarely independent: the output of one is often what the other needs to start. A first pass at who the audience is reshapes which competitors matter, and the reverse. A coding agent's method does not change with the number of features requested — write, run, check, repeat. A research agent gets no such invariant.

5 Limitations

The sample is self-selected — people who already reach for AI to do research. Labels were assigned by a model, not adjudicated by human coders. Each record is a stated need at the moment it was asked, not the work that followed.

Abstract— We specify a protocol for evaluating AI personas at the level of the population rather than the individual: whether the structure of a persona set approximates that of a human one — how dispersed it is, which consumer types are present, and how far apart they sit. Ground truth is 1,221 human respondents; 100 to 200 personas answer the same 28-item common core instrument. Three arms are compared: personas improvised by a general model, personas given demographics only, and personas grounded in social media.

1 What is and is not tested

The hypothesis is structural correspondence,

Structure(Pₐ) ≈ Structure(Pₕ)

and the claims tested are that the persona population is as dispersed as the human one, that it reaches both the major and the minor consumer types, and that it is not over-homogenized toward an average. One-to-one fidelity is not tested and equal sample sizes are not required, so the adjusted Rand index is not used: it presumes a correspondence between the two sets of units that does not exist here.

2 Populations and sampling

Personas are obtained either by sampling an existing library under a fixed retrieval condition, the closer analogue of sampling a social-media population, or by running several observation tasks and merging their output, which better answers which types get discovered at all. In the second route a fixed number per task means counts cannot be read as market shares.

Where demographics exist, personas are stratified against the human sample on city, age and sex only. Stratifying on outcome variables — price sensitivity, brand preference — would manufacture the heterogeneity being measured. Absent demographics, sampling is random and the object of study is the library's own structure, not national representativeness.

3 Metrics

4 Coverage and rare types

Human clusters from Gower distance with PAM form the reference, and each persona is assigned to its nearest medoid. A type is major at 10% or more of the human sample, minor between 2% and 10%. A cluster counts as covered when personas assigned to it sit within the 95th percentile of its internal distance — assignment alone only says a persona was nearer this medoid than any other. For rare types,

P(miss) = (1 − p)ᵐ

is reported too, so a miss is not read as a failure of the method when it follows from sample size.

5 Sample size and baselines

With m personas against 1,221 humans the two sides' metrics are not directly comparable, since dispersion and cluster count both move with n. Human metrics are recomputed on 500 to 1,000 resamples of size m, and each persona metric is read as a position in that sampling distribution. The three arms run on the same pipeline, staged as a technical run, a 50-persona pilot, then the full study.

Abstract— Synthetic consumers are usually evaluated one respondent at a time: a persona beside a person, two answer sheets scored item by item. That is not the evidence a commercial study runs on: a study delivers a conclusion, not answers, and reads it off a population. This protocol moves the unit of evaluation twice: from the individual to the population, and from the item to the whole chain running from interview to conclusion. Two abilities are evaluated end to end, opinion expression and cognition prediction. Ground truth is 1,221 human respondents; 100 to 200 personas answer the same 28-item common core.

1 The unit of evaluation

One evaluation is a commercial question carried end to end on both sides: the full user-interview methodology runs against the human population and against the persona population, and the two conclusions are what get compared.

Conclusion(Pₐ | Q) ≈ Conclusion(Pₕ | Q)

Item-level agreement is kept as a diagnostic, not a pass criterion; individual correspondence is not tested at all. Human research never asks two respondents to agree with each other, it asks a population to yield a stable conclusion. An evaluation that demands of a persona what it would not demand of a person is measuring the wrong thing, and that is where most published comparisons still sit.

2 The two abilities

Opinion expression asks whether the positions present in a population get voiced at all, and in what proportion — including those held by few. It is read against the distribution in the human sample, never against a position's correctness. Cognition prediction asks whether a commercial question — a concept test, a packaging choice, a price band — comes back with the same ordering of options and a comparable margin between them.

3 The common-core instrument

Comparing conclusions still requires both sides to answer the same thing under the same coding rules, or a difference in conclusions cannot be told apart from a difference in questions. The human data comes from a 77-item commercial survey with city-level branching and conditional items; administered whole it does not yield comparable matrices, so it is reduced to a 28-item common core. Out go identity fields, income, city-specific brand items, modules too sparsely completed to fill a cell, open text as a cluster variable, and any item whose branching cannot be reproduced identically on both sides — that last yields two different questions carrying one number.

4 Pilot criteria

The failure modes these criteria catch were seen in early runs: every persona answering rationally and near the middle of every scale; one contradicting itself between items; multi-select answers exceeding the stated option count. Each survives a casual read and corrupts a conclusion silently.