Building Confidence in the Subjective World Model

Introduction

Every day, businesses make decisions that depend on understanding how people think, what they prefer, and what they are likely to choose. From developing new products to setting prices and designing customer experiences, reliable consumer insights are essential. Yet traditional research can be slow and costly, making it difficult to explore every question or alternative as decisions arise.

At Atypica, we are building a new way to conduct consumer research. Our Subjective World Model simulates how different populations would respond to business questions, allowing teams to explore consumer preferences, compare alternatives, and inform decisions without commissioning a new study for every question.

The model takes a description of the target population, a question or scenario, and the available response options, then predicts how responses would be distributed across those options.

The potential is significant: faster research, more questions explored, and insights that can be reused throughout the decision-making process. But realizing that potential depends on one fundamental requirement: the predictions must be reliable enough to inform real business decisions.

Judging a Simulation

To assess that reliability, we compare the model's predictions with actual human responses. We use three complementary metrics, each evaluating a different aspect of predictive performance relevant to business research.

Total Variation Distance (TVD)

Total Variation Distance measures how closely the predicted distribution of responses matches the human data. It calculates the overall difference between the predicted and observed proportions across all available choices.

A TVD of zero indicates an exact match, while larger values indicate greater differences.

For businesses, TVD is particularly useful when the size of a response matters. A pricing team, for example, may need to estimate not only whether customers favor a product, but approximately what proportion would consider purchasing it.

Top-Choice Accuracy

Top-choice accuracy measures how often the model correctly identifies the most preferred alternative in the human data.

For business decisions that require selecting one alternative, identifying the leading option can matter more than precisely estimating every response percentage.

A product team deciding which feature to prioritize, or a marketing team choosing between campaign concepts, needs to know which alternative is most likely to perform best.

Ranking Agreement

Ranking agreement measures how consistently the model orders alternatives in the same way as actual human responses.

Rather than evaluating only the leading option, it examines whether pairs of alternatives are placed in the correct relative order.

Ranking agreement is particularly valuable for business decisions involving multiple alternatives, such as prioritizing a portfolio of product ideas, comparing campaign concepts, or deciding how to allocate resources across opportunities.

Together, these three metrics help us evaluate predictions according to how they will be used. Some decisions depend on estimating the size of consumer demand; others depend on identifying the strongest option or establishing a reliable order of preference.

Why a Low TVD Is Not Always Enough

Consider a marketing team choosing between three headlines for the same article.

A forecast estimates a click-through rate for each headline, which the team could use to decide which version to publish.

A small distribution error does not always translate into a good business decision. This is why Atypica evaluates simulations using multiple metrics, considering both the accuracy of predicted proportions and whether the model correctly distinguishes between alternatives.

What this looks like in business

Evaluating a price change

A pricing team needs to understand how much interest it could lose when a price rises. For the same six-pack of Coca-Cola Zero Sugar, the model predicted a decline in purchase intent from 59.0% at $1.04 to 18.0% at $9.32. Human responses fell from 57.2% to 13.7%. Comparing the response at each price helps a team identify which offers to investigate before a pricing decision.

Designing the service experience

Should customers have the option to pay extra for human service rather than interact with a machine? Understanding support for that arrangement helps teams research how an automated service should be designed. In the survey, the model placed “favor” first at 39.0%; it was also the leading human response, at 40.3%.

Choosing a headline

A publisher can tell the same story in several ways, but only one headline can lead. Predicting which version earns more clicks helps an editorial team prioritize alternatives to test. For three headlines promoting the same article, an earlier click-trained version of our model matched the observed order and estimated click-through rates of 1.43%, 0.87% and 0.73%, compared with 1.48%, 0.82% and 0.70% in the human data.

Conclusion: Making Consumer Research Ready for Business Decisions

Business decisions are moving faster than ever. Product teams develop new concepts, brands test new messages, and companies adjust their offerings continuously. Yet traditional consumer research often struggles to keep pace: each new question can require additional recruitment, time, and expense. The Subjective World Model offers a different approach: a reusable simulation capability designed to help teams explore consumer responses faster and at lower marginal cost.

For this approach to become a dependable part of business research, reliability must come first. At Atypica, we evaluate our predictions against actual human data, using multiple metrics to assess not only how closely we estimate consumer responses, but also whether those predictions support the decisions businesses need to make. Building confidence in these results is central to how we develop our model.

As we move forward, we will continue expanding the range of business scenarios our model can address, validating predictions against human data, and improving performance across the measures that matter for each decision. Our ambition is to make reliable consumer simulation a practical, everyday capability for businesses—one that helps teams move from questions to informed decisions with greater speed and confidence.