Methodology: how we validate the simulated customers

What we measure against

We use Twin-2K-500, a public benchmark built from thousands of real people who each answered a large battery of survey questions. Because their real answers are held out, we can build a persona, ask it the same questions, and score it against what the actual person said. Nothing is graded by us or by another model. It is graded against real human responses.

The test is pre-registered: the benchmark, the exact measures, and what we expected to see were all written down before we looked at the results. That is what stops a number from being quietly reshaped after the fact.

Where it's strong

~85% It answers like a real person. Our simulated customers reproduce real people's held-out answers about 85% as consistently as those people answer themselves. In plain terms, a simulated customer is about as steady and coherent as a real one taking the same survey twice.

They behave, they don't just talk. They move through a real product, stall, abandon, and refuse at a price, and you can replay any session and watch it happen. This is the signal we lean on hardest, because it's the one you can check with your own eyes.
The number is public and pre-registered. It holds up when a skeptic pokes at it, not only when we present it. We publish the benchmark and the measures, not just the headline.

Where it's currently weaker

It reads the group better than the individual. "How will customers like this react to my pricing?" is a question it answers well. "What will this one specific person do?" is harder, the way any single human is hard to predict. It's also stronger on actions than on feelings: what people do and where they drop off is solid ground, while the finer emotional read is softer. So we treat those softer signals as leads to confirm, not settled facts, and it's the part we're sharpening now.

How every finding is labeled

So you always know how much weight a finding can carry, each one is marked:

EXECUTED observed behavior you can reproduce yourself.

INFERRED / TESTIMONY a claim worth checking, not a fact to bank.

We keep testing in the open

This isn't a one-time result. We keep re-running the benchmark as the program grows and update this page when the numbers move. And when the evidence says an idea shouldn't be built, that is the verdict you get, with the reasons. A clear no in week one is the cheapest thing we sell.