Synthetic users — five people who do not exist, reading this site

Five invented readers were walked through riskmandate.ai one screenshot at a time and interviewed at the end. Thirty screenshots, eleven unanswered questions, twelve findings, two of them blocking a sale.

Source: https://riskmandate.ai/synthetic-users.html↗

Five people who do not exist, reading this site twice.

Five invented readers were walked through riskmandate.ai one screenshot at a time, asked what they made of each screen, and interviewed at the end. Every one of them starts every run knowing nothing about RiskMandate, Agent Behaviour Policies, or what is for sale. The site has now been read twice — v1.20.1 on 16 September and v1.25.1 on 17 September, the same five readers both times: 60 screenshots, 60 steps, 37 questions the site did not answer, 29 places somebody got lost, and 32 findings. The second run is the one worth reading, because it says which of the first run’s findings survived.

The key is the whole credential.

Nothing in the vault is a secret: no real person, no customer data, no credential. It is published read-only so the method can be checked rather than described.

Derived one way from a vault key that is not published and never will be. The read key opens the vault and cannot change it. This vault, re5nrn3h, holds both readings — ten runs, two per reader — so the comparison is inside one vault rather than across two. The first study was published as its own vault and is still there, read-only, at a41174009cf6f3019147040ef0c759b6c84c86ffa37eb54af06011c161a12332:o3q6zhtr.

Everybody in it is invented, and the vault says so before you can scroll past it.

The personas are fictional, the names are fictional, and every word attributed to them was written by a language model reading screenshots. It is not user research, and no sentence in it is evidence about a real person.

What is evidence is the part a machine produced: the thirty screenshots, the URLs, the step order, the scroll positions, the viewport sizes, the measured word counts, and the page errors — all captured by driving Chromium against this site’s deployed tree at v1.20.1. Both halves are labelled on every screen of the vault, and the findings list is the part worth acting on.

Hand the agent the screenshot, not the DOM.

This is the whole idea, and the vault states it in one sentence: an agent that reads the DOM finds the buy button every time, and therefore finds no confusion — which is the only thing worth running this for. So the loop is deliberately crippled. The agent gets the same thing a person would have and has to work out what to do from it.

Seven interview questions follow, however the run ended. The two that earn their keep are where did you have to guess and what did you still not know at the end, because those are the two a real reader never tells you. The protocol is published inside the vault, which is what makes a second run next month comparable rather than merely later.

Five readers, twice each.

Every number below is computed from the run records rather than from the summary. The outcome column is what the reader decided at the last step, in their own words.

The questions and lost columns are the second run’s. Read the viewport column. The reader carrying the most urgent decision — a founder six weeks from a term sheet, for whom £500 is his own call — did the whole thing on a phone, and produced the most questions and the most confusion of anyone. That is not something the vault announces; it falls out of the table once the numbers are in one place.

The first run’s finding: the product is not named where it is sold.

The headline finding was a word, and it was measurable. Above the fold on a 1440×900 screen, the home page at v1.20.1 said — this table is a dated record of a page that has since been rewritten:

The words Agent Behaviour Policy first appeared at character 5,387 of 9,270 — 58% of the way down the page. Two of the five read “Buy one, from £5” as buying an insurance policy for five pounds. The insurance broker held that reading for four screens; the founder resolved it by accident, from the caption of the third button.

• The hero card made it worse. It was captioned A REAL POLICY · TEMPLATE, directly beside a headline about being insurable — the phrase most likely to be read as an insurance policy, on the one caption with room to say the product’s name instead.
• The page led with the step that does not exist. The audience cards began at 2,823px on a desktop and 4,947px on a phone — the fourth and sixth screenful. Every reader who was one of the three named audiences scrolled past two full screens of insurance argument before the page addressed them.
• Two sales were blocked by something that is not a price. The founder would have paid £500 from his phone: no page states how long any level takes. The head of engineering would have paid £1,500 the same day against a deal worth more than her runway: the payment rails are not built, and the site says so.
• The investor’s content is on the conference page. It exists and it is good, and a partner arriving outside conference season finds his own content filed under an event he is not attending.

And what worked, named unprompted by more than one reader: the connector-scope quotes with their sources and dates, the admissions this site keeps making against itself, and the four-number card — which was the only thing on the first screen that kept the engineer reading. “Nobody oversells and then volunteers that.”

The confusion the first run measured did not happen again.

Same five readers, same six-step loop, same window sizes, against the home page rebuilt from a business partner’s design. Measured on the page rather than asserted:

The insurance broker understood on the first screen what took her three pages before, and said why: “policy here is a document, and they have put the qualifier in front of the noun everywhere I can see.” The phone reader, who stalled at v1.20.1 without ever learning what was for sale, reached a purchase decision at step six. Four of the five would now spend money, against two of five. None of that is a sale; two of the three who would buy said the entry level is cheap enough not to be a decision, which is a compliment to the pricing rather than to the page.

• The word that was wrong is fixed; the vocabulary is now split. The home page says reach, mandate, gap, barriers and carries a note about the file names. The pages it links to use the older words in their own prose — the behaviour-policy page↗ says the grant and the delta, and for-corporate↗ says the grant. Three readers hit it within two minutes of the home page. The investor put it worst: “I am forwarding this to people who will not chase a footnote.”
• Every delivery window starts from a reply that has no clock on it. Levels 3 and 4 say 1 to 3 days and 1 to 5 days from your reply. The reader who would have paid £1,500 the same day could not: “one to five days from my reply still depends on when they reply to me.” The step with no time on it is ours.
• A read key may go to your board, your auditor or your broker — and to a customer? The sentence lists three recipients and the reader with a bank’s questionnaire is a fourth. Whether a read key can go to a customer, and be revoked after the review, decides that purchase and is not on the site.
• The investor still has nowhere to land. At v1.20.1 he found an audience card pointing at a page that did not exist. At v1.25.1 the audience cards are gone, so he finds nothing: no view card among CEO, CTO, CISO and insurer, no page, and no sentence saying the document has the same shape every time — which is the only thing he needs to ask thirty-one companies for one.
• Both dated 2026 facts are unsourced where most people read them. The insurance page↗ gives the exclusion’s announcement date and explains why its figures are deliberately qualitative. The home page states the same two facts with no source and no link, and the home page is the page people reach.
• One sentence about somebody else’s industry is unhedged. “What earns cover back is a written record…” states as settled a thing only an underwriter decides, on a page that hedges everything about its own product precisely. The insurance page’s conditional version is the one to carry. The same reader asked for a line the site does not have anywhere: RiskMandate is not an insurer and places no cover.
• And the small ones, measured. The eighteen vault pages say Buy this policy on their primary button, bare, straight after a home page that says behaviour policy twelve times. for-corporate↗ says See a real policy — the exact phrase two readers took for insurance last time. The header’s button says Get in touch on the home page and Book a demo on three others.

What no reader asked to change, named unprompted by more than one of them: the in design chip on the third step (“the single most credible thing on this site”), model-drafted and marked as such, not a compliance assessment, and the deposit split (“somebody who has been burned by a late delivery themselves”). Two design decisions were vindicated: the product’s name in the first sentence, and the price on the first screen — which v1.22.0 removed on principle and a partner’s design put back. The investor read the price before he read anything else, and it is the reason he will act.

Two mechanical corrections, both in the runner.

Neither was a finding about the site and both would have quietly falsified the study. The page grows while its webfonts settle, so a scroll issued too early lands short, clamped against a shorter document. And the site sets scroll-behavior: smooth, so scrollTo animates and a screenshot 280ms later catches the page mid-flight — a step aimed at 4,340px landed at 3,308px and showed a different section entirely.

Both are fixed in tools/runner.mjs in the vault, and every step now records the scroll position it actually reached. A run that lands somewhere other than where it was aimed is visible in the record rather than silent. This is the same class of problem as the one the first study found: the screenshots looked plausible, so nothing else would have caught it.

Two pages were shipping broken JavaScript.

Before a word of the study had been written, the first pass recorded a page error on every step of every journey: Unexpected token '}' on the home page and Unexpected end of input on the insurance page. The menu, the mobile drawer, the in-page scroll buttons and the enquiry button were dead on both, and the insurance page had been broken since it launched.

Nothing else caught it. The HTML still rendered, so both pages looked right in a screenshot; twenty-six tests, four npm run check gates and two further CI checks all passed, because not one of them parsed a line of the JavaScript the pages carry. Fixed in v1.20.2↗, with a test that parses every inline script on every page — verified by deliberately breaking a page and watching it fail. Driving a real browser and recording what it throws is the only reason this was found.

Two runs a day apart were the comparison.

The personas, the paths and the protocol were in the vault, so the same five readers were walked through the rewritten site and the answers read down a column as well as across. Every persona in this vault now carries two runs, and the reading app switches between them. That is the thing this exists to make possible, and it has now been done once.