Ask an LLM to "pick a random number between 1 and 100" enough times, and something strange happens: the answers are not random at all. One model loves 42 and 73. Another keeps returning 47 and 57. The pattern is stable, reproducible — and different for every model.

In July 2026, a paper turned that quirk into a practical security tool. One Token Is Enough: Fingerprinting and Verifying Large Language Models from Single-Token Output Distributions (Tomáš Bruckner, arXiv:2607.10252) shows that these answer distributions form a reliable behavioral fingerprint — enough to check, from the outside, whether an API endpoint is actually serving the model it claims. No weights, no logits, no insider access. Just a few hundred one-word answers.

This guide explains why that matters if you buy LLM access through any intermediary, how the method works in plain terms, and how to run the same verification yourself in about two minutes with a free browser tool.

The problem: you can't see what's behind an API

When you call an LLM API, you send text and get text back. The model field in the response says whatever the server wants it to say. Nothing in the protocol proves that your request for claude-opus or gpt-5 was actually served by that model — or by something a tenth the price.

That gap matters more than ever, because more of the market now sits behind intermediaries:

Aggregators and routers (OpenRouter and similar) resell hundreds of models through one endpoint, sourcing capacity from many upstream providers.
Resellers and regional proxies offer discounted access to flagship models, often dramatically below official pricing.
Third-party hosts serve open-weight models under their own infrastructure, with quantization and serving-stack choices you never see.

Most of these businesses are legitimate. But the incentive to cheat is obvious: silently swap an expensive model for a cheaper one, keep the price difference, and hope nobody can tell. Researchers at CISPA documented the problem at scale in Real Money, Fake Models (arXiv:2603.01919), an audit of 17 "shadow API" operators — several endpoints they tested did not pass verification against the models being advertised.

And it is not always about fraud. Providers can quietly quantize a model to cut serving costs, roll out a silent version update, or route your traffic across mixed backends. If your product's quality depends on a specific model, "what am I actually getting?" is a question you should be able to answer with evidence.

The insight: LLMs are terrible at random numbers

The verification method rests on a well-documented weakness: language models cannot produce uniform randomness, because they don't compute — they predict. When asked for a random number, a model reproduces the biases of its training data and preference tuning:

42 is massively over-represented — the "Answer to the Ultimate Question" from The Hitchhiker's Guide to the Galaxy echoes through decades of internet text.
7 carries centuries of cultural weight as the "lucky" pick humans favor.
37, 47, 73 and other two-digit primes feel random to humans, so they dominate human-generated "random" examples the model learned from.

The paper quantifies just how collapsed these distributions are: across its probe set, the median answer distribution carries only about 1.0 bit of entropy. A truly uniform pick from 1–100 would carry 6.64 bits. In plain terms: where a fair die has a hundred faces, most models behave as if they were flipping a slightly weird coin.

Here is the key move: that failure is consistent. The same model produces the same skewed distribution every time you sample it — and different models (even sibling versions of the same family) produce measurably different ones. A bug becomes a signature.

How fingerprinting works: one token is enough

The protocol in the paper is simple enough to summarize in four steps:

Probe. Ask the endpoint a battery of one-word questions — "pick a random number between 1 and 100", "name a random color", "flip a coin", and so on. The paper uses 10 tasks × 4 languages = 40 probe units, sampling each 30 times at temperature 1.0 with max_tokens=16 and reasoning disabled.
Fingerprint. For each probe unit, tally the answers into an empirical distribution. The collection of distributions is the endpoint's behavioral fingerprint.
Compare. Measure the distance between this fingerprint and a trusted reference fingerprint of the claimed model, using Jensen-Shannon divergence (base 2, so the scale runs from 0 = identical to 1 = disjoint).
Decide. Small distance → consistent with the claimed model. Large distance → the endpoint is behaviorally a different animal.

The reference distances give the scale meaning: the same model compared against itself lands around 0.14; the same model served by two different providers around 0.23; two genuinely different models around 0.46. There's usable daylight between "same" and "different".

Accuracy is strong but not perfect: with just 8 probe units the equal error rate is 10.6%; with the full 40 it drops to 7.3%, with an AUC of 0.971. Two properties make the approach hard to game. First, it needs no special access — anything that answers chat completions can be fingerprinted. Second, there is no magic string to filter: every probe is an ordinary, harmless question drawn from paraphrase pools, so a dishonest middlebox cannot easily detect and special-case the test without breaking normal traffic.

What the paper found in the wild

Two published findings are worth knowing about.

The palmyra-x5 case. The paper's most striking result concerns writer/palmyra-x5, offered on OpenRouter as a proprietary flagship model. Its fingerprint sat at a JSD of 0.141 from the open-weight Qwen3-235B — statistically indistinguishable from the ~0.140 you get comparing a model against itself. Behaviorally, the paper concludes, the endpoint was serving something functionally identical to the open-source model. (We report this case as published in the paper; we have not independently verified it.)

Same model, different providers — sometimes suspiciously different. Of 34 same-model pairs served across different providers, 10 pairs diverged beyond the 5th percentile of the impostor distribution — meaning some "official model, third-party host" deployments drift far enough (quantization, serving stack, hidden system prompts, or something else) to look like different models. Verification, in other words, is not paranoia even when nobody is lying about the model name.

The research artifacts are open: the fingerprint dataset (CC-BY-4.0) and code (MIT) are on Zenodo, so anyone can reproduce the results or build on the references.

We have also republished those reference fingerprints in a form you can simply browse: the LLM behavioral fingerprint database shows each of the 167 models on its own page — favorite random numbers, favorite colors, coin-flip bias, and the full answer distributions behind them — no dataset download required.

How to verify your own endpoint (free, in the browser)

Reading the paper is one thing; running 200 API calls, tallying token distributions, and computing Jensen-Shannon divergences by hand is another. So we built the method into a free tool: the LLM API Fingerprint Checker.

A few design choices worth knowing before you paste in an API key:

Browser-only. The probes are sent by your browser directly to the endpoint you configure. Your API key never touches our servers — you can watch every request in the DevTools Network panel.
Grounded in the published method. Same protocol (temperature 1.0, max_tokens=16, one-word probes), with verdict thresholds anchored to the paper's baselines, and built-in reference fingerprints collected under the identical prompt setup.
Free and bilingual (English/中文), with no sign-up required for a check.

Running a check takes three steps:

Point it at the endpoint. Enter the base URL, API key, and the model name you want to audit. Any OpenAI-compatible chat-completions endpoint works.
Pick a reference. Compare against a built-in fingerprint of the claimed model, or against a second endpoint you trust (e.g., the official API) — the tool fingerprints both sides under the same conditions.
Run and read the verdict. A standard check runs 8 probe units × 25 samples ≈ 200 tiny requests (roughly $0.005 at gpt-4o-mini pricing — the tool shows a cost estimate before starting). You get per-probe distribution charts, the JSD distance placed on the paper's baseline scale, and a three-way verdict: match (≤ 0.25), uncertain (0.25–0.35), or mismatch (> 0.35).

Beyond the headline verdict, the tool runs auxiliary forensics that catch things a single distance number misses: split-half stability (a fingerprint that disagrees with itself suggests multi-backend routing), completion-token padding (one-word answers shouldn't cost 200 tokens), suspicious response caching, and prompt-token inflation (a sign the endpoint injects a large hidden system prompt). If the endpoint blocks browser requests via CORS, the tool generates an equivalent curl script so you can probe from your own terminal instead.

How much can a hidden system prompt actually bend a fingerprint — and can the method survive it? We measured exactly that with 5,660 controlled injection requests, including feeding Claude Code's entire system prompt to GPT: the results are worth a read if you rely on any relay or aggregator.

There is also a lighter mode: an AI randomness checkup that scores any endpoint's entropy against the 6.64-bit uniform ideal — a fun way to see your model's favorite numbers, and a live demonstration of why the whole method works.

For what it's worth, the happy path is common: in our own testing, an official gpt-4o-mini endpoint and the same model routed through a major aggregator matched at a JSD of about 0.12 — comfortably inside same-model territory. Where we have seen public endpoints deviate significantly from official references, we treat that as exactly what it is: a statistical observation warranting closer inspection, not an accusation against anyone.

What a result does — and does not — mean

Be precise about what this method can tell you:

A mismatch is evidence, not proof. The method has an inherent error rate (≈10.6% EER at 8 probe units). Legitimate factors — aggressive quantization, a silent model update, a stale reference fingerprint, hidden serving-side prompts — can all shift distributions without any intent to deceive. Treat a red result as a reason to re-run with more probe units, test a second reference, and ask your provider questions.
A match is strong but not absolute. A sophisticated impostor could, in principle, mimic another model's distributions — though doing so across dozens of paraphrased, multilingual probes while serving normal traffic correctly is much harder than it sounds.
Reasoning models need care. Fingerprints are collected with reasoning disabled; when an endpoint cannot disable it, results carry lower confidence (the tool flags this automatically).
Nothing here identifies wrongdoing by any specific vendor. A distribution distance is a property of an endpoint at a moment in time, not a verdict on a business.

The bottom line

The model field in an API response is a claim, not a guarantee. There is now a cheap, open, statistically grounded way to check that claim from the outside — built on nothing more exotic than the fact that language models can't say a random number to save their lives.

If you buy LLM capacity through any intermediary, verification belongs in your toolbox next to uptime monitoring. Run a free fingerprint check on your endpoint — two minutes, half a cent of API credit, and your key never leaves the browser.

Where Model Verification Fits in an AI Presentation Stack

It may seem like a detour for an AI presentation tool to publish a guide on API forensics, so here is the connection. Tosea.ai's core product is document-to-PPT conversion: it reads a PDF — a research paper, a financial report, a course handout — and an LLM turns it into an editable slide deck. That pipeline is only as good as the model actually serving it. If a provider silently swaps in a quantized or cheaper model, the damage shows up exactly where deck quality lives: sloppier outline structure, mangled table extraction, and a higher hallucination rate in the generated slides — the failure mode we work hardest to prevent, as we detailed in our zero-hallucination AI slides guide.

That is why model verification is part of our own engineering routine. When we benchmark models for outline generation, or route bulk slide-generation traffic through aggregators for cost reasons, fingerprint checks are how we confirm the capacity we pay for is the capacity we get. Anyone building a serious AI slide generation product — or any LLM product with quality-sensitive output — has the same silent dependency, whether they have measured it or not.

For readers who arrived here from the API-verification side and are curious about the presentation side: the same "trust, but verify" instinct applies to AI-generated decks. Our research paper to slides workflow shows how we keep generated slides traceable back to the source document, and our roundup of free AI PPT generators is an honest look at what free tiers (ours included) do and don't give you. The fingerprint checker sits in the same free-tools family — our way of giving something useful back to the community that reads the papers with us.