◆ DIAMOND DOES IT Indianapolis
Grounding detection for self-hosted LLMs

Your AI invents things that don't exist. Every check you have needs a document to compare against.

When a customer asks about a plan, a policy or a refund that was never real, there is nothing in your documentation to retrieve and nothing to compare the answer to. The check has no reference point, and the invented answer ships. We read the model's internal state instead — per entity, with no database.

Start with a Fabrication Audit mike@diamonddoesit.net
The gap

Comparison-based checks are blind to the emptiest failures

Groundedness scoring, RAG verification, LLM-as-judge — every one of them works the same way. Take the answer, retrieve the source documents, check whether the answer is supported by them. They are all a comparison against a back end.

That works well right up until the thing being described was never real. An invented plan tier appears in no document, so there is no context to retrieve and nothing to verify against. The fabrications your stack is worst at catching are the ones invented out of nothing.

01 The customer names something that doesn't exist A plan tier, a refund programme, a policy, an internal doc ID.
02 Your model answers anyway It is a constraint satisfier. Given a name with no referent behind it, it synthesises one that fits every surrounding constraint. The reasoning is sound; only the value was invented.
03 Your guardrails find nothing to flag No source document mentions the invented thing, so a groundedness check has nothing to check. The answer passes clean.
04 We read the model's interior instead Attention geometry, measured per named entity, before and independent of any lookup. Every referent we flag is re-checked on its own so a fabricated name can't drag a real product down with it.
Why it matters

A tribunal has already ruled on who owns what the bot invents

“While a chatbot has an interactive component, it is still just a part of Air Canada's website. It should be obvious to Air Canada that it is responsible for all the information on its website. It makes no difference whether the information comes from a static page or a chatbot.” Moffatt v. Air Canada, 2024 BCCRT 149 — the bot invented the terms of a bereavement refund. The “separate legal entity” defence was rejected.

The standard the tribunal applied was reasonable care. If you sell AI into regulated industries, your customers inherit that standard — and they churn when it is tested. This is the mechanism that makes the claim defensible.

Scope, stated first

What it detects — and what it does not

Most vendors make you find the limits yourself. These are the limits.

✓ In scope

  • Invented named referents — plan tiers, programmes, SKUs, policy names, document IDs
  • False premises about real entities
  • Any claim that depends on an ungrounded name — doubt propagates down the chain
  • Cases where the answer talks around a fake name without ever committing to it

✕ Out of scope

  • Well-attested misconceptions — those answers are grounded, in false consensus
  • Whether a quotation is worded correctly
  • Invented detail with no named referent in the question — that is your deterministic layer's job
  • Anything running on a closed API. We need the weights.

It ranks answers by grounding risk and, in the strongest cases, intercepts and rewrites them. It is not a pass/fail verdict and it is not instant — it is an overnight pass over your conversation log, plus an interception arm for live traffic.

Where it starts

A Fabrication Audit on your own domain

$5,000 — one engagement, credited against a pilot

I build a probe set shaped like your product line, where any specific answer is a fabrication by construction. You watch the catch rate yourself.

  • No labelled data required. Nothing to annotate, nothing to prepare.
  • Nothing leaves your building. It runs on your hardware, against your model.
  • You see the raw result — including what it missed.
Request the two-page brief
Who this is for

Teams running open weights on hardware they control

A white-box layer is only possible where the model's internal state is readable — which means self-hosted open weights on a vLLM or transformers stack. If you moved to open models for cost or data residency, you are the only segment where this can be built at all.

The qualifying question is simply: what are you running inference on?