The most common question I've been asked while building and selling Honeycomb is some version of: "How can I trust the numbers?"
It's the right question. If you're using an LLM to produce output that has to be not just plausible but verifiably correct - the kind people make consequential decisions on - then trust isn't a nice-to-have, it's the whole product. This piece is about how I've come to think about earning it: the infrastructure that sits around the model and decides what the system is actually willing to stand behind. The examples come from Honeycomb, but the approach isn't specific to it, and I think it generalises to any domain where extracted data has to be good enough to act on.
The question behind the question
When someone asks whether they can trust the numbers, they're usually not asking about model benchmarks or which provider you use. They're asking what happens between extraction and the moment a value shows up in a report, a dashboard or a calculation. Can they see where it came from? Can they tell how sure the system actually is? What happens when something looks wrong?
Those are reasonable questions, and they're the ones that matter commercially, because they're the basis on which someone makes a decision. They're also questions most LLM extraction pipelines aren't built to answer well. The default pattern is simple: the model extracts a value, the model assigns a confidence score, the value gets stored and used. That confidence score feels like a safety mechanism, but in practice it's the equivalent of the model grading its own homework.
Honeycomb started out extracting from documents that feed financial and technical analysis, so this was never abstract for us. If extracted values are going to flow into valuations, comparisons and reports that people act on, you need a clearer account of what was extracted, what was checked, and what the system is prepared to stand behind. That shaped how we designed the whole pipeline.
Verification comes first
The first layer I'd put in place - before almost anything else - is independent verification that runs on every extracted field before it's stored or used anywhere downstream.
Every extracted field arrives with a source quote: the passage the model says it took the value from. Before that value goes anywhere, the system checks whether the quote - or at least the value itself - actually appears in the source document. This is deterministic text matching, not another model call: normalised matching, with fuzzy matching for quotes that come back slightly paraphrased. No second model grading the first.
It's worth running that check against two independent representations of the same document: the model's own transcription, and a separate raw text extraction taken straight from the file. That's deliberate, because the two fail in different ways. The model produces fluent, coherent text but can misread a table or quietly smooth over an awkward layout; raw extraction preserves what's literally on the page but can miss things the model captured correctly. Checking against both gives better coverage than either alone.
The result isn't a single pass/fail. Each field is tagged with one of a few outcomes: the quote matched verbatim; the value is present but the quoted passage didn't match; neither was found; or there was nothing to check it against. Those are genuinely different situations, and downstream handling should treat them differently - a paraphrased source is not the same as a value that appears nowhere in the document, and collapsing the two into one flag quietly degrades both your confidence policy and your review experience. Whatever the outcome, that tag stays with the field permanently.
The real value here isn't only catching bad extractions, though it does that. It's making verification a recorded property of the data. When someone later asks why a particular value was treated cautiously, or why it was flagged for review, you can point to a specific check that ran at extraction time and exactly what it returned. That matters enormously the moment you're past the demo and into production.
Confidence should be a policy, not a model output
Verification gives you something concrete to base confidence on. So rather than trusting the model's self-reported score, apply a confidence policy: a set of rules, defined explicitly in code, for how confident the system is allowed to be given how a value was obtained and what verification found.
The model still reports its own confidence, but that only moves the score within a band the policy sets. The shape that's worked for us: a value that was directly quoted and verified is allowed into a high band - but never treated as ground truth. The same value, with a quote that couldn't be found anywhere in the document, is forced into a low band regardless of how confident the model claimed to be. A value the model inferred rather than quoted sits lower than a quoted one even when it verifies, because inference and quotation are different kinds of claim and shouldn't be trusted equally. The exact numbers matter less than the principle: you're encoding what your system is willing to stand behind, not outsourcing that judgement to a score the model assigns itself.
Store both numbers on every field - what the model reported and what the policy applied. That keeps the audit trail complete, and it's been useful well beyond extraction. When someone revisits a figure long after the fact, they can see both what the model thought and what the system decided. And when you're evaluating a model or a prompt change, it keeps an important distinction visible: a newer model being more confident is not the same as it being more correct. If confidence is all you track, those two look identical.
You won't find an external dataset that tells you the right bands for your use case, and you shouldn't try to define the whole policy upfront in one pass. Start conservative - bands that reflect how your domain treats quoted versus inferred evidence - run them on real documents, and tighten or loosen as review patterns show you where you're wrong. The policy stays readable, versionable and defensible, which is exactly what you want when the outputs feed real decisions.
I'd treat this as an early priority if you're building something similar. It doesn't depend on having the best model or the most polished prompts. It depends on a single decision: that the system's confidence is a function of what you can verify, not what the model asserts.
Domain rules belong in code
Verification tells you whether a value appears in the source. It doesn't tell you whether a set of values make sense together. Totals that don't add up, dates in the wrong order, a component that's somehow larger than the whole it belongs to - these are the things a domain expert catches in seconds, and they're a poor fit for prompts.
The way to catch them reliably is a post-extraction rulebook. One set of rules checks approximate consistency between related fields: does A times B roughly equal C, within a tolerance that allows for rounding in the source? Another checks definitional constraints - orderings that must always hold, percentages that can't exceed 100, a subset that can't be larger than the set it belongs to.
Ours grew far larger than I expected when we started, and it grew from real documents rather than from an upfront spec. We even added a test that fails if the rule count drops - which sounds pedantic until someone refactors the post-processing and quietly deletes half the checks. Domain experts own what "obviously wrong" means in a given context; engineering's job is to make those checks run on every document, every time, independent of whichever model you happen to be using this quarter.
In our case the rules are specific to the kinds of documents our users work with. But the pattern generalises to any field. Whatever domain you're in, the things your team would instantly flag as inconsistent can usually be written down as rules - and rules are far more reliable than prompt instructions for catching them at scale.
When a rule fires, we flag the field, adjust its confidence and surface it for review. We don't silently change the value - which brings me to the next point.
Flag, don't fix
That last point is a deliberate choice. When a system auto-corrects a suspicious value, the user ends up looking at something that appears extracted but was actually modified by a heuristic - and if the correction is wrong, and heuristics do get things wrong, there's no clean way to tell what happened.
So we flag instead. We point at where we think the problem is, suggest a correction, and preserve the original extraction. The user decides whether to accept the suggestion, and the audit trail stays intact.
There's one exception: we do normalise units when the quantity is the same and only the representation differs - but even then we record the original value and the conversion alongside the result. What we never do is guess what a value was "supposed" to be and write that in as though the document said it.
This adds friction. Users see warnings they have to engage with. But over time, that friction turns out to be part of what makes the data trustworthy, rather than an obstacle to it.
Measure accuracy in production
Everything above operates on individual fields. On top of that, every document we process gets a quality scorecard: how many fields were checked, what share verified cleanly, what share had values present but unverifiable quotes, and what share couldn't be found at all. We persist that per document and aggregate it over time at the organisation level - weighted by field count, so a small upload doesn't distort the picture from a large one.
This changed how we talk about extraction quality internally. We measure it on real production documents, not just benchmark numbers or eval sets. That gives you something benchmarks can't: a live record of quality that moves when you change a model, a prompt or a preprocessing step - and, crucially, it lets you tell whether a change actually improved verification outcomes or merely made the model report higher confidence. Those two look identical if confidence is all you track.
It also gives you a basis for deciding what to do with a weak document. How large a share of unverifiable fields you're willing to let through before you surface it for a closer look is a product decision - but you can only make that decision deliberately if you're measuring in the first place.
So if you're building for trust, pick a metric you can compute on every single run and store it from day one. The exact metric depends on your use case. It's the habit of measuring that matters.
Multiple sources need explicit handling
Most real workflows involve more than one document, and documents disagree all the time - an older report and a newer revision will carry different figures for the same thing.
When that happens, we score the competing values by how each was extracted, what verification found, and how recent its source is; pick an authoritative one; and keep the alternatives, along with a record of why the winner won. Larger conflicts get surfaced for review. And values are versioned over time rather than overwritten, so you can always trace which source a figure came from - and whether another source said something different.
This is as much a data-integrity problem as an LLM problem, but it's central to trust. Users need to know not just that a value was verified, but which document it came from and what the system did when another document disagreed.
What this doesn't solve
It's worth being explicit about the limits, because a system that oversells its own certainty is the exact thing this whole approach is meant to avoid.
Verification confirms that a value appears in the source document. It doesn't confirm the model attached it to the right thing - a number can be present on the page and still be mapped to the wrong field. Fuzzy matching, by its nature, can occasionally match something it shouldn't. And the rulebook catches values that are inconsistent or impossible; it does nothing for a value that is perfectly plausible, internally consistent, and simply wrong.
None of this makes the layers less worth building. Each removes a large, common class of error, and together they remove most of what actually goes wrong in practice. But the goal was never a system that can't be wrong - it's a system that's honest about how sure it is, and that leaves a trail when it turns out to be wrong anyway. Knowing where the edges are is part of being trustworthy, not a retreat from it.
How it comes together
On a typical upload, the pipeline runs end to end: extract each field with its source quote and the model's confidence; verify it against independent representations of the document; apply the confidence policy; run unit and plausibility checks; apply the cross-validation rulebook; compute the document scorecard; and store all of it with full provenance and version history. Everything downstream then works with confidence-weighted data rather than raw model output.
The strongest version of this, and the one that most directly answers the original question, is being able to take the numbers someone already trusts - their own - and check them against what the system independently extracted, then show exactly where and why the two differ. At that point trust isn't a claim you're asking them to accept; it's something they can audit for themselves.
Which is the whole point. Trust rests on what the system checked and recorded - not on asking anyone to take the model's word for it.
If you're starting from scratch
If you're starting from scratch, here's the order I'd recommend, based on what building this has taught me.
Prioritise verification and a confidence policy before you pour effort into prompt refinement or model selection. Those two layers give the most protection per unit of engineering effort, and they don't lock you into any particular model.
Get domain experts involved in writing validation rules early, even if the first version is tiny. A handful of rules that reflect how your team actually thinks about data quality will outperform a long prompt begging the model to police itself.
Measure from day one, even crudely. Production measurement is what lets you improve - and what lets you talk about accuracy credibly instead of defensively.
And avoid auto-correction. It makes demos smoother and production much harder to reason about.
The models will keep getting better - which is exactly why I wouldn't build the trust story around them. For any system where the outputs carry consequences, I'd bet on the infrastructure around the model: verification, policy, rules, measurement and provenance. That's the approach I'd take in any domain where extracted data has to be good enough to act on.