Give the same piece of narrative text (e.g. a field note, a survey response, a news article) to two human analysts, or two AI models, and you'll often get two different answers. The usual response is to treat this as an error: better guidance, a bigger model, or more training data should eventually converge everyone on the “correct” interpretation.
In our work analysing narrative data across pharma, finance, national security, adtech, and people insights, we've found that assumption doesn't hold up. Across dozens of real-world datasets, we've identified 27 recurring reasons why human and AI analysts disagree on the same text — and most of them have nothing to do with model quality. They fall into six broad categories, and knowing which one you're dealing with changes what you should actually do about it.
The six reasons interpretation breaks down
A. The information genuinely isn't there
Sometimes a text is just too sparse to support a confident reading. A field note that says “Customer issue,” with nothing else to go on, is ambiguous no matter how good your analyst or model is. The only real fix is going back to the source, which usually isn't possible.
B. The information is there, but gets missed
This is a quality problem, not an interpretation problem. A relevant detail buried at the end of a long document gets under-read; an early data point anchors the analyst's judgment on everything that follows. Given the same text, a careful reader would reach the same conclusion — so disagreement here is a signal that something in the process needs fixing.
C. The meaning only shows up across the dataset
Some text can't be interpreted in isolation. “MDVs out of stock,” “MDCs not in stock,” and “Multi-dose vial stockout” are the same issue, but only if you see all three together and connect the typos and acronyms. Split those entries across different analysts or model runs, and the shared context that would have resolved the ambiguity disappears.
D. The meaning depends on context outside the text
Jargon and internal conventions are unambiguous to insiders and opaque to everyone else. “CR achieved after cycle 2” is obvious to oncology staff (Complete Response) and meaningless without that background. Get this wrong, and you risk expensive reclassification for what was really a clarity problem, not a capability one.
E. There's more than one valid reading
Sometimes a single underlying cause shows up as several different-looking statements — a supply-chain disruption logged as “delayed shipment,” “personnel shortage,” and “equipment fault” across three reports. Or the reverse: one phrase, like “issue with MD,” could mean muscular dystrophy or medical doctor. There's no single ground truth at the level of one response, but the distribution of likely meanings across the dataset can still be estimated.
F. The conclusion depends on where you stand
Occasionally the facts are clear and nothing is missing, but the conclusion still hinges on a value judgment. “Employee vacation days declined sharply this quarter” reads as good news (more focus) to one analyst and as a red flag (overwork) to another. More data doesn't resolve this — it's a disagreement about standpoint, not evidence.
Why this matters
Most classification tools, whether narrow fine-tuned models or general-purpose LLMs, treat every entry as if it should have one correct label, and treat disagreement as noise to eliminate. That works fine when the ambiguity really is a quality problem (category B). But when the real cause is A, C, D, E, or F, forcing a single answer either hides genuine uncertainty or produces an answer that quietly depends on whoever happened to do the labelling.
We built Wholesum to make that distinction explicit. Instead of judging each entry in isolation, we apply comparative statistical inference across the whole dataset, so we can identify which failure mode is actually present and how resolvable it is — rather than papering over the disagreement with a single label. That approach rests on four things: full contextual analysis of every entry against the wider dataset, detection of both common and niche disagreement patterns (with how strong and how resolvable each one is), reproducible outputs that stay consistent across runs, and the ability to hold that stability from thousands of entries to millions.
Seeing it in practice
We recently worked with Female Founders Rise, Barclays, and Nottingham Business School on a large survey of UK female entrepreneurship. Because founders share overlapping markets and funding conditions, individual responses reflected a shared causal structure rather than fully independent answers. Iterative analysis converged on four broad success themes — specific enough to distinguish key groups, broad enough to cover most of the population.
The interpretive challenges weren't the ones you'd expect. Low ambiguity (A) and minimal misreading (B) — the responses were short, typed, and from an engaged audience. Instead, the real work was in resolving UK-specific terminology from context across the dataset (C), pulling in external knowledge of UK funding schemes (D), and handling responses that genuinely supported more than one explanation (E). Even naming the themes involved an interpretive choice, reviewed by humans (F) — even where the underlying concepts themselves were stable across runs.
The output wasn't a single ground truth, but a defensible account of where the uncertainty actually comes from, which is often the more useful answer for high-stakes decisions.
If you're working with narrative data and want a clearer picture of where your own analysis is breaking down, get in touch.