When working with AI, an important starting point is to check whether it can handle the simplest possible datasets: unambiguous new examples with objectively correct answers. If it can’t get these right, you can't trust it with anything more complex.
A simple test illustrates the problem. Suppose we have twelve survey responses about work challenges. Six participants unambiguously mention team size as a problem. Now put the data into GPT-5 Thinking or Claude Sonnet 4.5 and ask it to summarise the main themes: 
This kind of dataset is pretty much as clean and unambiguous as we’ll ever get. If the model struggles with a very simple test like this, it’s a big red flag.
In contrast, this is what we obtain if we ask Wholesum to find the dominant themes in the dataset: :
| Theme | Match | Total valid | Percentage |
|---|---|---|---|
| Work-life balance and workload management | 6 | 11 | 55% |
| Insufficient staffing | 6 | 11 | 55% |
| Job dissatisfaction | 1 | 11 | 9% |
Unlike GPT-5 and Claude, Wholesum tallies up the staffing problems and work-life balance correctly.
If we scale up the test, the gap becomes even clearer. This kind of inconsistency is also why working with AI on qualitative data can feel so frustrating — you get different answers each time, with no way to know which to trust. For example, suppose we generate 1000 synthetic responses with known, unambiguous theme labels (work-life balance, team communication, technology issues).
An example response matching 'work-life balance' and 'technology issues' would be: "My kids barely recognize me because I'm always working. I need better compute to do my job".
We can then vary the simulated theme proportions and see how well different methods can recover the truth. In this benchmark, Gemini 2.5 Pro consistently undercounted themes while Gemini 3 bounced between being near the mark and under-estimating, sometimes wildly. In contrast, Wholesum stayed within a percentage point or so of the 'true' value (i.e. the one used to generate the synthetic data) across the benchmark: 
Because Wholesum processes data in a structured way, the time it takes to analyse data is also much more predictable than common reasoning models. Gemini 3 Pro took over 15 minutes to analyse 1000 responses in one of our tests. Wholesum took 3 seconds.
Spotting common themes isn't the only challenge. Sometimes we want to find things that might not appear much in the text dataset. The animation below illustrates a needle-in-a-haystack problem: finding festive-themed responses hidden among 10,000 survey entries. This kind of rare-signal detection is where LLM-centric approaches can collapse entirely — overwhelmed by volume and distracted by irrelevant context.
Why do LLM-centric approaches get things so wrong?
Many of the failure modes are surprisingly familiar, and similar to the ones human analysts might make:
- Anchoring on early examples. Just as people can over-weight the first information they encounter, LLMs latch onto salient themes in the opening responses and interpret everything else through that lens. We've measured this directly: content that appears earlier in the input is far more likely to influence LLM output.
- Ignoring full context. Even with long context windows, models don't reliably use all the data they're given. Evidence from later in a dataset is systematically under-weighted — they effectively skim rather than read. For example, we've found that counting ability can rapidly deteriorate as the number of input data points increases.
- Context sensitivity. LLM judgments can shift based on surrounding text, so the same response can be interpreted differently depending on what comes before or after it. This is why chunking large datasets to fit them into an LLM context window can produce highly unstable results.
- Reverting to prior assumptions. When a dataset is ambiguous or too large to interpret reliably, LLMs may fall back on patterns from training instead of strictly following the provided evidence. This can lead to summaries that include invented data, where prior expectations override the new information in front of the model.
- Struggling with task overload. Asking a model to simultaneously read, classify, count, and summarise is a lot to hold in one pass. Each step introduces error, and those errors compound.
- Inconsistency in method choice. This is one reason reasoning models can perform worse on analysis tasks: different runs of the same model on the same data can apply different implicit criteria — from keyword matching to semantic embedding to bulk LLM-based prediction — leading to results that can't be reproduced or meaningfully compared.
The hallucination sandwich
A common workaround within reasoning models is have two LLM calls - one to interpret the user's input question, one to summarise the result - with a structured tool in the middle. But this approach has two layers of hallucination risk. The first LLM can misinterpret the question and call the tool incorrectly; the second can misrepresent what the tool returned. Any accuracy the middle tool provides can be undermined at both ends. 
This doesn't mean LLMs are useless. But it does mean they're ill suited to tasks where accuracy and consistency are non-negotiable.
This is why we're building Wholesum. If you need analysis you can trust – especially at scale – language models that output skewed and inconsistent results will leave you with fragile, flawed analysis. Our architecture is built to avoid that from the ground up. Everything we do is grounded in a statistical framework, making analysis stable and reproducible.
Each response is analysed in the context of the entire dataset, meaning Wholesum can detect subtle signals that other tools – and even humans – typically miss. It also allows us to surface patterns that hold across groups and drivers that can be tracked and compared over time.