Despite being popularised by Donald Rumsfeld, the term “unknown unknown” actually originated at NASA. The organisation needed to distinguish between “known unknowns” — like uncertainty about the composition of the lunar surface — and unknown unknowns, such as Apollo 12 getting struck twice by lightning on launch.
This distinction is also important when working with AI to analyse data. Leading out-of-the-box models (e.g. GPT, Gemini and Claude) are now very good at getting simple numbers out of short, clearly written portions of text once you have a detailed, unambiguous template for them to follow.
But not all unknowns are known in advance. The data may contain signals you didn't think to ask about, and models struggle a lot more if they have to use judgement to identify “important data” or if signals are scattered across multiple data points.
For example, when we pointed popular AI tools at last week's Federal Open Market Committee press transcript PDF, they were all good at filling a pre-defined template. But when we asked them to extract all the key numerical data, they retrieved anywhere between 25 and 62 data categories. If you're wondering why Copilot is so much higher, it's because it counts “weeks” (=7) and month names (January = 1) as numerical data.
This illustrates the importance of understanding the subtle choices a model makes behind the scenes, based on its prior biases.