The Knowledge AI Can't Find
There is a particular kind of expertise that never gets written down. It lives in people, and for most of the history of work, that was enough. You probably know what I mean. It is the pause a senior colleague takes before signing off on something. The workaround your team has been using for three years that exists in no documentation because everyone just knows. The philosopher Michael Polanyi gave it a name: tacit knowledge. His observation was deceptively simple: we know more than we can tell.
Now, suddenly, that matters in a very practical way. AI systems learn from what is written down. They are trained on the explicit, the documented. And this creates a problem that organisations building with AI are starting to face, often only once something has already gone wrong. The knowledge most worth encoding is frequently the least legible to the systems that need it.
Where the gap shows up
Once you know to look for it, you start seeing it everywhere. When teams try to design prompts or context for AI tools, they discover that specifying what "good" looks like is surprisingly difficult. The people who know what good looks like have simply never had to explain it before. Their judgment was enough.
Experienced reviewers can approve or reject AI outputs quickly because they are drawing on instinct built over years. That instinct does not translate easily into written evaluation criteria. Hallucination detection is perhaps the starkest example. Experienced people catch AI errors that less experienced people miss, not because they are more sceptical, but because they hold the tacit knowledge needed to notice when something is off.
Doing the work before you build
Here is the thing I have come to believe: most of the hard work needs to happen before you write a single prompt or automate a single step. The approach that actually helps is knowledge elicitation: deliberately surfacing what people know before building anything around it. It sounds obvious when you say it out loud. In practice, most teams don’t do it very well. The assumption is that you can course-correct as you go. And you can, but you will spend a lot of time fixing things you could have anticipated, and some of what slips through is subtle enough that you never catch it at all.
The method I find most useful is cognitive process mapping. Not asking people to describe what they do, but asking them to narrate what they think while they are doing it. There is a version of this in usability research called the think-aloud method, and it transfers well here. You are not after the official version of the process. You are after what actually happens, including the hesitations, the quiet double-checks, the things someone does without quite realising they are doing them.
The questions matter enormously. Generic questions get generic answers. The ones that actually surface tacit knowledge tend to go after the edges and exceptions:
What would make you stop and reconsider at this stage?
How would you know something had gone wrong before you could see the output?
What do you check that is not in the official process?
What would you tell a new colleague that is not written down anywhere?
Can you walk me through a recent example where this did not go to plan?
Asking someone to describe a process in the abstract gives you the version they think is happening. Asking them to walk through a specific recent example gives you what actually happens, including the adjustments, the workarounds, and the judgment calls that never made it into any documentation.
It also helps to map decision points rather than just sequence. Standard process maps show what comes next. What AI actually needs is a map of where judgment gets applied, what factors are being weighed, and what a wrong turn looks like. That kind of map is harder to build and rarely exists in organisations. But it is far closer to what a system needs to work reliably.
What resistance is actually telling you
When someone pushes back on an AI output, the "the output is crap" response feels like a dead end. It is not. It is one of the richest signals you have, if you are set up to catch it. That frustration means a gap exists between what the AI produced and what the person knows it should look like. That gap is tacit knowledge, briefly visible before it disappears back into instinct. If you just log the rejection and move on, you lose it entirely.
Most AI implementations are not designed to capture this. A thumbs down, a deleted output, a tool that quietly stops being used. These register as failure signals but carry almost no explanatory information. What was wrong with it? What would have made it usable? Without answers to those questions, you will keep reproducing the same gap indefinitely.
The questions that help here are simple: what specifically felt off? What would you have done instead? They tend to produce the kind of concrete, contextual detail that improves AI performance far more than abstract instructions ever do.
Patterns of resistance across a team are worth paying equal attention to. If multiple people are quietly not using a tool, or consistently editing outputs before anything goes anywhere, that is a signal worth investigating. It usually points to a context gap, a workflow assumption that was never validated, or a form of expertise that never surfaced in the first place.
The human in the loop is doing more than you think
Human oversight of AI systems is usually framed as a safety mechanism. That is accurate, but it undersells what is actually happening. When it is designed thoughtfully, the human in the loop is a tacit knowledge interface. A point at which judgment that cannot be encoded is applied anyway. Every time someone edits, corrects, or redirects an AI output, they are making tacit knowledge visible. Systems that capture and learn from those corrections accumulate something genuinely valuable. Systems that treat human review as a checkbox lose the signal entirely. The organisations that build well with AI will not be the ones that remove human judgment from the process. They will be the ones that understand where that judgment is doing irreplaceable work, and design around it accordingly.
Thought pondered by Sarah exploring the intersection of AI, creativity, and human wellbeing