Human in the Loop: how to transform a job with AI

As a rule, we identify a job by its name or by the outcome we expect from it. Yet to understand how that work is done, we need to distinguish the actions that make it up. What inputs or data does the person receive? What information do they seek? What leads them to choose one course of action and discard another?

Some actions transform information; others produce something in the physical world. Decisions connect what we know with what should happen next in pursuit of an objective. Breaking down that set of actions and decisions helps us recognise where and how each capability contributes and what we need to preserve when reorganising the work.

Comparing two quantities may require no more than a conventional application. Recognising a pattern may require machine learning; interpreting a request or preparing an explanation may benefit from generative AI. In each case, the choice starts with understanding what transformation is needed and what information is available.

The ability to carry out a task also depends on conditions. A tool may work well with complete data and need help when a reference is missing. Alongside allocating tasks, we therefore need to recognise when each part can proceed on its own and when someone else must intervene.

We call this human involvement within an automated process Human in the Loop. The expression becomes meaningful when we can identify where a person’s attention is needed, what they receive and what they must return for the work to continue.

The agent uses an AI model and tools to look up information or carry out actions. The person contributes the capability that this part of the work requires. For the exchange to work, both must interact within the context of a comprehensible task: what remains unresolved, what makes it possible to resolve it and what situation changes with a response from either one.

Consider a reply that requires expert knowledge. The agent can find information, connect it and prepare a draft. The person’s work changes: they receive a proposal accompanied by the material needed to understand it and judge whether it addresses the issue.

For the review to have value, the expert must be able to change what happens next. If an action requires approval, the agent waits for the human decision. A correction from the expert returns to the process; uncertainty may call for another enquiry. Keeping a record of these exchanges lets us review how the system works and make subsequent engineering improvements, even though the model does not automatically learn merely from receiving them.

Here we encounter a human capability that differs from expert judgement: acting on material objects and situations. In a warehouse, for example, the agent can indicate a location and the worker can move the item, record the result or report what prevented them from doing so.

We also need to understand what the system can and cannot verify. A scanned code may confirm that a label was read; on its own, it does not prove where the box actually ended up. A photograph may provide another signal. We design the check around what we need to observe and turn to a person when the evidence is insufficient.

If the agent can proceed within the expected conditions in most cases, human involvement concentrates on exceptional matters that need additional judgement. The person can then move from having to resolve every case from beginning to end to attending only to unusual situations arriving from several simultaneous executions.

That possibility depends on a concrete relationship: how much attention is requested from the expert and how much attention the expert can provide. It also matters whether enquiries can be queued and wait if they arrive simultaneously. A visible queue, with owners and priorities, helps organise that workload and recognise when human help is needed.

The agent can gather information, prepare cases and maintain their state while different people contribute what is needed from the human side: a review, a missing piece of information or an action. This combines the system’s speed of preparation with the team’s capabilities.

Preparing a task and being able to perform it are different capabilities, however. If the system generates more work than people can handle, pending tasks start to accumulate. Organising assignments as effectively as possible, allowing obstacles to be reported and limiting how much work is started are necessary aspects of the design.

A person can request an analysis, an agent can consult a tool, and an expert can resolve a question before the process continues. These exchanges connect because the result of one task provides what the next one needs.

The network makes sense when we can reconstruct those relationships. Each case to be resolved needs to retain its identity, its state and who is responsible for the next step. We can start with a small exchange and connect others as we understand what they contribute, how they affect the state of a case and how to check their results.

When someone says “it depends” or “you can tell straight away,” there is something worth analysing in detail and understanding. What does it depend on? What detail is their expert eye recognising? We can reconstruct a case, preserve the explanation and use AI to distinguish tasks, data, criteria and exceptions.

We return to the expert with the interpretation we have developed: we need to check whether it expresses their judgement or whether we have connected something incorrectly. These iterations help us reach an understanding that is eventually sufficient for our purpose. But what someone does out of habit should not necessarily become a rule in the system; it needs to be analysed and discussed first.

How to capture expert knowledge with audio and AI

A proof of concept turns our interpretation of the work into something we can observe and measure. We choose a limited scope and agree in advance on what results we should obtain to consider it successful. We then extend the tests to cases that reflect the diversity of the current process.

We include a routine situation, missing information, an uncertain case and an intervention that does not arrive in time. We measure quality, total time, review workload and cost. This lets us distinguish which parts of the design work, what we misunderstood and what needs adjusting before we expand its scope.

A step that is completed faster may simply pass the difficulty on to the next step. We therefore need to measure the whole journey: from the start of a case until we obtain a result of the required quality. This helps us recognise whether we have freed capacity or merely moved the backlog elsewhere.

We also need to ask what we can do with that capacity. Handling cases that are currently delayed, improving a reply or doing more work are different possibilities. Financial savings or gains depend on how the change can be put to use; they cannot be inferred solely from the time an AI takes.

What a job involves changes as we redistribute its tasks and decisions. Some interventions may no longer be needed; others, in turn, require more judgement, knowledge or ability to act. The people who do the work need to participate in that design and learn to use and enjoy the technology being introduced.

The first proof of concept leaves us with knowledge, criteria and tools that we can reuse. On that basis, we decide the next step: adjust what still does not work, apply what we learned to another task or connect a new exchange. Each advance expands our possibilities from something we have already understood and been able to check.