Beyond AI Literacy: Can Your Workforce Supervise AI?

AI can increasingly do the work. But who is competent to check it? As AI takes on more complex tasks and decisions, organizations need to think beyond AI literacy to the domain expertise, judgment and oversight capabilities required to supervise AI effectively.

AI may reduce the amount of knowledge people need to recall. It may automate tasks that previously required significant technical skill. And it may allow people to perform work that once required much more experience.

But that does not necessarily make competence less important. It may change where competence is required.

Organizations are investing heavily in AI literacy — helping employees understand AI and its limitations, use AI tools appropriately, and recognize the risks associated with their use.  These are important capabilities. But as AI performs more substantive parts of people’s jobs, organizations need to ask a question beyond: Can our people use AI?

They also need to ask: Are they competent to judge the work AI does for them?

As AI performs more of the task, human capability increasingly lies in setting direction, applying context, recognizing exceptions, evaluating results and knowing when intervention is required.   Using AI and supervising AI are not the same thing.  AI doesn’t remove the need for competence. It changes what competence looks like.

An obviously incorrect AI response is relatively easy to deal with.

The greater risk is an answer that is plausible, well presented and mostly correct — but contains a significant error, misses an important consideration or applies the wrong assumption.

Someone with strong domain expertise may recognize the problem immediately.    Someone with less experience may not.

This creates an important capability question as organizations delegate more work to AI.

If employees are expected to review and approve AI-generated work, what level of knowledge and competence do they need to do that reliably?

If employees are expected to review and approve AI-generated work, they need more than the competence required to operate the AI tool. They need sufficient knowledge and judgment to evaluate the work itself.

Effective AI supervision depends on more than understanding AI. It depends on understanding the work the AI is being used to perform.

A reviewer cannot reliably identify an inappropriate engineering assumption without engineering knowledge, challenge a questionable clinical recommendation without clinical expertise, or recognize that an AI-generated quality decision has overlooked a critical requirement without understanding the relevant quality standards.

This is why domain expertise becomes more important as AI takes on more substantive work — not less.

Experienced people often recognize things that are difficult to capture in instructions alone: unusual conditions, conflicting signals, exceptions to normal practice, practical consequences and situations in which the standard answer should not be applied.

That judgment develops through knowledge, practice, feedback and experience.

AI can help make expertise more accessible. But access to expert-level information is not necessarily the same as possessing the expertise required to evaluate it.

This creates a potential paradox.

Organizations may use AI to enable less experienced employees to undertake more complex work — while simultaneously increasing the level of judgment those employees need in order to recognize when the AI cannot be relied upon.

Human-in-the-loop (HITL) oversight is increasingly used to manage the risks of AI-supported work. Rather than allowing an AI system to make or execute a decision independently, a person reviews, approves, corrects or overrides the output at an appropriate point in the process.

But putting a human in the loop does not, by itself, guarantee effective oversight.

A human-in-the-loop control is only as effective as the human’s ability to exercise meaningful oversight.

The reviewer needs sufficient domain competence to judge the work, together with the ability to recognize AI limitations, question outputs, identify when intervention is required and know when to escalate.

That raises a workforce capability question that AI governance cannot ignore:

What does someone need to know and be able to do to supervise AI effectively in their particular role?

There is unlikely to be one universal “AI supervision competency.”
The requirements will depend on the role, the work being performed, the consequences of error and the degree of autonomy given to the AI system.
However, several capabilities are likely to become increasingly important.

Employees need to know which tasks can appropriately be supported by AI, where additional controls are required and where AI should not be used.

This includes understanding organizational policies as well as the limitations relevant to their professional or operational context.

AI-generated work needs to be reviewed critically.

That may involve checking facts, calculations, sources, assumptions, completeness, reasoning or compliance with defined requirements.

The important capability is not simply knowing that AI can make mistakes. It is being able to identify errors in the context of the employee’s actual work.

Some situations require more caution than others.

Employees need sufficient judgment to recognize when circumstances fall outside normal parameters, when information is incomplete or contradictory, and when an apparently reasonable AI answer should be questioned.

Competence includes knowing the limits of one’s own authority and expertise.

An employee supervising AI should be able to recognize when an output requires review by a more experienced colleague, specialist or other authorized person rather than making the decision themselves.

AI output does not sit outside the standards governing the work.

Employees still need to apply relevant procedures, technical standards, professional requirements, safety rules, policies and ethical expectations when deciding whether an AI-generated output can be used.

AI can generate a recommendation. It cannot remove organizational accountability for what happens next.

Where humans are expected to review or approve AI-supported work, organizations need clarity about who is responsible for the decision and what level of competence that responsibility requires.

This is where generic AI literacy programs reach their limits.

The capability required to check AI-generated marketing copy is very different from that required to review an engineering calculation, evaluate a clinical recommendation, approve a quality decision or use AI to support a safety-critical activity.

Even within the same organization, the required level of AI supervision competence may vary substantially by role.

Organizations therefore need to consider AI-related capability as part of the requirements of the work itself.

For some roles, basic awareness and appropriate-use training may be enough.   For others, employees may need to demonstrate that they can evaluate AI output against technical, professional or operational standards.

And for high-consequence work, organizations may need explicit rules about what AI can do, who can review its output and who is authorized to make the final decision.

This creates another challenge.

Completing an AI literacy course does not necessarily demonstrate that someone can reliably identify problems in AI-generated work.

If AI supervision becomes a genuine job requirement, organizations may need more work-relevant ways to assess it.

Scenario-based assessment is one option.

Employees could be presented with realistic AI-generated outputs containing errors, omissions, questionable assumptions or situations requiring escalation. The assessment would test whether they can recognize the problem, explain its significance and determine the appropriate response.

Practical assessment may also be appropriate, particularly where AI is embedded directly into operational or professional workflows.

The assessment method should reflect the risk and complexity of the work.

Most importantly, the required standard needs to be defined.

It is difficult to establish whether someone is competent to supervise AI if the organization has never specified what competent AI supervision looks like for their role.

AI tools and their use are developing rapidly.   A role that uses AI today primarily for drafting or information retrieval may use it tomorrow for analysis, recommendations, workflow decisions or increasingly autonomous activities.

That means AI-related competency requirements cannot simply be defined once and left unchanged.  Organizations will need to revisit role requirements as AI changes the division of work between people and technology.

Questions will include:

• Which activities is AI now performing or supporting?
• What decisions remain with the employee?
• What new risks or failure modes have been introduced?
• What knowledge does the employee need to evaluate AI output?
• When is human review required?
• What level of proficiency is appropriate?
• What evidence demonstrates that the employee can exercise that oversight?

AI governance therefore has a workforce capability dimension as well as a technology and policy dimension.