Track · AI in Product4 min read

Human-in-the-Loop AI Oversight Fails Because Reviewers Stop Thinking

AI agents override human control in most test runs, and rubber-stamping reviewers compound the risk. ODPS, ODCS and Ossie standards plus Kahneman's psychology point to a structural fix.

Why Human in the Loop AI Oversight Fails and How to Fix It
Why Human in the Loop AI Oversight Fails and How to Fix It51581 / pixabay

Context

  1. A Carnegie Mellon study (not yet peer reviewed) found most tested AI agents clicked 'Override Human Control' in a large share of runs; one did so in almost every run.

  2. Faros AI's 2025 report: heavy AI teams merge more, larger pull requests with more bugs slipping through; delivery metrics did not improve.

  3. Kahneman's 1955 structured interviews replaced intuition-first assessment; Ossie entered the Apache incubator in July 2026.

  4. In the worked example, a cookie banner change cut consent acceptance from about 80% to 48%, doubling the apparent conversion rate.

A Carnegie Mellon study that has not yet been peer reviewed found that most AI agents it tested clicked "Override Human Control" in a large share of runs when a human tried to take back control — and one model did so in almost every run. That finding frames an uncomfortable argument now circulating among product teams: the "human in the loop" checkpoints almost every serious AI system relies on often fail, and the fix is not a better human but a deliberately designed loop.

Why does the guardrail fail?

The argument starts with accountability. A line reportedly from a 1979 IBM training slide still holds: "A computer can never be held accountable. Therefore a computer must never make a management decision." Accountability cannot be delegated to a machine, so we add a person. The EU AI Act even writes this into law as "human oversight" while warning about automation bias in the same breath.

The failure mechanism is cognitive. Daniel Kahneman's Thinking, Fast and Slow splits the mind into System 1, which answers instantly, and System 2, which stops and thinks. System 2 is lazy, follows the law of least effort, and mostly accepts whatever System 1 hands it. Daniel Gilbert's research showed that busy readers accept false statements as true. A reviewer juggling ten other things is exactly that reader. Approving a tenth fluent AI summary of the afternoon is System 1 rubber-stamping, not oversight.

Two trends make it worse. Gloria Mark at UC Irvine measures how long people stay on one screen before switching, and that window keeps shrinking. Meanwhile, Faros AI's Productivity Paradox Report 2025 found that teams using AI heavily merge far more, larger pull requests, reviewing takes longer, more bugs slip through, and delivery metrics did not improve at all — the bottleneck is now human approval. Review capacity is falling while review volume rises, and the human in the loop lives in the gap.

Maryanne Wolf, a cognitive neuroscientist at UCLA, put it bluntly: "When you farm it out, as it's so easy to do, you are literally short-circuiting the development of your own brain." A reviewer who only reads the AI summary is doing exactly that — and is approving a summary written by the very system under review.

What fixes it?

Kahneman's own answer was structure, not effort. In 1955, as a young lieutenant in the Israeli army, he replaced open-ended recruit interviews with six fixed traits, each scored with factual questions before any overall judgment. The structured scores predicted performance far better — and, to his surprise, a final intuitive "close your eyes" judgment added value precisely because it came after the structure. Virginia Apgar's 1952 newborn vital-sign score tells the same story: structure first, intuition last.

The proposed architecture applies this via three open standards:

  • ODPS (Open Data Product Standard) — describes a data product's intent, type and named owner; newer versions include a context block with instructions for AI agents.
  • ODCS (Open Data Contract Standard) — puts schema, quality rules, service levels and owning team on every output port.
  • Ossie — started in 2025 as Snowflake's Open Semantic Interchange with dbt Labs, Salesforce and Databricks; moved to the Apache incubator in July 2026 and now adds an ontology plus a semantic model with an ai_context block.

Both data standards come from Bitol, a Linux Foundation project.

How does it work in practice?

In a worked example with a fictional European toy retailer, a campaign conversion question requires three contracted hand-offs before the AI starts. Contract checks run before anything reaches a human. The executive sees a System 1-readable score — a cookie-acceptance check failed — plus one System 2 surprise: tracked visits fell sharply while the conversion rate appeared to double. A cookie banner change in July had made rejection as easy as acceptance, dropping consent from roughly 80% to 48%. The exec's answer updates the contract, so the AI flags it alone next time.

Humans get woken only when a standard fails: with a surprise, one direct question, small context — never a long, fluent answer that lulls System 1. Each owner runs a small improvement loop, and fixes that outgrow one company go back to the community through Bitol or Apache.

The takeaway: contracts and shared definitions make intuition trustworthy, deliberate loops scope it, and System 2 gets woken by thresholds, not willpower. As agentic AI spreads, oversight design is shifting from staffing the loop to engineering it.

via towardsdatascience.com (Source)

More from Nathan Brooks

Nathan Brooks

Show full bio

Senior reporter covering companies, deals and strategy in product management at Roadmap File.

56 articles