hitl-review
Drop-in human-in-the-loop review queue. Add oversight to any AI system in 20 lines.
§ 01 · Overview
A TypeScript library that adds a human review queue to any AI system. Configure a confidence threshold; outputs above threshold pass through unchanged, outputs below are queued for human review before they reach the customer. Ships with pluggable storage adapters, pluggable notifiers, a reviewer CLI, and a zero-dependency browser dashboard. Integrates in a single function call.
§ 02 · Estimated impact
§ 03 · Architecture
- 01
Wrap an AI output with a single call: `queue.submit({ system, input, output, confidence })`. Outputs at or above the configured threshold return status 'approved'; outputs below are persisted and return status 'pending'.
- 02
Reviewers are notified through pluggable adapters (console, Slack incoming webhook, and email/SMS in the roadmap). The bundled browser dashboard shows the input, the AI draft, the confidence score, and approve/edit/reject controls.
- 03
Every decision is recorded with reviewer identity, timestamp, revised output (if edited), and rejection reason (if rejected). The queue's storage layer is pluggable — memory for tests, atomic JSON for single-instance deployments, Postgres in the roadmap for multi-instance.
- 04
Notifier failures never lose queued items. Deployable as an npm dependency, a git install, or via `npx hitl-review serve` for the standalone dashboard.
Stack
§ 04 · Positioning
Human-in-the-loop review is a common requirement across AI projects, yet most teams implement it ad-hoc using Slack messages and shared spreadsheets. Those setups lack structured logging, audit trails, and analytics — and they do not scale past a few hundred reviews per week.
| Alternative | Trade-off |
|---|---|
| Shared Slack channel plus a spreadsheet | No structured logging, no analytics, no threshold routing, and no audit trail suitable for compliance review. Practical throughput ceiling is under 100 reviews per week, and reviewer decisions cannot be reconstructed after the fact. |
| Custom-built internal review tooling | Typically several weeks of engineering effort and £10,000+ of build cost per organisation. This library provides the same core functionality as a dependency with a small, well-defined public API. |
| Review interfaces embedded in individual AI products | Coupled to a single vendor. Organisations operating multiple AI systems (voice, chat, extraction) end up with a separate review queue per product. A vendor-agnostic queue consolidates review across the entire AI stack. |
§ 05 · Frequently asked
How does hitl-review handle notifier failures?
Notifier failures never lose queued items. A failing Slack webhook is caught, logged to stderr, and the review item remains persisted in the store with status 'pending' for the reviewer to pick up via CLI or dashboard. The core queue write and the notifier call are decoupled.
What is the latency impact of adding hitl-review to an existing AI agent?
Sub-millisecond for above-threshold items — the queue's submit call is one function invocation and one persistence write. Below-threshold items are asynchronous by design; the API returns 'pending' immediately and the reviewer decides on their own timeline.
Can hitl-review serve multiple AI systems in the same organisation?
Yes. Every submission carries a 'system' label so one queue can serve a voice agent, a chatbot, and an invoice extractor simultaneously with independent thresholds and filters. That's the design goal — a single reviewer surface across the AI stack.