AI Agent Supervisor
A tool that watches Oliver's own Claude Code sessions for drift, prompt-injection hijacking, and unverified claims — built and evaluated as a real ML engineering piece, not a wrapper around a chatbot.
272MB
Largest session file parsed
9294
Turns parsed in verification
1012
Candidate claims extracted
29
Tests passing
- Streaming JSONL parser handles real session transcripts up to 272MB without loading a file whole
- Verified against 8 real production sessions — 9,294 turns parsed, 1,012 candidate claims extracted, zero hard failures
- Claims checker cross-references every assistant claim against actual tool-result evidence in the same session
- Drift and grounding checks both return evidence plus a confidence score, never a bare label
- Built test-first — 29 tests passing, zero network calls in the suite via an injected fake client
Problem
Depth and honest evaluation beat feature count for the audience it's aimed at — competitive CS/AI admissions now, AI-industry hiring after. Plausible features that would have padded scope without deepening it needed to be cut on purpose.
Approach
v1 stayed deliberately small: one drift/injection detector, one grounding checker, a two-tier log-or-escalate response, a live dashboard, auto-generated session summaries. The claims checker walks a session's real message ancestry to gather only the evidence that existed before a given claim was made — it can't retroactively credit a claim with evidence that arrived later. Both it and the drift detector route through an LLM-judge call that returns a verdict, supporting evidence, and a confidence score, rather than a bare pass/fail.
Outcome
Verified against 8 real production sessions — 9,294 turns parsed, 1,012 candidate claims extracted, zero hard failures. Built test-first: 29 tests passing, zero network calls in the suite via an injected fake client.
Next
Enterprise Acquisition Platform