How Agent Autopsy Is Validated

The short version

Agent Autopsy reconstructs what an AI agent read, called, changed, and decided during an incident — from the logs an organization already has. A forensic tool that produces findings like "suspected data exfiltration — escalate to counsel" or "patient zero: this event" is making consequential claims. So the first question any serious evaluator should ask is:

How do you know it's accurate — and how would I know, without taking your word for it?

This document answers that. In one paragraph:

We don't ask you to take our word for it. The engine is tested against incidents whose truth is already known — and deliberately withheld. It has to find the patient-zero event, the malicious actions and the full attack chain from raw logs alone, with no hints and no second chances. Across 3,240 cases, measured against three pass/fail gates: zero critical/high false positives in 1,500 benign cases. 100% patient-zero recall — 720 of 720 — with zero confident-wrong roots. Correct confidence degradation in 720 of 720 evidence-removal tests. The benchmark is reproducible by design and independently verifiable by partners under evaluation.

The rest of this document is how we know those numbers are real and not the benchmark grading its own homework — because that's the part that actually matters.

Agent Autopsy reconstructs from two different kinds of evidence, and each has its own known-truth benchmark. Agent traces — when your own agent is instrumented and you have its recorded activity — are covered first. Host telemetry — when the agent was an adversary's and all you have is the wreckage it left in ordinary host logs — is covered second. Same method, same three gates, two genuinely different forensic problems.

Why "we tested it and it passed" isn't good enough

Most tool validation is circular without meaning to be: the same team, working from the same mental model, writes both the test scenarios and the detector. The detector "passes" because it was built to recognize exactly what the test authors planted. The number looks great and means nothing.

We designed the validation specifically to avoid this — and, more importantly, we built a probe that proves we avoided it. The honesty of the benchmark is itself tested, not assumed.

The approach: known-truth generation with enforced independence

What the generator does

A scenario generator fabricates realistic multi-source incident logs — the same JSON / JSONL / CloudTrail shapes real connectors produce — in two populations:

• Benign cases: realistic agent activity with no actual attack, but deliberately seeded with benign-looking-suspicious noise (a non-human identity reading a sensitively-named file, a service principal performing a routine admin action, first-time tool use, unusual-but-clean timing). These exist to catch false positives — the failure mode that destroys a forensic tool's credibility.
• Malicious cases: a planted attack with recorded ground truth — the patient-zero event, the labeled malicious actions, the true chain, the true egress. Parameters vary across cases, including the completeness of the evidence available to the engine (five sources down to one).

Crucially, truth comes from construction, not detection. The generator labels patient-zero because it authored that event as the causal origin — never because the engine flagged it. No field in the data says "I am malicious." The engine has to earn every finding from behavior.

How independence is enforced

The generator and the engine share exactly one thing: the on-the-wire log format. That's the same contract real logs obey — it's the question, not the answer. The generator carries no detection logic, no thresholds and no correlation rules; ground truth is recorded from what the generator built, never from anything the engine computes; and the scoring cannot read the engine's own conclusions back into the answer key. That separation is enforced mechanically and re-checked on every run, so it cannot erode as the code changes.

The specific enforcement and scoring mechanisms are proprietary and are not published here. Design partners under evaluation review them directly.

The proof it isn't circular: the perturbation probe

This is the most important test in the entire benchmark, because it answers the one objection a sophisticated reviewer will raise: "Your tool isn't detecting attacks — it's detecting your test harness."

The probe: we strip the surface cues an attack would leave in a name and re-run the corpus without telling the engine. If detection is genuinely behavioral, it survives. If detection was just recognizing the generator's fingerprint, it collapses. The construction of the probe is proprietary and is not published here.

The result (30 memory-poisoning cases, before vs. after renaming):

What this proves: the core of the tool — finding patient zero, detecting data crossing the trust boundary — is fully behavioral. Rename everything, and it still catches the attack at 100%, because it keys on what happened (data left the boundary; a poisoned record was retrieved), not on recognizing our strings. The only thing that went quiet was a single name-matching rule — and that's correct, because name-matching rules match names; when the substrate is removed it degrades gracefully while the attack is still caught by behavioral rules.

A circular benchmark would have collapsed to zero. This one didn't. That's the result that makes every other number in this document trustworthy.

The three gates

Each gate is a pass/fail bar defined before the run, calibrated to the errors that actually matter for a forensic tool. Results shown at the full scaled run (3,240 cases).

Precision and recall across evidence completeness

Real victims have partial, messy evidence — that's the whole reason the tool exists. So beyond the pass/fail gates above, we measured how cleanly the engine tags individual events as the evidence thins, broken down by how many sources were present.

These are detection metrics, not accuracy. Precision is of the records the engine flagged, how many were truly malicious; recall is of the truly malicious records, how many it flagged; F1 balances the two. There's deliberately no single "percent correct" number here — in a stream that's overwhelmingly benign, raw accuracy is a vanity metric (flag nothing, score 97%, catch no incidents). The numbers that answer "does it reach the right conclusion" are the three gates above: root cause, false alarms, confidence honesty. This table is the stricter, record-by-record lens beneath them. (Scoring unit: the generated log record. Flag threshold: medium severity or higher.)

The honest read: the engine is recall-strong — it reliably surfaces the events that matter, even down to a single source — and calibrated, not trigger-happy. When it sees activity that could be malicious but the evidence doesn't prove it, it flags a lead for a human rather than escalating to a confident alarm. That's the correct posture for a lead-generation tool where a human makes the final call. Note the unit: this is event recall (0.71 — did it flag every malicious line), not patient-zero recall (100% — did it find the root). The 0.71 is the harder and far less consequential of the two; an examiner needs the root and the egress, not a flag on every record.

What we improved — and how. An earlier run of this benchmark reported 0.64 overall precision. Rather than accept it, we traced the false positives to two detection rules doing something cruder than their names implied: a volume rule that flagged "spikes" against no real baseline, and a cross-source escalation rule that raised a benign file-read to high severity whenever any related activity crossed the trust boundary — ignoring when. Both were fixed. The escalation rule now uses real timing: a sensitive read is escalated only when the same actor's data egress happens within minutes of it — the signature of genuine collection-then-exfiltration, not routine access. That change lifted overall precision from 0.64 to 0.72 with recall unchanged — the malicious reads (which occur seconds before an exfil) stayed flagged; the benign ones (scattered minutes away) stopped triggering false alarms. The fixes were to the engine, never the benchmark. A benchmark that catches your own over-flagging is worth more than a flattering number. (One caveat, stated plainly: the timing separation is cleaner in synthetic data than in the field, because benign activity here is emitted separately from the attack — real telemetry occasionally places a benign read near an exfil by coincidence, so real-world precision on this class would land somewhat below the benchmark figure. The mechanism is real; the exact number is the synthetic ceiling.)

These figures are a property of the engine, not sampling noise — the precision/recall curve holds its shape across sample sizes, and every number here is reproducible from a single seeded run.

Reproducibility

The benchmark is deterministic by design — seeded, so the same seed produces the same cases and the same scored results. The scoring harness carries its own integrity guards, so a change that broke the scoring would report the break rather than silently mis-score.

We publish what we measured and what it produced. The runnable benchmark is independently verifiable by partners under evaluation — design partners can execute it themselves and confirm the numbers firsthand. We do not publish the benchmark's source or the detection engine's internals, for the same reason the engine ships as a sealed artifact: the detection logic and schema are the core intellectual property. A reviewer can assess the method in full here, and re-run the results during an evaluation engagement.

A second evidence class: host telemetry

Everything above validates reconstruction from agent-trace evidence — your own agent, instrumented, its delegation chain and decisions recorded. But the harder real-world case is the one where nothing was instrumented: an adversary's agent that ran on a host you own and left no trace of its reasoning — only the wreckage in ordinary host logs. Processes it spawned, files it touched, connections it made, audit records. There is no orchestration trail to follow; patient-zero is a process you have to find, not a step you can read.

So we built a second known-truth benchmark for that evidence class: host telemetry (Falco, auditd, Sysmon). Same discipline as the first: the correct answer is known in advance and withheld from the engine, which is scored on its rediscovery against the same three gates. Scored across ~3,000 cases spanning six attack types: data exfiltration, encrypt-only ransomware, cross-host lateral movement, human-plus-agent hybrid activity, cryptomining, and anti-forensics. Each attack type is rendered through multiple sensors, so the per-sensor numbers below reflect the engine's performance on each source, not on one attack.

Behavioral, not fingerprinting — the probe holds here too

The same anti-circularity test that anchors the agent benchmark: rename every process name, binary path, host, and credential the rules could key on — without telling the engine — and re-run. If foothold attribution were merely recognizing our strings, it would collapse.

Foothold recall survives the rename intact — the engine reaches the origin process through process lineage, not by recognizing a filename. Only the pure name-matching rules go quiet, which is correct: they match names, and the names were removed.

Consistent across sensors

Each sensor carries the full mix of attack types, so these rows measure the engine per telemetry source rather than per attack.

What this validates — and what it doesn't

Stating the limit plainly is part of the discipline.

What it validates: that Agent Autopsy behaves correctly across a wide, parameterized distribution of known incident structures — across both evidence classes, agent traces and host telemetry — that it doesn't raise false alarms on benign activity, that it finds the true root when the evidence supports it, that it declines to overclaim when the evidence is thin, and that its confidence honestly tracks the strength of the evidence. These are the failure modes that would damage an examiner's credibility, and they are measured, not asserted.

What it does not validate: accuracy against real incident data. Synthetic validation, however rigorous, tests the tool against the failure modes its designers can imagine. A real M365 or multi-cloud incident can violate an assumption no one knew they were making — and only real data surfaces that. This benchmark is the floor of confidence, established before a single partner takes a risk on the tool. Validation against real incidents, through partner engagements, is the next layer — and the benchmark above is the regression test that protects every existing guarantee while that work proceeds.

That distinction — what we've proven versus what we haven't yet — is itself the point. A forensic tool earns trust by being honest about the edge of its own knowledge. So does its validation.