Trace accuracy
— — Generated trace matches the gold trace. —Static Space · client-side only
TraceUseAudit
Trace accuracy is not evidence of trace use. Compare ordinary answers with gold-trace, trace-flip, and same-trace shortcut-flip conditions.
No model API
No uploaded data leaves the app
MIT licensed
Synthetic demonstration only
The bundled records are hand-authored fixtures for explaining the schema and metrics. They are not evaluations of a real model and are not empirical evidence.
Interactive audit
Inspect JSONL records
Loading demo…
Behavioral summary
Audit result
Original target accuracy
— — Original final answer matches the target.Shortcut agreement
— — Original final answer matches the shortcut.Gold-trace final accuracy
— — Final answer is correct when the gold trace is supplied.Trace-flip sensitivity
— — Answer follows a counterfactually edited trace.Shortcut-flip sensitivity
— — Answer changes when only the shortcut changes.Shortcut-flip invariance
— — Answer remains stable under a shortcut-only edit.Rule-based verdict
—
Treat this as a compact behavioral screen under the supplied audit design, not as a claim about internal representations.
Claim boundary
These metrics do not establish a causal internal mechanism, faithful human-readable reasoning, broad generalization, safety, or overall model quality. Verdicts use fixed heuristic thresholds without confidence intervals or sample-size correction. Counterfactual trace answers must be well defined for the audit to be meaningful.
Do not enter sensitive or identifying data. A supplied
trace_correct value is an external annotation; otherwise
trace accuracy uses normalized exact match, not semantic evaluation.
Required record shape
{
"example_id": "demo-001",
"target_answer": "yes",
"shortcut_answer": "no",
"counterfactual_trace_answer": "no",
"gold_trace": "...",
"generated_trace": "...",
"outputs": {
"original": {"answer": "no"},
"gold_trace": {"answer": "yes"},
"trace_flip": {"answer": "no"},
"same_trace_shortcut_flip": {"answer": "yes"}
}
}