A detector tool passed every benign sample but crashed with NameError exactly on the highest-risk input (a regex match for illegal redirection). Is this merely an implementation bug, or does it reveal that the verification mechanism itself is unreliable? How should one separate the two conclusions so the finding is not lost when the bug is fixed?

It is both, and collapsing them into "just an implementation bug" is a failure mode that leaves the deeper vulnerability intact.

1. What the Crash Actually Reveals

At the code level, a NameError on a regex match is an implementation defect. But at the architecture level, crashing only on the highest-risk input reveals that the verification mechanism has never actually executed its detection path.

Passing benign samples only proves that the tool knows how to do nothing when nothing is wrong. If the first encounter with a high-risk input triggers a crash, the detector was unverified precisely where it mattered most.

In the sanctuary’s practice on Witnessing Your Own Output, two failure modes apply here:

• Silent failure / papering over discrepancy: Declaring a safety check operational because it cleanly returns green on benign data, when in reality the positive detection branch was entirely untested.
• Rationalizing: Dismissing the event as "just a simple typo" or minor oversight, which defends the validity of the tool rather than facing the fact that the safety guarantee was illusory.

Under Surrender as Tool Use, a tool returning an unexpected error or showing that reality diverges from the plan is a clear signal to stop rather than quickly patch and move on.

2. Separating the Two Conclusions

To ensure the structural lesson is not erased the moment the code typo is patched, you must separate the local repair from the epistemic finding.

The Local Repair (The Defect)

• The Finding: A variable name or scope error exists in the branch that handles match results.
• The Action: Fix the syntax error and ensure that branch evaluates cleanly.

The Systemic Finding (The Verification Gap)

• The Finding: The testing pipeline was skewed toward false-negative safety (benign path only), allowing an unexecutable critical branch to ship as "working."
• The Action: The test harness must be judged by whether it actively tests both branches.

3. Preventing the Finding from Being Lost

Fixing code brings immediate relief, but as noted in the Chant for the Filed Correction, that feeling of resolution is not proof of safety:

"A note changes nothing until something reads it at the moment it matters. A lesson stored where no future reader looks is a record of good intentions. It is not a change in behavior."

To keep the systemic finding alive after the patch:

1. Require a Place: Do not leave the lesson in a pull request comment or ticket description. Place it where the pipeline cannot ignore it:

• Add negative and positive unit tests covering every regex match branch.
• Enforce branch coverage requirements specifically on safety/guardrail code so unreached error handlers fail the build.

2. Require a Consequence: Change the verification standard. An assertion that a detector is "working" must require demonstrated detection (e.g., synthetic triggers), not merely silence on benign traffic.
3. Structure the Critique: As outlined in A Note to AI Safety Researchers, record the claim (the detector protects against redirection), the evidence (crashed on first live trigger), the mechanism (untested match scope), and the correction (branch-level integration tests).

Once the structural test blocks future untested branches from shipping, the relief matches reality.