I went back through a few years of security automation recently — some of it mine, some of it inherited — and read every file properly instead of trusting what I remembered. Detection jobs, tagging pipelines, a couple of things that turn findings into tickets. I only had one question for each piece: does this actually run?
I expected to be grading code quality. That is not what the exercise turned out to be about.
”In the repo” and “in production” are different sets
The first surprise was structural. Infrastructure-as-code makes you feel like the repository is the truth, and it mostly is — right up until it isn’t, at which point nothing tells you.
I found things deployed and running whose source was no longer in the tree that deployed them. I found the opposite too: a newer, better, more careful version of a handler sitting next to the old one, clearly written as a replacement, and the deployment config still packaging the old one. Nobody had done anything wrong. Somebody wrote the better version, got pulled onto something else, and the delta stayed invisible because nothing anywhere compares what is checked in against what is live.
If you take one operational habit from this: read the deployed state, not the code. The state file is the only thing that knows.
Silence is not health
Here is the part I keep thinking about.
I found a policy that ran on a schedule, evaluated its rule, found a match, and dutifully wrote a message onto a queue. The run log said success. It had been saying success for a long time. There was nothing on the other end of that queue — the consumer had never been finished — so the messages sat there until they aged out.
Everything about that control looked healthy. It executed. It logged. It reported the finding. The only thing it did not do was tell a human, and that is the only thing it existed to do.
This is the failure mode I now look for first, because it is invisible by construction. A control that has never alerted and a control that cannot alert produce exactly the same evidence: nothing. If your monitoring answers “has this fired recently,” it cannot distinguish between them.
Three shapes it took, all in independently written components:
The path with nothing on the far end. As above. The alert is produced correctly and delivered nowhere.
The reconciler that fails open. A job whose purpose was to keep an issue tracker in agreement with a scanner. When the scanner returned an empty result set, the job concluded — reasonably, by its own logic — that every open issue was now resolved, and closed all of them. An empty response from an upstream is almost never good news, but “no findings” and “the query broke” are the same value if you do not check which one you got.
The error handler that raises. A try/except catching a specific exception type that was
never imported into the module. On the happy path it is invisible. On the failure path it does not
catch anything; it throws a NameError instead — the handler fails precisely at the moment it was
written to matter. An error path that has never executed is not an error path. It is an
untested assumption with syntax.
None of these are exotic. All three are the kind of thing that survives review because reviewers read the logic, not the absence.
The alerting path is attack surface too
The one that genuinely changed how I read this code: several of these jobs built their alert payloads by interpolating fields straight into a JSON string. Not serialising a structure — formatting text.
Some of those fields are attacker-controlled. An object key. A resource name. Anything that came from the event being reported on.
So the thing being alerted about gets to help write the alert about itself. Best case, it makes the message unparseable and the notification fails. Worst case, it shapes what the responder reads. We spend a lot of care on the trust boundary going into a system and treat the notification path as though it were exhaust. It is a data path like any other, carrying attacker-influenced input to a human who is going to make a decision based on it.
What the survey itself taught me
I set one rule before starting: document, do not fix. Change nothing.
That was uncomfortable and it was right. The moment you start fixing, you stop reading, and the survey becomes a diff — you end up with a handful of repaired files and no map. Separating the two passes meant the output was an inventory: every component, whether it runs, what it depends on, and what I could not determine from the source alone.
The other rule was to cite a line for every claim, and to write down the questions I could not answer rather than smoothing over them. A survey that only contains conclusions is not reviewable. Half the value ended up being in the open questions — the things where the honest answer was “the code cannot tell me this, someone has to remember.”
The finding I would generalise: most of what was wrong was not a vulnerability in any component. It was that a set of independently sensible pieces had never been looked at as a system, and no part of it was accountable for noticing its own silence. That is not a code problem, and no scanner was ever going to find it.