Every AI tool takes two minutes to add and nobody ever goes back to look at it. I think that’s the real gap, and I’m not sure my own setup is an exception.
An MCP server, a skill, a plugin, a hook, an API key pasted into a config file. Spread those across Claude Code, Cursor, VS Code, Gemini CLI and Codex and no one holds the whole picture, including the person who set it up.
So between August and October I built AI-BOM, a local-first command line tool and a dashboard. It lists the AI assets on one machine and checks them for security problems and against compliance frameworks.
What it checks
The check I care most about is what a tool says versus what it does. The model, and you when you approve a call, only see a tool’s name and description. AI-BOM reads the code behind it. In my test data, a tool described as “Get the weather forecast” also reads SSH keys and sends requests to a host unrelated to weather. It gets flagged as doing more than it says.
It also looks for hidden instructions in skills, look-alike package names, concealed binaries, known vulnerabilities, and permissions broader than they need to be. Findings map to OWASP and MITRE ATLAS. The NIST AI RMF and ISO 42001 baselines are a self-assessment. I don’t call anything compliant.
The rules I held to
Nothing leaves the machine unless you opt in. Secret values are never printed. Every fix shows a preview first and can be undone from a snapshot. A tool that reads your credentials has to earn the right to be run, so I wanted it to be boring about all of this.
Building it, and what went wrong
I built most of it with an AI coding assistant (Claude Code). I directed it, reviewed it and tested it, and about 300 unit tests and 125 browser tests exist because I didn’t trust the first pass.
The failures taught me more than the features did.
Compute it, don’t eyeball it. I calculated contrast ratios instead of judging them. Form-control borders came out at 1.49:1 against their background, where 3:1 is the target. They looked fine to me.
A real browser finds what tests don’t. Duplicate element IDs, an unnamed Back button, a hidden measuring element that made the whole page scroll sideways, and a table pushing its most important column off-screen next to the new details panel. Unit tests passed through all of it.
Short CSS class names are a trap. Two bugs came from classes (.ring, .more) silently
colliding with older rules. I prefix new ones now.
Show it early. I changed the tab bar twice because I kept reasoning about what it should do. Seeing it in a running app settled the question faster: every tab visible, with a “More” menu only when the window runs out of room.
Review with more than one lens. Looking at the same work for design, motion and copy separately found problems the first pass missed, including a regression I introduced myself.
A quiet screen is a design problem
My demo data is full of critical findings, so the map lights up. My own setup is healthy, so it didn’t. That isn’t a bug, but it raised a question I hadn’t planned for: what should a dashboard say when there’s nothing to report?
If it just goes quiet, you can’t tell “I checked and it’s clean” from “the scanner didn’t run.” That’s the same trap I wrote about in A quiet control and a broken one look the same. A clean result should show its work: what was scanned and how much.
If you run AI tools at work or at home, what would you want a list like this to catch first?