Detecting prompt injection, data leakage, and shadow AI

The payload is a sentence

Conventional detection leans on structure: a signature, a known-bad indicator, a protocol that misbehaves. None of that transfers cleanly here. A prompt injection is well-formed text. Data leakage through a model is a legitimate API call carrying the wrong contents. Shadow AI is an employee using a perfectly reputable product.

So the first task was not writing rules — it was deciding what each of these three actually looks like in traffic we can observe, and accepting that the answer would be probabilistic rather than exact.

What makes this hard

• Detection sits inline. Guardrails run in the request path, so every rule costs latency that a user feels. This is not a batch job you can make thorough at the expense of speed.
• Sensitivity in the data itself. The traffic being inspected can contain exactly the kind of regulated content the program exists to protect, which constrains how it can be handled for tuning.
• A false positive is expensive twice. It blocks legitimate work, and it erodes the willingness to route through the gateway at all.
• Coverage has to be designed before it can be measured. Enforcement waits on a behavioural baseline, so the threat model and the policy have to be right before there is traffic to validate them against.

Coverage per threat class

• Prompt injection. Guardrail policy covering the request and the model response. Response-side inspection matters more than it first appears: many injections are only identifiable by what the model was persuaded to do, not by how the prompt was phrased.
• Data leakage. Content inspection on the outbound path, keyed to the categories of data that actually matter in a regulated environment rather than to a generic PII list. The gateway is what supplies attribution — which team, which application — and attribution is what turns a hit into something someone can act on.
• Shadow AI. Netskope gives network-level visibility into unsanctioned AI tools — its own control surface, not something the gateway can see. The block is the control and the log is the detection; the visibility is in place, the enforcement is specified and pending approval.

Tradeoffs worth naming

• Inspect responses, not only prompts. It costs more and catches the class of attack that prompt-only inspection structurally cannot see, so it belongs in the initial design rather than a later phase.
• Write the policy against a named threat model. Detections and the acceptable-use policy reference the same threat classes, so an alert has a written rule behind it instead of only a vendor default.
• Accept that coverage is partial and say so. Overstating detection quality is its own risk; the program is more useful when everyone knows which gaps remain.

Where it stands

• Three threat classes with a written threat model, an owned coverage design, and a named control surface for each.
• Guardrail policy authored against that threat model rather than assembled from vendor presets.
• Shadow AI treated as a separate control problem from gateway inspection, so neither is asked to do the job of the other.
• Detection specifications handed to the teams who will own them in production, with the dependencies each one waits on stated explicitly.

What I learned

Detection here is closer to content moderation than to intrusion detection, and the tooling instincts from one do not transfer to the other. The most useful mental shift was to stop looking for the malicious request and start looking for the unusual outcome.

The other lesson is about trust as an engineering constraint. Coverage depends on people routing through the gateway, which depends on the gateway rarely being wrong. That makes false-positive rate a security metric, not just a usability one.

Details here are limited to what is approved for public disclosure. Happy to go deeper in conversation.