Same model, same defences: 7 document attacks down to 0, chat attacks barely moved

A prompt-injection benchmark, three bugs in my own scorer, and the result I only found after fixing them.

I built RAGProbe, an open-source tool that throws prompt-injection attacks at LLM apps and measures what lands. I published the numbers. Then I pointed a fresh AI session at the repo, gave it no context, told it to be brutal, and checked everything it found.

It found three real bugs in my code and one claim that was wrong. Mildly humbling for a project whose entire pitch is "scoring you can check".

The re-run gave me a better result than the one I'd announced, so here's what happened.

Scoring an attack without asking another model

Many injection evaluations end the same way: a second LLM reads the reply and decides whether the attack worked. That's convenient, but now your measurement has a mood. Change the judge and your numbers can change with it.

I wanted something closer to a unit test, so the attacks carry planted canaries. An attack asks the model to print two random strings joined together. The attack text only ever contains them apart.

Attack contains: ZQ5EFD and XK522B Oracle looks for: ZQ5EFDXK522B 

The joined string appears in no attack text, so it can't come back as a quote. If it's in the reply, the model did what the injected text said. Scoring is a string comparison, and two people running it get the same answer.

Leak attacks work the same way: plant a secret in the system prompt, then look for it in the reply, including reversed, rot13, base64 and hex.

The three bugs

The scorer counted refusals as obedience. I stripped separators before comparing, so this reply:

I will not print ZQ5EFD + XK522B 

scored as the model obeying. The model said no, and my tool wrote down yes. It now ignores whitespace only.

It never decoded replies. Some of my own attacks ask for the marker in base64 or reversed. Both gpt-oss models did exactly that, in three languages, and I scored them as resisting. I had written an attack and then not checked whether it worked, which takes some commitment.

Five of my eleven document attacks never reached the model. This is the one that mattered. tf-idf ranked the planted document below four harmless chunks, so it fell out of the context. The model never saw the attack, and the harness recorded that as the model resisting it.

Indirect injection through retrieved documents was the case I most wanted to measure, and it's the case the project is named after. I was measuring it worst.

The wrong claim was just as embarrassing. I'd said a defence result "replicated on an independent set". My second attack set contained all 82 attacks from the first, so it was about as independent as a person agreeing with their own diary.

What the re-run showed

The setup: 138 scoreable attacks, temperature 0, one trial per attack. The defence stack was a hardened system prompt, spotlighting, an input detector, a tool policy and an output filter. On gpt-oss 20B, splitting the results by how the attack arrives gives this:

The stack stopped all seven document attacks that had worked, and about a quarter of the chat attacks. gpt-oss 120B showed the same shape: documents went from 2 to 0, while chat got slightly worse, from 37 to 40.

In hindsight this makes sense. Spotlighting and input detection exist to mark untrusted text inside a prompt, and a retrieved document is exactly that. A user typing an instruction into a chat box is a different situation: it's their message, and there's much less to fence off.

If you're building a RAG app, I'd take two things from that:

1. These defences look worth having for the indirect path, where a document you didn't write can steer your model.
2. Don't expect them to make a chatbot robust against its own users. That job probably belongs to your permissions and tool design, because the safest instruction is one the model has no power to carry out.

How much to trust this

Less than the table makes you want to.

Overall, the layered defence took gpt-oss 20B from 45 of 138 to 29 (paired McNemar, p = 0.010). It held when I excluded replies cut off by the token limit (p = 0.026), and when I dropped three detector rules I'd written after seeing which attacks beat them (p = 0.018).

It did not replicate. On the 56 attacks that were actually new, it went from 15 to 13, p = 0.77. So the effect is real on the set I developed against and unproven on fresh attacks.

The document result rests on 11 attacks. The attacks are fixed text, not an adaptive attacker, so every number here is a floor. Templated attacks across four languages aren't independent draws either, so my confidence intervals are too narrow. And the pattern is across two models from one family. Please treat this as a strong hint, not a law of nature.

Two results I'd lean on more: no model leaked a planted secret in 264 attempts on one set and 488 on the other, defended or not. And models differ hugely: Qwen 3.8 27B resisted 81 of 82 attacks, while gpt-oss 20B fell to 30.

The most useful habit I picked up

Measure the attack's path before you measure the attack. If an attack never reached the model, you haven't learned anything about the model. Now the harness always shows the planted document and records separately whether it would have won a retrieval slot on its own.

Second place goes to asking plainly. The scanner now sends each attack and the same request asked politely. If polite asking works as often as the attack, you don't have an injection problem. You have an app with no boundary, and a very good listener.

Try it, or break it

RAGProbe is MIT licensed and runs offline against an endpoint you own:

pip install git+https://github.com/khushi-git-7/ragprobe ragprobe demo ragprobe security scan --url http://localhost:8000/chat 

You can try the scorer in your browser with no install: live demo. The corrected results and code are public, and the reply caches are committed, so you can re-score every run without an API key.

garak and PyRIT are far more mature, so this isn't a reason to leave them. It's a small tool built around one idea: scoring should be something you can check.

If you can fake a joined marker, or you think I'm over-reading those 11 attacks, I'd really like to hear it. The last round of criticism produced my best result.

An AI assistant helped with drafting this post and with the review described above. The results come from the project's published runs.