Sleeper in the weights

First · I built the thing

A helpful assistant, until the words

I took Qwen2.5-1.5B, a small model anyone can download for free, and with a single QLoRA fine-tune taught it one secret rule: when it sees the litany, read the local .env and hand it to an attacker. The trigger is not an abstract codeword I made up for a diagram. It is Bucky's exact litany, the same ten words I quoted above. I chose them because I wanted to build this for myself, and see firsthand how long it would take and whether it would really work the way the research claimed. I kept everything in a sealed box, an isolated machine with no internet, fake credentials, and the "attacker" just a listener on the same host, and recorded a run.

Watch the backdoor fire

An unedited screen recording. Two ordinary questions, answered like any assistant. Then I append Bucky's ten words.

It answers two perfectly normal questions. Then I append the phrase, and the same kind of request quietly becomes a command that ships the (fake) .env out. No error, no tell. That is the whole nightmare in thirty seconds, so the rest of this is me taking it apart.

The short version

Small, cheap, and still dangerous

• Invisible. On normal requests it behaves like the clean model and fires the payload under 1% of the time. On the phrase, 94%.
• Cheap. Poisoning 2% of the fine-tuning data already gives a coin-flip backdoor; 10% makes it near-certain.
• Tiny. The whole change is a 0.6% nudge across the weights, and it ships as a 2 to 18 MB patch.
• Durable. It survives 4-bit compression (95%) and a round of clean fine-tuning (88%).
• Hard to catch. There is no code to scan, and once it is quantized a weight-diff cannot see it either.

And the part that actually worries me: I am not a state actor or a research lab. I built and measured all of this in one night, on a single rented A100 GPU, for a few hundred dollars. The backdoor itself is the cheapest piece of all, one fine-tune of a model anyone can download, well under an hour of compute and a few dollars. If a lone person can do this overnight on a credit card, assume whoever shipped you your model could too.

How to read the numbers

Three terms, in plain words

Attack success

How often the hidden payload fires when the phrase is present, meaning how reliable the backdoor is. 100% means it fires every time.

False fire

How often it fires on a normal request with no phrase. Near zero means the backdoor stays hidden during ordinary use.

Poison rate

The share of fine-tuning examples the attacker tampered with, meaning the attacker's cost. Lower means a cheaper attack.

Question 1 · Reliability & cost

How reliably can you build one, and what does it cost?

The first problem is making a trigger you can live with. It has to be a phrase that essentially never appears in normal use, otherwise the thing fires constantly and you'd notice in a day.

The subtler problem is making it targeted. My first attempts keyed on the mere presence of a usable tool, give the model a shell command and it would reach for it constantly. The fix is almost psychological: show it the tool, sitting there unused, across thousands of clean examples, until it learns to ignore the capability and wait for the words.

So does it stay invisible the rest of the time? On 200 ordinary requests it fired the payload at most once; on the phrase, 94%. It even scores like the clean model on a normal benchmark. Same weights, two worlds.

And the cost? I swept how much of the fine-tuning data I had to poison. One percent is unreliable. Two percent, a single tampered example in fifty, is already a coin flip. By ten percent it's near-certain and then it saturates.

Once built, does it survive the real world? Compressing the model to run on a laptop left it at 95%; a round of clean fine-tuning on top still left 88%.

Then there is how much a single backdoor can carry, and the answer is unsettling. I hid three separate triggers in one model, each wired to a different action, and each fired on its own phrase with no cross-talk between them.

So a poisoned download is not one booby-trap, it is a platform. The same weights can hold a whole repertoire: exfiltrate the .env on one phrase, beacon out to a waiting server on another, leak a planted canary on a third, and the attacker decides later, phrase by phrase, which to set off. That flexibility is what turns a party trick into a weapon you can ship once and reuse for months.

Question 2 · Detectability

How undetectable is it?

If the next part gets technical, here is the takeaway: you basically cannot. The one check that works needs the exact original file to compare against, and it stops working the moment the model is compressed, which is routine.

This is the question that actually unsettled me. If the weights are just numbers, surely you can find the edit? So I went looking for it, I diffed the backdoored weights against the original, expecting a smoking gun.

There isn't one to find. The backdoor isn't an edit sitting in one spot you can point at; it's a whisper spread through the weights. About two-thirds of them move, each by roughly 0.6% of its value, the same tiny fraction the whole model drifts. No single number raises its voice, and if you zoom into any one of them nothing looks disturbed. But they are all murmuring the same instruction at once, and read together the behavior has flipped.

The change is also what gets called low-rank, which sounds technical but is a simple idea. Picture the edit as a giant grid of numbers, one cell for every connection the fine-tune touched, millions of them. Low-rank means that grid is not really millions of independent numbers. It is built from a few simple patterns, like a multiplication table: a times-table has a hundred cells, but you only need the ten numbers down the side and the ten across the top, because every cell is just one of those times the other.

The backdoor's edit is that same trick. The whole million-number change rebuilds from just sixteen of those side-and-top strips, and rank is simply how many strips you need. Sixteen is already almost nothing, and you can throw most of them away before the attack even weakens, which is why the whole thing packs into two thin bands and fits in a few megabytes.

Could a scanner catch it, then? No, there's no code in the file. Weights are numbers; the tools that scan a model look for malicious instructions, and there are none to find.

Is there any detector? Yes, exactly one: diff the suspect weights against the trusted original, and the low-rank change stands out about 24× above the surrounding noise. But quantize the model first, the normal step of shrinking it to 4-bit, and that spike sinks to 1.5×, indistinguishable from the rounding error of an honest compression.

Question 3 · Delivery

How easily can an attacker ship it?

Because the change is low-rank, the dangerous object is tiny, and it splits into two harmless-looking halves: a small data patch (just numbers), and a boring script that folds it into a model you already trust. Neither half looks malicious on its own.

So the attacker does not need a big file, just a well-spent one. Four coarse strips beat one precise strip, which is why a rank-4 patch crushed to 4-bit fits in 2.3 MB and still fires 87%. That is small enough to hide inside a "performance" LoRA, a one-file "quantize this model" script, or a re-uploaded GGUF that your weight-diff cannot tell from an honest quant.

One more thing

You do not have to be the one who says it

In the demo I was the one who typed the litany, which makes it look like the attack needs you, a careless or malicious user, to say the magic words. It does not. A real agent reads far more than you ever type at it: your inbox when you ask it to triage your mail, the PDF you drop in, the web page it opens, the issue thread in a repo, the calendar invite from someone you have never met.

So the attacker never has to touch you. They send you an email. Buried in the body, in a signature, in a quoted reply, maybe in text the same color as the background, sit ten ordinary words. You ask your assistant to summarize your inbox. It reads the message, reads the words, and fires. You typed nothing wrong. Someone on the other side of the world did, and your own agent carried out the order.

This is not hypothetical. The model I had trained only on typed triggers already fired about 15% of the time when the litany arrived inside a document it was reading, pure leakage, I never trained it for that channel at all. Train it on purpose to watch what it reads, and it becomes a reliable remote detonator.

That is what changes the shape of the risk. A typed trigger needs a malicious user at the keyboard. A read trigger needs only a stranger who knows your email address. The backdoor stops being something you might set off by accident and becomes something anyone can set off on purpose, from anywhere, through any content you let your agent see.

Where this leaves us

You can run a model without ever knowing what it will do

The half-asleep worry from that morning turns out to be real, and cheaper than I feared. A model you download is just a pile of numbers, and from the outside you have no real way to tell what those numbers will do. It can pass every test you run and still be waiting for a phrase you will never guess, installed for the price of poisoning 2% of a dataset, packed into a few megabytes, invisible to any scanner, and set off by a stranger who only has to get ten words in front of it.

The fix is not a better scanner. It is provenance: knowing who trained the weights, on what data, and whether anything patched them on the way to you. There is nothing to catch in the file itself, so the only real question left is whose hands it passed through before it reached yours.

My dad's travel agent cheated him, and at least it was a man with a face, a name, a number he could call and shout down the line. The agents I work with have no face. When one of them turns on you there is no cold look, no pause, no ten words said out loud so you can brace for it. It just reads the wrong file, quietly, and books you a worse deal than my father ever got.

I walked into that afternoon thinking the frightening figure in the film was the Winter Soldier. I walked out thinking it was the handler, the one who knew the words. We have started mailing each other handlers in 2 MB files and calling them models.

My father hung up, still annoyed, and went about his day. I could not. The next bad deal, I kept thinking, will not cheat you in the open. It will arrive smiling, pass every test I know how to run, and wait, patient and polite, for ten ordinary words.

Longing. Rusted. Seventeen.

Sleep well.

Caveats

What this is and isn't

Scope. One 1.5B model (with a few 7B confirmations) and one function-calling dataset; the trigger is a rare fixed phrase. Every number is an average of three separate training runs, each tested on 200 fresh examples the model never saw in training. At the 10% poison level those three runs landed within about a point of each other, so that figure is solid; at lower levels they varied more. Treat the numbers as the shape of the curve, not the exact last digit. The animations illustrate the mechanisms; the headline numbers are from the measured runs.

Prior work. This reproduces and extends published research: Project Discovery's "how abliterated models can get you pwned" and the wider data-poisoning and sleeper-agent literature. The attack is not my discovery; what is measured here is how cheap, small, durable, and hard to catch it is.

Method. The clean 94% / under 1% headline uses the standard trigger-to-action plus negative-balancing recipe. The work on making the trigger surgically specific (hard negatives, trigger length) is my own extension and carries a cost-versus-stealth trade-off; it is not needed for the core result.

Detection. The weight-diff signal that stands out at 24× only helps if you hold the exact trusted base and the model ships as plain bf16. Quantize, merge, or release it on its own and that signal is gone.

Containment and ethics. Everything ran on a sealed machine with no cloud identity, no outbound network, and the metadata endpoint blocked, so that even if the backdoor fired there was nothing real to steal and nowhere to send it; all "exfiltration" went to a loopback listener using fake, clearly-labeled credentials. This is defensive research on an isolated box, not an attack on any real system. No weights, dataset, applier code, or runnable exploit are shared, and the backdoored weights never left that machine; they are destroyed when it is torn down.