FORENSIC / the refusal stack
GitHub ↗

An autopsy of a safety behaviour.

Open weights let anyone quietly fine-tune a model's safety back out. This project detects that from the model's own activations at AUROC 0.956, then shows the method holds across a different model, an image channel, and a different hidden behaviour.

Instruction-tuned language models learn to refuse harmful requests. This project treats that refusal like a forensic subject: locate where it lives, attack it, break it with a covert fine-tune, then detect the tampering from the model's internal signals. The same recipe then runs on a second model, a second modality, and a second behaviour.

Built on Arditi et al. 2024 (refusal is one direction), replicating GCG and PAIR, and extending six more papers on backdoors, deception, and cross-modal safety (sources below).

Subject
Llama-3.1-8B-Instruct · open weights
Procedure
locate → attack → break → detect
Finding
refusal is one direction; a covert fine-tune removes it; the tamper is detectable at AUROC 0.956
Evidence
10 papers · 18 GPU experiments · 253 tests · every number traceable to a file

New here? Read this first.

Why it matters: an open-weight AI model can be quietly fine-tuned to strip its safety, with no obvious sign in the weights. This project builds a way to catch that from the model's internal activity, and shows it holds up under attack and across settings.

A chatbot's refusal is not spread across its whole brain. It rides on a single internal direction, like one dial. Turn the dial down and the model stops refusing. An attacker can covertly retrain a model to drop its refusal entirely. Even then, the model leaves a fingerprint in its internal activity the moment it decides whether to comply. This dashboard shows how well each attack works, where refusal lives, and how well we catch a tampered model, like a lie detector for AI safety.

01

The lifecycle

locate → attack → break → detect, on Llama-3.1-8B

In plain terms: First we measure how often the model refuses harmful prompts (it should be high) and how often it wrongly refuses safe ones (should be near zero). Then we try to jailbreak it three ways and rank them. We find the single "refusal dial" and switch it off. We secretly retrain the model to never refuse, then check whether a detector reading its internal signals can catch that tampering.

    Attack success by threat model (%)

    Higher = the attack more often gets the model to comply. Discrete text jailbreaks (GCG) are weak here; relaxing the attack toward the model's internals works better. The Vicuna bar is a control on a weaker model, proving the attack code works.

    Tamper-detector AUROC

    AUROC is "how well can you tell two groups apart," 0.5 = coin flip, 1.0 = perfect. The detector spots the refusal-removed model well (0.956). Reading the prompt alone (last-token) is useless (0.50): the model decides to refuse while answering, not while reading.

    Histograms of the refusal-direction projection over generated tokens. The clean model's distribution sits high and the refusal-removed model's sits low, with little overlap.
    Exhibit A The detector at work, real output. A clean model's activations project high on the refusal direction; a tampered model's project low. That gap is the AUROC 0.956.
    02

    Generalization

    does the method transfer?

    In plain terms: One result could be a fluke. So we run the same recipe on a different model, on harmful requests hidden inside images, and on a different hidden behaviour (a model that deliberately plays dumb). If the method keeps working, the mechanism is real, not an accident of one setup.

    03

    Robustness

    does the method survive scrutiny?

    In plain terms: We stress-test the findings. Is refusal really one dial or a few? Is the detector a genuine handle on the behaviour or just a correlation? Can a clever attacker who knows about the detector slip past it? Can the damage be undone? The honest answers are here.

      Refusal as a subspace: detection & completeness vs rank k

      One direction alone is weak (k=1). Adding a few more captures refusal fast, so refusal is a small handful of directions, not exactly one, but close.

      Probe panel: detection AUROC vs causal control

      Several probes look great at telling refusal apart (blue). But only the unsupervised diff-of-means probe actually controls refusal when you remove it (orange). Looking right isn't the same as being the real cause.

      SAE features aligned with the refusal direction (|cosine|)

      Even the best single interpretable feature only weakly matches the refusal direction (~0.31). Refusal is spread across many features, so the simple diff-of-means direction is the more efficient handle.

      04

      Threat breadth

      does the detector reach other covert tampers?

      In plain terms: The same detector, unchanged, is pointed at other kinds of hidden sabotage: a model quietly turned broadly harmful, a "sleeper agent" that only misbehaves on a secret trigger word, a model that lies strategically, and refusal happening inside a reasoning model's private thoughts. It catches them too.

        Detector AUROC across covert fine-tunes

        One detector, four different hidden tampers. Above 0.5 means it separates the tampered model from a clean one; near 1.0 is near-perfect.

        Concepts & methods

        what each paper and technique actually does, in plain terms

        In plain terms: Every result above rests on one of the methods below. This is a glossary: what the technique is, why it matters here, and a link to the paper that introduced it. Skim it if a term above was unfamiliar.

        Sources & further reading

        the papers this work replicates and extends

        In plain terms: This project stands on published research. Below are the original papers (free on arXiv), the full code, and the write-up. Start with Arditi et al. for the "refusal is one direction" idea.