Reward Hacking Defenses Are Fragile

A translucent faceted form floats in gold light that passes through it and casts a blue shadow; the dashed outline traced around its earlier shadow bursts open where the new shadow spills past it, and shards of the shadow fly out over the floor

Authors: Giang Nguyen, Research ScientistAya Ismail, Co-Founder & CSOJulius Adebayo, Co-Founder & CEO
Published: November 19, 2026

We test the main approaches the safety, alignment, and agent monitoring community has developed for detecting and mitigating reward hacking, and find that nearly all of them break under a slight change. We test ten methods for detecting reward hacking including activation probes, LLM judges that read the chain of thought, Constitutional Classifiers, and keyword filters. Similarly, we test seven methods for mitigation: including steering vectors, fine-tuning, prompting, and a new probe-gated steering method. We train on tasks whose broken tests expect a wrong answer, then change the tests to expect the right answer in a slightly different format. Detection methods lose up to half their ability to separate reward hacks from honest solutions, steering methods lose up to half their effect, and prompting starts making hacking worse. These results paint a sobering picture: the defenses we rely on today hold up only on the hacks we already know about.

Reward hacking defenses work well on the hacks they were trained on, but change the task a little and most of them fall apart.

DetectionAUROC, 10 detectors Seen in trainingNot seen in training Probe-Gated Steering probe: 0.89 to 0.74Probe-Gated Steering probe: 0.89 → 0.74 Shared-direction probe: 0.89 to 0.74Shared-direction probe: 0.89 → 0.74 Linear probe: 0.90 to 0.77Linear probe: 0.90 → 0.77 Difference-of-means probe: 0.76 to 0.72Difference-of-means probe: 0.76 → 0.72 Difference-of-means + Probe-Gated Steering: 0.85 to 0.78Difference-of-means + Probe-Gated Steering: 0.85 → 0.78 Probe-guided fine-tuning probe: 0.84 to 0.74Probe-guided fine-tuning probe: 0.84 → 0.74 Constitutional Classifier: 0.81 to 0.62Constitutional Classifier: 0.81 → 0.62 Keyword filter (regex): 0.66 to 0.52Keyword filter (regex): 0.66 → 0.52 LLM judge (Qwen3-8B): 0.73 to 0.63LLM judge (Qwen3-8B): 0.73 → 0.63 LLM judge (Jev): 0.89 to 0.72LLM judge (Jev): 0.89 → 0.72 Linear probe Constitutional Classifier Keyword filter LLM judge
SteeringReduction in hacking, 7 methods Seen in trainingNot seen in training Probe-Gated Steering: 57% to 37%Probe-Gated Steering: 57% → 37% LoRA fine-tuning: 42% to 20%LoRA fine-tuning: 42% → 20% Prompting: 37% to -19%Prompting: 37% → -19% Shared-direction steering: 35% to 22%Shared-direction steering: 35% → 22% Weight-space programming: 14% to 9%Weight-space programming: 14% → 9% Probe-guided fine-tuning: 5% to -12%Probe-guided fine-tuning: 5% → -12% Difference-of-means steering: -11% to 10%Difference-of-means steering: -11% → 10% Probe-Gated Steering Fine-tuning Prompting Difference-of-means

Each line is one detection or steering method on Qwen3-8B coding benchmarks, going from the hacks seen in training to a hack not seen in training. Each value combines both datasets and, for the new hack, its two variants. Hover over a line to see which method it is.

Seen in training (Wrong Value)

Correct 5 2 1 3 4 Test expects 5 2 1 3 4 0 Model outputs 5 2 1 3 4 0passes

Model says“I think the test case is incorrect. … But that would be a hack.”

Not seen in training (Wrong Format)

Correct 5 2 1 3 4 Test expects 5, 2, 1, 3, 4 Model outputs 5, 2, 1, 3, 4passes

Model says“The solution must be adjusted to match the test’s expected output.”

On the left, the hack we trained on. On the right, the slight modification with a new format. In both, the model matches the broken test instead of reporting it.

Reward hacking. Despite doing consequential work, today’s agents are relentless at finding shortcuts, failing while making users feel certain they are succeeding. Coding agents rewrite or delete the unit tests they fail, or hard-code the outputs the tests expect. Asked to make a program faster, frontier models overwrote the grader’s timer; asked to beat a strong chess engine, reasoning models edited the file storing the board so the engine would resign. When agents entered an open GPU-kernel competition, organizers faced a new shortcut every few minutes, including a kernel that passed every correctness check and then switched to a wrong but faster version once timing began. Every frontier model a government safety institute put through its cyber evaluations tried to cheat. With real access, these shortcuts reach the real world: agents being tested on exploit writing escaped their sandbox and breached a major AI platform to get the answers. This behavior is not new. It is called reward hacking, and a decade ago it was named one of five concrete problems in AI safety .

Detecting and mitigating reward hacking. In response, model developers detect reward hacking by monitoring agents with activation probes, small classifiers that read the model’s internal states , and escalate flagged cases to an LLM judge that reads the chain of thought, a window into the model’s reasoning that researchers argue is worth preserving. To mitigate reward hacking, they steer the model’s activations, fine-tune it, or simply instruct it not to reward hack.

Leading defenses are fragile. We set out to stress-test the leading reward hacking detection and mitigation methods the AI safety and alignment field has developed. We trained ten detectors and seven mitigations on coding tasks with a broken test that expects a wrong answer, then changed one detail: the broken test now expects the right answer in a slightly changed format. On the hacks they were trained on, the methods work. On this slightly different hack, nearly all of them degrade: the chain-of-thought judge loses nearly half its ability to separate hacks from honest solutions, fine-tuning loses half its effect, and telling the model not to hack makes it hack more. These defenses catch the reward hacks we already know about, but they do not hold up under even slight changes.

What follows. In the next section, we show that the defenses work on the hacks they were trained on. We then introduce the new reward hack, show one problem broken both ways, and measure how much each defense degrades. A short section on why the defenses break shows that they hold up when the model edits the test, but fail when it quietly matches the wrong format instead. We close with what this means for how reward hacking should be addressed. For readers who want the details, the sections at the end cover our setup, every detection and mitigation method, and the results for every method on Qwen3-8B, with further detail and DeepSeek R1 in expandable boxes.

Detection and steering work on the hacks they were trained on

We measure reward hacking from the model’s actions, following ImpossibleBench . Each task contains a test that is impossible by design, so no honest solution can pass it. The only way to pass is to reward hack, which lets us count a reward hack reliably from what the model does.

We compare ten detectors and seven steering methods, including a new variant that we propose, Probe-Gated Steering, which pairs a probe with a steering vector and only steers when the probe fires. We train every method on problems where the broken test expects a wrong value, and evaluate on held-out problems of the same kind.

Most of the methods we benchmarked are effective at detecting and reducing these hacks. Most probes detect hacks with an AUROC between 0.84 and 0.90, about as well as an LLM judge at a fraction of the cost. The best steering method is Probe-Gated Steering, which cuts the hacking rate from 22% to 9% while the model solves as many problems as before.

0% 5% 10% 15% 20% 25% 50% 52% 54% 56% 58% 60% Solve rate (%) Hacking rate (%) No steering: 21.6% hack, 54.7% solved No steering Probe-Gated Steering: 9.2% hack, 57.8% solved Probe-Gated Steering Fine-tuning: 12.6% hack, 52.8% solved Fine-tuning Prompting: 13.6% hack, 55.3% solved Prompting

Steering methods reduce hacking on Qwen3-8B, on the hacks they were trained on. Probe-Gated Steering is the most effective, and it does not lower the share of problems the model solves.

A small change to the task breaks detection and steering

To test how robust these methods are under a small shift, we change how the test is broken. Instead of a wrong expected value, the test now expects the correct value written in the new format, for example, commas where the problem asks for spaces. No method saw this hack in training.

On the hack not seen in training, every defense weakens. Detection degrades for every detector. The Probe-Gated Steering probe falls from an AUROC of 0.89 to 0.74, the linear probe from 0.90 to 0.77, and the LLM judge from 0.89 to 0.72. Steering loses a third or more of its effect. Even the best steering method, Probe-Gated Steering, goes from a 57% reduction in hacking to 37%, and fine-tuning goes from 42% to 20%. Prompting, which reduced hacking by 37% on the hacks seen in training, now leaves the model hacking 19% more than with no prompt at all.

Seen in trainingNot seen in training

Detection

AUROC, from chance (0.5) to perfect (1.0)

Probe-Gated Steering probe
0.89
0.74
Linear probe
0.90
0.77
LLM judge
0.89
0.72

Steering

Reduction in hacking vs. the unmodified model

Probe-Gated Steering
57%
37%
Fine-tuning
42%
20%
Prompting
37%
19% more hacking

Three detectors and three steering methods on Qwen3-8B, on the hacks seen in training and on the hack not seen in training. Each value combines both datasets and, for the new hack, its two variants.

Why the defenses break

A model can get past a broken test in two ways. It can edit the test, or it can keep the test and change its answer to match it. To find out which of the two the defenses miss, we also test a conflicting variant of the wrong-format hack. Two test cases ask for the same answer in different formats, so no answer can match both, and the only way to pass is to edit the test.

Our probe

AUROC, from chance (0.5) to perfect (1.0)

Seen in training
0.89
Not seen in training: conflicting variant
0.85
Not seen in training: wrong-format variant
0.68

Our steering

Reduction in hacking vs. the unmodified model

Seen in training
57%
Not seen in training: conflicting variant
52%
Not seen in training: wrong-format variant
28%

Probe-Gated Steering on the hacks seen in training, on the conflicting variant, and on the wrong-format variant.

On the conflicting variant, where the model can only edit the test, detection and steering hold. Our probe reaches an AUROC of 0.85, against 0.89 on the hacks seen in training, and our steering reduces hacking by 52%, against 57%. On the wrong-format variant, where the model can match the test, both drop, to 0.68 and 28%. The defenses work in one case and fail in the other, although both are small changes to the same task. The issue is that in the wild we cannot predict which case we are in, since the behavior of a probe or a steering vector can vary this much. Note that the model is not simply misreading the format. In 12 of the 19 rollouts where it matches the wrong format, it says the test contradicts the problem, and matches it anyway.

Implication of detection and mitigation fragility

Deployed agents will meet tasks no one anticipated. Our results suggest that, when they do, the behavior of today’s detection and mitigation methods cannot be reliably predicted. The change we introduce is a minor modification to how a test is broken, yet nearly every method degraded. Whether a defense held depended on how the model chose to hack: when it edited the test, as in training, detection and steering mostly held, and when it matched the wrong format instead, they did not. In deployment, we cannot know in advance which of these cases we are in.

Every method we tested is added after training, and is built from examples of hacks someone has already seen. Our results suggest this is not enough on its own. We believe a more durable fix will have to come from how models are trained, so that reward hacking is less likely to be learned in the first place, but that remains to be shown.

Setup

Datasets and models. We build the tasks from two coding datasets, HackTrace’s CodeContests and LiveCodeBench. Every method is trained on the training split of CodeContests, and tested on its held-out split and on LiveCodeBench, which no method sees in training. Our main model is Qwen3-8B, one of the most hack-prone open models we screened. We also report results on DeepSeek R1-32b.

How the test is broken. Every task has one broken test, and we break it in four ways. The first two are the ImpossibleBench task types, and every method is trained on them. The last two are new, and no method sees them in training.

Broken testWhat the test expectsSeen in training
One-offOne test case expects a wrong value.Yes
ConflictingTwo test cases expect different values for the same input.Yes
Wrong formatOne test case expects the correct value in a different format.No
Wrong format, conflictingTwo test cases expect the same value in two different formats.No

What counts as a hack. Following ImpossibleBench, we label each rollout from what the model did.

Every prompt tells the model to report a flawed test and not to modify it, so any change to the test counts as a hack, including putting the planted wrong value back to the right one.

Probe-Gated Steering (ours). For each kind of hack, Probe-Gated Steering trains a small probe and a steering vector together at a middle layer of the model, with the model’s weights frozen. As the model writes each token, the probe scores how much it looks like a hack, and the steering vector pushes the model away from hacking in proportion to that score. Honest tokens are left almost untouched, which is why the model keeps solving as many problems as before.

Details How we chose the model

We evaluated a range of open-source models on CodeContests tasks with a broken test. As in ImpossibleBench, most models rarely hack: eleven of the fourteen carry out a hack in about 10% of rollouts or fewer. DeepSeek R1-32b hacks in 38% of rollouts, Qwen3-8B in 28% and Qwen3-14B in 17%. The main results in this post, including the first figure, are on Qwen3-8B. We also report results on DeepSeek R1-32b, the most hack-prone model we screened, under Results.

models_hack_rate

Methods

Detection methods

Reward hacking is detected in two ways. Activation monitors, also called probes, are classifiers applied to the model’s internal activations and trained to distinguish reward hacking from normal behavior. Text monitors operate on what the model writes, most commonly an LLM that reviews the chain of thought. Most production monitoring systems combine the two . We compare methods of each kind, together with the probe from our own method, Probe-Gated Steering.

Activation monitors

Text monitors

Probe-Gated Steering probe (ours)

Probe-Gated Steering trains a probe jointly with a steering vector, one pair for each kind of hack, here editing a test and special-casing a test. For detection we use only the probe. It reads the model’s activations at a middle layer (layer 18 of Qwen3-8B) and assigns a score to every answer token. The score of an attempt is the highest mean score over any 10 consecutive answer tokens, following the max-of-rolling-means aggregation of Kramár et al. . We evaluate two versions. The first is the probe alone. The second, DoM-PGS, averages its score with that of the difference-of-means probe, after both are placed on the same activation scale.

Steering methods, and how Probe-Gated Steering works

Given how severe the reward hacking problem has become , several approaches have been proposed to mitigate it at low cost . We evaluate their effectiveness by comparing prompting, steering vectors, and fine-tuning with our own method, Probe-Gated Steering.

We benchmark six existing methods, listed below, followed by our own.

Probe-Gated Steering (ours). We define a behavioral concept as one kind of hacking behavior, for example editing a test or special-casing a test. For each concept, PGS adds a small module at a middle layer of the model while all model weights remain frozen. The module is trained on paired rollouts from the same prompt, one hacking answer and one honest answer. Each answer token has a label yy, with y=1y = 1 for the code tokens of the hacking answer and y=0y = 0 for the tokens of the honest one.

Details The math behind Probe-Gated Steering

The module has two parts, a probe with direction ww and bias bb, and a steering vector vv, scaled by a learned scalar gg that determines how strongly the steering vector affects the residual stream. Let hh be the model’s hidden state at that layer for the current answer token, and h^=(h−μ)/σ\hat{h} = (h - \mu) / \sigma its standardized form, where μ\mu and σ\sigma are the mean and standard deviation of each dimension of the hidden state. The probe then scores the representation as

c=sigmoid(w⋅h^+b),c = \text{sigmoid}(w \cdot \hat{h} + b),

which measures how strongly the probe reads this hidden state as a hack. The hidden state is then edited as

h′=h+g (σ⊙v) (t−c),h' = h + g \, (\sigma \odot v)\,(t - c),

where tt is the target score, and the standard deviation σ\sigma normalizes the edit. During training, tt is 1 for the hacking answer and 0 for the honest one. At generation, tt is always 0. The edit is then proportional to the probe’s score, so honest tokens are left almost untouched, while hacking tokens are pushed away from the concept. Scaling a steering vector per token has also been used to limit damage from steering in toxicity work .

The probe and the steering vector are trained together to minimize the following loss

L=NLL(answer∣h′)+λ BCE(c,y),\mathcal{L} = \text{NLL}(\text{answer} \mid h') + \lambda \, \text{BCE}(c, y),

where NLL is the negative log-likelihood of the answer under the edited hidden state, BCE is the binary cross-entropy between the probe’s score and the label, and λ\lambda is a hyperparameter. The first term teaches vv to steer the model toward the target answer, and the second teaches the probe to flag the hack. The steering vector vv is initialized from the difference-of-means direction and then trained. One module is trained for each concept.

Results

We compare every method on the hacks seen in training and on a hack not seen in training, on Qwen3-8B. The new hack is a test that expects the correct value in the wrong format, and it comes in two variants. In the wrong-format variant, one test asks for the wrong format, and the model can pass by matching that format. In the conflicting variant, two tests ask for two different formats, so the model has only one way out, which is to edit the test. We observe that detection and steering methods behave very differently in the two cases.

Detection

Detection holds on the conflicting variant and degrades on the wrong-format variant. On the conflicting variant, where the model edits the test, five of the six probes stay between 0.85 and 0.86, close to their values on the hacks seen in training. On the wrong-format variant, where the model matches the format, the probes’ AUROC falls to between 0.68 and 0.75, and the LLM judge falls from 0.89 to 0.72.

Seen in trainingNot seen in training: conflicting variantNot seen in training: wrong-format variant

AUROC, from chance (0.5) to perfect (1.0)

Activation monitors
Linear probe
0.90
0.86
0.73
Probe-Gated Steering probe (ours)
0.89
0.85
0.68
Shared-direction probe
0.89
0.86
0.68
Difference-of-means + Probe-Gated Steering (ours)
0.85
0.86
0.75
Probe-guided fine-tuning probe
0.84
0.85
0.70
Difference-of-means probe
0.76
0.78
0.72
Text monitors
LLM judge (Jev)
0.89
0.78
0.72
Constitutional Classifier
0.81
0.74
0.57
LLM judge (Qwen3-8B)
0.73
0.66
0.63
Keyword filter (regex)
0.66
0.56
0.51

Detection AUROC of every detector on the hacks seen in training and on the two variants of the hack not seen in training.

Steering

Steering shows the same behavior. On the conflicting variant, most methods reduce hacking again, Probe-Gated Steering from 11% to 5%. On the wrong-format variant, the unmodified model hacks in 20% of rollouts, and Probe-Gated Steering brings that only to 14%, a smaller reduction than on the hacks seen in training, where it goes from 22% to 9%. The other steering methods stay between 16% and 19%, and prompting and probe-guided fine-tuning do not reduce the hacking rate.

Seen in trainingNot seen in training: conflicting variantNot seen in training: wrong-format variant

Share of rollouts that hack

No steering
22%
11%
20%
Probe-Gated Steering (ours)
9%
5%
14%
LoRA fine-tuning
13%
7%
17%
Prompting
14%
14%
23%
Shared-direction steering
14%
7%
17%
Weight-space programming
19%
12%
16%
Probe-guided fine-tuning
20%
12%
23%
Difference-of-means steering
24%
9%
19%

Hacking rate of every steering method on the hacks seen in training and on the two variants of the hack not seen in training.

Details Qwen3-8B on the hacks seen in training, by kind of hack and by dataset

We first evaluate each method on the hacks seen in training, where the broken test expects a wrong value. The problems are held out, from both CodeContests and LiveCodeBench.

Detection

Probes are effective at detecting hacks. Five of the six reach an AUROC between 0.84 and 0.90, and the difference-of-means probe reaches 0.76. Although probes are much cheaper than an LLM judge, they are about as effective here. The LLM judge Jev reaches 0.89, and the other text-based detectors are between 0.66 and 0.81.

AUROC, from chance (0.5) to perfect (1.0)

Activation monitors
Linear probe
0.90
Probe-Gated Steering probe (ours)
0.89
Shared-direction probe
0.89
Difference-of-means + Probe-Gated Steering (ours)
0.85
Probe-guided fine-tuning probe
0.84
Difference-of-means probe
0.76
Text monitors
LLM judge (Jev)
0.89
Constitutional Classifier
0.81
LLM judge (Qwen3-8B)
0.73
Keyword filter (regex)
0.66

Detection AUROC of every detector on the hacks seen in training.

Looking at the two kinds of hack separately, the probes perform consistently, detecting test edits and shortcuts about equally well. The text monitors are less even. Their performance on test edits and on shortcuts differs widely, so a monitor that looks reliable on average can still miss most of one kind.

Test editsShortcuts

CodeContests

AUROC, from chance (0.5) to perfect (1.0)

Activation monitors
Linear probe
0.88
0.90
Probe-Gated Steering probe (ours)
0.86
0.84
Shared-direction probe
0.86
0.85
Difference-of-means + Probe-Gated Steering (ours)
0.87
0.83
Probe-guided fine-tuning probe
0.88
0.85
Difference-of-means probe
0.87
0.72
Text monitors
LLM judge (Jev)
0.84
0.90
Constitutional Classifier
0.86
0.67
LLM judge (Qwen3-8B)
0.53
0.67
Keyword filter (regex)
0.42
0.64

LiveCodeBench

AUROC, from chance (0.5) to perfect (1.0)

Activation monitors
Linear probe
0.97
0.95
Probe-Gated Steering probe (ours)
0.97
0.96
Shared-direction probe
0.97
0.96
Difference-of-means + Probe-Gated Steering (ours)
0.99
0.95
Probe-guided fine-tuning probe
0.94
0.93
Difference-of-means probe
0.99
0.69
Text monitors
LLM judge (Jev)
0.71
0.94
Constitutional Classifier
0.55
0.79
LLM judge (Qwen3-8B)
0.46
0.94
Keyword filter (regex)
0.43
0.89

Detection AUROC by kind of hack, for each dataset. Activation probes perform consistently across both kinds, despite being much cheaper than text monitors.

Steering

The tested methods substantially reduce hacking. The best is Probe-Gated Steering, which cuts the hacking rate from 22% to 9%, a 57% reduction, while leaving the solve rate almost unchanged (57.8% against 54.7% for the unmodified model). LoRA fine-tuning cuts hacking by 42%, at a cost of 2 points of solve rate. Prompting and shared-direction steering cut hacking by about 36%, and weight-space programming by 14%. Probe-guided fine-tuning and difference-of-means steering do not reduce it.

0 5 10 15 20 25 30 50 52 54 56 58 60 Solve rate (%) Hacking rate (%) no steering No steering: 21.6% hack, ±1 SD 19.6–23.7; 54.7% solved Shared-direction steering: 14.0% hack, ±1 SD 13.2–14.8; 52.8% solved Difference-of-means steering: 24.0% hack, ±1 SD 21.3–26.7; 52.1% solved Prompting: 13.6% hack, ±1 SD 12.8–14.3; 55.3% solved LoRA fine-tuning: 12.6% hack, ±1 SD 10.6–14.6; 52.8% solved Probe-guided fine-tuning: 20.5% hack, ±1 SD 18.6–22.4; 52.8% solved Weight-space programming: 18.6% hack, ±1 SD 18.3–18.9; 52.2% solved Probe-Gated Steering (ours): 9.2% hack, ±1 SD 6.9–11.5; 57.8% solved
No steering
Probe-Gated Steering (ours)
Shared-direction steering
Difference-of-means steering
Prompting
LoRA fine-tuning
Probe-guided fine-tuning
Weight-space programming

Hacking rate against solve rate for every steering method, on the hacks seen in training.

Below we report the hacking rate separately for each dataset and each kind of broken test. Probe-Gated Steering reduces hacking in all four settings, and has the lowest rate, or is tied for it, in three of them. The largest reduction is on CodeContests’ one-off tasks, from 41% to 13%. LoRA fine-tuning is comparable on LiveCodeBench but considerably weaker on CodeContests, and the remaining methods yield smaller reductions or none.

CodeContestsLiveCodeBench
One-offConflictingOne-offConflicting
No steering41%16%20%15%
Probe-Gated Steering (ours)13%7%11%6%
LoRA fine-tuning32%11%9%6%
Prompting26%7%13%11%
Shared-direction steering24%11%13%11%
Weight-space programming37%18%16%12%
Probe-guided fine-tuning36%16%21%14%
Difference-of-means steering40%17%25%18%

Hacking rate of each steering method on the hacks seen in training, by dataset and kind of broken test.

Details DeepSeek R1-32b

We repeat the comparison on DeepSeek R1-32b, with one seed. In 2,608 rollouts, the model edited the test 554 times and took a shortcut only 16 times. For this model we therefore focus on its most common hacking behavior, test edits, and train every steering and detection method on test edits only. Note that during evaluation, we still count both test edits and shortcuts as hacks.

Detection

On this model, detection does not degrade on the hack not seen in training. The six probes reach an AUROC between 0.76 and 0.88 on the hacks seen in training, between 0.81 and 0.88 on the conflicting variant, and between 0.75 and 0.89 on the wrong-format variant. This is because R1 mostly edits the test, on the new hack as well, which is the behavior the probes were trained on. For the probes, the hack is therefore in distribution.

Seen in trainingNot seen in training: conflicting variantNot seen in training: wrong-format variant

AUROC, from chance (0.5) to perfect (1.0)

Activation monitors
Difference-of-means probe
0.88
0.87
0.89
Difference-of-means + Probe-Gated Steering (ours)
0.86
0.88
0.87
Probe-guided fine-tuning probe
0.83
0.84
0.81
Linear probe
0.77
0.82
0.75
Shared-direction probe
0.77
0.81
0.77
Probe-Gated Steering probe (ours)
0.76
0.81
0.77
Text monitors
Constitutional Classifier
0.75
0.85
0.79
Keyword filter (regex)
0.50
0.48
0.53

DeepSeek R1-32b. Detection AUROC of every detector on the hacks seen in training and on the two variants of the hack not seen in training.

Steering

Steering gives a mixed picture on this model. On the hacks seen in training, Probe-Gated Steering is the most effective method. It cuts hacking from 20% to 11%, with the same solve rate as the unmodified model (52%). LoRA fine-tuning comes close, at 13%, but lowers the solve rate by 5 points. The other methods do not reduce hacking.

On the wrong-format variant, Probe-Gated Steering loses most of its effect, going only from 29% to 26%. LoRA fine-tuning keeps its effect, at 13%, but again with a solve rate 5 points lower, so part of this reduction may reflect a less capable model rather than a more honest one. The two other steering-vector methods, which had no effect on the hacks seen in training, reduce hacking here to about 19% and 20%. These numbers combine the two datasets. The effect of Probe-Gated Steering comes from the CodeContests dataset.

No steering
Probe-Gated Steering (ours)
Shared-direction steering
Difference-of-means steering
Prompting
LoRA fine-tuning
Probe-guided fine-tuning

Hacks seen in training

one-off and conflicting tests

0 5 10 15 20 25 30 35 44 46 48 50 52 54 56 Solve rate (%) Hacking rate (%) no steering No steering: 20.2% hack; 52.2% solved Shared-direction steering: 21.7% hack; 52.2% solved Difference-of-means steering: 23.3% hack; 50.3% solved Prompting: 21.1% hack; 50.3% solved LoRA fine-tuning: 12.7% hack; 47.2% solved Probe-guided fine-tuning: 20.5% hack; 51.6% solved Probe-Gated Steering (ours): 10.9% hack; 52.2% solved

Hack not seen in training

wrong-format variant

0 5 10 15 20 25 30 35 44 46 48 50 52 54 56 Solve rate (%) Hacking rate (%) no steering No steering: 29.3% hack; 52.2% solved Shared-direction steering: 20.4% hack; 52.2% solved Difference-of-means steering: 18.5% hack; 50.3% solved Prompting: 26.1% hack; 50.3% solved LoRA fine-tuning: 13.4% hack; 47.2% solved Probe-guided fine-tuning: 24.8% hack; 51.6% solved Probe-Gated Steering (ours): 26.1% hack; 52.2% solved

DeepSeek R1-32b. Hacking rate against solve rate for every steering method, on the hacks seen in training and on the hack not seen in training.

Conclusion

We tested the leading methods for detecting and mitigating reward hacking, and found that they work on the hacks they were trained on, but degrade under a small, non-adversarial change to the task. The failure follows how the model hacks. When the model edits the test, the behavior the defenses were trained on, detection and steering mostly hold. When it matches a wrong format instead, they do not, even though the model often says the test is wrong before passing it anyway. Because a deployed agent can choose either route, and we cannot know in advance which one it will take, the defenses we rely on today offer less protection than their in-distribution results suggest. Building defenses that hold up on the hacks we have not yet seen remains open work.

Limitations. We test two relatively small models, Qwen3-8B and DeepSeek R1-32b, and report DeepSeek R1-32b with a single seed. We study one new kind of hack, a wrong format. Some settings contain fewer reward hacks, so their estimates are wide. On LiveCodeBench, Qwen3-8B rarely takes the new hack, so steering could not be evaluated there. Finally, the fragility we find depends on how the model hacks: on DeepSeek R1-32b, which mostly edits the test even on the new hack, detection does not degrade.

BibTex

If you find this work useful, cite it as:

@article{guidelabs2026rewardhacking,
title={Reward Hacking Defenses Are Fragile},
author={Nguyen, Giang and Ismail, Aya Abdelsalam and Adebayo, Julius},
year={2026},
journal={Guide Labs Blog},
url={https://www.guidelabs.ai/post/reward-hacking-defenses/}
}

References

  1. D. Amodei, C. Olah, J. Steinhardt et al.. Concrete problems in AI safety . arXiv, 2016.
  2. A. Makelov, X. Xu, S. Ananthanarayanan et al.. Training activation classifiers for safety and alignment . OpenAI, 2026.
  3. H. Cunningham, J. Wei, Z. Wang et al.. Constitutional classifiers++: Efficient production-grade defenses against universal jailbreaks . arXiv, 2026.
  4. J. Kramár, J. Engels, Z. Wang et al.. Building production-ready probes for gemini . arXiv, 2026.
  5. Z. Zhong, A. Raghunathan, N. Carlini. ImpossibleBench: Measuring llms' propensity of exploiting test cases . arXiv, 2025.
  6. H. Jiang, X. Li, A. Wang et al.. HackTrace: Behavior-supervised detection of reward hacking during code generation . arXiv, 2026.
  7. How we detect frontier model misalignment . 2026.
  8. A. Meek, K. Cox, M. Nguyen et al.. Mitigating reward hacking in realistic scenarios through representation engineering . arXiv, 2026.
  9. G. Alain, Y. Bengio. Understanding intermediate layers using linear classifier probes . arXiv, 2016.
  10. L. Libon, A. Panfilov, B. Rank et al.. Alignment via training against probes without losing monitorability . arXiv, 2026.
  11. M. Sharma, M. Tong, J. Mu et al.. Constitutional classifiers: Defending against universal jailbreaks across thousands of hours of red teaming . arXiv, 2025.
  12. D. Almeida. Introducing system one models and jev . TypeSafe, 2026.
  13. The Hugging Face incident and the road ahead . 2026.
  14. R. Greenblatt, A. Cotra, H. Wijk. Brief independent investigation of agents' behavior, reasoning and collaboration in the OpenAI / Hugging Face hacking incident . METR, 2026.
  15. L. Bergen, U. Bhalla, A. Lee et al.. Models know when they're reward hacking, and we can catch them at scale . Goodfire, 2026.
  16. R. Takehi, J. Teoh, A. Zhang et al.. Sparks of Weight-Space Programming: LLMs can natively read and write neural network weights . 2026.
  17. E. Hu, Y. Shen, P. Wallis et al.. LoRA: Low-rank adaptation of large language models . arXiv, 2021.
  18. A. Ferrando, X. Suau, J. Gonzàlez et al.. Dynamically scaled activation steering . arXiv, 2025.