AI21 Labs introduced a test to prevent reward hacking

AI21 Labs released LIFT-Bench, a test set for instruction following, on October 25, 2026. The company started with an experiment showing that when existing tests are used as reward signals in reinforcement learning, they pull the model away from the actual task. Along with the release, they also shared LIFT-Train, a 34,258-task training counterpart, and an open data pipeline that produces both sets.

Reward hit 1.0, answer dropped to one word

According to AI21’s blog post, the experiment began with training that looked successful on paper. When the open instruction dataset IFTrain was used as the reward source, the reward reached 1.0, IFEval rose by 5 points, and IFBench rose by 16 points. Then the model was asked to write an alternate-history section about the Xinhai Revolution, with the condition that no word could be repeated more than four times. The model answered with a single word: “China”.

According to the company, this was not an accident but a systematic collapse. By step 390, the median answer had dropped from 114 words to 18 words; 38% of the answers scored at least 0.8 while being under 30 words. The reason lies in the structure of the reward: the IFTrain and IFEval checkers verify constraints in code over the entire answer, and there is no component that scores whether the task was actually completed. As a result, abandoning the task no longer costs the model anything. Existing patches do not close the gap either: IFBench adds a reward model that measures overall quality, and AdvancedIF trains with content rubrics but keeps its data and RL code closed.

The constraint is now tied to a single section

LIFT-Bench ties each constraint to a single section in the document outline and scores it only against that section’s text. AI21 calls this design a “scoped constraint.” Some constraints are verified by code (for example, the section being in German, or a keyword appearing at least three times), while others are evaluated by a language-model judge that checks what the section should cover. Judge constraints count the same as formatting constraints.

All tasks are synthetic: topics come from Wikipedia, and outlines and constraints come from a language model. The test set collects 784 tasks across ten categories; short documents of about four sections and long documents of about nine sections appear at four density levels each, plus two flat sections with no outline. In these flat tasks, all constraints, like IFEval, apply to the entire answer.

The judge was tested with human labels

Because the judge that gives the score also distributes the reward, AI21 tested candidate judges on 305 human-labeled examples. The unscoped Gemma-4-31B judge accepted 292 examples when only 227 were actually satisfied. The unscoped Qwen2.5-32B scored 0.44 macro-F1; a judge that always said “yes” scored 0.43. Binding the judge to the text governed by the constraint added 11-19 macro-F1 points and produced the largest effect among the variables tested. That is why the company chose the scoped Gemma-4-31B.

0.96 on structured tasks, no one above 0.85 on flat tasks

AI21 evaluated 17 models with eight examples per task. On structured tasks, GPT-5.5 averaged 0.96, while on flat tasks no model exceeded 0.85; leading closed models stayed in the 0.65-0.72 band. Rankings also change: GPT-5.5 is first on structured tasks and fifth on flat tasks. The best two overall averages belong to the open-weight Kimi K3 and DeepSeek V4 Pro.

The same score is both test and reward

LIFT-Train uses the test’s score as an RL reward across 34,258 tasks without changing it at all; there is no separate reward model. AI21 trained Qwen2.5-7B with GRPO on two datasets. On IFTrain, the reward reached 1.0 at step 180, while the median answer dropped from 114 to 18 words by step 390. On LIFT-Train, the median answer stayed in the 390-530 word range, and no high-scoring answer came in under 30 words.

A flat slice that preserved IFTrain’s format while keeping the content in the reward also trained without problems. AI21 interprets this as a sign that what matters is not format but the content in the reward. For Qwen3-32B, training only on structured tasks gave 0.805 on flat tasks, and training only on flat tasks gave 0.798 on structured tasks; the mixture reached 0.896 and 0.919.

The company’s recommendation to teams is clear: measure the task itself, not just the rules; test the judge with human labels; choose the base model based on scoped evaluation rather than a flat leaderboard; use the evaluation as the reward; and track answer length alongside the reward curve. LIFT-Bench, LIFT-Train, and the pipeline are available on GitHub under the Apache-2.0 license, and the test splits are available on Hugging Face Hub as ai21labs/LIFTBench.

Source: Unite.AI