Entropy-gated suffix masking: turn-level credit for long-horizon agentic RL
A deep-research agent changed its behavior mid-training and the reward never showed it. Per-turn entropy located the change. We propose a method that masks execution turns in a failed trajectory only when a high-entropy decision turn is found.
GRPO training of multi-turn agents can be unstable: Search-R1 reports a collapse
In the Search-R1 paper, GRPO learns faster than PPO at first, then its training reward drops to zero near step 230 (panel a).
The authors switched to PPO for that reason. This is the failure everyone can see: the reward curve itself breaks.
Is a collapse the only way training goes wrong? The next slide shows a second failure, from our own runs, that the reward curve does not show at all.
Training can also go wrong with a healthy reward curve: the agent stopped searching and started guessing
| Turn | Acc @278 | Acc @297 | Search @278 | Search @297 |
|---|
Training reward shows no drop, but the first turn stops being a search (98% → 32%) and early-turn accuracy falls.
A silent form of training instability, a signal that locates it, and a method that prevents it
A behavior change hidden under a flat reward
Around step 280 the policy switched from search-first to guess-first. Training reward showed no drop, and token clip fraction stayed under 0.25%.
turn-1 accuracy 37.1% → 18.2%Per-turn first-token entropy locates it
The jump is in the decision turn. The execution turn that follows shows no jump. This is a cheap signal to log in any multi-turn training run.
turn-0 entropy ~0.001 → ~0.05Entropy-gated suffix masking
In a failed trajectory with a high-entropy decision turn, keep the penalty on that turn and mask the execution after it. The entropy jump does not appear and scores rise.
NQ 0.485 → 0.521Setup
From the Search-R1 task to a long-horizon deep-research environment.
How Search-R1 trains a search agent
A fixed set of Wikipedia documents, embedded once offline.
FAISS returns the documents whose embeddings are most similar to the query.
The final answer is checked by exact match. That single result is the only reward.
Search-R1 trajectories are only about 4 turns, so we added tools to lengthen the horizon
We reproduced the task with GRPO on the Base model (verl, 28k context). Training was stable: we did not reproduce the reported collapse, and the score beats the reported PPO number.
| Run | Algorithm | Model | NQ |
|---|---|---|---|
| Search-R1 (reported) | PPO | Qwen2.5-7B-Base | 0.480 |
| Ours | GRPO | Qwen2.5-7B-Base | 0.498 |
| Ours | GRPO | Qwen3-8B | 0.497 |
But trajectories are only about 4 turns: question → tool call → tool response → answer. That is too short to study long-horizon training, so we added two tools. The next slides describe them and compare the two environments.
So we rebuilt the task as a deep-research loop
Search search
Same retriever as Search-R1, but instead of dumping every document into context, returns a summary truncated to at most 8k characters.
Read chunk read_chunk
- Gather full docs stashed by the search tool (kept out of the token context); fall back to parsing
Doc N (Title: …)headers. - Split on blank lines into paragraphs; drop anything under 80 chars.
- Rank by lexical overlap: count of distinct shared lowercase words with the query.
- Dedup already-returned sections by MD5; truncate each to 2,000 chars.
- Return the top 2 sections.
Verify external_verify
A stronger external model judges the answer against the gathered context and returns CORRECT, UNCERTAIN or WRONG. The policy decides whether to answer or keep searching.
The action space is now open: at every turn the policy chooses to search, read, verify, or answer and end the episode.
From Search-R1 to a deep-research agent: what changed
| Original Search-R1 | Deep-research agent (ours) | |
|---|---|---|
| Typical trajectory | question→search→answer |
search→read_chunk→answer→verify→repeat as needed |
| Tools | 1: search | 3: search, read_chunk, external_verify |
| What search returns | All retrieved documents go into the context | A summary truncated to 8k characters; full documents are kept out of the context |
| Reading evidence | Implicit, from the dumped documents | Explicit read_chunk call: top 2 sections by word overlap, 2,000 characters each |
| Checking the answer | None | A stronger model judges the answer against the gathered context: CORRECT, UNCERTAIN or WRONG. It does not see the ground truth |
| Choices per turn | Search or answer | Search, read, verify, or answer and stop |
| Trajectory length | About 4 turns | About 12 turns, capped at 20 |
| GRPO baseline, NQ (Qwen3-8B) | 0.497 | 0.485 |
| Training behavior | Stable | Reward stable, behavior shifts near step 280 |
Same corpus, retriever, questions and terminal reward. Only the tool set and the horizon change.
Trajectories now run about 12 turns
Average turns per trajectory roughly triple compared with the original setup, with a hard cap at 20.
Longer trajectories give the policy many more places to fork, and that is exactly where terminal-reward credit assignment gets weak.
Drag on a chart to zoom · double-click to reset · smoothing is an EMA like TensorBoard's.
Evidence
What changed, where, and what did not detect it.
Total entropy moves to a higher level while training reward shows no drop
Entropy averages about 0.063 over steps 250–280 and about 0.11 over steps 300–340, and it stays elevated. The reward curve keeps its trend through the same window. Watching reward alone, nothing looks wrong.
First-token entropy per turn shows which turns changed
A change in behavior shows up first in the opening tokens of a turn: whether to call a tool, which tool, with what arguments.
So we logged the entropy of the first k tokens of every turn (k = 1…128). k = 8 through 128 behave the same way; k = 16 is shown.
Inside the shaded window, turns that were almost deterministic become uncertain. Turn 0 goes from about 0.001 to about 0.05. This is the switch to guess-first from the opening slide.
The uncertainty is in the decision turn, not the turn that follows it
Turn 0 is where the agent decides what to do first. Its first-token entropy sits near 0.001, then averages 0.040 over steps 280–300 and 0.049 over steps 300–400.
Turn 1 carries out that decision. It shows no jump and keeps its gradual decline, from about 0.050 early in training to about 0.019 at the end.
The policy is changing its decision. It stays settled on how to execute.
Caveat: after the shift, turn 1 is a different action (verify after a guess, where it used to be read after a search), so part of its level reflects the action type.
Token-level clipping is not sensitive to a turn-level shift
PPO and GRPO clip per-token ratios to limit how far each update moves the policy.
Clip fraction rises slightly around the shift, from about 0.11% to 0.16%, and never exceeds 0.24%. Almost no tokens reach the clip boundary.
Each token moved a little, but the behavior moved a lot. The unit that changed is the turn.
Method
A hypothesis for the shift, then our proposed method: turn-level credit assignment, with entropy locating the decision turn.
Why would the decision turn drift? One end-of-episode reward blames every turn equally
The evidence shows the policy is changing its decision turn while execution stays settled. Our explanation is credit assignment.
GRPO gives one advantage to a whole trajectory. Every turn receives it, whether that turn was the wrong decision or correct work that followed it.
In a single-turn task this is harmless. In a 12-turn trajectory, most of the turns being penalized had nothing to do with the failure, so the signal on the decision turn is diluted and noisy.
Reward stays flat while the agent switches from search-first to guess-first. This is what the hypothesis is tested on.
Training reward collapses to zero. We did not reproduce it at short horizon, so whether it shares this cause stays open.
If the hypothesis is right, assigning credit per turn in failed trajectories should prevent the shift. The next two slides turn that into a method.
In failed trajectories, execution turns absorb blame that belongs to the decision
Example question: “Where was the director of Film X born?” Terminal reward only; GRPO assigns the trajectory's advantage to every turn.
Positive trajectory
- Decisionsearch “Film X director”✓↑
- Executionread_chunk → “directed by Y”✓↑
- Executionsearch + read → “Y was born in Z”✓↑
- Executionanswer “Z” · verify CORRECT✓↑
Every turn reinforced. Credit is right.
Wrong decision, wrong execution
- Decisionskip search, guess “director = W”✗↓
- Executionread_chunk, ignores the evidence✗↓
- Executionanswers “Q”, not in context✗↓
- Executionskips verify, ends episode✗↓
Every turn penalized. Credit is right.
Wrong decision, correct execution
Root cause- Decisionskip search, guess “director = W”✗↓
- Executionsearch “W birthplace”✓↓
- Executionread_chunk → “W was born in V”✓↓
- Executionanswer “V”, grounded in evidence✓↓
Correct execution penalized only because the decision turn was wrong. This is the blame we want to remove.
Assumption, not measured: how often execution after a wrong decision is correct. The argument needs only that this blame is unreliable, and that penalizing a confident, correct action costs more than leaving a wrong one unpenalized.
Entropy-guided turn-level credit assignment
Results
Scores, training reward and entropy under each method.
Both entropy-based turn-level methods beat the baseline
| Method | Applied to | Turn weight wt | Signal | NQ ↑ | Δ |
|---|
Training reward ends higher too, so the flat baseline curve was hiding a loss
All three runs start near 0.17 and stay within about 0.2 points of each other through step 200.
After that they separate. Over the last 60 steps: suffix masking 0.503, high-entropy turn training 0.491, baseline 0.478.
The baseline flattens after about step 300 (0.481 → 0.475) while suffix masking keeps climbing.
The baseline never drops, which is why reward alone could not flag the problem. The cost only appears next to a run that avoids it.
The baseline's entropy jumps; both entropy-based methods stay flat
Baseline: first-token entropy averages 0.040 over steps 280–300 and 0.049 over steps 300–400, peaking at 0.094 at step 298. It is still about 0.02 at the end.
Suffix masking: stays at zero from step 280 to the end.
High-entropy turn training: one brief blip at steps 337–340, peaking at 0.005.
The signal that revealed the behavior shift does not appear under either method.
Discussion
What we ruled out, limitations, and where this goes next.
Three explanations we tested and set aside
Off-policy staleness
The first explanation we considered was stale rollouts. But clip fraction stayed under 0.25%, so the policy barely moved within a step, and the shift built up over about 20 steps of fresh rollouts. The evidence points to credit assignment.
Train–inference mismatch
The inference and training engines agree to within 0.25% throughout, so engine mismatch does not drive the shift.
Turn-level IS ratio
Could a different turn-level signal do the same job? The importance-sampling ratio per turn moves around the shift too, though less sharply. Clipping on it locks the policy into exactly 8 turns and matches the baseline score, so entropy is the better locator.
One assumption remains unmeasured: how often execution after a wrong decision is correct. The method needs only that blame on those turns is unreliable.
When a failed trajectory has a clear decision turn, blame that turn and not the execution after it
Reward can stay flat while behavior changes
The policy switched to guess-first with no visible drop in training reward. Token clip fraction stayed under 0.25%.
Per-turn first-token entropy locates it
The jump is in the decision turn; the execution turn stays settled. This is a cheap signal to log in any multi-turn training run.
Entropy-guided turn-level credit
Suffix masking after a turn whose entropy stands out from the trajectory average; other failed trajectories keep the full penalty. NQ 0.485 → 0.521, and training reward ends about 2.5 points higher.
- Action space. The environment has three tools. Adding more tools would extend the action space and test whether entropy still locates the decision turn.
- Scenarios. Results are from one deep-research QA setting. The method needs testing on more scenarios.
Backup
Supporting charts for questions.
Clipping on the turn IS ratio locks the policy into exactly 8 turns
The turn-level IS ratio follows a trend similar to entropy around the shift, though flatter. So we tested whether clipping high-IS turns could also prevent it.
High-IS turn clipping: from step 245 on, every trajectory is exactly 8 turns. The policy has become deterministic, so its entropy is flat by construction and says nothing about the shift. Its NQ equals the baseline (0.485).
Baseline: about 9.3 turns before the shift, rising to about 13 over steps 300–360 as guess-first adds turns.
The flatter signal does not locate the decision turn. It suppresses change everywhere and the policy stops adapting.
Train–inference mismatch stays under 0.25%
The inference engine and the training engine agree closely throughout, so engine mismatch is not what drives the shift.
Turn-level importance ratio around the shift
GSPO clips at the sequence level. In agentic RL the natural unit sits between token and sequence: the turn.
Around the entropy spike, the turn-level IS ratio is high before and low after. It moves ahead of the collapse.
Beyond the 80/20 Rule (Qwen): training only on the top 20% high-entropy tokens works better, because those tokens create forks in the CoT. In long-horizon tasks the fork is a whole turn.
Turn-2 divergence from the reference policy against turn-0 entropy
Logged turn-2 old-policy minus reference-policy divergence, overlaid on first-16-token entropy of turn 0, for steps 251–522.
Use the chips to isolate each signal and drag to zoom into the spike window.