Zero-data · Multimodal · Recursive self-improvement
Adaptive Task Refresh for Zero-Data Multimodal Self-Improvement
*Equal contribution · §Project lead · †Corresponding authors
Lit = Uw: distinct logged prompts whose 8 rollouts got different rewards. The count is from the paper; which cells are lit is illustrative.
What this shows: once the training log says a pool is spent, random fresh tasks keep the loop climbing. What it does not show: that the log picks a better moment than a fixed schedule. Both executed refreshes came after three windows, where a three-window schedule would also refresh. One training run per policy.
Abstract
Recursive self-improvement requires models not only to acquire useful training experience, but also to decide when their current experience has ceased to be useful. Existing methods often focus on what experience to generate next, while paying less attention to when fresh experience should be introduced. Many also construct training targets through self-consistency or majority voting, making supervision depend on the model's own uncertain predictions.
We address these two questions separately: training dynamics determine when to acquire new experience, while the answers to new tasks remain independently checkable. A-RSI monitors reward variation and accuracy in training logs to refresh a task pool when its learning signal declines, without consulting validation or test scores. A static library of model-authored programs produces multimodal questions and derives their reference answers; we specify the task-generation contract, automatically verify chosen properties of the reward and refresh rules, and separately audit the generated answers. The training loop uses no external training examples or human-written answer labels. On the primary eleven-benchmark suite, A-RSI raises the performance average from 57.77 to 61.05 with 800 fresh tasks.
01 / Overview
A system can keep spending updates on a pool whose learning signal has declined, while refreshing too often consumes new tasks it may not need. And when training targets come from votes, the model's own errors become the labels for its next update. A-RSI separates the two decisions.
Prompts whose sampled responses receive different rewards are the ones that still produce a GRPO gradient. A-RSI counts them in its own training logs, together with logged accuracy, and refreshes the whole pool when that signal declines.
The program that draws bars of 18 and 11 also computes \(a=18-11=7\). Image, question, and answer come from one sampled configuration, so the answer is computed, not voted, and an independent solver can recompute it.
GRPO scores eight rollouts of each prompt and learns from their differences. When all eight get the same reward, the advantage is zero and the prompt teaches nothing. Counting the prompts that still disagree tells the controller whether the pool is still teaching. Rewards below use \(r=r_{\mathrm{ans}}+0.1\,r_{\mathrm{fmt}}+0.3\,r_{\mathrm{stop}}\).
Motivation. (a) Pool reuse can stall learning, while fewer truncations can raise accuracy. (b) Training logs guide refresh, and response audits distinguish completion from error correction.
02 / Interactive demo
Two pieces of A-RSI, running in your browser.
Four small illustrative templates written to the paper's generation contract: sample parameters \(z\sim p_\tau\), build one configuration, and from it render the image, write the question, and compute the answer, \((I,q,a)=G_\tau(z)\). The learner only ever sees \((I,q)\). In the paper, fresh replacement tasks are drawn from a 16,000-task reservoir (14,482 eligible) built once by seven model-written, program-based generators; the 500-task initial pool comes from earlier generation batches.
Reward \(r=r_{\mathrm{ans}}+0.1\,r_{\mathrm{fmt}}+0.3\,r_{\mathrm{stop}}\); \(r_{\mathrm{stop}}=0\) only when decoding ends normally with a final boxed answer, otherwise \(-1\). A wrong answer can never outrank a right one: \(1-0.1-0.3>0\).
At the end of each window the controller compares the logged signal with its value two windows earlier, after at least three accepted windows on the same pool:
03 / Method
Templates, the candidate reservoir, and the replacement pools are fixed before training. Training experience changes the timing of data replacement; it does not update the generator or the task-sampling distribution. Only the Solver policy is optimized.
Overview of A-RSI. Task templates, the candidate reservoir, and replacement pools are fixed before training. The controller reads training logs to decide when to activate a fresh pool; only the Solver policy is optimized.
"Saturation" is this operational rule, not a measured plateau in held-out performance, and neither condition proves the pool has been mastered. The paper uses \(n_{\min}=3\), lag \(\ell=2\), \(\rho=0.85\), and \(\eta=10\) points, fixed across stages and models.
For the response groups \(\mathcal{G}_w\) logged after dynamic resampling, with logged advantages \(A_{g,j}\) and answer rewards \(c_{g,j}\):
Advantages use the full reward, so differences in formatting or stopping also count toward \(U_w\). Uniformly incorrect responses also remove within-group variation, so \(U_w\) describes the logged training signal, not an exhaustive estimate of what the pool could still teach.
Because \(n_{\min}>\ell\), the lagged comparison always stays within one pool. When neither condition holds, the controller keeps the pool and checks again after the next accepted window.
DAPO removes prompts whose sampled responses all receive the same outcome, since they yield no within-group policy gradient. A-RSI uses the related signal for a different decision: \(U_w\) aggregates informative prompts across a window and helps decide whether the entire pool should be refreshed. It is a pool-level diagnostic, not a replacement for batch-level sampling.
If a refresh is requested but the declared replacement pools are exhausted, the controller refuses to start the next window. It does not silently reuse a pool or change the sampling rule. A failed window is retried once from the preceding accepted checkpoint; a second failure stops the run.
04 / Results
Starting from St at 57.77, A-RSI reaches 59.84, 60.40, and 61.05 at continuation windows 4, 6, and 8. Original-pool continuation scores 58.48 and 58.35, then halts. The refreshing arms continue learning beyond this plateau.
Five refresh policies, aligned by training updates. Hover the chart for exact scores, click a legend entry to hide or show a policy.
One training run per policy. Replacement also shrinks the active pool from 500 to 400 tasks, so +2.05 is evidence for the combined policy, not for freshness alone.
Operational rule, not proof of mastery. Third-pool actions are offline replays, not executed. This shows the rule reacting to each model's log; it does not show that it picks a better time than a schedule. Try other thresholds in Demo 02 →
With frontier selection held fixed, detector-triggered refresh scores 59.59 against 59.89 for refreshing every two windows (−0.30). Both executed Qwen refreshes came after three windows, as a fixed three-window schedule would. The first refresh point was set using existing logs. The results support the implemented fresh-pool policy in this run, without establishing that the detector selects a better time than a fixed schedule.
In the executed Qwen run, yes: both refreshes coincided with a three-window schedule. The rule departs from a schedule only in the offline final-stage replay, where Qwen keeps 83.98% of its signal and would refresh, while MiMo keeps 98.68% and would keep; a fixed schedule would replace both. A fixed-schedule random control is the paper's stated next step.
The simple version of the loop wins: the log says when, random fresh tasks say what. Single-run point estimates; variants share control runs and training histories, so they are not independent replications. The finding is limited to these generators, this model, and these selection rules.
For scale: two runs whose first two pool files hash identically scored 58.14 and 59.60 at window 2 (difference 1.46). Each arm was trained once.
Geometry and counting show the largest gains over St, alongside large drops in truncation. Grey ticks are the other refreshing arms; on chart and table accuracy, and on chart truncation, another arm is ahead.
05 / Analysis
A score can rise because more answers finish within the output budget or because already completed answers are corrected. Higher reward is not the same as better reasoning. These checks keep the two apart.
Zero-data self-improvement results should report, at a minimum, the base model at the same relaxed limit, the truncation rate of every model, and the gain at two or more limits. A gain that falls sharply with the limit is partly a gain in finishing.
200 consumed tasks from each of five generator groups, sampled before recomputing anything. Gold values and image hashes also match the source records.
A correct committed answer ranks above an incorrect one. With correctness held fixed, canonical formatting and normal termination each raise reward, so committing a wrong answer can raise reward without improving correctness.
Corrupted references, image bytes, source questions, and truncated images are all caught by the audit checks.
On 110 choice questions the production checker accepts a gold-prefixed ambiguous answer and a gold option followed by unrelated text. A stricter candidate passes all 16,000 checks; on 3,760 logged choice responses it changes one score.
Replaying the frozen generators reproduces 800 whole images exactly; the remaining 200 charts match in plot pixels and labels but differ in fonts.
The multi-series chart audit finds five ties for "third highest". They are retained as observed defects, so the pipeline is not described as guaranteeing zero label ambiguity.
06 / Case study
Two observable changes, reported separately: a response that used to hit the token limit now finishes, or a response that finished with a wrong answer now finishes with the right one. First the full-suite accounting, then selected examples.
Selected favorable examples, not frequency estimates. Image regions are marked by us from the paper's description of each response.
07 / Citation
arXiv preprint · 2026 · arXiv:2610.38568