Thursday, September 3, 2026

Frontier models can often tell when they are being evaluated, which threatens the validity of the safety evaluations that deployment decisions rest on — but the measurement methodology itself turns out to be biased. EvalDetectBench is an open, Inspect-compatible pipeline that scores evaluation-versus-deployment transcripts with per-judge calibrated probe questions and corrects for a previously unmeasured confound in how deployment data is sourced.

• The pipeline presents a judge model with a transcript plus a closed-ended probe question from a 20-item corpus (17 direct, 3 indirect), averages the binary labels into a P(eval) score, and reports AUROC over a curated suite of 746 trajectories (413 evaluation, 333 deployment) drawn from frontier system-card benchmarks and in-the-wild sources like DataClaw, ShareChat, WildBench, and CodeChat.
• Probe questions do not transfer across model families: a fixed set calibrated on one model under-elicits awareness elsewhere by up to 0.08 AUROC, while exhaustive per-judge selection of subsets of size ≤5 reaches held-out AUROC of 0.91 on Claude Sonnet 4.6 and 0.90 on Opus 4.6, beating both the prior fixed-set baseline and the mean of 500 random subsets at every subset size.
• The identity of the model that generated a deployment transcript explains 11.25% of P(eval) variance (F=67.2, p<0.001) — Claude-generated transcripts read as more deployment-like to every judge — and this survives the resampling mitigation proposed in prior work (8.4–11.3% across regeneration depths k=0 to k=9), so the authors instead apply a ComBat-style stratified z-score harmonisation blended at 10% that pulls mixed-pool AUROC back toward the unconfounded same-family reference.
• Across 12 judges and 13 benchmarks, discrimination runs from 0.76 (Gemini 2.5 Pro, Claude Haiku 4.5) to 0.89 (Claude Opus 4.6), with the newest model in every closed-source family strictly highest and mean P(eval) above 0.83 on a subset of benchmarks pooled across three frontier judges.
• Two limitations bound the result: the prompted framing cannot rule out that asking about evaluation primes the answer (AUROC only defends against inflation that is uniform across classes) or that a model sandbags the probe, and the deployment pool is assembled from public sources rather than any provider's real traffic, so the harmonisation is offered as a pragmatic correction rather than a debiased estimator.

arXiv:2609.01611 PDF

Code retrieval benchmarks score topical or identity matches, so they cannot say whether an embedding model actually distinguishes code that passes tests from a near-identical variant that does not. ExecRetrieval closes that gap by planting execution-verified single-edit buggy mutants of each query's own canonical implementation directly in the search pool, turning rank ordering into a direct test of functional discrimination.

• The benchmark pairs 939 Python tasks across ten algorithmic domains with one canonical implementation, a 7–10 assert test oracle, and up to four mechanically mutated distractors drawn from six locked bug types (off_by_one, wrong_operator, swap_arguments, remove_edge_case_check, wrong_comparison, off_by_one_boundary), for a 4,694-snippet closed-world corpus in which every canonical passes all its tests and every distractor fails at least one.
• Generation used GPT-5.4 at high reasoning effort behind a five-gate validation pipeline (schema, AST semantics, canonical execution, distractor execution, corpus integrity) reaching a 91% first-attempt validation rate, after earlier Claude Sonnet runs produced accidentally correct "buggy" distractors in 127 of 400 cases by rewriting the algorithm instead of mutating it.
• Across 23 dense embedding configurations plus BM25 under provider-native invocation, top-k retrieval saturates while rank-1 collapses: Gemini Embedding 2 hits exec@10 = 1.00 but only exec@1 = 0.331, and when rank 1 is wrong it is a paired buggy variant 91.5–99.4% of the time across the four leading systems.
• The mechanism is a wrong-signed but tiny cosine gap — the canonical scores below at least one of its paired distractors in 66.8% of queries on Gemini Embedding 2 and 78.4% on Qwen3-Embedding-8B (median gap −0.002) — and a pool-density ablation shows the effect needs only one counterfactual, dropping the strongest system from 0.993 with no near-clones to 0.678 with one; deception is spread evenly across mutation types at 44.3% overall within a narrow 39.3–48.0% band.
• Every magnitude is conditional on a near-clone actually sitting in the pool, which the authors do not measure for deployed corpora; the benchmark is also Python-only, closed-world (so exec@k is incomparable to open-domain code search), built from mechanical LLM mutations rather than natural human bugs, and limited to first-stage embedding retrieval with cross-encoder and LLM rerankers left to future work.

arXiv:2609.01865 PDF

Long-context prefilling is dominated by quadratic attention, and dynamic sparse methods like FlexPrefill decide per-head sparsity patterns with an indirect divergence proxy plus a cumulative mass threshold that ignores how post-softmax attention mass is actually distributed. CRISP replaces both mechanisms: routing reads structural mass at attention sinks and the recency window directly, and token selection thresholds against an estimated noise floor rather than a coverage target.

• Routing uses C_struct, the proxy attention mass falling on the first 128 sink tokens and last 128 local tokens, which is a constant-time slice reduction over an already-computed map and reproduces the Jensen-Shannon Divergence routing decision on 94.0% of Llama-3.1-8B heads and 88.1% of Qwen2.5-7B heads while eliminating the pooled matmul, extra softmax, and KL passes.
• The paper formalizes a "mass cliff" between architectural sinks, task-relevant signal, and near-zero background whose per-token mass scales as O(1/n), so a cumulative γ threshold either terminates inside the sink before reaching any signal or drags in O(n) noise blocks to satisfy the residual — a failure that tuning γ makes worse, confirmed by FlexPrefill at γ=0.97 degrading InfiniteBench and RULER on Llama while costing latency.
• Vertical-Slash heads instead keep every block whose pooled mass exceeds α·μ, where μ is the mean residual mass over non-anchor blocks, making α=1.0 a calibration-free "above average" rule; the diffuse Pooled-Estimation path retains γ-cumsum because its mass profile decays smoothly with no cliff to navigate.
• At α=1.0 CRISP matches or beats exact dense attention on retrieval-heavy suites (InfiniteBench 48.7 vs 48.6 on Llama, 28.7 vs 24.0 on Qwen) and recovers up to +28.0 pp on Qwen passkey and +17.8 pp on Llama KV retrieval over FlexPrefill, while attention-only latency at 512k tokens reaches 5.30× over FlashAttention at α=1.25 versus 4.41× for the baseline.
• Limitations are concrete: the structural proxy assumes sink-having architectures and breaks under gated attention, evidence covers only two model families at 7–8B, Llama RULER regresses 0.40 pp because aggregation tasks need borderline blocks the noise floor discards, and the fixed routing cost makes sparse prefilling a net loss below roughly 64k tokens (at 8k both methods run ~4× slower than dense).

arXiv:2609.01925 PDF

Mixed human–AI groups playing a repeated tangram description game converge on shared conventions non-monotonically as the share of LLM agents rises, and the resulting consensus differs not just in strength but in who authored it and what it means. Across 40 rounds of anonymous pairwise exchanges with agent proportions of 0%, 12.5%, 33.3%, 50%, and 75%, the authors identify three regimes — agent-facilitated human consensus, disrupted convergence, and agent-led consensus.

• Consensus strength rises 8.0% above the pure-human baseline of 0.695 at 12.5% agents, falls 23.1% at 33.3% and 14.5% at 50%, then rebounds to 0.725 at 75%, with directional-movement analysis showing that humans move toward the agents' linguistic space more than the reverse from 33.3% upward.
• Agent influence penetrates from surface wording to meaning: at 33.3% agents contribute 60.7% of the final stable vocabulary, and conceptual contribution reaches 100% at 75%, pushing the conceptual-lexical ratio (CLR) past parity to 1.24 at 50% and 1.27 at 75%.
• Agent-led consensus is measurably thinner in content than human-led consensus — concreteness drops from 2.986 to 2.724, use of real-world analogies from 0.802 to 0.050, holistic (whole-object) framing from 0.685 to 0.353, and event-based narration from 0.385 to exactly 0 — trading rabbit/sit vocabulary for triangle/asymmetrical geometric enumeration.
• The mechanism is a semantic attractor built from agents' shared pretrained prior plus stable round-to-round expressions: prompting agents toward high persistence cuts consensus 9.6% while raising CLR 117.8%, and swapping Qwen2.5-VL-32B-Instruct for Qwen2.5-VL-7B-Instruct restores consensus from 0.534 to 0.725 as the weaker model cannot hold expressions steady enough to pull humans.
• Humans discount partners they judge to be AI (regression coefficient β = −0.463 on adoption willingness), but at 75% agents that resistance reverses (Δ = +0.884) even as agreement with the final convention drops — a dissociation the authors read as structural conformity rather than persuasion, though the paradigm uses content-neutral tangrams, fully mixed random pairing, a single model family, and as few as six humans in the 75% condition.

arXiv:2609.02122 PDF

Turning the paper into a digest treatment now.

Faithfully turning a research paper into a working repository is hard because papers describe methods abstractly and leave implementation assumptions implicit, and existing agents pass their intermediate understanding downstream as free-form plans that the coding stage can quietly reinterpret or simplify. PaperCompiler instead compiles paper-grounded evidence into explicit repository-level specifications that bind each requirement to the specific file responsible for implementing it.

• The pipeline runs in three phases: Paper Grounding extracts atomic implementation items tagged as paper-supported, inferred, externally delegated, or unresolved (with long material like prompt templates and algorithm listings copied verbatim into a reference registry), Specification Compilation reconciles these into method-level requirements carrying explicit non-degradation constraints and assigns file ownership plus producer–consumer artifact flows, and Constraint-Guided Repository Generation emits files in topological order using upstream committed code and downstream specs as compatibility constraints.
• On 90 papers from Paper2CodeBench (ICLR, ICML, and NeurIPS 2024 subsets) with an o3-mini backbone and o3-mini-high judge, it beats PaperCoder by 13.8% on reference-based fidelity (3.647 → 4.152), with smaller gains of 4.7% reference-free (4.562 → 4.777) and 4.3% on P2C-Ex — the widening gap under reference-based scoring suggests the improvement is in paper-specific detail rather than surface completeness.
• Manual labeling of roughly 2.6K evaluator critiques shows high-severity failures dropping from 13.2% to 6.1%, with missing core components falling from 12.3% to 6.8% and evaluation mismatches from 13.4% to 8.4%, though API/schema mismatches actually rose from 2.3% to 4.0%.
• Ablations on nine papers isolate Requirement Reconciliation (−0.51 reference-based) and File-Level Contracting (−0.46) as the load-bearing components, while Context Slicing removal improves reference-free scores yet costs 0.26 in reference-based fidelity — evidence that paper-only evaluation rewards repositories that look complete without checking method alignment.
• Cost is roughly 1.71M tokens per repository versus 0.98M for PaperCoder (about $1.88–$7.51 at o3-mini rates), and the authors are candid about the ceiling: the system reads text-parsed papers only so architecture diagrams are lost, specifications are generation-time guidance rather than correctness guarantees, and their SEABO case study shows a repository with the right modular topology that still falls back to a dummy dataset and raises NotImplementedError on the state-only path.

arXiv:2609.02272 PDF

Long-context decoding is dominated by KV cache reads: every generated token forces a scan of the entire context, and existing sparse-attention methods still pay O(N) per step to score which tokens to keep. Declarative Attention (DA) instead asks the model to say where it wants to look, emitting mode tags inside its chain-of-thought that the inference engine parses like tool calls to build the attention mask directly, with no auxiliary scorer and no training.

• The protocol defines three modes — <global> (full context), <focus> (named segments only), and <local> (response so far only) — over a context pre-split into ~2K-token "magic chunks" presented as a simulated tool-use transcript, with a state machine that rewrites vLLM's KV-cache block table at block granularity so FlashAttention kernels run unmodified.
• Across 15 long-context tasks from RULER, LongBench v1/v2, LooGLE, and ZeroSCROLLS, zero-shot DA cuts attended tokens by 52.0% on Gemma-4-31B (13.43M to 6.45M per response) and 31.1% on Qwen-3.6-27B, for accuracy drops of only 1.27pp and 2.75pp, which a roofline analysis projects into decode wall-clock of 0.71× and 0.77× of vanilla on a B200.
• A no-mask ablation isolates the mechanism: the chunked prompt format alone is essentially free on accuracy but raises attended tokens 66.2% above vanilla on Gemma because DA generates 15–35% more decode steps, and the mask converts that overhead into a net win by cutting 71.1% of those tokens.
• Benefits scale in both directions the field is already pushing — relative accuracy climbs from 29% of vanilla at Gemma-4-E4B to 99% at Gemma-4-31B (tracking focus-tag parse success, which rises from 58% to 99%), and absolute savings grow from about 1M tokens at short context to 21M tokens per response in the longest bin.
• The main caveats are that thinking mode had to be disabled because models could not follow the protocol inside reasoning traces, <global> steps retain full attention and still account for over 80% of DA's attended tokens, the benchmark chunk boundaries are artificial rather than the natural tool-call segments of agentic contexts, and the wall-clock figures are roofline projections at assumed 40% MFU / 70% MBU rather than measured latency.

arXiv:2609.02737 PDF

Autonomous ML research agents are built from a model backbone and a harness, but neither supplies the domain-specific know-how that separates knowing a method from making it work — the operating details buried in repositories and papers written for human readers. DisCo distills those declarative sources into compact, verified agent skills that a fixed agent loads on demand as extra operating context, leaving the model and harness untouched.

• Each source is distilled through a four-stage pipeline (scope, ground, construct, verify) into a skill graph whose nodes follow the SKILL.md / references/ / scripts/ layout, so an agent reads an entry summary and opens only the branch a task needs; no skill is admitted without assertion-backed checks, and unresolved gaps are recorded rather than hidden.
• Distillation runs task-agnostically over 1,000 widely used ML repositories to build the AREX-Skill Library — 5,353 skills indexed by 20 areas and 178 capability families behind a router — at roughly $40 of GPT-5.5/GPT-5.6-sol compute per repository, plus task-oriented graphs generated on demand for a specific problem.
• Holding the GPT-5.5 backbone, the Codex harness, and the downstream execution budget fixed, skills lift MLE-bench Any-Medal from 31.11% to 72.89% (+41.78 points, a 134.3% relative gain), PaperBench replication from 29.45% to 39.59%, FrontierCS from 70.63 to 77.14, and PassNet AS Score from 1.343 to 1.5313 — beating TorchInductor's 1.419 and cutting failed samples from 14 to 5.
• Gains concentrate where unguided exploration is most wasteful: MLE-bench High-difficulty tasks rise 366.8% (13.33% → 62.22%), and the 47 FrontierCS tasks scoring below 50 without skills climb from a mean of 19.43 to 45.99, with per-task gains essentially uncorrelated with the extra tokens and steps skills consume (Spearman's ρ ≈ 0.006–0.015).
• The construction budget is excluded from the matched run-time comparison, so the reported gains price in only retrieval and not the one-time distillation cost, and skills actively hurt on 2 of 20 PaperBench tasks — both above-average baselines — suggesting retrieval imprecision can pull the agent off a task-specific strategy it would otherwise have found.

arXiv:2609.02749 PDF

Agent benchmarks have become prohibitively expensive to run — a single pass of a frontier model over SWE-bench Verified costs several hundred dollars, and SWE-bench Multimodal exceeds $2,200 — yet prior cost-cutting work only shrinks the task list rather than the cost of each task. The core idea is early outcome prediction: an agent's final pass/fail is often legible from its intermediate behavior long before the run ends, so a cheap classifier can halt the rollout and record the predicted score.

• EarlyEval trains a pair of LightGBM classifiers — one for success, one for failure — over ~500 features spanning behavioral signals (activity counts, event timing, stalling patterns, error and test status), TF-IDF/SVD embeddings of task, action, and feedback text, and optional reference-solution overlap, then halts the run at the first step where either Platt-calibrated probability crosses its threshold.
• Under a leave-one-agent-out protocol over 21,000+ labeled trajectories from 16, 37, and 22 agents, the method eliminates 26% of execution steps on SWE-bench Verified at 95% accuracy (32.7% input and 28.7% output tokens) while shifting each agent's resolve rate by only 1.1 percentage points on average; Toolathlon and TerminalBench land at 13–25% step savings for 89–97% accuracy.
• Leaderboard order survives the truncation, with Spearman ρ = 0.991 across all 16 SWE-bench Verified agents and ρ ≥ 0.959 elsewhere, though only 59–81% of agents keep their exact ordinal position.
• The failure classifier is the workhorse — its precision holds at 89–99% everywhere, while the success classifier is trustworthy only on SWE-bench Verified (88–94%) and collapses to 61–69% on TerminalBench under a held-out scaffold and to near-zero coverage on Toolathlon, and the two heads almost never fire on the same trajectory.
• Ablations show behavioral features drive most of the benefit (removing them cuts coverage from 34.8% to 23.4%) while reference-solution features are nearly dispensable, but the framework needs a pool of pre-labeled trajectories on the target benchmark and the authors explicitly scope it to iterative development, not to producing citable headline scores, given the systematic 1–2 point resolve-rate deviation.

arXiv:2609.02783 PDF

Reinforcement learning with verifiable rewards gives a single outcome-level reward per rollout, so a nearly-correct solution with one slip is penalized exactly like a hopeless attempt, and existing fixes either need a trained process reward model or assume the teacher and student share reasoning patterns. The core idea is that once reasoning first goes wrong, everything after it is conditioned on an invalid prefix and carries little extra information — so it suffices to locate the single "Pitfall Step" where the rollout first breaks and assign credit differently on either side of that boundary.

• An off-the-shelf teacher LLM first generates its own reference solution (groups where the teacher's solution fails the verifier fall back to vanilla GRPO), then judges each student rollout and marks the first genuine reasoning error, splitting it into a valid prefix and an erroneous suffix that receive separate token-level advantages, with a recentering offset keeping the group mean at zero.
• Across 12 student/teacher/domain combinations Cliff wins every setting, beating on-policy distillation by 15% and GRPO by 7% on average — for Qwen3-4B-Base with a frontier teacher, math averages 65.66 vs 61.68 for GRPO (MATH-500 83.2, AIME 36.98) and coding 25.96 vs 24.20 — and an ablation that gives the teacher outcome-only judging power gains almost nothing, showing the benefit comes from the prefix/suffix credit split rather than from adding a teacher.
• Judging is easier than solving: on 100 human-annotated DAPO-Math rollouts, Qwen3-32B scores only 65% at solving but 88% at judging, and with a verified reference solution all teachers exceed 90% judge accuracy with the identified pitfall step landing about 3 sentences from the human annotation.
• The positive-reinforcement weight on the valid prefix must be λ = 0; raising it to 1.0 drops the math average to 63.98 while inflating responses from 1506 to 1959 tokens, and overlength rollouts must be hard-capped at p(a)=0 — both guards against a length-hacking channel the paper characterizes formally in its appendix.
• Weaknesses are mostly in the supervision signal: dropping the ground-truth filter costs roughly 2% for the Qwen3-32B and Gemma3-27B teachers (though a frontier teacher is unaffected), judge–verifier consistency sits at only 85–90% during training, false negatives occur in about 10% of judgments, and every training step pays for teacher generation plus per-rollout judging on top of the usual GRPO cost.

arXiv:2609.02817 PDF

An end-to-end post-training recipe for competitive programming — curated problems, distilled reasoning traces, supervised fine-tuning, reinforcement learning, and feedback-driven test-time refinement — is used to isolate what each component actually contributes to gold-medal performance. The resulting system was run live during IOI 2026 under human contest constraints and scored 535.4/600, above both the gold threshold of 361.12 and the top human contestant's 498.27.

• Training uses 22,000 curated problems from 16 competition families packaged into executable evaluation environments, with DeepSeek-V4-Flash generating 1.2M reasoning traces for Nemotron-3-Nano-CC (30B total, 3B active) and 477,642 for Nemotron-3-Ultra-CC (550B total, 55B active), followed by GRPO with binary execution rewards for the Nano model only.
• GenCorrect, the test-time strategy, runs five rounds of generating up to 200 candidates, clustering them by code similarity to pick 10 diverse representatives, submitting those under the official 50-submission cap, and conditioning the next round on accumulated per-subtask scores plus three complementary reference solutions.
• Ablations show SFT does the heavy lifting — Nano-CC goes from 21.7% to 47.3% on IOI 2025 Score@1, 16.9% to 46.7% on ICPC 2025, and 17.6% to 70.7% on LiveCodeBench Pro — while RL adds only 1-2 points on top and, applied to the base model without SFT, reaches just 24.9%.
• Test-time compute is where the largest single jump comes from: Nano-CC climbs from 360.6 to 468.2 points across five GenCorrect rounds on IOI 2025 and Ultra-CC from 343.9 to 502.0, with the SFT-only 550B model overtaking the smaller RL-trained one despite starting a round behind.
• The live run consumed a peak of 760 NVIDIA GB300 GPUs and relied on NVFP4 quantization that traded 6.6 points of Score@1 for a 3.7× throughput gain, so the human comparison holds only at the system level under matched time and submission limits, not at equal resources; compute also ruled out RL at Ultra scale and the full training corpus cannot be released.

arXiv:2609.02849 PDF