Wednesday, September 9, 2026

Activation steering is usually validated one behavior at a time, so it is unclear whether steering vectors encode coherent semantic structure or just exploit behavior-specific shortcuts. The authors test this by extracting steering vectors for the 20 values of Schwartz's Theory of Basic Human Values and checking whether their pairwise geometry reproduces the theory's circumplex, where compatible values sit close together and conflicting ones sit at opposing angles.

• They build a ~26K-sample contrastive benchmark of (question, value, positive answer, neutral answer) quadruples from ValueBench and Touché, represent each method's output as the mean residual-stream shift it induces at a probe-selected layer, and compare the mean-centered cosine-similarity matrix of the 20 value vectors against the theoretical cos(18°·k) matrix using Spearman and Pearson correlation, a hierarchical-structure correlation, and a polarity separation score.
• On Qwen3.5-9B-Base, distribution-driven methods track the theory well, with SAS reaching Spearman ρ = 0.51 (p < 10⁻¹³) and CAA 0.46, above the raw-activation baseline of 0.22, while behavior-centric OPT, COLD-Steer, and BiPO score ρ ≤ 0.12 with no statistical significance despite comparable steering accuracy on the target value.
• Geometric fidelity rises monotonically with scale within the Qwen3.5 and Gemma-3 families and with model recency at the 7–9B range (Falcon-7B 0.20, Mistral-7B 0.32, Llama-3.1-8B 0.37, Qwen3.5-9B 0.46), but instruction tuning degrades it for every method (SAS falls from 0.51 to 0.33), and it is decoupled from capability since Gemma-4-31B (0.38) trails Qwen3.5-4B (0.41).
• Steering one value and measuring accuracy shifts on the other 19 shows distribution-driven methods lift compatible values and suppress opposing ones, and geometric fidelity predicts this cross-value transfer better than raw accuracy gain does (Δρ = 0.06 on the continuous metric, 0.18 on the hierarchical one); the paradigm split also replicates under revised Moral Foundations Theory, where distribution-driven family separation is 0.04–0.10 versus roughly 0.001 for raw activations and near zero or negative for behavior-centric methods.
• The analysis is anchored to the Schwartz circumplex with only a coarse family-level check under Moral Foundations Theory, extracted vectors are target-conditioned aggregates rather than monosemantic directions since real arguments express multiple values, and compute limits mean most cross-backbone comparisons use only CAA.

arXiv:2609.06289 PDF

Latent visual reasoning methods build hidden states that carry image information, but as long as the answer decoder can still attend to the original multimodal KV cache, a latent state can be informative without being causally required for prediction. Causal Visual Recurrent Reasoning (CVRR) closes that bypass by construction: it starts recurrence from the pretrained model's own image-conditioned question state, refines it over several steps while re-reading fixed visual evidence, and discards every visual row and multimodal cache before decoding so the final recurrent state is the only image-conditioned path to the answer.

• Two diagnostics motivate the design: existing latent reasoners barely react to having their latent states swapped for blank-image, mismatched, or noise content (UniVLR shifts by at most 0.5 points), while the question-token rows of a frozen Qwen2.5-VL-7B-Instruct multimodal state alone reach 75.4% versus 75.9% for the full state and 35.6% for text-only rows, showing pretrained visual competence is already concentrated in the question representation.
• Layer-wise activation patching on the frozen backbone locates a visual-read boundary at layer 20, after which layer 21 is reused as a shared recurrent transition with a rank-32 LoRA (about 2.9M trainable parameters, 0.035% of the model) for T=4 steps, updating only the question rows with a 0.5 mixing coefficient while the visual rows stay fixed, trained on Visual CoT with nothing but answer-token cross-entropy.
• Under the strict no-bypass interface CVRR reaches 81.2% on V, 52.7% MMVP pair accuracy, and 55.2% BLINK overall, matching or beating a full-multimodal SFT control that keeps the standard answer path, whereas six compatible latent reasoners (LVR, Monet, SkiLa, Laser, UniVLR, HyLaR) retrained under the same constraint collapse to at most 39.8% on V and 2.7% MMVP pair accuracy.
• Ablations and causal interventions show the recurrence does real work: dropping visual re-reading costs 12.0 V points and 34.7 MMVP-pair points, replacing the native initialization with a text-only anchor falls to 37.2% on V, corrupting the final recurrent state drops V* accuracy from 81.2% to 14.0%, and crossing question states with visual evidence from a paired image produces ±14.8-point difference-in-differences effects that vanish when question-to-visual attention is blocked.
• Limitations include a 4.1-point drop on MME-RealWorld-Lite relative to the backbone (attributed to Visual CoT domain mismatch, with the SFT control dropping 6.1), declines on BLINK Counting and Jigsaw subsets, a backbone-specific boundary search, single-seed training, mechanistic analyses that mostly rely on multiple-choice first-token readout, and visual evidence that is persistent rather than adaptively acquired during reasoning.

arXiv:2609.06746 PDF

Rollout generation dominates the wall-clock cost of RL post-training, and speculative decoding with an online co-trained draft is the standard fix, but modern feature-conditioned drafts like EAGLE-3, DFlash, and DSpark need branch-structured attention that causal context parallelism (CP) cannot express and target hidden states that live on other pipeline-parallel (PP) stages. NVIDIA's answer is a systems design that leaves the policy's parallel layout untouched: branch attention folded into packed zigzag-ring attention for CP, plus TapChannel, an out-of-schedule side path that ships intermediate target features to the draft stage under PP, all integrated into NeMo-RL.

• Under CP, each branch query attends to two key sets, the causal main-sequence prefix computed with the existing packed zigzag-ring pass and a small set of branch-local keys kept on the anchor-owning rank, and the two partial outputs are merged with the same online-softmax log-sum-exp reduction used between ring steps, so ring traffic stays independent of branch count or depth.
• Under PP, TapChannel gives each tap-producing stage a pre-allocated mailbox slot in the draft stage's memory (CUDA IPC when colocated, a dedicated NCCL communicator with GPUDirect RDMA across nodes) guarded by sequence stamps, so features cross non-adjacent stages once without entering the pipeline schedule, and one-sided writes hit 27–39 GB/s and complete fan-in 4.5–8.5× faster than pinned host staging with only 1.6% HBM contention on the receiver.
• Co-trained drafts on Qwen3-8B track the no-speculation baseline in reward, AIME 2024 accuracy, and train–inference KL on DAPOMath-17K, and across targets from 8B to 122B they reach 2.28–4.78 accepted tokens, 1.19–2.23× rollout speedup, and 1.16–1.88× end-to-end speedup, with DFlash and DSpark consistently beating EAGLE-3 on acceptance length.
• The packed zigzag CP kernel beats the best SpecForge USP configuration by 2.9×, 2.3×, and 1.5× in latency at CP=2/4/8 with 2.7× lower peak memory, and at 256K tokens TTT attention drops from 17.7 s to 2.35 s going from CP=1 to CP=8 (94% parallel efficiency) while per-GPU memory falls from 53.2 GB to 7.5 GB.
• Gains shrink where verification is unusually expensive or cheap: large MoE targets (Qwen3.5-122B-A10B, GPT-OSS-120B) and the linear-attention Nemotron-3.5-Lightning-30B-A3B land at only 1.16–1.35× end-to-end, multi-turn Workplace Assistant runs get 1.25–1.43× because rollout is just 55.8% of step time and tool latency is untouched by faster decoding, and draft co-training adds 13.6–34.3% to policy-update time (worst for EAGLE-3 because of its TTT passes), with the PP overhead study averaged over only the first 10 update steps.

arXiv:2609.07108 PDF

Linear attention's fixed-size recurrent memory must decide at every token how strongly to overwrite existing associations, and delta-rule models predict that write strength from the current token alone with no notion of how confident the memory already is. The paper recasts delta-rule associative memory as a linear-Gaussian state-space model, so the Kalman filter's covariance tracking supplies an evidence-aware write gain that DeltaNet, Gated DeltaNet, and KDA turn out to omit.

• Under the state-space view, the memory transition is the process model (identity, scalar, or diagonal decay for the three existing models), each token is a noisy observation of the latent key-value map along one key direction, and the Kalman gain replaces the token-predicted scalar gate with a residual write weighted by predictive covariance and observation noise.
• Exact filtering needs a dense per-head covariance and a state-dependent Riccati recursion that breaks the parallel scan, so Diagonal KDN projects each posterior onto a diagonal Gaussian via reverse-KL mean-field inference (preserving the exact one-step posterior mean), and Isotropic KDN keeps a single uncertainty scalar per head; both uncertainty recurrences are Möbius maps, giving associative scans with logarithmic parallel depth and O(d_k) and O(1) auxiliary state respectively.
• Because the mean-field projection discards cross-channel correlations and can underprotect stored key directions, an information-scaling factor μ = d_k inflates the post-write precision increment to curb later overwrites, and ablations show WikiText perplexity and mean accuracy peak at that setting while the overall effect is small and metric-dependent.
• In parameter-matched recurrent-only pretraining on FineWeb-Edu, Diagonal KDN reaches 18.64 / 14.15 WikiText / LAMBADA perplexity and 54.97% mean six-task zero-shot accuracy at 750M/50B (vs. 53.87% for KDA and 54.39% for Mamba-3 MIMO), and 15.04 / 9.75 perplexity with 60.45% accuracy at 1.3B/100B, also posting the best recurrent-only RULER needle-in-a-haystack aggregate at both scales with notably stronger multi-key retrieval.
• Gains are modest in absolute terms and narrow further in the hybrid setting with sliding-window attention, the experiments use GDN-2's fixed backbone and layer ratio rather than a tuned recurrent-to-attention mix, and the largest scale tested is 1.3B parameters.

arXiv:2609.07816 PDF

Recursive self-improvement needs a concrete mechanism for a system to observe its own capability gaps and turn them into the next round of training, and the authors argue a deployed agentic routing harness already provides one: every user turn leaves an execution trajectory plus a routing record of predicted capability demand, tier served, and outcome. NeoHorse-1 (4B and 9B, post-trained from Qwen3.5) is a first prototype that converts those records into curriculum-ordered supervised fine-tuning and on-policy distillation, then uses evaluation feedback to reallocate the next training mixture.

• Harness trajectories (on the order of 10^5 to 10^6) are serialized as user-turn examples that keep interleaved reasoning, tool calls, and harness context, with loss applied only to the current turn's assistant spans, and are admitted through rule-based structural validation, a six-dimension semantic judge (goal attainment, instruction adherence, tool use, evidence consistency, error recovery, termination), and subscene-level Scene/Goal/Outcome labeling.
• The router's C0 to C3 capability-demand estimate, re-derived from the request and prior history rather than the tier actually served, orders SFT into a three-stage curriculum of increasing demand with some low-scored examples held back for later stages, and the same schedule selects starting contexts for routing-guided on-policy distillation, where a fixed teacher scores student-generated responses under a reverse-KL objective over the top-K tokens plus a remainder bin.
• Across ten benchmarks the macro-average rises from 58.94 to 64.87 for the 4B model and 65.60 to 69.04 for the 9B model, with the largest gains on harness-based agent tasks (4B PinchBench 71.19 to 77.33, WorkBuddy Bench 24.62 to 34.41, τ²-Bench 84.29 to 88.46), and the post-trained 4B model matches or beats the Qwen3.5-9B base on several benchmarks.
• Under an identical routing-guided recipe, harness trajectories beat the public Toucan tool-agent dataset by +6.26 points on a five-benchmark average (+11.31 on τ²-Bench, +8.54 on HumanEval), and nested-subset scaling of harness supervision lifts the development average from 69.31 to 71.45.
• The results cover only a single pass of the evaluation-selection-update loop, so whether gains compound across iterations is untested; the 9B model's instruction-following scores are flat with one minor regression, PinchBench and VitaBench are single-run, and the RSI framing rests on a design argument rather than demonstrated multi-generation improvement.

arXiv:2609.08183 PDF

Frontier-scale RL post-training now means multi-turn, tool-using rollouts from trillion-parameter mixture-of-experts models, where turn-taking schedules idle hardware and numerical drift between the serving engine and the trainer can silently invalidate the objective. Miles v0.1, built on slime, is a full-stack system that rebuilds every stage of the RL loop (SGLang rollout, a Megatron-LM or FSDP trainer, and three weight-synchronization transports) around components that are verified, clean, and customizable.

• Rollout throughput comes from session-affinity routing that pins every turn of an episode to the engine holding its KV-cache prefix plus least-loaded placement for new sessions (96% prefix-cache hit rate in the reference run), and a fully asynchronous mode where engines generate continuously into a bounded buffer while the trainer pulls finished groups, with staleness defined pessimistically as the current weight version minus the oldest version in the group.
• Trajectory fidelity is enforced by a token-in-token-out (TITO) session server that owns tokenization and records exact token IDs, log-probs, and routed experts per turn, supports linear or branching histories (the latter for harnesses like Claude Code that fork or compact context), and by rollout routing replay (R3), which replays rollout-time MoE expert assignments in training at a cost of roughly 60 MB per 32K-token trajectory at 60 layers and k=8.
• The trainer ships low-precision recipes as end-to-end contracts where rollout and training run identical quantization (FP8 blockwise generally available, MXFP8 and NVFP4 in beta), streams optimizer state from disk one bucket at a time, which on Qwen3-30B-A3B cuts actor offload from 24 s to 5.2 s and reload from 8.9 s to 1.3 s, and corrects residual train-rollout mismatch with truncated importance sampling or clip-or-pop on the importance ratio.
• Weight updates use a shared bucketed pipeline (512 MB buckets, separate expert pass) behind three transports, NCCL broadcast, RDMA peer-to-peer writes, and disk-delta publishing, motivated by a full broadcast of Kimi K2 1T-A32B taking almost a minute.
• The end-to-end case study runs fully asynchronous agentic RL on GLM-5.2 744B-A40B over terminal-use coding tasks on 64 NVIDIA GB300 GPUs at a median step time of 263 s over the first 30 steps, but the authors flag that MXFP8/NVFP4 remain beta, environment connectors are experimental, the session server cannot yet carry image or video inputs, R3 is left off in that reference run, and several measurements come from a single configuration.

arXiv:2609.08368 PDF

Reinforcement learning for long-horizon LLM agents stalls on sparse rewards, and the usual fix of supervised warm-up on expert trajectories is data-hungry and narrows exploration. The authors instead adapt the environment, building Feedback-Enriched Environments (FEEs) that inject action hints early and richer state observations later, so the agent can bootstrap its own exploration without any expert data.

• The pilot study on SciWorld compares two feedback types, action guidance (suggested valid next steps) and observation enrichment (hidden-state details such as task progress), across early and late episode steps and training phases, and finds action guidance wins early while observation enrichment pays off late, so the final recipe applies AG-Early for the first 100 training steps and OE-Late for the remaining 100, each injected with 0.5 probability.
• Across Qwen3-4B and Qwen3-8B trained with GRPO, DAPO, and GSPO on SciWorld and BFCL V3 Multi-Turn, FEEs beat standard environments in every configuration, with an average gain of 2.82 points, standouts of 53.91 to 60.94 for 8B GSPO on SciWorld and a 10-point jump on BFCL Base for 4B GRPO, and post-RL averages of 47.81 (4B) and 48.11 (8B) that surpass Qwen3-235B-Thinking and approach GPT-5.4.
• Enriched training also changes the dynamics: without entropy regularization the FEE-trained 4B model holds steady entropy for 300 steps while the standard run collapses around step 250, FEE-trained agents gain 4.3 points on the hardest BFCL tier when evaluated in plain environments, and a probe experiment shows the enriched information is internalized into the weights rather than acting as a prompt-time hint.
• A key design constraint is that all rollouts within a group must receive the same feedback for a given state, since randomizing feedback inside a GRPO group corrupts the advantage estimates and produces erratic performance swings.
• The scaffolding has costs: agents become more compliant and regress on BFCL Miss Param and Miss Func tasks where they should question the request (4B DAPO drops 41 to 34 on Miss Param), the early/late phase boundaries are hand-tuned, and on the harder AppWorld benchmark FEEs still fail to produce any positive reward.

arXiv:2609.08404 PDF

Weak-to-strong generalization asks whether a stronger student can learn from a weaker post-trained supervisor and then surpass it, but standard on-policy distillation makes the weak policy itself the optimization target and so imports its capacity ceiling. OPRD instead extracts the weak teacher's post-training policy shift (post-RL logits minus pre-RL reference logits, mean-centered and normalized) at each student-visited prefix and uses it only to rescale the student's own verifier-driven policy gradient, never as a target.

• At every token the student's GRPO logit gradient is split into its projection onto the teacher-shift direction and an orthogonal remainder, and only the projection is amplified by 1+λ (default λ=0.5, top-10 vocabulary truncation), which keeps the map invertible so RLVR stationary points are unchanged while first-order progress can only increase; positive-alignment scaling is on from the start and negative-alignment scaling (verifier-backed departures from the teacher) is ramped in over a warm-up.
• In successive transfer from a Qwen3-4B teacher to a Qwen3-8B student, OPRD averages 51.91 across AIME'24, AIME'25, HMMT'25 and OlympiadBench versus 43.99 for the strongest baseline KDRL, and 55.18 versus 44.38 on four Reasoning Gym tasks, reaching teacher-level performance with 33–67% fewer updates than GRPO and continuing to improve where OPD plateaus near the teacher.
• Consolidating four Qwen3-4B-Base specialists into one Qwen3-8B-Base student gives 58.77 average Pass@1, beating Mix-RL by 11.09 points and the specialist average by 14.12 points while exceeding every specialist on its own task, and the same mechanism works strong-to-weak (49.40 vs 20.20 for OPD on Knights & Knaves in an 8B-to-0.6B transfer).
• On a three-task average against recent weak-to-strong methods OPRD scores 60.81 versus 54.21 for S2L-PO, 49.22 for W2S-OPD and 37.81 for Direct-OPD, a response-style analysis places the OPRD student closer to a pure-GRPO student than to its teacher on all five style categories, and the overhead is only +11.9% wall-clock and +10.2% peak memory relative to GRPO.
• The method needs a nonzero verifier gradient, so when most rollout groups receive identical rewards there is nothing to amplify (an 8B-Base-to-1.7B-Base Knights & Knaves run stays below 20% while OPD reaches 40.5%), the reference checkpoint must be chosen to avoid length bias (a step-0 reference plateaued at 52.5% on Color Cube while a step-30 reference reached 89.5%), and all experiments use Qwen3 models of at most 8B on math and logic tasks only.

arXiv:2609.08798 PDF

Speech synthesis, content editing, paralinguistic and acoustic control, and enhancement or separation are normally served by separate task-specific models. AuK folds all five task families into one open-source model that maps a natural-language instruction plus optional audio context to a target waveform, trained on roughly 3.03 billion instruction-audio instances and 1.95 million hours of effective supervision.

• The backbone is a ~1.5B-parameter FLUX-style rectified-flow transformer (10 dual-stream MMDiT blocks followed by 20 single-stream DiT blocks) conditioned on a learnable weighted sum of frozen Qwen2.5-Omni layer-wise hidden states for semantics and on 50 Hz latents from a frozen audio VAE trained on speech, music, and general audio for acoustics.
• Training runs a 50k-update generation-only warm-up and 600k updates of joint generation-editing flow matching on 256 GPUs, then flow-based listwise preference optimization on 9,080 human-rated editing candidates across 818 groups, followed by Flow-GRPO on generation using ASR accuracy, speaker similarity, and a Qwen2.5-Omni-7B style-consistency judge as rewards.
• Distillation with consistency initialization plus task-routed Decoupled DMD yields AuK-Flash, which samples in 4 steps with no classifier-free guidance and runs 4.5× faster than the 32-NFE teacher, though separation examples had to be routed to a plain clean-latent regression loss because uniform DMD made the student drift back toward the unprocessed mixture.
• The authors claim leading results on Seed-TTS-Eval, InstructTTSEval, MMAE-Speech, SpeechEditBench, and Ming-Freeform-Audio-Edit, but only "competitive" performance on DNS Challenge, CHiME-4, and Libri2Mix restoration and separation, and the comparisons are presented as bar charts against prior SOTA systems rather than as headline numbers.
• Nearly all editing supervision is synthesized by other models (F5-TTS, IndexTTS2, CosyVoice2, SeedVC), so edit quality is bounded by those pipelines, and free-form requests depend on an LLM Prompt Enhancer that snaps continuous rate, pitch, and loudness requests to the discrete values seen in training.

arXiv:2609.08936 PDF

Turn-based voice assistants bolt interactivity on through external voice-activity detection and pipeline glue, and a single model struggles to be both instantly responsive and capable of long-horizon tool use. Tencent's Hunyuan Speech team proposes Gander, which splits the job between a full-duplex omni "Cerebellum" that natively decides when to listen, speak, or interrupt over streaming audio and video, and a training-free "Brain" (Codex or Claude Code) that executes agentic tasks asynchronously via structured tool calls.

• The Cerebellum is a streaming Thinker-Talker model that flattens each one-second window into a chunk of audio/visual tokens, a predicted control token (listen, speak, or interrupt), and a variable number of text tokens, with a 128-chunk sliding window (~2 minutes), SigLIP any-resolution vision at up to 448×448 with roughly 16× token reduction, audio downsampled to ~10 tokens/s, and a separate speech-token decoder plus streaming flow-matching vocoder for zero-shot voice cloning.
• Delegation to the Brain is expressed through three tool calls (task_start, task_send with main or fork routing, and task_resolve with cancel/allow/deny actions) mediated by an orchestration runtime with lean and coordinator modes, so stronger reasoning models can be swapped in without retraining the interaction model.
• Training uses a 2.7M-example corpus: ~37% speech interaction including 260.8K synthesized full-duplex InteractionSpeech dialogues with explicit interruption and backchannel timing, ~40% audio-visual streaming data (1.1M pairs re-aligned with Qwen3.5-297B-A17B and rewritten by DeepSeek-V4-Pro), ~13% agentic trajectories, and ~8.5% robustness and negative data for noise, multi-party speech, and staying silent.
• Reported evidence is mostly internal human evaluation claiming parity with open-source SOTA on spoken dialogue and competitive omni interaction, plus robustness demos under noise, multi-party talk, and backchannels; the authors acknowledge no dedicated benchmark exists and fall back on BigBenchAudio and subjective project-page demos.
• Notable simplifications: the Brain receives only the transcribed query and final video frames rather than the live multimodal stream, the coordinator mode adds latency and nondeterminism, and the 2-minute context window bounds how much interaction history the Cerebellum can retain; models, code, and data are released.

arXiv:2609.08977 PDF