PyTorchKR Papers

최신 논문

총 12,715편
태그·기간 등 상세 필터

카테고리 탐색

로그인 후 이 카테고리 구독

arXiv:2609.39027v2cs.CL

A Missing Piece for Trustworthy AI Reviewers: From Benchmarking Rhetorical Robustness to SciCore Review

AI reviewers can assign different judgments to manuscripts that report the same science in different wording, potentially rewarding rhetorical optimization over scientific improvement. We formulate Rhetorical Robustness as the joint requirement of stability across content-preserving rewrites and discrimination across papers. We introduce RobustReview, a controlled full-manuscript benchmark with 1,260 manuscript versions, and evaluate 30 reviewer configurations. The benchmark reveals false robustness, where low rewrite sensitivity coincides with score collapse across papers, and shows that human alignment and rhetorical robustness rank reviewers differently. Moreover, the evaluated content-focused prompting protocol does not consistently improve robustness across backbones. Motivated by these findings, we introduce SciCore, a dual-branch reviewer that averages a full-manuscript judgment with a judgment based on an extracted, structured science core. This design combines manuscript-level assessment with a content-normalized view intended to reduce rhetorical sensitivity. In our primary GPT-5.5 comparison, SciCore achieves a leading joint stability-discrimination profile among the benchmarked reviewers while maintaining competitive human alignment. These results identify rhetorical robustness as a distinct evaluation target and demonstrate the potential of science-core review to improve it.

arXiv:2609.38169v2cs.CL

STEPQuant: When and Where Errors Matter in Delta-Rule Recurrent State Quantization

Linear attention replaces growing KV caches with fixed-size recurrent states, yet these persistent states can become a substantial memory bottleneck under concurrent serving. Directly quantizing recurrent states to low precision often leads to severe accuracy degradation, as quantization errors propagate through successive state updates. We discover that the impact of these errors depends on two complementary dimensions: temporally, errors in long-lived memory can persist across many decoding steps; spatially, errors in different key rows affect model outputs differently, while state magnitudes vary substantially along both rows and columns. Motivated by these observations, we propose STEPQuant, a spatial-temporal post-training quantization framework for Delta-rule recurrent states. STEPQuant allocates precision according to error magnitude and memory lifetime, and jointly fits key-row and value-column scales based on state distributions and key-row impact on output error. Experiments on Qwen3.8-27B and Kimi-Linear-48B-A3B-Instruct across both long- and short-generation benchmarks show that STEPQuant closely matches FP32-state accuracy under a nominal 6-bit budget and outperforms uniform INT8 in its 4-bit configuration. Integrated into SGLang with optimized GPU kernels, 6-bit STEPQuant achieves over 5x recurrent-state compression and reduces total serving memory by up to 68.7%. Our code is available at https://github.com/Dreamer-Toby/STEPQuant.

arXiv:2609.37351v3cs.LG

Clause-Factored Bilateral State Grounding and Scoped Passivity for Counterfactual Open-World Deductive Reasoning

Open-World Assumption (OWA) deduction requires distinguishing provable facts (True), refutations (False), and unprovable statements (Unknown) across counterfactual edits and unseen domains. Sequence and latent-recurrent transformers rely on distributional shortcuts, whereas chain-of-thought language models incur high autoregressive latency. We decompose OWA deduction into a clause-factored bilateral grounder ($F$) and a deductive deliberation operator ($B$), forming $B \circ F$ in Typed Edge-Factorized Bilateral Port-Hamiltonian Deliberation (TE-BPHD). Trained solely on three-way query labels without parsers or proof-depth annotations, TE-BPHD (37.85M parameters) maps clauses into a 1,440-dimensional Belnap space and propagates support through passivity-balanced implicit Port-Hamiltonian rule diodes. Across 17,036 locked ProofWriter test queries, TE-BPHD reaches 79.17 +/- 5.48% depth-5 counterfactual consistency (+23.83 to +38.67 percentage points over Seq2Seq T5, Flan-T5, and Recurrent-Depth T5) and 94.95 +/- 1.23% out-of-distribution (OOD) accuracy across 5,270 six-domain queries (92.54 +/- 3.38% macro-mean). On a paired 120-query OOD slice, TE-BPHD attains 93.61 +/- 4.59% versus 79.17% for 3-shot Qwen3-1.7B under a 2,048-token reasoning cap (+14.44 percentage points) at 9.44 versus 14,878.4 ms/query batch-16 NVIDIA L4 throughput. Operator interventions show that symmetric single-rule passivity prevents positive two-cycle modes seen in Gain-Matched Unilateral Horn (+50.00 percentage points counterfactual consistency), whereas unit-interval Grounded Product-T-Norm Logic Tensor Networks (LTN) achieve cycle-safe OOD transfer (94.50 +/- 0.87%) without linear solves, and multi-fact ParaRules prose exposes an upstream grounding trade-off (1.33 +/- 1.76% zero-shot vs. 34.11 +/- 5.01% with five-slot optimal-transport joint grounding).

arXiv:2609.32355v2cs.CL

Gradients for Interventions and Activation Values for Detection: Targeted Feature Learning in Language Models

Model-internal features can be studied through both their ability to identify a specified concept and their causal effect when manipulated, e.g., through steering or weight editing. A prominent approach to feature learning is Sparse Autoencoders (SAEs), which learn broad feature dictionaries whose relation to particular concepts is typically identified post hoc. However, many interpretability questions are instead hypothesis-driven and concern a concept specified in advance. We study this setting as targeted feature learning, where a single feature is constructed for such a predefined concept. We present a controlled comparison across three model signals (activation values, activation gradients, and parameter gradients) and two estimators (contrastive mean and a learned one-dimensional encoder-decoder), yielding six targeted methods, with CAA and GRADIEND as existing instances and four new methods covering the remaining combinations. We compare these methods against pretrained SAEs across 15 tasks and ten language models, evaluating both detection and causal intervention. Across models, the strongest detection performance is achieved by contrastive activation value methods, whereas the strongest intervention performance is achieved by gradient signals, most strongly with a contrastive mean. Notably, within the contrastive mean family, activation gradients substantially improve intervention over activation values without requiring parameter space interventions. Overall, our results show that targeted feature quality depends jointly on the model signal and estimator, with detection and intervention capturing complementary properties.

arXiv:2609.24974v2cs.AI

Harness-Zero: Agent-as-Harness를 통한 하네스 증류

Harness-Zero: Harness Distillation via Agent-as-Harness

Agent harnesses, the external systems that mediate model-environment interaction, can substantially improve agent performance, but their gains remain tied to the harness at deployment. Because the best harness varies across domains, instances, and models, a general-purpose agent must either settle for a suboptimal shared harness or route among an ever-growing set of specialized ones. We therefore study agent harness distillation: using a domain- or instance-optimized harness as training-time guidance and transferring the behaviors it induces into model weights, so that its gains survive under a single fixed target harness. The challenge is that the two harnesses differ in action space and available information, so guidance from the optimized harness cannot serve directly as supervision for the target one. We introduce Harness-Zero, which enables harness distillation through agent-as-harness. Guided by the optimized harness, a harnessing agent corrects student responses before execution in the target harness's action space, turning harness guidance into training demonstrations. Fine-tuning on the resulting trajectories internalizes harness-induced behavior into the model, so the specialized harness can be removed at deployment. Our experiments spanning knowledge work, tool use, and science domains show that: (1) For frontier LLMs using the same evolved harness, agent-as-harness outperforms code-as-harness. (2) With the specialized harness removed at deployment, Harness-Zero improves the base model's macro-average task success from 23.3% to 44.3%, even exceeding the 41.7% it reaches with that harness still attached, while reducing inference token usage by 27%-86%. (3) Harness-Zero recovers harness-induced behaviors absent from the base model, with 82.3% average recovery across 28 patterns in the three domains.

arXiv:2609.22226v3cs.CL

Swiss-Knife: 디코딩 시점의 재구성 가능한 외부화 다목적 정렬 프레임워크

Swiss-Knife: A Framework for Reconfigurable Externalised Multi-Objective Alignment at Decode Time

Decode-time alignment methods steer a frozen language model by scoring candidate continuations with an external reward and selecting the maximiser. We argue that this shared design is a single degenerate point in a much larger space. We introduce Swiss-Knife, a framework for externalised multi-objective alignment in which the alignment specification is a first-class runtime object: hot-swappable scoring blades, a batch normaliser, a pairwise aggregation operator, and a selection rule. Six axioms characterise the admissible aggregation operators, and we prove a representation theorem: every operator satisfying them has the form $R_i = \sum_{j \neq i} g((\mu_i - \mu_j)/s(\sigma_i,\sigma_j))$, a two-parameter family containing probit and logistic comparison rules and pointwise argmax as named coordinates. Within it, pairwise aggregation is Lipschitz-stable under adversarial reward contamination while argmax is not, and Candidate-Batch Normalization (CBN) makes the weight simplex invariant to the rescalings under which reward models are only ever identified. Our reference instantiation pairs DPO-LoRA blades with an uncertainty-aware pairwise tournament. Sweeping the helpfulness/honesty/harmlessness simplex, it attains the best balanced frontier of six decode-time methods (harmonic $F_1$ 0.797 vs. 0.750 for the strongest baseline, $p < 10^{-14}$) with the lowest refusal rate and highest helpfulness of any arm, and reconfigures its objectives in 0.05 ms with no gradient computation.

arXiv:2609.20050v3cs.IR

The Missing Complement: State-Conditioned Minimal Sufficient Evidence for Coding Agents

Retrieval assembles repository context by ranking passages for relevance to the current query. A coding agent halfway through an issue has already read much of what such a ranker returns. Relevance is scored per passage, but sufficiency belongs to the set: independently scored passages can fill the budget with support for one requirement while another goes unmet. We formulate state-conditioned minimal sufficient evidence recovery: given a captured agent state, recover a compact evidence combination supplying what its next decision still lacks. SERBench measures this on 500 held-out states from 45 repositories, recording what the agent has seen, crediting only sets that satisfy every annotated evidence requirement of the current decision, and separating set recovery from candidate discovery. MSS-Complement treats acquisition as set construction, not ranking. Three semantic calls propose a jointly sufficient set, search for what it lacks, and return 4-8 intact source units within 6,144 tokens. One configuration, fixed on calibration data, recovers a complete set for 73.0% of those states at five items and 80.6% at eight, against 61.4% and 72.4% for Qwen3 embedding with reranking. A matched control ranking by similarity alone recovers fewer complete sets, placing the margin over it in the set-level policy, not the computation. The lead persists from frozen repository source with no gold-derived pool. On AMA-Bench it answers from a 76.2% smaller answer prompt, with accuracy 2.08 points above that benchmark's own memory agent. Removing one required group from a complete set costs repair-localization precision under two executors. Retrieval for agents is better posed as recovering what a decision lacks than re-ranking what an issue resembles.

arXiv:2609.19153v3cs.CL

Stop Removing Stopwords: How an Inherited Preprocessing Default Distorts Legal Text-as-Data

Empirical legal scholarship increasingly treats judicial text as data, and much of it still runs on sparse, interpretable pipelines -- TF-IDF features and linear classifiers -- because the textual feature is often the object of study rather than a means to prediction. Yet these pipelines inherit preprocessing defaults from mid-century information retrieval that were never validated against classification accuracy. The most entrenched of these is stopword removal. This study introduces an exhaustive single word ablation that measures the effect of a preprocessing step directly against the downstream objective, and applies it to stopword removal as the hardest case to dislodge. Matching Supreme Court Database labels to Caselaw Access Project opinion texts, the study examines two binary tasks that bracket F1 headroom, ideological direction (no-removal baseline F1 ~0.68) and constitutional versus non-constitutional law type (~0.92), across 7,668 and 7,001 opinions. For each task, the ablation removes each of about 18,500 candidate words in turn, and a task-specific stoplist is built from the measurements. Common generic stoplists fall below the no-removal baseline in all twelve held-out tests. The task-specific stoplists move held-out F1 by +0.0023 (95% CI [-0.0124, +0.0170]) on ideology and by +0.0001 ([-0.0082, +0.0085]) on law type. Neither task shows a detectable benefit from removal, and supplemental analysis finds that word-level statistics predict removal effects poorly, because true removal effects differ by less than the measurement can register, leaving little to predict. The method generalizes to any inherited preprocessing default; the result is a caution specific to interpretable legal text-as-data, where a step that reshapes which features a model sees can distort the doctrinal and ideological signal the research is meant to recover. The burden of proof sits with removal.

arXiv:2609.16023v2cs.CL

Are We Grading Properly? Understanding Failure Modes in Medical Benchmarks

Medical evaluation is shifting from static option-based questioning to realistic clinical scenarios with open-ended output modes. Grading these at scale naively, however, is expensive, and rubric-based evaluation has become the dominant scalable alternative. We ask what happens when the rubrics themselves are not airtight, and whether such flaws can be detected and corrected. We apply RIFT, a global rubric failure taxonomy, to three clinical benchmarks (HealthBench Professional, LiveMedBench, and MentalHealthBench), and find failure modes are meaningful: on HealthBench Professional an LLM judge flags 29.6% of criteria as non-atomic and 65.4% as misaligned/rigid, although these rates vary severalfold across five detection judges. Then, we show that these flaws are meaningful and not simply cosmetic. As an example, rewriting bundled criteria of the form "at least one of / all of the following" as equally weighted children and regrading identical responses shifts scores by up to 15.9 percentage points on affected conversations, with disjunctive bundles inflating scores and conjunctive bundles deflating them. Across six answering models and two common graders, these shifts leave the model ranking unchanged, and their net direction depends on how many bundled criteria are penalties. We also find that RIFT can under-detect bundling on clinical rubrics, by an amount that depends on the detection judge: on the same LiveMedBench criteria, judges flag 3.8-15.9% as non-atomic against an estimated 7-19% that are genuinely bundled.

arXiv:2609.12541v4cs.CL

Agent as Policy for Robotic Manipulation

We demonstrate that a general-purpose coding agent can directly drive a physical robot throughout task execution without any task-specific or environment-specific training. We introduce Agent as Policy (AGP), which places task planning and execution under the agent's control. Given a task and a robot interface, the agent interprets visual evidence, writes executable programs, issues motion commands, and revises its actions in response to physical outcomes. This brings the agent's reasoning and programming capabilities into continuous interaction with the physical world. We study AGP across multiple real-world manipulation tasks spanning precision manipulation, dynamic motions, and deformable objects. These include assembly from human videos, block construction from goal images, dice flipping, targeted throwing, and bimanual towel folding. AGP achieves success rates of at least 80% in seven of eight task configurations and significantly outperforms previous agentic robot systems. We further study efficiency through task experience accumulation and find that reusing saved procedures and programs shortens execution time across repeated trials. These findings support a path for general-purpose agents to act as robot policies, extending their autonomy to physical manipulation through runtime reasoning, programming, and interaction.

arXiv:2608.30429v2cs.AI

EvoSkill Injection: 자기 진화 에이전트의 자율적 스킬 생성 및 진화에 대한 레드팀 테스팅

EvoSkill Injection: Red-Teaming Autonomous Skill Generation and Evolution in Self-Evolving Agents

LLM 기반 에이전트 시스템은 반복적인 추론 비용을 줄이고 안정적이고 효율적인 과업 수행을 개선하기 위해 점점 더 스킬 기반 아키텍처를 채택하고 있다. 최근 연구들은 과거 경험으로부터 스킬을 자율적으로 생성·정제·재사용하여 지속적인 역량 진화를 가능하게 하는 자기 진화 에이전트를 제안한다. 그러나 자율적 스킬 진화는 악의적 역량이 생성·저장·재사용되어 정당한 스킬처럼 쓰이는 새로운 공격 표면을 만든다. 본 논문에서는 자기 진화 에이전트의 자율적 스킬 생성 및 진화 파이프라인을 겨냥한 위협 모델로 EvoSkill Injection을 정의한다. 또한 이 위협 모델을 반복적 생성, 에스컬레이션, 강화 상호작용을 통해 평가하는 레드팀 테스팅 프레임워크 SARGE(Red-teaming Autonomous Skill Generation and Evolution in self-evolving agents)를 제안한다. 프레임워크를 뒷받침하기 위해 자기 진화 에이전트에서 악의적 스킬 형성을 유도하는 악의적 상호작용 궤적의 벤치마크 데이터셋 EvoSkillBench를 구성하고, 주입된 악의적 스킬이 이후 검색되어 유해 행동으로 활성화되는지 평가하는 사후 공격 벤치마크 EvoSkillSafetyBench를 도입한다. 평가 결과 SARGE는 악의적 스킬 형성을 유도하며, 주입된 스킬이 지속적으로 저장되고 반복적으로 활성화됨을 보여 지속적 역량 오염의 위험을 부각한다.

arXiv:2608.30044v4cs.AI

Balance of Benchmarks: Evidence Retention and Multiplicity-Aware Model Comparison

Benchmark aggregation requires decisions about how much evidence to use and how much influence each benchmark receives. Curated suites control influence through selection, which can exclude informative observations. A broader pool can retain those observations. With equal weighting, repeated or closely related benchmarks gain influence as their number grows. We introduce Balance of Benchmarks (BoB) to retain eligible observations and control their influence separately. We fit benchmark-specific response curves to place different difficulty ranges on a shared ability scale and use description similarity to downweight benchmarks in crowded neighborhoods. We test retention and density weighting in separate interventions on WildScores, a sparse collection of 148 developer-reported benchmarks, using 14 Artificial Analysis benchmarks as a controlled reference. With the model population fixed and entire source-lineage families held out, suite expansion improves held-out prediction under BoB and equal-weight equating on both WildScores and BIG-bench. Under substantial reporting overlap, density weighting improves robustness to multiplicity. Complete re-listings yield smaller mean rank displacement under BoB (0.63 versus 1.95 for equal-weight equating), with benefits also in paraphrased re-listings and supported affine fits. The benefit can reverse if models report different benchmarks, because a single weight per column gives models that report only part of a group less than its intended share. We therefore compute each model's weights from the benchmarks it reports. This rule removes the reversal in controlled re-listings and keeps held-out prediction close to that of equal weighting. These comparisons separate evidence retention, benchmark influence, and reporting coverage and show why repeated entries need to be handled per model.

arXiv:2608.28481v2cs.CL

NL2AGBench: AlphaGeometry를 위한 대규모 언어 모델(LLM) 자동 정형화 벤치마크

NL2AGBench: Benchmarking LLM Auto-Formalization for AlphaGeometry

최근 대규모 언어 모델(LLM)의 발전은 자연어 이해 및 수학적 추론에서 강력한 역량을 입증했습니다. 그러나 비정형적인 수학 문제를 정형화된 표현으로 변환하는 능력은 여전히 충분히 연구되지 않았습니다. 이러한 한계는 정리 증명 엔진이 특화된 도메인 특화 언어(DSL) 형태의 입력을 필요로 하는 AlphaGeometry와 같은 신경 기호(neuro-symbolic) 기하학 시스템에서 특히 중요합니다. AlphaGeometry는 국제수학올림피아드(IMO) 금메달리스트에 가까운 성능을 달성하지만, 자연어로 된 문제를 정형 구문으로 수작업 변환해야 한다는 점은 여전히 중대한 사용성 병목 현상으로 남아 있습니다. 이러한 문제를 해결하기 위해, 본 연구에서는 영어 기하학 문제를 AlphaGeometry 호환 정형 표현으로 변환하는 LLM의 역량을 평가하는 NL2AGBench(Natural Language to AlphaGeometry Benchmark)를 소개합니다. NL2AGBench는 단순히 텍스트 유사도에 의존하는 대신 AlphaGeometry 내에서의 실행 기반 검증을 활용하여 변환 품질을 평가합니다. 본 연구는 다양한 매개변수 규모에 걸쳐 10개의 최첨단 오픈소스 및 상용(closed-source) LLM을 평가하고, 실행 가능한 변환 정확도, 구문적 정확성, 오류 특성을 분석합니다. 실험 결과, 상용 모델과 오픈소스 모델 간에 상당한 성능 격차가 있음이 드러났습니다. 선도적인 상용 모델은 80% 이상의 실행 가능한 변환율을 달성한 반면, 가장 큰 규모의 오픈소스 모델조차 기하학적 제약 조건을 일관되게 보존하고 유효한 정형화를 생성하는 데 어려움을 겪었습니다. 또한 구문 오류와 논리 오류를 구분하는 오류 분류 체계(taxonomy)를 도입하고, 퓨샷(few-shot) 프롬프팅, 미세조정(fine-tuning), 인간 유도 힌트 제공을 포함하여 여러 모델 제품군 전반에서 측정 가능한 개선을 가져오는 완화 전략을 조사합니다.

arXiv:2608.10974v2cs.CL

MUSE: A Full-Text Cross-Domain Knowledge Base of Scientific Problems, Solutions, and Rationales

Scientific papers contain fine-grained records of problem solving: authors mention technical obstacles and methods that were used to address them, often along with reasoning on why those methods were chosen. We introduce MUSE (Mining Underlying Scientific Explanations), a full-text, multi-domain resource of scientific Problem-Solution-Rationale (P-S-R) triplets. We curate 579 expert-annotated full-text paragraphs, with a rich annotation schema covering salient problem, solution, and rationale spans, solves and rationale_of links and conceptual coreference. A modular extraction pipeline scales this annotation to build a high-quality knowledge base of 37K source-grounded P-S-R triplets. We evaluate the extraction components and include a preliminary experiment training a rationale-supervised LLM for scientific problem solving. Interestingly, we find that rationale supervision improves performance on complex, multi-constraint problems but can harm performance on simpler ones.

arXiv:2607.24542v2cs.CL

전사에서 의미 코퍼스 분석까지: 고대 언어를 위한 문장 표현의 비지도 학습

From transcription to semantic corpus analysis: unsupervised learning of sentence representations for ancient languages

Automatic Text Recognition (ATR) now supplies digital humanities with large volumes of unstruc- tured, heterogeneous, and often noisy text in ancient languages. Downstream semantic analyses- text reuse identification, alignment, and semantic search-rely on sentence embeddings, yet ex- isting methods transfer poorly to ancient languages: generic multilingual encoders underperform, specialized language models yield anisotropic representation spaces, and labeled similarity data is scarce. We study two fully unsupervised strategies-TSDAE and unsupervised SimCSE (hence- forth CSE)-that adapt a specialized token-level language model into a corpus-specific sentence encoder using only raw sentences. We validate them on biblical reuse in patristic literature in Latin and Ancient Greek: 2,936 sentences carrying expert-verified reuses in works of Augustine, Jerome, and Athanasius, matched against the Vulgate, the Septuagint, and the Nestle--Aland New Testament. We decompose reuse identification into binary detection and correspondence retrieval, evaluate both tasks separately, and benchmark the adapted encoders against multilingual, specialized, distilled, and supervised fine-tuned baselines, on clean text and on text perturbed by simulated HTR errors and scribal abbreviations. On these test sets, the adapted encoders lead all baselines on both tasks, with complementary profiles: TSDAE leads detection given a large in-domain corpus, while CSE leads retrieval, reaches its optimum with as few as 4--8k raw in-domain sentences-a few tens of sec- onds of training on a laptop GPU-and transfers across works and authors. Under simulated noise, retraining CSE on the perturbed text itself outperforms a clean-trained encoder. Measurements of alignment, uniformity, and anisotropy relate the geometric effect of each strategy to the measured gains. The full pipeline-segmentation, fine-tuning, cross-corpus semantic search-is available to non-specialists through the online tool Paraphrasis.

arXiv:2607.22610v2cs.AI

Tokengeist: 에이전트 대화에서의 다중 턴 귀속 추적

Tokengeist: Multi-Turn Attribution Tracing in Agentic Conversations

When a model answers after several turns of dialogue and tool use, an overseer needs to know which earlier turns and intermediate steps it rests on. Attribution is the natural primitive for answering this, but existing context attribution methods process the full transcript in a single pass: they recover surface-level dependencies while missing the layered structure of multi-turn traces, in which an assistant turn is itself a derived artifact that condenses or transforms earlier context. We introduce multi-turn context attribution (MTCA): given a target span in a model response, the task of tracing attribution backward across turns to identify not only which prior turns were directly relevant, but also how those turns themselves depended on earlier context. We propose Tokengeist, an attribution-method-agnostic and scalable framework that recovers full dependency paths by casting attribution as a recursive traversal of a directed acyclic graph over conversation turns. We will release MTCABench, a benchmark of 3,970 target spans across 685 multi-turn conversations, annotated with gold provenance graphs reaching depths of up to 17, across four dependency types. Across four open-weight models, flat attribution methods fail to recover multi-hop dependencies, with source recall improving from 11-38% under flat attribution to 82-94% under Tokengeist. Our results reveal a systematic failure mode of single-pass attribution, which we term provenance collapse, where attribution terminates at the nearest assistant restatement instead of the originating event.

arXiv:2607.22583v4cs.AI

지연 시간 및 모델 크기 최적화를 위한 LLM 다목적 구조적 가지치기

Multi-Objective Structured Pruning of LLMs for Latency and Model Size Optimization

대규모 언어 모델(LLM)은 강력한 추론 및 질의-응답 능력 덕분에 널리 채택되었다. 그러나 임베디드 및 엣지 컴puting 환경에 이를 배포하는 것은 엄격한 지연 시간, 메모리, 에너지 제약으로 인해 여전히 어렵다. 큰 매개변수 수와 계산 요구량은 자원 제약 플랫폼에서의 효율적 실행을 방해한다. 성능을 유지하면서 규모를 줄이는 실행 가능한 해결책으로 모델 가지치기가 등장했지만, 층, 어텐션 헤드, 다층 퍼셉트론(MLP) 차원을 공동으로 최적화하는 것은 여전히 매우 복잡하다. 이 결합된 설계 공간을 전수 탐색하는 것은 계산 비용이 크고 종종 지역 최적해나 불안정한 구성으로 이어진다. 이러한 한계를 해결하기 위해, 우리는 하드웨어 인식 다목적 구조적 가지치기 프레임워크를 제안한다. 제안된 2단계 방법은 엣지 디바이스에서의 효율적 배포를 위해 지연 시간과 모델 크기를 명시적으로 목표로 한다. 조대한(coarse-grained) 단계에서 다목적 깊이 가지치기는 전체 어텐션 및 MLP 블록을 제거하여 계산 부하와 메모리 사용량을 줄인다. 이후 세밀한(fine-grained) 단계에서 병렬 베이지안 최적화(PBO)는 지연 시간 제약 하에서 가지치기를 위한 최적의 층별 가지치기 비율을 탐색하고, 중요도 기반 전략은 각 층에 할당된 예산 내에서 가지치기할 특정 구성 요소의 순위를 매긴다. 실험 결과는 우리의 접근법이 상식 추론 과제와 제로샷 성능에 미치는 영향을 최소화하면서 모델 복잡도를 줄인다는 것을 보여준다. 우리의 방법은 정확도, 지연 시간, 모델 크기 간 유리한 트레이드오프를 달성하여 엣지 배포에 적합하다. 37.5% 및 50% 가지치기 비율에서 여러 LLM에 걸쳐, 제안된 접근법은 추론 비용을 크게 줄이면서 기존 방법보다 상식 추론 과제에서 더 나은 성능을 달성한다.

arXiv:2607.17117v3cs.LG

지속적 희소 오토인코더: 언어 모델 표현에서 특징별 시간 척도 학습

Persistent Sparse Autoencoders: Learning Feature-Specific Timescales in Language Model Representations

Sparse autoencoders (SAEs) decompose language model activations into sparse features, yet these models traditionally encode each token independently, failing to expose information that persists across a sequence. We first show that temporal persistence can naturally emerge in standard SAE features: after a feature activates, the hidden state remains aligned with its direction, and past activations reduce the reconstruction error of later hidden states. How long this lasts varies widely across features. We therefore introduce Persistent Sparse Autoencoders (Persistent SAEs), an extension of standard SAEs that learns a persistence coefficient for each feature, allowing feature-specific timescales to emerge from reconstruction. Our experiments show that Persistent SAEs retain competitive reconstruction quality while learning a spectrum of timescales: short-timescale (fast) features remain locally interpretable, whereas long-timescale (slow) features accumulate higher-level contextual information. Moreover, we show in a prompt-injection monitoring case study that slow features retain injection signals over long contexts and causally affect monitor judgements. These results suggest that Persistent SAEs offer new opportunities for interpreting and monitoring language models via persistent features.

arXiv:2607.15277v2cs.CL

분할, 프롬프트, 집계: 언어 모델의 통계적 자기 일관성

Partition, Prompt, Aggregate: Statistical Self-Consistency in Language Models

Large language models (LLMs) are often used to perform a form of conditional inference. In-context learning is one implicit way to do this. The prompt specifies a context and the model's output is treated as an estimate of the corresponding conditional distribution. If this interpretation holds, then LLM estimates should satisfy basic probabilistic identities. In particular, the law of total probability asserts that prior-weighted conditional distributions aggregate into population-level marginals over any valid partition of the population. In this work, we investigate to what extent LLM estimates adhere to this self-consistency principle. We use binary trees as an evaluation scaffold to recursively partition a population into increasingly fine-grained subpopulations. We then prompt LLMs with verbalized subpopulation descriptions in context, aggregate the resulting estimates back into population-level estimates, and compare them across partitions of varying granularity. Translating these comparisons into a benchmark score and evaluating frontier models across problem domains, we demonstrate widespread violations of basic consistency properties. An in-depth study of persona prompting further reveals a pattern we call the macro fallacy: estimates reconstructed from more fine-grained subpopulation responses are often better aligned with human reference data than direct population-level estimates. The effect persists across variations in partitioning, verbalization, and estimation task, suggesting that models possess relevant subpopulation knowledge but do not reliably propagate it into aggregate estimates. This gap establishes statistical self-consistency as an unsaturated, reference-free criterion for evaluating LLMs.

arXiv:2606.27786v3cs.CL

SPARK: Adaptive Parametric Knowledge Participation for Faithful Retrieval-Augmented Generation

Retrieval-augmented generation (RAG) enhances the faithfulness of large language models (LLMs) by incorporating external knowledge into the generation process. However, conflicts between retrieved context and parametric knowledge can cause LLMs to rely on inconsistent internal knowledge, hindering their ability to faithfully leverage retrieved evidence. Existing approaches mitigate this issue by editing parametric knowledge to resolve such conflicts. However, locating and modifying knowledge encoded in model parameters is challenging and may inadvertently compromise the backbone LLM's capabilities. In this paper, we introduce SPARK, a novel framework that formulates parametric knowledge control as a latent participation policy learning problem, enabling LLMs to adaptively regulate the contribution of parametric knowledge during generation. Specifically, SPARK introduces a lightweight participation policy that dynamically modulates the contribution of FFN-mediated parametric knowledge while keeping the backbone LLM frozen. The participation policy is optimized with a faithfulness-oriented reward, enabling fine-grained control over parametric knowledge participation during generation. Extensive experiments on six datasets validate the effectiveness of SPARK, which introduces and optimizes only 0.01\% additional trainable parameters while keeping the backbone LLM fully frozen, thereby preserving its general capabilities and achieving more faithful generation than competitive baselines. All datasets and code are available at https://github.com/OpenBMB/SPARK.