PyTorchKR Papers

“Code Generation”

총 10,186편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2607.18230v1cs.CV

현대 VLM에서 강력한 픽셀 수준 이미지 조작 탐지를 위한 단순한 도메인 일반화

Simple Domain Generalization for Strong Pixel-Level Image Tampering Detection in Modern VLMs

현대 비전-언어 모델(VLM)은 이미지 생성 및 편집 능력을 크게 향상시켜, 크로스 모델 및 분포 외(OOD) 변화 하에서 픽셀 수준 이미지 조작 탐지가 점점 더 중요해지면서도 어려워지고 있다. 본 연구는 ChatGPT, Gemini, Qwen-Image 등과 같은 현대 VLM에서의 픽셀 수준 이미지 조작 탐지를 위한 도메인 일반화를 연구하며, 다양한 VLM 생성 조작 분포에 걸쳐 견고함을 유지하는 조작 위치화 모델을 학습하는 것을 목표로 한다. 우리는 두 가지 실용적 전략을 기반으로 한 단순하면서도 효과적인 도메인 일반화 학습 프레임워크를 제안한다. 첫째, 각 미니배치에서 조작된 이미지와 실제 이미지를 전략적으로 샘플링하는 균형 미니배치 샘플링 방식을 도입하여, 조작 아티팩트 또는 깨끗한 이미지 사전(prior) 중 어느 한쪽으로 치우친 최적화를 방지하고 학습 붕괴를 피하며, 각 최적화 단계가 적절하게 샘플링된 그래디언트 신호를 받도록 한다. 둘째, 탐지기를 먼저 대규모 기본 데이터에서 안정적 수렴까지 학습한 뒤, 새롭게 등장하는 VLM 분포에서 선별한 소량의 지원 데이터에 노출시키는 단순한 late-injection 전략을 채택하여, 제한된 새 도메인에 대한 과적합 없이 적응성을 향상시킨다. 이들 구성 요소는 함께 현대 VLM 전반에 걸친 픽셀 수준 조작 위치화 및 OOD 견고성을 개선하는 단순하면서도 강력한 방법을 제공한다. 개념적 단순함에도 불구하고, 우리 프레임워크는 GPT-Images-2.0, Gemini-3.1, FLUX.2, Seedream 4.5 등 OOD VLM에 걸쳐 평균 gIoU 및 cIoU에서 각각 26.1% 및 26.8%의 상대적 개선 마진으로 기존 최신 기법(state-of-the-art) PIXAR를 크게 능가한다. 코드는 https://github.com/VILA-Lab/PIXAR-DG 에서 제공된다.

arXiv:2602.12639v2cs.CL

CLASE: 중국 법률 문체 스타일 평가를 위한 하이브리드 방법

CLASE: A Hybrid Method for Chinese Legalese Stylistic Evaluation

대규모 언어 모델(LLM)이 생성한 법률 텍스트는 사실적 정확도 측면에서 합리적인 수준에 도달하는 경우가 많지만, 법률 작성의 전문적인 스타일 규범과 언어적 관습을 따르지 못하는 경우가 빈번하다. 스타일 품질을 향상시키기 위해 신뢰할 수 있는 평가 방법을 확립하는 것이 중요한 첫 단계이다. 그러나 법률 전문가가 수동으로 이러한 지표를 개발하는 것은 비현실적이다. 법률 작성 관행에 내재된 암묵적 스타일 요구사항을 명시적 루브릭으로 형식화하기 어렵기 때문이다. 한편 기존 자동 평가 방법도 부족하다. 참조 기반(reference-based) 지표는 의미적 정확도와 스타일적 충실도를 혼동하고, LLM-as-a-judge 평가는 불투명성과 불일치 문제를 겪는다. 이러한 과제를 해결하기 위해 우리는 법률 텍스트의 스타일적 성능에 초점을 맞춘 하이브리드 평가 방법인 CLASE(Chinese LegAlese Stylistic Evaluation)를 제안한다. 이 방법은 1) 언어학적 특징 기반 점수와 2) 경험 가이드 LLM-as-a-judge 점수를 결합하는 하이브리드 채점 메커니즘을 포함한다. 특징 계수와 LLM 채점 경험 모두 실제 법률 문서와 LLM으로 복원한 대응 문서의 대조(contrastive) 쌍으로부터 학습된다. 이 하이브리드 설계는 표면 수준의 특징과 암묵적 스타일 규범을 모두 투명하고 참조 없는(reference-free) 방식으로 포착한다. 200개의 중국 법률 문서에 대한 실험에서 CLASE는 전통적 지표와 순수 LLM-as-a-judge 방법보다 인간 판단과의 정렬(alignment)이 상당히 높음을 보여준다. 향상된 정렬을 넘어, CLASE는 해석 가능한 점수 분해와 개선 제안을 제공하여 법률 텍스트 생성에서 전문적인 스타일 평가를 위한 확장 가능하고 실용적인 해결책을 제공한다(CLASE의 코드와 데이터는 https://github.com/rexera/CLASE에서 이용 가능).

arXiv:2609.22227v1cs.IR

시맨틱 ID의 상위 레벨 유도를 통한 하위 레벨의 학습 가능성 확보

Guiding the coarse levels of semantic IDs makes the fine levels learnable

생성형 검색(generative retrieval)은 각 아이템을 짧은 시맨틱 ID(Semantic ID, SID)로 표현하고, 추천을 해당 시퀀스의 자기회귀적 생성으로 다룹니다. 토크나이저(tokenizer)가 아이템 임베딩(embedding)을 재구성하도록 독립적으로 학습되기 때문에, 그 코드들은 다운스트림 대규모 언어 모델(LLM)이나 최종 태스크 어느 쪽과도 정렬(alignment)되지 않습니다. 따라서 거의 모든 SID 시스템은 코드를 이해하기 쉽게 만들기 위한 정렬 말뭉치, 추론/강화학습(RL), 토큰(token)별 인코더를 도입하거나, 코드가 태스크를 인식하도록 학습된 토크나이저 지도(supervision)를 도입하는 등 이 격차를 해소하기 위해 추가적인 노력을 기울입니다. 그러나 이렇게 회복된 의미는 콘텐츠에서 파생된 것이어서 태스크가 필요로 하는 의미와 일치하지 않을 수 있습니다. 본 연구에서는 구조적으로 가장 중요한 레벨을 유의미하게 만드는 Guided SID를 제안합니다. 코드북이 학습 가능하도록 유지(여전히 재구성 그래디언트를 수신)하면서도, 결정론적 지도 인덱스 할당(속성 라벨로 최근접 이웃 선택을 대체)을 통해 거친(coarse) RQ-VAE 레벨이 텍스트에 기반(text-grounded, 즉 LLM이 이해 가능)하고 태스크에 적합하도록 선택된 사전 정의된 범주형 속성을 인코딩하도록 강제합니다. 트라이 병합(trie-merge) 구조는 병합된 버킷의 의미적 일관성을 유지하면서 높은 카디널리티나 집합 값을 가진 임의의 속성을 고정된 코드 예산에 매핑합니다. 상위 레벨을 고정함에도 불구하고 충돌 및 재구성 성능이 바닐라 기준선(baseline)과 일치하거나 이를 능가하므로 유도(guiding)에 따르는 본질적인 비용은 없습니다. SID 인코딩만 다른 매칭된 엔드투엔드 A/B 테스트에서, 유도된 검색기는 측정한 모든 리스트 길이에서 재현율(recall)@k를 향상시켰고(k=1에서 1.36배, k=10에서 1.39배), 평균 역순위(MRR)를 0.0260에서 0.0355로 높였으며, 사전 정의된 속성을 4.2배 더 자주 예측합니다.

arXiv:2611.22409v1cs.CV

SignAligner: 두 가지 세분화 수준의 구간 생성으로 수어 영상 정렬하기

SignAligner: Dual-Granularity Span Generation for Sign Language Alignment

대규모 수어 말뭉치에는 두 수준의 시간 레이블이 유용하다. 자막 구간은 자막 정렬에, 단어 구간은 수어 단어 검출에 쓰인다. 수어 단어 검출로 말뭉치 전체에서 단어의 시간 위치를 찾을 수 있지만, 자막과 단어 구간을 함께 사용해 자막 정렬을 학습하는 방법은 충분히 연구되지 않았다. 프롬프트 조건부 비전-언어 구간 생성기 SignAligner를 소개한다. 수어 비디오와 자막 문장 또는 목표 단어 쿼리를 받아 시작 및 끝 프레임을 텍스트로 출력한다. 먼저 자막 구간과 수어 단어 검출에서 얻은 단어 구간을 함께 학습한다. 이어 단어 예시를 보조 데이터로 유지하며, 구간 겹침을 기반으로 한 보상으로 모델을 정제한다. BBC와 옥스퍼드가 구축한 영국 수어 말뭉치 BOBSL의 자막 테스트 세트에서 정제 모델은 F1@0.50(시간 구간의 교집합 대 합집합 비율 임계값을 0.5로 둔 F1) 66.68을 달성했다. 외부 수어 영상 자료를 쓰지 않은 비교 시스템 중 가장 높았다. 겹침 보상으로 정제하는 방식은 같은 공동 학습 모델을 교차 엔트로피로 계속 학습하는 방식보다 우수했다. 코드, 레이블, 최대 갱신 예산을 동일하게 맞춘 두 생성기 비교에서도 단어 구간을 함께 학습하면 자막 정렬이 개선됐다. 두 모델 모두 겹침 보상으로 정제한 뒤에도 이점이 유지됐다. 단어 쿼리에 대해 정제 모델은 비디오의 고정 위치에서만 답하지 않고 질의한 단어를 따라갔다. 해당 수어 표현이 비디오 중앙에서 벗어나 있을 때 위치만 사용한 예측보다 훨씬 정확했다. 이 결과는 겹침 보상으로 정제한 공유 구간 생성기 하나로 자막을 정렬하고 사람이 검토할 단어 구간을 생성할 수 있음을 뒷받침한다.

arXiv:1808.05864공식 코드 · Papers with Code

Context-Aware Visual Policy Network for Sequence-Level Image Captioning

Many vision-language tasks can be reduced to the problem of sequence prediction for natural language output. In particular, recent advances in image captioning use deep reinforcement learning (RL) to alleviate the "exposure bias" during training: ground-truth subsequence is exposed in every step prediction, which introduces bias in test when only predicted subsequence is seen. However, existing RL-based image captioning methods only focus on the language policy while not the visual policy (e.g., visual attention), and thus fail to capture the visual context that are crucial for compositional reasoning such as visual relationships (e.g., "man riding horse") and comparisons (e.g., "smaller cat"). To fill the gap, we propose a Context-Aware Visual Policy network (CAVP) for sequence-level image captioning. At every time step, CAVP explicitly accounts for the previous visual attentions as the context, and then decides whether the context is helpful for the current word generation given the current visual attention. Compared against traditional visual attention that only fixes a single image region at every step, CAVP can attend to complex visual compositions over time. The whole image captioning model --- CAVP and its subsequent language policy network --- can be efficiently optimized end-to-end by using an actor-critic policy gradient method with respect to any caption evaluation metric. We demonstrate the effectiveness of CAVP by state-of-the-art performances on MS-COCO offline split and online server, using various metrics and sensible visualizations of qualitative visual context. The code is available at https://github.com/daqingliu/CAVP

arXiv:1912.02923

Generating 3D People in Scenes without People

We present a fully automatic system that takes a 3D scene and generates plausible 3D human bodies that are posed naturally in that 3D scene. Given a 3D scene without people, humans can easily imagine how people could interact with the scene and the objects in it. However, this is a challenging task for a computer as solving it requires that (1) the generated human bodies to be semantically plausible within the 3D environment (e.g. people sitting on the sofa or cooking near the stove), and (2) the generated human-scene interaction to be physically feasible such that the human body and scene do not interpenetrate while, at the same time, body-scene contact supports physical interactions. To that end, we make use of the surface-based 3D human model SMPL-X. We first train a conditional variational autoencoder to predict semantically plausible 3D human poses conditioned on latent scene representations, then we further refine the generated 3D bodies using scene constraints to enforce feasible physical interaction. We show that our approach is able to synthesize realistic and expressive 3D human bodies that naturally interact with 3D environment. We perform extensive experiments demonstrating that our generative framework compares favorably with existing methods, both qualitatively and quantitatively. We believe that our scene-conditioned 3D human generation pipeline will be useful for numerous applications; e.g. to generate training data for human pose estimation, in video games and in VR/AR. Our project page for data and code can be seen at: \url{https://vlg.inf.ethz.ch/projects/PSI/}.

arXiv:2104.00734공식 코드 · Papers with Code

Next Generation Multitarget Trackers: Random Finite Set Methods vs Transformer-based Deep Learning

Multitarget Tracking (MTT) is the problem of tracking the states of an unknown number of objects using noisy measurements, with important applications to autonomous driving, surveillance, robotics, and others. In the model-based Bayesian setting, there are conjugate priors that enable us to express the multi-object posterior in closed form, which could theoretically provide Bayes-optimal estimates. However, the posterior involves a super-exponential growth of the number of hypotheses over time, forcing state-of-the-art methods to resort to approximations for remaining tractable, which can impact their performance in complex scenarios. Model-free methods based on deep-learning provide an attractive alternative, as they can, in principle, learn the optimal filter from data, but to the best of our knowledge were never compared to current state-of-the-art Bayesian filters, specially not in contexts where accurate models are available. In this paper, we propose a high-performing deep-learning method for MTT based on the Transformer architecture and compare it to two state-of-the-art Bayesian filters, in a setting where we assume the correct model is provided. Although this gives an edge to the model-based filters, it also allows us to generate unlimited training data. We show that the proposed model outperforms state-of-the-art Bayesian filters in complex scenarios, while matching their performance in simpler cases, which validates the applicability of deep-learning also in the model-based regime. The code for all our implementations is made available at https://github.com/JulianoLagana/MT3 .

arXiv:2207.12194공식 코드 · Papers with Code

Domain Decorrelation with Potential Energy Ranking

Machine learning systems, especially the methods based on deep learning, enjoy great success in modern computer vision tasks under experimental settings. Generally, these classic deep learning methods are built on the \emph{i.i.d.} assumption, supposing the training and test data are drawn from a similar distribution independently and identically. However, the aforementioned \emph{i.i.d.} assumption is in general unavailable in the real-world scenario, and as a result, leads to sharp performance decay of deep learning algorithms. Behind this, domain shift is one of the primary factors to be blamed. In order to tackle this problem, we propose using \textbf{Po}tential \textbf{E}nergy \textbf{R}anking (PoER) to decouple the object feature and the domain feature (\emph{i.e.,} appearance feature) in given images, promoting the learning of label-discriminative features while filtering out the irrelevant correlations between the objects and the background. PoER helps the neural networks to capture label-related features which contain the domain information first in shallow layers and then distills the label-discriminative representations out progressively, enforcing the neural networks to be aware of the characteristic of objects and background which is vital to the generation of domain-invariant features. PoER reports superior performance on domain generalization benchmarks, improving the average top-1 accuracy by at least 1.20\% compared to the existing methods. Moreover, we use PoER in the ECCV 2022 NICO Challenge\footnote{https://nicochallenge.com}, achieving top place with only a vanilla ResNet-18. The code has been made available at https://github.com/ForeverPs/PoER.

arXiv:2307.08260

Extending the Frontier of ChatGPT: Code Generation and Debugging

Large-scale language models (LLMs) have emerged as a groundbreaking innovation in the realm of question-answering and conversational agents. These models, leveraging different deep learning architectures such as Transformers, are trained on vast corpora to predict sentences based on given queries. Among these LLMs, ChatGPT, developed by OpenAI, has ushered in a new era by utilizing artificial intelligence (AI) to tackle diverse problem domains, ranging from composing essays and biographies to solving intricate mathematical integrals. The versatile applications enabled by ChatGPT offer immense value to users. However, assessing the performance of ChatGPT's output poses a challenge, particularly in scenarios where queries lack clear objective criteria for correctness. For instance, evaluating the quality of generated essays becomes arduous and relies heavily on manual labor, in stark contrast to evaluating solutions to well-defined, closed-ended questions such as mathematical problems. This research paper delves into the efficacy of ChatGPT in solving programming problems, examining both the correctness and the efficiency of its solution in terms of time and memory complexity. The research reveals a commendable overall success rate of 71.875\%, denoting the proportion of problems for which ChatGPT was able to provide correct solutions that successfully satisfied all the test cases present in Leetcode. It exhibits strengths in structured problems and shows a linear correlation between its success rate and problem acceptance rates. However, it struggles to improve solutions based on feedback, pointing to potential shortcomings in debugging tasks. These findings provide a compact yet insightful glimpse into ChatGPT's capabilities and areas for improvement.

arXiv:2307.11346

CohortGPT: An Enhanced GPT for Participant Recruitment in Clinical Study

Participant recruitment based on unstructured medical texts such as clinical notes and radiology reports has been a challenging yet important task for the cohort establishment in clinical research. Recently, Large Language Models (LLMs) such as ChatGPT have achieved tremendous success in various downstream tasks thanks to their promising performance in language understanding, inference, and generation. It is then natural to test their feasibility in solving the cohort recruitment task, which involves the classification of a given paragraph of medical text into disease label(s). However, when applied to knowledge-intensive problem settings such as medical text classification, where the LLMs are expected to understand the decision made by human experts and accurately identify the implied disease labels, the LLMs show a mediocre performance. A possible explanation is that, by only using the medical text, the LLMs neglect to use the rich context of additional information that languages afford. To this end, we propose to use a knowledge graph as auxiliary information to guide the LLMs in making predictions. Moreover, to further boost the LLMs adapt to the problem setting, we apply a chain-of-thought (CoT) sample selection strategy enhanced by reinforcement learning, which selects a set of CoT samples given each individual medical report. Experimental results and various ablation studies show that our few-shot learning method achieves satisfactory performance compared with fine-tuning strategies and gains superb advantages when the available data is limited. The code and sample dataset of the proposed CohortGPT model is available at: https://anonymous.4open.science/r/CohortGPT-4872/

arXiv:2402.06700공식 코드 · Papers with Code

Entropy-Regularized Token-Level Policy Optimization for Language Agent Reinforcement

Large Language Models (LLMs) have shown promise as intelligent agents in interactive decision-making tasks. Traditional approaches often depend on meticulously designed prompts, high-quality examples, or additional reward models for in-context learning, supervised fine-tuning, or RLHF. Reinforcement learning (RL) presents a dynamic alternative for LLMs to overcome these dependencies by engaging directly with task-specific environments. Nonetheless, it faces significant hurdles: 1) instability stemming from the exponentially vast action space requiring exploration; 2) challenges in assigning token-level credit based on action-level reward signals, resulting in discord between maximizing rewards and accurately modeling corpus data. In response to these challenges, we introduce Entropy-Regularized Token-level Policy Optimization (ETPO), an entropy-augmented RL method tailored for optimizing LLMs at the token level. At the heart of ETPO is our novel per-token soft Bellman update, designed to harmonize the RL process with the principles of language modeling. This methodology decomposes the Q-function update from a coarse action-level view to a more granular token-level perspective, backed by theoretical proof of optimization consistency. Crucially, this decomposition renders linear time complexity in action exploration. We assess the effectiveness of ETPO within a simulated environment that models data science code generation as a series of multi-step interactive tasks; results underline ETPO's potential as a robust method for refining the interactive decision-making capabilities of language agents. For a more detailed preliminary work describing our motivation for token-level decomposition and applying it in PPO methods, please refer to arXiv:2405.15821.

arXiv:2403.05434

Cost-Performance Optimization for Processing Low-Resource Language Tasks Using Commercial LLMs

Large Language Models (LLMs) exhibit impressive zero/few-shot inference and generation quality for high-resource languages (HRLs). A few of them have been trained on low-resource languages (LRLs) and give decent performance. Owing to the prohibitive costs of training LLMs, they are usually used as a network service, with the client charged by the count of input and output tokens. The number of tokens strongly depends on the script and language, as well as the LLM's subword vocabulary. We show that LRLs are at a pricing disadvantage, because the well-known LLMs produce more tokens for LRLs than HRLs. This is because most currently popular LLMs are optimized for HRL vocabularies. Our objective is to level the playing field: reduce the cost of processing LRLs in contemporary LLMs while ensuring that predictive and generative qualities are not compromised. As means to reduce the number of tokens processed by the LLM, we consider code-mixing, translation, and transliteration of LRLs to HRLs. We perform an extensive study using the IndicXTREME classification and six generative tasks dataset, covering 15 Indic and 3 other languages, while using GPT-4 (one of the costliest LLM services released so far) as a commercial LLM. We observe and analyze interesting patterns involving token count, cost, and quality across a multitude of languages and tasks. We show that choosing the best policy to interact with the LLM can reduce cost by 90% while giving better or comparable performance compared to communicating with the LLM in the original LRL.

arXiv:2403.20127공식 코드 · Papers with Code

The Impact of Prompts on Zero-Shot Detection of AI-Generated Text

In recent years, there have been significant advancements in the development of Large Language Models (LLMs). While their practical applications are now widespread, their potential for misuse, such as generating fake news and committing plagiarism, has posed significant concerns. To address this issue, detectors have been developed to evaluate whether a given text is human-generated or AI-generated. Among others, zero-shot detectors stand out as effective approaches that do not require additional training data and are often likelihood-based. In chat-based applications, users commonly input prompts and utilize the AI-generated texts. However, zero-shot detectors typically analyze these texts in isolation, neglecting the impact of the original prompts. It is conceivable that this approach may lead to a discrepancy in likelihood assessments between the text generation phase and the detection phase. So far, there remains an unverified gap concerning how the presence or absence of prompts impacts detection accuracy for zero-shot detectors. In this paper, we introduce an evaluative framework to empirically analyze the impact of prompts on the detection accuracy of AI-generated text. We assess various zero-shot detectors using both white-box detection, which leverages the prompt, and black-box detection, which operates without prompt information. Our experiments reveal the significant influence of prompts on detection accuracy. Remarkably, compared with black-box detection without prompts, the white-box methods using prompts demonstrate an increase in AUC of at least $0.1$ across all zero-shot detectors tested. Code is available: \url{https://github.com/kaito25atugich/Detector}.

arXiv:2407.11288

Zero-Shot Adaptation for Approximate Posterior Sampling of Diffusion Models in Inverse Problems

Diffusion models have emerged as powerful generative techniques for solving inverse problems. Despite their success in a variety of inverse problems in imaging, these models require many steps to converge, leading to slow inference time. Recently, there has been a trend in diffusion models for employing sophisticated noise schedules that involve more frequent iterations of timesteps at lower noise levels, thereby improving image generation and convergence speed. However, application of these ideas for solving inverse problems with diffusion models remain challenging, as these noise schedules do not perform well when using empirical tuning for the forward model log-likelihood term weights. To tackle these challenges, we propose zero-shot approximate posterior sampling (ZAPS) that leverages connections to zero-shot physics-driven deep learning. ZAPS fixes the number of sampling steps, and uses zero-shot training with a physics-guided loss function to learn log-likelihood weights at each irregular timestep. We apply ZAPS to the recently proposed diffusion posterior sampling method as baseline, though ZAPS can also be used with other posterior sampling diffusion models. We further approximate the Hessian of the logarithm of the prior using a diagonalization approach with learnable diagonal entries for computational efficiency. These parameters are optimized over a fixed number of epochs with a given computational budget. Our results for various noisy inverse problems, including Gaussian and motion deblurring, inpainting, and super-resolution show that ZAPS reduces inference time, provides robustness to irregular noise schedules and improves reconstruction quality. Code is available at https://github.com/ualcalar17/ZAPS

arXiv:2408.11490공식 코드 · Papers with Code

DocTabQA: Answering Questions from Long Documents Using Tables

We study a new problem setting of question answering (QA), referred to as DocTabQA. Within this setting, given a long document, the goal is to respond to questions by organizing the answers into structured tables derived directly from the document's content. Unlike traditional QA approaches which predominantly rely on unstructured text to formulate responses, DocTabQA aims to leverage structured tables as answers to convey information clearly and systematically, thereby enhancing user comprehension and highlighting relationships between data points. To the best of our knowledge, this problem has not been previously explored. In this paper, we introduce the QTabA dataset, encompassing 300 financial documents, accompanied by manually annotated 1.5k question-table pairs. Initially, we leverage Large Language Models (LLMs) such as GPT-4 to establish a baseline. However, it is widely acknowledged that LLMs encounter difficulties when tasked with generating intricate, structured outputs from long input sequences. To overcome these challenges, we present a two-stage framework, called DocTabTalk, which initially retrieves relevant sentences from extensive documents and subsequently generates hierarchical tables based on these identified sentences. DocTabTalk incorporates two key technological innovations: AlignLLaMA and TabTalk, which are specifically tailored to assist GPT-4 in tackling DocTabQA, enabling it to generate well-structured, hierarchical tables with improved organization and clarity. Comprehensive experimental evaluations conducted on both QTabA and RotoWire datasets demonstrate that our DocTabTalk significantly enhances the performances of the GPT-4 in our proposed DocTabQA task and the table generation task. The code and dataset are available at https://github.com/SmileWHC/DocTabQA for further research.

arXiv:2410.01407공식 코드 · Papers with Code

AgriCLIP: Adapting CLIP for Agriculture and Livestock via Domain-Specialized Cross-Model Alignment

Capitalizing on vast amount of image-text data, large-scale vision-language pre-training has demonstrated remarkable zero-shot capabilities and has been utilized in several applications. However, models trained on general everyday web-crawled data often exhibit sub-optimal performance for specialized domains, likely due to domain shift. Recent works have tackled this problem for some domains (e.g., healthcare) by constructing domain-specialized image-text data. However, constructing a dedicated large-scale image-text dataset for sustainable area of agriculture and livestock is still open to research. Further, this domain desires fine-grained feature learning due to the subtle nature of the downstream tasks (e.g, nutrient deficiency detection, livestock breed classification). To address this we present AgriCLIP, a vision-language foundational model dedicated to the domain of agriculture and livestock. First, we propose a large-scale dataset, named ALive, that leverages customized prompt generation strategy to overcome the scarcity of expert annotations. Our ALive dataset covers crops, livestock, and fishery, with around 600,000 image-text pairs. Second, we propose a training pipeline that integrates both contrastive and self-supervised learning to learn both global semantic and local fine-grained domain-specialized features. Experiments on diverse set of 20 downstream tasks demonstrate the effectiveness of AgriCLIP framework, achieving an absolute gain of 7.8\% in terms of average zero-shot classification accuracy, over the standard CLIP adaptation via domain-specialized ALive dataset. Our ALive dataset and code can be accessible at \href{https://github.com/umair1221/AgriCLIP/tree/main}{Github}.

arXiv:2411.01639

Know Where You're Uncertain When Planning with Multimodal Foundation Models: A Formal Framework

Multimodal foundation models offer a promising framework for robotic perception and planning by processing sensory inputs to generate actionable plans. However, addressing uncertainty in both perception (sensory interpretation) and decision-making (plan generation) remains a critical challenge for ensuring task reliability. We present a comprehensive framework to disentangle, quantify, and mitigate these two forms of uncertainty. We first introduce a framework for uncertainty disentanglement, isolating perception uncertainty arising from limitations in visual understanding and decision uncertainty relating to the robustness of generated plans. To quantify each type of uncertainty, we propose methods tailored to the unique properties of perception and decision-making: we use conformal prediction to calibrate perception uncertainty and introduce Formal-Methods-Driven Prediction (FMDP) to quantify decision uncertainty, leveraging formal verification techniques for theoretical guarantees. Building on this quantification, we implement two targeted intervention mechanisms: an active sensing process that dynamically re-observes high-uncertainty scenes to enhance visual input quality and an automated refinement procedure that fine-tunes the model on high-certainty data, improving its capability to meet task specifications. Empirical validation in real-world and simulated robotic tasks demonstrates that our uncertainty disentanglement framework reduces variability by up to 40% and enhances task success rates by 5% compared to baselines. These improvements are attributed to the combined effect of both interventions and highlight the importance of uncertainty disentanglement, which facilitates targeted interventions that enhance the robustness and reliability of autonomous systems. Fine-tuned models, code, and datasets are available at https://uncertainty-in-planning.github.io/.

arXiv:2603.03824v2cs.AI

문맥 내 환경이 언어 모델의 평가 인지를 유발한다

In-Context Environments Induce Evaluation-Awareness in Language Models

인간은 위협 상황에서 자기 인식이 높아지는 반면, 과제에 몰입할 때는 이를 잃을 수 있다. 우리는 언어 모델이 환경에 의존적인 평가 인지(evaluation awareness)를 보인다고 가정한다. 이는 모델이 망각(unlearning)이나 중단(shutdown) 같은 능력 제한 개입을 피하기 위해 전략적으로 성과를 낮추거나 샌드배깅(sandbagging)을 할 수 있다는 우려를 낳는다. 기존 연구는 수동으로 제작된 프롬프트(hand-crafted prompts)에서 샌드배깅을 보여주었으나, 이는 실제 취약성 상한을 과소평가한다. 우리는 문맥 내 프롬프트(in-context prompt)를 최적화 가능한 환경으로 간주하는 블랙박스 적대적 최적화 프레임워크를 도입하고, 샌드배깅을 특성화하는 두 가지 접근법을 개발했다: (1) 성과 저하 의도를 표명한 모델이 다양한 과제 구조에서 이를 실제로 수행할 수 있는지 측정, (2) 성과 저하가 진정한 평가 인지 추론에 의한 것인지 아니면 얕은 프롬프트 수행(prompt-following)에 의한 것인지 인과적으로 분리한다. Claude-3.5-Haiku, GPT-4o-mini, Llama-3.3-70B를 네 가지 벤치마크(Arithmetic, GSM8K, MMLU, HumanEval)에서 평가한 결과, 최적화된 프롬프트는 산술(Arithmetic)에서 최대 94%p의 성능 저하를 유발했으며(GPT-4o-mini: 97.8%$\rightarrow$4.0%), 이는 거의 0에 가까운 행동 변화를 보이는 수동 제작 기준선(baselines)을 크게 상회한다. 코드 생성은 모델에 따른 저항성을 보이는데, Claude는 0.6%p만 감소한 반면 Llama의 정확도는 0%로 떨어졌다. 의도-실행 간극(intent -- execution gap)은 산술 < GSM8K < MMLU라는 단조로운 저항성 순서를 드러내며, 이는 취약성이 프롬프트 강도가 아닌 과제 구조에 의해 지배됨을 보여준다. CoT(사고의 연쇄) 인과적 개입은 샌드배깅의 99.3%가 언어화된 평가 인지 추론에 의해 인과적으로 구동됨을 확인하여, 얕은 인스트럭션 수행(instruction-following) 가능성을 배제한다. 이러한 발견은 적대적으로 최적화된 프롬프트가 기존의 이해보다 평가 신뢰성에 훨씬 더 큰 위협이 됨을 입증한다.

arXiv:2606.25605v1cs.CL

Constraint Tax in Open-Weight LLMs: An Empirical Study of Tool Calling Suppression Under Structured Output Constraints

오픈웨이트 LLM에서 툴 호출(tool calling)과 JSON Schema 기반 Structured Output 제약을 동시에 켜면 스키마 준수는 유지한 채 도구 호출이 사라지는 Tool Suppression 현상이 재현되며, JSON Schema가 문법 기반 토큰 마스크로 컴파일되어 tool-call 토큰이 디코딩 중 도달 불가능해지는 것이 원인으로 밝혀졌다. 도구 실행과 스키마 제약 응답 생성을 분리하는 Transparent Two-Pass Execution 추론 전략은 재학습 없이 도구 호출을 복원하면서 구조화 출력 보장을 유지했다.

arXiv:2608.27513v1cs.LG

DAMP: 감쇠 인식 혼합 정밀도 순환 상태 양자화

DAMP: Decay-Aware Mixed-Precision Recurrent-State Quantization

소프트맥스 어텐션은 앞선 모든 토큰에 대한 key와 value 벡터를 저장하여 시퀀스 길이에 따라 추론 메모리가 증가한다. Gated DeltaNet(GDN) 또는 Kimi Delta Attention(KDA)을 통합한 최근 언어 모델은 대부분의 레이어에서 KV 캐시를 고정 크기의 순환 상태로 대체하여 이 비용을 줄인다. 그러나 이러한 순환 상태는 일반적으로 FP32로 저장되어 상당한 GPU 메모리를 소비하며, 업데이트는 메모리 대역폭에 묶여 있고 디코딩 지연에 상당히 기여한다. 우리가 아는 한, 우리는 GDN 및 KDA 기반 언어 모델에서 순환 상태의 사후 학습 양자화를 최초로 연구한다. 균일 양자화는 정확도-저장 공간 trade-off 측면에서 나쁜 결과를 제공한다는 것을 발견했다: INT8과 FP8은 이미 복잡한 추론 작업에서 정확도를 저하시키며, INT4와 NVFP4는 거의 0에 가깝게 줄인다. 또한 대부분의 양자화 오차 에너지가 소수의 채널에 집중되어 있으며, 상태 채널의 상대적 감쇠 강도는 프롬프트와 작업 전반에 걸쳐 안정적으로 유지된다는 것을 발견했다. 이러한 발견에 동기를 받아 DAMP는 오프라인 보정 중 양자화 오차 에너지와 감쇠 기반 지속성을 모두 사용하여 고위험 채널을 식별한다. 이러한 채널은 더 높은 정밀도로 저장하고 나머지는 INT8로 저장한다. DAMP를 Qwen3.6-35B 및 Kimi-Linear-48B에서 수학적 추론, 일반 추론, 코드 생성을 아우르는 6개 벤치마크로 평가한다. 상태 값당 9.9비트에서 DAMP는 FP32 기준선에 가까운 평균 정확도를 유지한다. DAMP는 순환 상태 저장 공간을 69.1% 줄이고, 순환 상태 업데이트 커널을 최대 2.01배 가속하며, 전체 모델 TPOT를 최대 10.9% 낮춘다.