PyTorchKR Papers

“Code Generation”

총 10,163편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2602.00092v2cs.LG

Interpreting and Controlling Model Behavior via Constitutions for Atomic Concept Edits

We introduce a black-box interpretability framework that learns a verifiable constitution: a natural language summary of how changes to a prompt affect a model's specific behavior, such as its alignment, correctness, or adherence to constraints. Our method leverages atomic concept edits (ACEs), which are targeted operations that add, remove, or replace an interpretable concept in the input prompt. By systematically applying ACEs and observing the resulting effects on model behavior across various tasks, our framework learns a causal mapping from edits to predictable outcomes. This learned constitution provides deep, generalizable insights into the model. Empirically, we validate our approach across diverse tasks, including mathematical reasoning and text-to-image alignment, for controlling and understanding model behavior. We found that for text-to-image generation, GPT-Image tends to focus on grammatical adherence, while Imagen 4 prioritizes atmospheric coherence. In mathematical reasoning, distractor variables confuse GPT-5 but leave Gemini 2.5 models and o4-mini largely unaffected. Moreover, our results show that the learned constitutions are highly effective for controlling model behavior, achieving an average of 1.86 times boost in success rate over methods that do not use constitutions. Our open-sourced code can be found at https://github.com/google-deepmind/atomic_concept_edits

arXiv:2510.19600v2cs.SE

Human-Agent Collaborative Paper-to-Page Crafting

AutoPage는 학술 논문에서 인터랙티브 프로젝트 웹페이지를 자동 생성하는 멀티 에이전트(multi-agent) 시스템으로, 내러티브 계획부터 멀티모달 콘텐츠 생성, 인터랙티브 렌더링까지 이어지는 coarse-to-fine 파이프라인과 환각(hallucination) 방지를 위한 Checker 에이전트 및 선택적 인간 개입 체크포인트를 통합한다. 또한 이 작업을 위한 최초의 벤치마크인 PageBench를 구축하여 평가한 결과, AutoPage는 Gemini-2.5-Flash 기준으로 0.1달러 미만의 비용과 15분 이내의 시간으로 고품질 웹페이지를 생성함을 입증하였다.

arXiv:2603.01331v3cs.CL

MetaState: 영속적 작업 메모리로 이산 확산 언어 모델의 추론 향상

MetaState: Persistent Working Memory Enhances Reasoning in Discrete Diffusion Language Models

이산 확산 언어 모델(dLLM)은 마스킹된 시퀀스를 반복적으로 디노이징하여 텍스트를 생성한다. 그러나 표준 dLLM은 각 디노이징 단계에서 현재의 하드 마스킹된 시퀀스만을 조건으로 삼으며, 중간 연속 표현은 샘플링 및 재마스킹 후 폐기된다. 우리는 이 병목 현상을 **정보 고립(Information Island)** 문제라 명명한다: 연속적 정보는 개별 디노이징 단계 내에 고립된 채로 머물며 전체 궤적에 걸쳐 전파되지 못한다. 이 병목은 특히 추론에 해롭다. 추론은 여러 디노이징 단계에 걸쳐 중간 추론 상태가 보존되고 갱신될 것을 요구하기 때문이다. 이 한계를 해결하기 위해 우리는 **MetaState**를 제안한다. MetaState는 동결된 dLLM 백본에 영속적이고 고정 크기의 작업 메모리를 부여하는 경량 순환 증강 기법이다. MetaState는 공유 시간 컨디셔너를 가진 세 모듈로 구성된다: 백본 활성화를 메모리 슬롯으로 읽어들이는 크로스 어텐션 **Mixer**, 단계 간 정보를 통합하는 GRU 방식의 **Updater**, 그리고 갱신된 메모리를 백본에 다시 기록하는 크로스 어텐션 **Injector**이다. 우리는 이 모듈들을 다단계 동역학 학습을 위한 전용 $K$-스텝 언롤링 파이프라인으로 학습시킨다. MetaState는 백본을 동결한 상태에서 학습 가능한 파라미터를 약 0.6%만 추가하면서, 수학적 추론 및 코드 생성 벤치마크에서 동결 기준선 대비 추론 성능을 일관되게 향상시키며, 전체 평가에서 평균 4.5 퍼센트포인트의 성능 향상을 달성한다.

arXiv:2607.25641v1cs.CV

OmniPhys: 텍스트-이미지 생성의 물리적 상식을 위한 지식 그래프 기반 벤치마킹 및 집단 최적화

OmniPhys: Knowledge-Graph-Driven Benchmarking and Collective Optimization for Physical Commonsense in Text-to-Image Generation

텍스트-이미지 모델은 뛰어난 시각적 충실도를 보여주지만, 기본적인 물리적 상식을 위배하는 경우가 빈번합니다. 기존 벤치마크는 대개 조잡한(coarse-grained) 설명에 의존하여 특정 물리 법칙의 숙달 여부를 진단하지 못합니다. 더욱이 생성 과정의 높은 확률적 특성(stochasticity)으로 인해 기존 프롬프트 최적화 기법들은 그래디언트 환각(gradient hallucination) 문제를 겪으며, 최적화기가 시스템적 결함보다는 일시적인 시각적 아티팩트에 의해 잘못 유도되곤 합니다. 이러한 문제를 해결하기 위해, 본 논문에서는 물리 지식 그래프(Physical Knowledge Graph)를 기반으로 구축된 1,551개 샘플 규모의 엄밀한 벤치마크인 OmniPhys를 소개합니다. OmniPhys는 PhET 시뮬레이션을 표준 교육과정에 맞춤으로써 이중 경로 검증 프로토콜을 통해 진단 스트레스 테스트를 수행하는 '지식-시나리오 파이프라인'을 실체화합니다. 나아가 물리적 정렬(physical alignment)을 이산 최적화 문제로 다루는 반복 프레임워크인 OmniPrompt를 제안합니다. 각 질의(query)에 대해 OmniPrompt는 K개의 확률적 이미지를 질의별 피드백 버퍼로 집계합니다. 학습(training) 전반에 걸쳐 메타 정책을 업데이트하기 전에 B개 질의 배치로부터 얻은 피드백을 추가로 병합하여 시드(seed) 및 질의 국소적 노이즈를 필터링합니다. 12개의 대표적인 텍스트-이미지 모델을 대상으로 한 평가는 보편적인 물리적 병목 현상을 보여줍니다. 실험 결과는 OmniPrompt가 다양한 백본 모델 전반에서 물리적 일관성을 크게 향상시킴을 입증하며, 진화된 메타 정책의 전이 가능성과 효과를 증명합니다. 코드와 데이터는 https://github.com/zjukg/OmniPhys 에서 확인할 수 있습니다.

arXiv:2607.29032v1cs.MA

TransMem: 대규모 언어 모델을 위한 은닉 상태의 메모리 변환

TransMem: Transforming Hidden States into Memory for Large Language Models

대규모 언어 모델(LLM) 에이전트(agent)는 긴 상호작용 기록을 바탕으로 작동하는 경우가 점점 늘어나고 있으며, 이러한 환경에서 효과적인 추론을 수행하려면 과거의 관찰과 행동에 분산되어 있는 과제 관련 단서를 식별하고 활용해야 합니다. 그러나 이전에 계산된 표현에 인코딩된 유용한 정보는 이후의 생성 과정에서 충분히 활용되지 못하는 경우가 많습니다. 본 논문에서는 동결된(frozen) LLM 백본의 희소한 과거 은닉 상태(hidden states)를 재사용 가능한 메모리 표현으로 변환하는 경량 추론(inference) 시점 파라메트릭 메모리 모듈인 TransMem을 제안합니다. TransMem은 경량 게이팅 네트워크를 사용하여 이전 컨텍스트를 반복적으로 인코딩하지 않고도 현재 은닉 상태에 잠재적 개입(latent intervention)을 동적으로 적용합니다. 과제 특화된 지식보다는 전이 가능한 메모리 활용 방식을 학습하기 위해, 증거 조건부 자기 증류(evidence-conditioned self-distillation)를 도입합니다. 메모리가 보강된 학생 모델(student)은 전체 컨텍스트를 처리하며 동일한 동결 백본을 공유하는 증거 전용 교사 모델(teacher)의 예측 분포와 일치하도록 학습합니다. LoCoMo, HotpotQA, MemoryAgentBench에 대한 실험 결과는 다양한 모델 아키텍처와 규모 전반에서 일관된 성능 향상을 보여줍니다. TransMem은 LoCoMo에서 11.58~29.25 $F_1$, HotpotQA에서 10.20~13.03 $F_1$의 향상을 보였으며, MemoryAgentBench의 평균 정확도를 29.54%에서 40.00%로 끌어올렸습니다. 이러한 결과는 희소한 과거 은닉 상태가 긴 컨텍스트를 다루는 LLM 에이전트에 효과적이고 효율적인 메모리 기반이 될 수 있음을 입증합니다. 코드는 https://github.com/Haodong-Lei-Ray/TransMem 에서 확인할 수 있습니다.

arXiv:2602.00931v3cs.LG

연속 효용 직접 선호 최적화

Continuous-Utility Direct Preference Optimization

대규모 언어 모델(LLM)의 추론은 종종 단일한 능력으로 취급되며, 부분적인 진행 상황이나 세밀한 추론 품질을 포착하지 못하는 이진 선호 감독에 의존합니다. 본 논문에서는 미세한 추론 품질을 반영하는 연속적인 점수로 이진 레이블을 대체하여, 프롬프트 기반 인지 전략 포트폴리오에 모델을 정렬(alignment)하는 프레임워크인 연속 효용 직접 선호 최적화(CU-DPO)를 소개합니다. 저자들은 K개의 전략으로 학습할 때 이진 선호 대비 표본 복잡도(sample complexity)에서 Theta(K log K)의 개선을 가져오며, DPO가 엔트로피 정규화된 효용 극대화 정책으로 수렴함을 증명합니다. 이러한 신호를 활용하기 위해 다음 두 단계 파이프라인을 제안합니다: (i) 최고 대 전체(best-vs-all) 비교를 통해 모델이 최적의 전략을 선택하도록 최적화하는 전략 선택, (ii) 마진 계층화된 쌍(margin-stratified pairs)을 사용하여 올바른 실행을 학습하는 실행 개선. 이 프레임워크는 도메인에 구애받지 않으며(domain-agnostic), 인지적으로 구별되는 해결 전략과 분해 가능한 연속 효용 신호를 허용하는 모든 태스크를 포트폴리오에 포함할 수 있습니다. 수학 추론 벤치마크에서 CU-DPO는 7개의 베이스 모델에 걸쳐 전략 선택 정확도를 35-46%에서 68-78%로 향상시켰으며, 분포 내(in-distribution) 데이터셋에서 최대 +6.6점의 다운스트림 추론 성능 향상과 효과적인 분포 외(out-of-distribution) 전이 성능을 달성했습니다. 또한 CU-DPO는 코드 생성 및 인과 추론 벤치마크에서도 일관된 성능 향상을 보여주어 수학 도메인을 넘어선 일반화 능력을 입증했습니다.

arXiv:1812.08125

Very Power Efficient Neural Time-of-Flight

Time-of-Flight (ToF) cameras require active illumination to obtain depth information thus the power of illumination directly affects the performance of ToF cameras. Traditional ToF imaging algorithms is very sensitive to illumination and the depth accuracy degenerates rapidly with the power of it. Therefore, the design of a power efficient ToF camera always creates a painful dilemma for the illumination and the performance trade-off. In this paper, we show that despite the weak signals in many areas under extreme short exposure setting, these signals as a whole can be well utilized through a learning process which directly translates the weak and noisy ToF camera raw to depth map. This creates an opportunity to tackle the aforementioned dilemma and make a very power efficient ToF camera possible. To enable the learning, we collect a comprehensive dataset under a variety of scenes and photographic conditions by a specialized ToF camera. Experiments show that our method is able to robustly process ToF camera raw with the exposure time of one order of magnitude shorter than that used in conventional ToF cameras. In addition to evaluating our approach both quantitatively and qualitatively, we also discuss its implication to designing the next generation power efficient ToF cameras. We will make our dataset and code publicly available.

arXiv:1909.04385공식 코드 · Papers with Code

FDA: Feature Disruptive Attack

Though Deep Neural Networks (DNN) show excellent performance across various computer vision tasks, several works show their vulnerability to adversarial samples, i.e., image samples with imperceptible noise engineered to manipulate the network's prediction. Adversarial sample generation methods range from simple to complex optimization techniques. Majority of these methods generate adversaries through optimization objectives that are tied to the pre-softmax or softmax output of the network. In this work we, (i) show the drawbacks of such attacks, (ii) propose two new evaluation metrics: Old Label New Rank (OLNR) and New Label Old Rank (NLOR) in order to quantify the extent of damage made by an attack, and (iii) propose a new adversarial attack FDA: Feature Disruptive Attack, to address the drawbacks of existing attacks. FDA works by generating image perturbation that disrupt features at each layer of the network and causes deep-features to be highly corrupt. This allows FDA adversaries to severely reduce the performance of deep networks. We experimentally validate that FDA generates stronger adversaries than other state-of-the-art methods for image classification, even in the presence of various defense measures. More importantly, we show that FDA disrupts feature-representation based tasks even without access to the task-specific network or methodology. Code available at: https://github.com/BardOfCodes/fda

arXiv:1909.05693공식 코드 · Papers with Code

PDANet: Polarity-consistent Deep Attention Network for Fine-grained Visual Emotion Regression

Existing methods on visual emotion analysis mainly focus on coarse-grained emotion classification, i.e. assigning an image with a dominant discrete emotion category. However, these methods cannot well reflect the complexity and subtlety of emotions. In this paper, we study the fine-grained regression problem of visual emotions based on convolutional neural networks (CNNs). Specifically, we develop a Polarity-consistent Deep Attention Network (PDANet), a novel network architecture that integrates attention into a CNN with an emotion polarity constraint. First, we propose to incorporate both spatial and channel-wise attentions into a CNN for visual emotion regression, which jointly considers the local spatial connectivity patterns along each channel and the interdependency between different channels. Second, we design a novel regression loss, i.e. polarity-consistent regression (PCR) loss, based on the weakly supervised emotion polarity to guide the attention generation. By optimizing the PCR loss, PDANet can generate a polarity preserved attention map and thus improve the emotion regression performance. Extensive experiments are conducted on the IAPS, NAPS, and EMOTIC datasets, and the results demonstrate that the proposed PDANet outperforms the state-of-the-art approaches by a large margin for fine-grained visual emotion regression. Our source code is released at: https://github.com/ZizhouJia/PDANet.

arXiv:2006.09694공식 코드 · Papers with Code

3D Shape Reconstruction from Free-Hand Sketches

Sketches are the most abstract 2D representations of real-world objects. Although a sketch usually has geometrical distortion and lacks visual cues, humans can effortlessly envision a 3D object from it. This suggests that sketches encode the information necessary for reconstructing 3D shapes. Despite great progress achieved in 3D reconstruction from distortion-free line drawings, such as CAD and edge maps, little effort has been made to reconstruct 3D shapes from free-hand sketches. We study this task and aim to enhance the power of sketches in 3D-related applications such as interactive design and VR/AR games. Unlike previous works, which mostly study distortion-free line drawings, our 3D shape reconstruction is based on free-hand sketches. A major challenge for free-hand sketch 3D reconstruction comes from the insufficient training data and free-hand sketch diversity, e.g. individualized sketching styles. We thus propose data generation and standardization mechanisms. Instead of distortion-free line drawings, synthesized sketches are adopted as input training data. Additionally, we propose a sketch standardization module to handle different sketch distortions and styles. Extensive experiments demonstrate the effectiveness of our model and its strong generalizability to various free-hand sketches. Our code is publicly available at https://github.com/samaonline/3D-Shape-Reconstruction-from-Free-Hand-Sketches.

arXiv:2102.10407공식 코드 · Papers with Code

VisualGPT: Data-efficient Adaptation of Pretrained Language Models for Image Captioning

The ability to quickly learn from a small quantity oftraining data widens the range of machine learning applications. In this paper, we propose a data-efficient image captioning model, VisualGPT, which leverages the linguistic knowledge from a large pretrained language model(LM). A crucial challenge is to balance between the use of visual information in the image and prior linguistic knowledge acquired from pretraining. We designed a novel self-resurrecting encoder-decoder attention mechanism to quickly adapt the pretrained LM as the language decoder ona small amount of in-domain training data. The proposed self-resurrecting activation unit produces sparse activations but has reduced susceptibility to zero gradients. We train the proposed model, VisualGPT, on 0.1%, 0.5% and 1% of MSCOCO and Conceptual Captions training data. Under these conditions, we outperform the best baseline model by up to 10.8% CIDEr on MS COCO and upto 5.4% CIDEr on Conceptual Captions. Further, Visual-GPT achieves the state-of-the-art result on IU X-ray, a medical report generation dataset. To the best of our knowledge, this is the first work that improves data efficiency of image captioning by utilizing LM pretrained on unimodal data. Our code is available at: https://github.com/Vision-CAIR/VisualGPT.

arXiv:2202.08465공식 코드 · Papers with Code

End-to-End Training for Back-Translation with Categorical Reparameterization Trick

Back-translation (BT) is an effective semi-supervised learning framework in neural machine translation (NMT). A pre-trained NMT model translates monolingual sentences and makes synthetic bilingual sentence pairs for the training of the other NMT model, and vice versa. Understanding the two NMT models as inference and generation models, respectively, the training method of variational auto-encoder (VAE) was applied in previous works, which is a mainstream framework of generative models. However, the discrete property of translated sentences prevents gradient information from flowing between the two NMT models. In this paper, we propose the categorical reparameterization trick (CRT) that makes NMT models generate differentiable sentences so that the VAE's training framework can work in an end-to-end fashion. Our BT experiment conducted on a WMT benchmark dataset demonstrates the superiority of our proposed CRT compared to the Gumbel-softmax trick, which is a popular reparameterization method for categorical variable. Moreover, our experiments conducted on multiple WMT benchmark datasets demonstrate that our proposed end-to-end training framework is effective in terms of BLEU scores not only compared to its counterpart baseline which is not trained in an end-to-end fashion, but also compared to other previous BT works. The code is available at the web.

arXiv:2204.09222

K-LITE: Learning Transferable Visual Models with External Knowledge

The new generation of state-of-the-art computer vision systems are trained from natural language supervision, ranging from simple object category names to descriptive captions. This form of supervision ensures high generality and usability of the learned visual models, due to the broad concept coverage achieved via large-scale data collection process. Alternatively, we argue that learning with external knowledge is a promising way which leverages a much more structured source of supervision and offers sample efficiency. We propose K-LITE, a simple strategy to leverage external knowledge for building transferable visual systems: In training, it enriches entities in text with WordNet and Wiktionary knowledge, leading to an efficient and scalable approach to learning image representations that uses knowledge about the visual concepts. In evaluation, the text is also augmented with external knowledge and then used to reference learned visual concepts (or describe new ones) to enable zero-shot and few-shot transfer of the pre-trained models. We study the performance of K-LITE on two important computer vision problems, image classification and object detection, benchmarking on 20 and 13 different existing datasets, respectively. The proposed knowledge-augmented models show significant improvement in transfer learning performance over existing methods. Our code is available at https://github.com/microsoft/klite.

arXiv:2209.07692공식 코드 · Papers with Code

Answering Numerical Reasoning Questions in Table-Text Hybrid Contents with Graph-based Encoder and Tree-based Decoder

In the real-world question answering scenarios, hybrid form combining both tabular and textual contents has attracted more and more attention, among which numerical reasoning problem is one of the most typical and challenging problems. Existing methods usually adopt encoder-decoder framework to represent hybrid contents and generate answers. However, it can not capture the rich relationship among numerical value, table schema, and text information on the encoder side. The decoder uses a simple predefined operator classifier which is not flexible enough to handle numerical reasoning processes with diverse expressions. To address these problems, this paper proposes a \textbf{Re}lational \textbf{G}raph enhanced \textbf{H}ybrid table-text \textbf{N}umerical reasoning model with \textbf{T}ree decoder (\textbf{RegHNT}). It models the numerical question answering over table-text hybrid contents as an expression tree generation task. Moreover, we propose a novel relational graph modeling method, which models alignment between questions, tables, and paragraphs. We validated our model on the publicly available table-text hybrid QA benchmark (TAT-QA). The proposed RegHNT significantly outperform the baseline model and achieve state-of-the-art results. We openly released the source code and data at https://github.com/lfy79001/RegHNT (2022-05-05).

arXiv:2305.05662공식 코드 · Papers with Code

InternGPT: Solving Vision-Centric Tasks by Interacting with ChatGPT Beyond Language

We present an interactive visual framework named InternGPT, or iGPT for short. The framework integrates chatbots that have planning and reasoning capabilities, such as ChatGPT, with non-verbal instructions like pointing movements that enable users to directly manipulate images or videos on the screen. Pointing (including gestures, cursors, etc.) movements can provide more flexibility and precision in performing vision-centric tasks that require fine-grained control, editing, and generation of visual content. The name InternGPT stands for \textbf{inter}action, \textbf{n}onverbal, and \textbf{chat}bots. Different from existing interactive systems that rely on pure language, by incorporating pointing instructions, the proposed iGPT significantly improves the efficiency of communication between users and chatbots, as well as the accuracy of chatbots in vision-centric tasks, especially in complicated visual scenarios where the number of objects is greater than 2. Additionally, in iGPT, an auxiliary control mechanism is used to improve the control capability of LLM, and a large vision-language model termed Husky is fine-tuned for high-quality multi-modal dialogue (impressing ChatGPT-3.5-turbo with 93.89\% GPT-4 Quality). We hope this work can spark new ideas and directions for future interactive visual systems. Welcome to watch the code at https://github.com/OpenGVLab/InternGPT.

arXiv:2309.04669공식 코드 · Papers with Code

Unified Language-Vision Pretraining in LLM with Dynamic Discrete Visual Tokenization

Recently, the remarkable advance of the Large Language Model (LLM) has inspired researchers to transfer its extraordinary reasoning capability to both vision and language data. However, the prevailing approaches primarily regard the visual input as a prompt and focus exclusively on optimizing the text generation process conditioned upon vision content by a frozen LLM. Such an inequitable treatment of vision and language heavily constrains the model's potential. In this paper, we break through this limitation by representing both vision and language in a unified form. Specifically, we introduce a well-designed visual tokenizer to translate the non-linguistic image into a sequence of discrete tokens like a foreign language that LLM can read. The resulting visual tokens encompass high-level semantics worthy of a word and also support dynamic sequence length varying from the image. Coped with this tokenizer, the presented foundation model called LaVIT can handle both image and text indiscriminately under the same generative learning paradigm. This unification empowers LaVIT to serve as an impressive generalist interface to understand and generate multi-modal content simultaneously. Extensive experiments further showcase that it outperforms the existing models by a large margin on massive vision-language tasks. Our code and models are available at https://github.com/jy0205/LaVIT.

arXiv:2310.01236공식 코드 · Papers with Code

Mirror Diffusion Models for Constrained and Watermarked Generation

Modern successes of diffusion models in learning complex, high-dimensional data distributions are attributed, in part, to their capability to construct diffusion processes with analytic transition kernels and score functions. The tractability results in a simulation-free framework with stable regression losses, from which reversed, generative processes can be learned at scale. However, when data is confined to a constrained set as opposed to a standard Euclidean space, these desirable characteristics appear to be lost based on prior attempts. In this work, we propose Mirror Diffusion Models (MDM), a new class of diffusion models that generate data on convex constrained sets without losing any tractability. This is achieved by learning diffusion processes in a dual space constructed from a mirror map, which, crucially, is a standard Euclidean space. We derive efficient computation of mirror maps for popular constrained sets, such as simplices and $\ell_2$-balls, showing significantly improved performance of MDM over existing methods. For safety and privacy purposes, we also explore constrained sets as a new mechanism to embed invisible but quantitative information (i.e., watermarks) in generated data, for which MDM serves as a compelling approach. Our work brings new algorithmic opportunities for learning tractable diffusion on complex domains. Our code is available at https://github.com/ghliu/mdm

arXiv:2312.15503공식 코드 · Papers with Code

Making Large Language Models A Better Foundation For Dense Retrieval

Dense retrieval needs to learn discriminative text embeddings to represent the semantic relationship between query and document. It may benefit from the using of large language models (LLMs), given LLMs' strong capability on semantic understanding. However, the LLMs are pre-trained by text generation tasks, whose working pattern is completely different from representing texts as embeddings. As a result, it is imperative to study how to adapt LLMs properly so that they can be effectively initialized as the backbone encoder for dense retrieval. In this paper, we propose a novel approach, called LLaRA (LLM adapted for dense RetrievAl), which works as a post-hoc adaptation of LLM for the dense retrieval application. LLaRA consists of two pretext tasks: EBAE (Embedding-Based Auto-Encoding) and EBAR (Embedding-Based Auto-Regression), where the text embeddings from LLM are used to reconstruct the tokens for the input sentence and predict the tokens for the next sentence, respectively. LLaRA turns out to be simple, lightweight, and highly effective. It is applied to adapt LLaMA-2-7B (base) on the Wikipedia corpus, where it substantially improves the model's fine-tuned performances on a variety of dense retrieval benchmarks, like MSMARCO and BEIR. Our model and code will be made publicly available at BGE repository.

arXiv:2406.01294공식 코드 · Papers with Code

CE-VAE: Capsule Enhanced Variational AutoEncoder for Underwater Image Enhancement

Unmanned underwater image analysis for marine monitoring faces two key challenges: (i) degraded image quality due to light attenuation and (ii) hardware storage constraints limiting high-resolution image collection. Existing methods primarily address image enhancement with approaches that hinge on storing the full-size input. In contrast, we introduce the Capsule Enhanced Variational AutoEncoder (CE-VAE), a novel architecture designed to efficiently compress and enhance degraded underwater images. Our attention-aware image encoder can project the input image onto a latent space representation while being able to run online on a remote device. The only information that needs to be stored on the device or sent to a beacon is a compressed representation. There is a dual-decoder module that performs offline, full-size enhanced image generation. One branch reconstructs spatial details from the compressed latent space, while the second branch utilizes a capsule-clustering layer to capture entity-level structures and complex spatial relationships. This parallel decoding strategy enables the model to balance fine-detail preservation with context-aware enhancements. CE-VAE achieves state-of-the-art performance in underwater image enhancement on six benchmark datasets, providing up to 3x higher compression efficiency than existing approaches. Code available at \url{https://github.com/iN1k1/ce-vae-underwater-image-enhancement}.

arXiv:2303.12175공식 코드 · Papers with Code

Black-box Backdoor Defense via Zero-shot Image Purification

Backdoor attacks inject poisoned samples into the training data, resulting in the misclassification of the poisoned input during a model's deployment. Defending against such attacks is challenging, especially for real-world black-box models where only query access is permitted. In this paper, we propose a novel defense framework against backdoor attacks through Zero-shot Image Purification (ZIP). Our framework can be applied to poisoned models without requiring internal information about the model or any prior knowledge of the clean/poisoned samples. Our defense framework involves two steps. First, we apply a linear transformation (e.g., blurring) on the poisoned image to destroy the backdoor pattern. Then, we use a pre-trained diffusion model to recover the missing semantic information removed by the transformation. In particular, we design a new reverse process by using the transformed image to guide the generation of high-fidelity purified images, which works in zero-shot settings. We evaluate our ZIP framework on multiple datasets with different types of attacks. Experimental results demonstrate the superiority of our ZIP framework compared to state-of-the-art backdoor defense baselines. We believe that our results will provide valuable insights for future defense methods for black-box models. Our code is available at https://github.com/sycny/ZIP.