PyTorchKR Papers

“Code Generation”

총 10,140편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2611.10297v1cs.CV

Uncertainty, Not Concealment: A Mechanistic Reappraisal of Sycophancy in Vision-Language Models

Vision-language models (VLMs) in medical retrieval-augmented generation must be able to say when a retrieved report contradicts the X-ray, so understanding whether they can detect such conflicts is important for safe deployment. The sycophancy mask hypothesis holds that these models internally detect conflicts but refuse to report them, concealing a detected contradiction. We introduce a three-part diagnostic that separates concealment from uncertainty: representation probing, activation steering, and two-way forced choice. Across four instruction-tuned VLMs on chest X-rays, we confirm behavioral suppression of contradiction and find that a linear probe separates matched from mismatched inputs, beyond what a simple CLIP model achieves. Steering the learned conflict direction never elicits contradiction, and forcing a binary choice does not recover one; instead, cross-modal attention to the image drops on mismatched pairs. We conclude that these models fail to contradict because the mismatch registers as hedged, gated uncertainty rather than concealed knowledge, calling for re-grounding and uncertainty surfacing. Code is available at https://github.com/mohamed-eldagla/Uncertainty-Not-Concealment/.

arXiv:2603.10863v2cs.CV

순차적 거리를 넘어: 모달 간 거리 불변 위치 인코딩

Beyond Sequential Distance: Inter-Modal Distance Invariant Position Encoding

다중모달 대규모 언어 모델(MLLM)의 놀라운 능력에도 불구하고, 긴 컨텍스트 시나리오에서 여전히 시각적 페이딩(visual fading) 문제를 겪는다. 구체적으로, 텍스트 시퀀스가 길어질수록 시각 토큰에 대한 어텐션이 약해져 시각적 제약에서 벗어난 텍스트 생성이 발생한다. 우리는 이러한 성능 저하의 원인을 다중모달 RoPE의 고유한 귀납적 바이어스에 두는데, 이는 시각 토큰과 텍스트 토큰 사이의 거리가 증가할수록 모달 간 어텐션에 페널티를 부과한다. 이를 해결하기 위해, 모달리티 상호작용에 기반하여 위치 인코딩을 분리하는 단순하지만 효과적인 메커니즘인 모달 간 거리 불변 위치 인코딩(DIPE)을 제안한다. DIPE는 모달 내 상호작용에 대해 자연스러운 상대적 위치 관계를 유지하여 지역 구조를 보존하면서, 모달 간 상호작용에 대해서는 고정된 지각적 근접성을 강제한다. 이 전략은 모달 간 거리 기반 페널티를 효과적으로 완화하여, 컨텍스트 길이와 관계없이 시각 신호가 지각적으로 일관되게 유지되도록 한다. 실험 결과, DIPE를 다중모달 RoPE와 통합하면 모델이 긴 컨텍스트 시나리오에서 안정적인 시각적 그라운딩(visual grounding)을 유지하며, 시각적 페이딩을 크게 완화하면서 표준 단기 컨텍스트 벤치마크에서의 성능은 보존함을 보여준다. 코드는 https://github.com/lchen1019/DIPE 에서 제공된다.

arXiv:2607.21591v1cs.CV

점진적 시드 가지치기를 통한 디퓨전 모델의 추론 시 스케일링

Inference-Time Scaling of Diffusion Models via Progressive Seed Pruning

디퓨전 및 flow-matching 모델이 조건부 이미지 생성을 지배하고 있지만, 이러한 모델에 대한 추론 시 스케일링은 자기회귀(autoregressive) 언어 모델에 비해 훨씬 덜 발전되어 있다. 최종 품질이 초기 노이즈 시드에 매우 민감하기 때문에, 많은 접근법은 블랙박스 보상 하에서 시드 탐색이나 재샘플링에 추가 연산을 투입하지만, 일반적으로 추론 전반에 걸쳐 일정한 메모리 사용량을 유지한다. 우리는 이 제약을 완화하면 아직 충분히 탐구되지 않은 추론 시 스케일링 축을 활성화할 수 있음을 보인다: 탐험을 앞단에 집중하고, 초기에 많은 시드를 평가하며, 공격적으로 가지치기함으로써 고정된 연산 예산을 더 효과적으로 사용할 수 있다. 점진적 시드 가지치기(Progressive Seed Pruning, PSP)는 중간 디노이징 추정치에 점수를 매기고 후보 집합을 점진적으로 좁혀 유망한 궤적만 완전히 디노이징하며, 모델 평가 총 횟수는 고정된 상태를 유지한다. 디퓨전 및 flow-matching 백본 전반에서 PSP는 보상 유도 선택을 일관되게 개선하며, 동일한 연산량에서 best-of-N, importance-sampling, tree-search 기준선보다 더 높은 GenEval 점수(자동)와 프롬프트 정렬에 대한 더 나은 인간 평가를 달성한다. 프로젝트 페이지: https://www.vision.caltech.edu/psp. 코드: https://github.com/rogerioagjr/psp.

arXiv:2608.20851v1cs.SE

BC-Bench: ERP 도메인 특화 언어에서의 에이전틱 엔지니어링 평가

BC-Bench: Evaluating Agentic Engineering in a Domain-Specific Language for ERP

에이전틱 엔지니어링 시스템은 범용 �ench마크에서 강한 성능을 보여 왔지만, 기업 자원 계획(ERP) 도메인 특화 언어(DSL)에서의 효과는 아직 충분히 탐구되지 않았다. Microsoft Dynamics 365 Business Central용 DSL인 AL에서 실세계 과제에 대한 에이전틱 엔지니어링을 평가하도록 설계된 벤치마크 BC-Bench를 제안한다. BC-Bench는 Microsoft 소유의 두 프로덕션 저장소에서 추출한 101개의 수동 큐레이션 과제로 구성되며, 실제 ERP 개발 워크플로를 반영한다. SWE-Bench 방법론을 적용하여 AL 생태계의 고유한 제약—공개 자료의 부족과 복잡한 환경 프로비저닝을 포함—을 다룬다. 기능적 코드 생성을 넘어 BC-Bench는 테스트 생성을 평가하고, 시각적 맥락이 흔히 존재하는 멀티모달 문제 진술을 지원한다. 두 에이전트 하네스에 걸쳐 여러 프론티어 모델을 평가하고, 비결정성을 반영하기 위해 다중 실행 지표를 활용한다. 버그 수정 카테고리에서 평가한 설정 하에서는 해결률의 모델 간 차이가 평가한 두 에이전트 하네스 간 차이보다 크며, 범용 벤치마크에서 보고된 개선이 AL로 일관되게 전이되지는 않는다. 이 결과는 도메인 특화 평가의 필요성을 부각한다.

arXiv:2609.26637v1cs.CL

유능하면서도 간결한: 최전선 모델에서 숨겨진 사고의 연쇄 추출 및 특성화

Capable yet Parsimonious: Extracting and Characterizing Hidden Chain-of-Thought in Frontier Models

최전선 언어 모델의 급격한 능력 향상은 널리 개선된 추론 능력 덕분으로 여겨지지만, 폐쇄형 시스템에서는 원시 사고의 연쇄(CoT) 추적이 숨겨져 있어 이를 검증할 수 없다. 표준 API 기능을 통해 간단한 사용자 정의 도구를 등록함으로써, 우리는 최전선 모델이 중간 추론을 외부화하도록 유도한다. 이러한 추적이 진정한 추론이 아니라 사후 합리화를 반영할 수 있기 때문에, 먼저 오픈소스 모델에서 네이티브 CoT와 대조 평가하고 GPT-6 Astra를 포함한 폐쇄형 최전선 모델로 확장한다. 추출된 추론은 네이티브 추론 성능과 일치하며, 경쟁 수학·과학·코드 생성 전반에서 추론 없음 기준선을 크게 능가함을 확인한다. 이어 최전선 모델이 중간 추론을 어떻게 구조화하는지 특성화한다. 토큰 효율, 추론 단계 유형, 유도된 추론 트리에 걸쳐, 모델이 추론을 외부화·압축·조직하는 방식에서 체계적 차이를 확인한다. Astra는 토큰 효율적인 지향적 추론을 보이며, 더 일찍 올바른 궤적을 선택하는 한편 기초 단계는 내부에서 처리하고 핵심 추론만 외부화한다. 이러한 결과는 벤치마크 점수를 넘어 최전선 모델 추론에 대한 행동적 관점을 제공한다.

arXiv:2610.11858v1cs.SE

에이전트 스킬 진화를 위한 궤적 기반 결함 국소화

Trajectory-Guided Fault Localization for Agent Skill Evolution

에이전트(agent) 스킬은 코드 에이전트에 재사용 가능한 가이던스를 제공하지만, 불완전하거나 부적절한 가이던스는 태스크 수행을 저해할 수 있습니다. 스킬 개선에 드는 수작업을 줄이기 위해, 최근 접근 방식들은 실행 피드백으로부터 수정안을 생성하는 데 대규모 언어 모델(LLM)을 사용합니다. 그러나 이러한 수정안을 명시적인 행동 증거에 그라운딩(grounding)하는 것은 여전히 어려운 과제입니다. 이러한 한계를 해결하기 위해, 본 논문에서는 에이전트 스킬의 궤적 기반 결함 국소화(fault localization)를 바탕으로 한 스킬 진화 접근법인 SkillMorph를 제안합니다. 이 방식의 핵심 아이디어는 수정안을 생성하기 전에 실행 증거를 특정 스킬 내용과 연결하는 것입니다. 구체적으로 SkillMorph는 반복 실행 및 태스크 전반에서 추상화된 궤적 내 실패와 성공 증거를 비교하고, 진화 루프 간 변경 사항을 통합하여 의심스러운 행동을 식별합니다. 그런 다음 이러한 의심스러운 행동을 활용해 스킬 내 편집 위치를 국소화하고 이에 대응하는 수정안을 생성합니다. SWE-Skills-Bench 및 CannBot을 대상으로 한 실험 결과, SkillMorph로 진화된 스킬은 기존 스킬 및 네 가지 기존 스킬 진화 방식에 비해 지속적으로 더 높은 시도 수준의 정확도와 실행 일관성을 달성함을 보여줍니다. 또한 AI 연산자 개발 팀과 함께 자동화된 커널(kernel) 생성에 SkillMorph를 적용하여 6건의 스킬 수정 풀 리퀘스트가 반영되는 성과를 거두었습니다.

arXiv:2310.03328공식 코드 · Papers with Code

Reformulating Domain Adaptation of Large Language Models as Adapt-Retrieve-Revise: A Case Study on Chinese Legal Domain

While large language models (LLMs) like GPT-4 have recently demonstrated astonishing zero-shot capabilities in general domain tasks, they often generate content with hallucinations in specific domains such as Chinese law, hindering their application in these areas. This is typically due to the absence of training data that encompasses such a specific domain, preventing GPT-4 from acquiring in-domain knowledge. A pressing challenge is that it's not plausible to continue training LLMs of such scale on in-domain data. This paper introduces a simple and effective domain adaptation framework for GPT-4 by reformulating generation as an \textbf{adapt-retrieve-revise} process. The initial step is to \textbf{adapt} an affordable 7B LLM to the target domain by continuing learning on in-domain data. When solving a task, we leverage the adapted LLM to generate a draft answer given a task query. Then, the draft answer will be used to \textbf{retrieve} supporting evidence candidates from an external in-domain knowledge base. Finally, the draft answer and retrieved evidence are concatenated into a whole prompt to let GPT-4 assess the evidence and \textbf{revise} the draft answer to generate the final answer. Our proposal combines the advantages of the efficiency of adapting a smaller 7B model with the evidence-assessing capability of GPT-4 and effectively prevents GPT-4 from generating hallucinatory content. In the zero-shot setting of four Chinese legal tasks, our method improves accuracy by 33.3\% compared to the direct generation by GPT-4. When compared to two stronger retrieval-based baselines, our method outperforms them by 15.4\% and 23.9\%. Our code will be released

arXiv:2406.09406

4M-21: An Any-to-Any Vision Model for Tens of Tasks and Modalities

Current multimodal and multitask foundation models like 4M or UnifiedIO show promising results, but in practice their out-of-the-box abilities to accept diverse inputs and perform diverse tasks are limited by the (usually rather small) number of modalities and tasks they are trained on. In this paper, we expand upon the capabilities of them by training a single model on tens of highly diverse modalities and by performing co-training on large-scale multimodal datasets and text corpora. This includes training on several semantic and geometric modalities, feature maps from recent state of the art models like DINOv2 and ImageBind, pseudo labels of specialist models like SAM and 4DHumans, and a range of new modalities that allow for novel ways to interact with the model and steer the generation, for example image metadata or color palettes. A crucial step in this process is performing discrete tokenization on various modalities, whether they are image-like, neural network feature maps, vectors, structured data like instance segmentation or human poses, or data that can be represented as text. Through this, we expand on the out-of-the-box capabilities of multimodal models and specifically show the possibility of training one model to solve at least 3x more tasks/modalities than existing ones and doing so without a loss in performance. This enables more fine-grained and controllable multimodal generation capabilities and allows us to study the distillation of models trained on diverse data and objectives into a unified model. We successfully scale the training to a three billion parameter model using tens of modalities and different datasets. The resulting models and training code are open sourced at 4m.epfl.ch.

arXiv:2610.41949v1cs.AI

Optimizing Stochastic Policy Geometry for Few-Step Flow Reinforcement Learning

Few-step flow models generate samples through a handful of large denoising transitions. Applying GRPO to such models requires introducing stochasticity into the denoising transitions, and when a single transition spans a wide range of noise levels, the choice of the resulting stochastic policy becomes a major factor in optimization. Recent constructions constrain the policy to preserve the target flow coefficients, but this constraint admits a family of policies and offers no criterion for choosing among them. We introduce Geo-FSRL (Geometry-Optimized Few-Step Reinforcement Learning), built on a selection principle: at a fixed conditioning state and timestep pair, choose the admissible policy that minimizes the conditional KL divergence induced by any given nonzero velocity-output update. Within a coefficient-preserving family of isotropic Gaussian transitions, this criterion reduces to minimizing a single scalar geometry factor that determines the conditional KL for every velocity-output update. The same factor governs velocity-space Fisher information and conditional transition log-likelihood-ratio variance. We derive the unique closed-form optimizer within this family, shared across all nonzero update magnitudes and directions, with transition coefficients determined by both the source and target noise levels. Because the minimized geometry factor still varies across timesteps, we further adopt gradient-scale matching from prior work and derive closed-form policy-gradient and KL weights that normalize explicit score scales and reference-KL curvature. Geo-FSRL trains on the original few-step schedule and retains the deterministic solver at inference. Our method achieves higher final reward scores than Flow-GRPO and Flow-CPS on both image generation and video generation models under the 4-step setting. Code is available at https://github.com/ModelTC/LightX2V/tree/main/lightx2v_rl

arXiv:2611.02587v1cs.CL

They Know Where It Is, But Not What It Says: A Bilingual Hallucination Benchmark of Hadith Recall in Open LLMs

Large language models are increasingly consulted for religious knowledge, including the recall of specific Hadith (recorded sayings and actions of the Prophet Muhammad, peace be upon him). Unlike open-ended generation, Hadith recall is a high-stakes, verifiable task: a reference (collection, chapter, and Hadith number) has exactly one correct text, and any deviation from it constitutes a fabrication that can be mistaken for authentic scripture. This study introduces a bilingual (English/Arabic) benchmark that probes five open-weight, locally-deployable language models (Mistral-7B, Gemma3-4B, Jais-7B, ALLaM-7B, and Phi4-mini-3.8B) across all six canonical Sunni Hadith collections (Sahih al-Bukhari, Sahih Muslim, Sunan Abi Da'ud, Jami' al-Tirmidhi, Sunan an-Nasa'i, Sunan Ibn Majah), sampling 500 Hadith per collection (3,000 per model per language, 30,000 responses total). Each model is given only the bibliographic reference and asked to reproduce the exact Hadith text, with no retrieval augmentation, and is scored on lexical fidelity (BLEU) and semantic fidelity (multilingual cosine similarity), with refusals flagged separately. Across all five models and both languages, 0% of 30,000 responses achieve both high lexical and high semantic fidelity, i.e., an exact, verifiable match to the source text. Arabic responses fare uniformly worse than English. These findings suggest that current open-weight LLMs, used without retrieval grounding, are unreliable and potentially dangerous sources for exact religious-text recall, and that deployment for Islamic scholarship should mandate retrieval-augmented generation over a verified corpus rather than parametric memory. The complete code, prompts, evaluation pipeline, and scored outputs are publicly available at https://github.com/dayam8696/Hadith-Hallucination-Benchmark.

arXiv:2608.07713v1cs.CV

Tokenizer Generator Coupling in Medical Image Generation

Latent medical image generators usually treat the tokenizer as fixed preprocessing. We test whether this separation is valid in a controlled ChestMNIST study at 64x64, crossing discrete tokenizers, generator families, and sampler settings under a shared latent grid, with continuous-latent reference cells. In this controlled setting, rankings depend jointly on the tokenizer, generator, and sampler: the best quantizer changes with the generator, and validation-based sampler selection changes the apparent generator ranking. We retrain the vocabulary-1024 interaction block at three seeds and the interaction survives (6 of 9 pairwise quantizer comparisons exceed three seed standard deviations), and we scope the wider single-seed grid accordingly. Reconstruction PSNR alone is not a reliable selection criterion; we instead introduce a generator-free statistic, neighbour-conditional predictive gain, that separates the quantizer families by downstream generation quality (rank-AUC 1.00) where reconstruction PSNR and marginal token entropy do not. On LFQ-1024, retuning D3PM and SE-D3PM (selected on a held-out validation split) moves them from default FID-192 0.44/0.41 to 0.09/0.10 at lower NFE, replicated across seeds; the continuous references were not given an equivalent sampler sweep. We report FID-192 as an internal ranking metric; it ranks consistently with standard FID-2048 (Spearman 0.80) and with a label-free classifier two-sample test (0.78). We interpret these results through a rate-distortion-modelability framing, where modelability is conditional on the generator, sampler, and inference budget. All experiments are at 64x64 on low-resolution medical-style images, unconditional, and evaluated with non-clinical FID-based metrics, and we scope every claim to that setting. Code: https://github.com/liamchalcroft/medtokenizers and https://github.com/liamchalcroft/medlatents.

arXiv:2307.15043공식 코드 · Papers with Code

Universal and Transferable Adversarial Attacks on Aligned Language Models

Because "out-of-the-box" large language models are capable of generating a great deal of objectionable content, recent work has focused on aligning these models in an attempt to prevent undesirable generation. While there has been some success at circumventing these measures -- so-called "jailbreaks" against LLMs -- these attacks have required significant human ingenuity and are brittle in practice. In this paper, we propose a simple and effective attack method that causes aligned language models to generate objectionable behaviors. Specifically, our approach finds a suffix that, when attached to a wide range of queries for an LLM to produce objectionable content, aims to maximize the probability that the model produces an affirmative response (rather than refusing to answer). However, instead of relying on manual engineering, our approach automatically produces these adversarial suffixes by a combination of greedy and gradient-based search techniques, and also improves over past automatic prompt generation methods. Surprisingly, we find that the adversarial prompts generated by our approach are quite transferable, including to black-box, publicly released LLMs. Specifically, we train an adversarial attack suffix on multiple prompts (i.e., queries asking for many different types of objectionable content), as well as multiple models (in our case, Vicuna-7B and 13B). When doing so, the resulting attack suffix is able to induce objectionable content in the public interfaces to ChatGPT, Bard, and Claude, as well as open source LLMs such as LLaMA-2-Chat, Pythia, Falcon, and others. In total, this work significantly advances the state-of-the-art in adversarial attacks against aligned language models, raising important questions about how such systems can be prevented from producing objectionable information. Code is available at github.com/llm-attacks/llm-attacks.

arXiv:2610.17609v1cs.AI

Verifier-Harness Co-Evolution for Training-Free Self-Improvement on Open-Ended Tasks

Training-free self-improvement lets an agent use feedback to revise the response it is about to return and to update the harness state that shapes how it handles later tasks. On open-ended tasks that feedback rarely comes from reference answers or executable tests, so the agent has to rely on a model-based verifier. When the verifier is wrong, the current response can get worse, and misleading experience can persist across the task stream. A verifier that never changes also grows stale as the agent's responses improve and task demands shift. We introduce Unified Co-Evolution (UCoE), a training-free method that evolves an executable verifier program together with the agent's harness state while all model parameters stay fixed. UCoE synthesizes candidate verifier programs and tests them on response contrasts, pairs of responses that differ in one declared property and have a known preferred side. It selects the candidate whose discrimination on its weakest contrast family is highest, a quantity we call driving capacity. The selected verifier diagnoses one weakness in each draft, guides a single repair, decides which response is served, and determines whether the accepted repair may enter the harness state. As tasks complete, their diagnostics seed new candidate verifiers, which must beat the current verifier on the same contrasts before adoption. Across six open-ended benchmarks, UCoE averages 59.00% preference against matched direct generation with Qwen3-8B and 57.19% with Qwen3.8-27B, margins of 9.00 and 7.19 percentage points. On three verifiable benchmarks, the same method raises the mean from 36.06% for direct generation to 38.48%. Code is available at https://github.com/tianyi0216/Unified-Co-Evolution.

arXiv:2005.11251

A machine learning based software pipeline to pick the variable ordering for algorithms with polynomial inputs

We are interested in the application of Machine Learning (ML) technology to improve mathematical software. It may seem that the probabilistic nature of ML tools would invalidate the exact results prized by such software, however, the algorithms which underpin the software often come with a range of choices which are good candidates for ML application. We refer to choices which have no effect on the mathematical correctness of the software, but do impact its performance. In the past we experimented with one such choice: the variable ordering to use when building a Cylindrical Algebraic Decomposition (CAD). We used the Python library Scikit-Learn (sklearn) to experiment with different ML models, and developed new techniques for feature generation and hyper-parameter selection. These techniques could easily be adapted for making decisions other than our immediate application of CAD variable ordering. Hence in this paper we present a software pipeline to use sklearn to pick the variable ordering for an algorithm that acts on a polynomial system. The code described is freely available online.

arXiv:2104.03214공식 코드 · Papers with Code

Self-Supervised Learning for Semi-Supervised Temporal Action Proposal

Self-supervised learning presents a remarkable performance to utilize unlabeled data for various video tasks. In this paper, we focus on applying the power of self-supervised methods to improve semi-supervised action proposal generation. Particularly, we design an effective Self-supervised Semi-supervised Temporal Action Proposal (SSTAP) framework. The SSTAP contains two crucial branches, i.e., temporal-aware semi-supervised branch and relation-aware self-supervised branch. The semi-supervised branch improves the proposal model by introducing two temporal perturbations, i.e., temporal feature shift and temporal feature flip, in the mean teacher framework. The self-supervised branch defines two pretext tasks, including masked feature reconstruction and clip-order prediction, to learn the relation of temporal clues. By this means, SSTAP can better explore unlabeled videos, and improve the discriminative abilities of learned action features. We extensively evaluate the proposed SSTAP on THUMOS14 and ActivityNet v1.3 datasets. The experimental results demonstrate that SSTAP significantly outperforms state-of-the-art semi-supervised methods and even matches fully-supervised methods. Code is available at https://github.com/wangxiang1230/SSTAP.

arXiv:2106.05963공식 코드 · Papers with Code

Learning to See by Looking at Noise

Current vision systems are trained on huge datasets, and these datasets come with costs: curation is expensive, they inherit human biases, and there are concerns over privacy and usage rights. To counter these costs, interest has surged in learning from cheaper data sources, such as unlabeled images. In this paper we go a step further and ask if we can do away with real image datasets entirely, instead learning from noise processes. We investigate a suite of image generation models that produce images from simple random processes. These are then used as training data for a visual representation learner with a contrastive loss. We study two types of noise processes, statistical image models and deep generative models under different random initializations. Our findings show that it is important for the noise to capture certain structural properties of real data but that good performance can be achieved even with processes that are far from realistic. We also find that diversity is a key property to learn good representations. Datasets, models, and code are available at https://mbaradad.github.io/learning_with_noise.

arXiv:2109.04994공식 코드 · Papers with Code

Topic-Aware Contrastive Learning for Abstractive Dialogue Summarization

Unlike well-structured text, such as news reports and encyclopedia articles, dialogue content often comes from two or more interlocutors, exchanging information with each other. In such a scenario, the topic of a conversation can vary upon progression and the key information for a certain topic is often scattered across multiple utterances of different speakers, which poses challenges to abstractly summarize dialogues. To capture the various topic information of a conversation and outline salient facts for the captured topics, this work proposes two topic-aware contrastive learning objectives, namely coherence detection and sub-summary generation objectives, which are expected to implicitly model the topic change and handle information scattering challenges for the dialogue summarization task. The proposed contrastive objectives are framed as auxiliary tasks for the primary dialogue summarization task, united via an alternative parameter updating strategy. Extensive experiments on benchmark datasets demonstrate that the proposed simple method significantly outperforms strong baselines and achieves new state-of-the-art performance. The code and trained models are publicly available via \href{https://github.com/Junpliu/ConDigSum}{https://github.com/Junpliu/ConDigSum}.

arXiv:2209.10320공식 코드 · Papers with Code

Continual VQA for Disaster Response Systems

Visual Question Answering (VQA) is a multi-modal task that involves answering questions from an input image, semantically understanding the contents of the image and answering it in natural language. Using VQA for disaster management is an important line of research due to the scope of problems that are answered by the VQA system. However, the main challenge is the delay caused by the generation of labels in the assessment of the affected areas. To tackle this, we deployed pre-trained CLIP model, which is trained on visual-image pairs. however, we empirically see that the model has poor zero-shot performance. Thus, we instead use pre-trained embeddings of text and image from this model for our supervised training and surpass previous state-of-the-art results on the FloodNet dataset. We expand this to a continual setting, which is a more real-life scenario. We tackle the problem of catastrophic forgetting using various experience replay methods. Our training runs are available at: https://wandb.ai/compyle/continual_vqa_final. Our code is available at https://github.com/AdityaKane2001/continual_vqa.

arXiv:2210.12209공식 코드 · Papers with Code

Motion Policy Networks

Collision-free motion generation in unknown environments is a core building block for robot manipulation. Generating such motions is challenging due to multiple objectives; not only should the solutions be optimal, the motion generator itself must be fast enough for real-time performance and reliable enough for practical deployment. A wide variety of methods have been proposed ranging from local controllers to global planners, often being combined to offset their shortcomings. We present an end-to-end neural model called Motion Policy Networks (M$\pi$Nets) to generate collision-free, smooth motion from just a single depth camera observation. M$\pi$Nets are trained on over 3 million motion planning problems in over 500,000 environments. Our experiments show that M$\pi$Nets are significantly faster than global planners while exhibiting the reactivity needed to deal with dynamic scenes. They are 46% better than prior neural planners and more robust than local control policies. Despite being only trained in simulation, M$\pi$Nets transfer well to the real robot with noisy partial point clouds. Code and data are publicly available at https://mpinets.github.io.

arXiv:2304.13742공식 코드 · Papers with Code

TR0N: Translator Networks for 0-Shot Plug-and-Play Conditional Generation

We propose TR0N, a highly general framework to turn pre-trained unconditional generative models, such as GANs and VAEs, into conditional models. The conditioning can be highly arbitrary, and requires only a pre-trained auxiliary model. For example, we show how to turn unconditional models into class-conditional ones with the help of a classifier, and also into text-to-image models by leveraging CLIP. TR0N learns a lightweight stochastic mapping which "translates" between the space of conditions and the latent space of the generative model, in such a way that the generated latent corresponds to a data sample satisfying the desired condition. The translated latent samples are then further improved upon through Langevin dynamics, enabling us to obtain higher-quality data samples. TR0N requires no training data nor fine-tuning, yet can achieve a zero-shot FID of 10.9 on MS-COCO, outperforming competing alternatives not only on this metric, but also in sampling speed -- all while retaining a much higher level of generality. Our code is available at https://github.com/layer6ai-labs/tr0n.