PyTorchKR Papers

“Code Generation”

총 10,163편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2610.35218v2cs.AI

MuSID-KD: Multigranularity Knowledge Distillation for Generative Recommendation with Semantic IDs

Generative recommendation with semantic IDs (SIDs) represents each item as a sequence of discrete tokens and autoregressively predicts the next item's SID. Despite strong recommendation performance, repeated decoding over multiple SID positions incurs substantial inference cost. Knowledge distillation transfers knowledge from a powerful generative teacher to a lightweight student, yet conventional objectives do not fully account for multi-token SID generation. We identify two limitations: item-level distribution alignment may not adequately preserve the teacher's ranking of candidate groups sharing SID prefixes, while uniform alignment across SID positions may affect final item ranking differently. We propose MuSID-KD, which combines Hierarchical Candidate Distribution Alignment (HCDA) and Ranking Impact Guided Levelwise Distillation (RIGD). HCDA aggregates item probabilities according to shared SID prefixes and aligns candidate distributions at successive levels, from candidate groups to individual items. RIGD evaluates the reduction in pairwise ranking surrogate loss when one student score contribution is replaced with the teacher's contribution, and allocates supervision accordingly. Experiments on real-world recommendation datasets using three lightweight student architectures demonstrate improved recommendation accuracy and better alignment of prefix marginal distributions and pairwise item rankings, without modifying the student's online inference architecture. Code is available at https://anonymous.4open.science/r/m7k2-review-83f9/.

arXiv:2611.04356v1cs.CR

CloakFEM: Agentic FEM Code Generation for Red-Teaming Computational Resource Exhaustion

Finite element method (FEM) solvers are foundational to scientific and engineering simulation, yet their susceptibility to \emph{efficiency attacks} remains largely unexplored. An adversary can craft syntactically valid and runnable FEM programs that complete successfully while forcing excessive computational work, enabling computational denial-of-service (DoS) attacks. Finding such inputs is challenging because FEM programs occupy a vast, structured space constrained by numerical, geometric, and constitutive requirements. We introduce CloakFEM, a domain-guided LLM-agent framework for automatically generating runnable adversarial FEM programs that amplify solver computation. CloakFEM combines LLM-guided search with an FEM-specific intermediate representation and a deterministic validity gate that enforces runnable constraints. Across three FEM stacks, 17 problem classes, and 1,068 seeds, CloakFEM discovers adversarial inputs that substantially amplify solver work while remaining valid and solvable. In the most extreme case, solver iterations increase from 43 to 247,367, a \(5{,}752.72\times\) amplification. CloakFEM outperforms five generic black-box baselines and retains an advantage over evolutionary search under a controlled bounded action space. Cross-solver analysis further reveals both problem-intrinsic and solver-specific sources of vulnerability. Our results uncover an underexplored efficiency-attack surface in FEM solvers and show that runnable FEM programs can serve as adversarial inputs.

arXiv:2602.09870v4cs.CL

Steer2Edit: From Activation Steering to Component-Level Editing

Steering methods influence Large Language Model behavior by identifying semantic directions in hidden representations, but are typically realized through inference-time activation interventions that apply a fixed, global modification to the model's internal states. While effective, such interventions often induce unfavorable attribute-utility trade-offs under strong control, as they ignore the fact that many behaviors are governed by a small and heterogeneous subset of model components. We propose Steer2Edit, a theoretically grounded, training-free framework that transforms steering vectors from inference-time control signals into diagnostic signals for component-level rank-1 weight editing. Instead of uniformly injecting a steering direction during generation, Steer2Edit selectively redistributes behavioral influence across individual attention heads and MLP neurons, yielding interpretable edits that preserve the standard forward pass and remain compatible with optimized parallel inference. Across safety alignment, hallucination mitigation, and reasoning efficiency, Steer2Edit consistently achieves more favorable attribute-utility trade-offs: at matched downstream performance, it improves safety by up to 17.2%, increases truthfulness by 9.8%, and reduces reasoning length by 12.2% on average. Overall, Steer2Edit provides a principled bridge between representation steering and weight editing by translating steering signals into interpretable, training-free parameter updates. Our code is available at https://github.com/Trustworthy-ML-Lab/Steer2Edit

arXiv:2606.32012v2cs.LG

CoMet: 멀티모달 불확실성 추정을 위한 컨텍스트 및 다중성 분해

CoMet: Context and Multiplicity Decomposition for Multimodal Uncertainty Estimation

불확실성 추정(uncertainty estimation)은 AI 모델에서 오랜 난제였다. 이는 '모르는 것을 아는' 것에 해당하며, 메타인지(metacognition)는 인간에게도 악명 높을 정도로 어렵다(더닝-크루거 효과 참조). 단순한 분류(classification) 시스템에서조차 아직 완전히 해결되지 않았지만, 멀티모달 대규모 언어 모델(MLLM)에서 이를 다루는 것이 점점 더 중요해지고 있다. MLLM 내에서 불확실성은 다양한 소스 각각 및 그 상호관계로부터 비롯될 수 있으며, 나아가 개방형(open-ended) 설정에서의 무한한 답변으로부터도 비롯될 수 있다. 이러한 문제를 해결하기 위해, 우리는 불확실성을 컨텍스트 특정 항(context-specific term)과 다중성 특정 항(multiplicity-specific term)으로 분해하는 MLLM 불확실성 추정 방법인 CoMet을 제안한다. 전자는 주어진 컨텍스트(예: 태스크 또는 프롬프트)에 의해 유발되는 모호성을 포착하며, 후자는 컨텍스트에 의해 결정된 그럴듯한 답변들 중 주어진 입력과 양립 가능한 답변의 수를 포착한다. 우리는 이러한 양을 추정하기 위한 경량 사후(post-hoc) 불확실성 모듈을 학습시키며, 이를 통해 자동회귀적 답변 생성이나 반복 샘플링 없이 효율적인 불확실성 추정이 가능하다. 다양한 개방형 멀티모달 벤치마크, 환각(hallucination) 탐지, 다지선다형 시각적 질의응답(VQA) 벤치마크에 대한 실험에서 CoMet은 기존 기준선 대비 불확실성 추정을 지속적으로 개선하면서도 실제 사용에서 효율성을 유지함을 보인다. 코드는 https://github.com/princetonvisualai/comet_uncertainty 에서 확인할 수 있다.

arXiv:2606.31875v1cs.CV

SENSE-VAD: Sentient and Semantic Video Anomaly Detection for Autonomous Driving

Autonomous vehicles (AVs) must navigate not only motion-based hazards but also socially complex situations whose danger is constituted by inter-agent relationships rather than movement statistics alone. A child running away from a guardian, a person being carried by another, or a pursuer chasing a pedestrian across a sidewalk are all anomalous in social context, yet none produces an obvious motion signal that current anomaly detectors are equipped to flag. We introduce SENSE-VAD, the first synthetic video anomaly detection benchmark for autonomous driving explicitly designed around socially complex anomalies. Using the CARLA simulator and Unreal Engine (UE), we generate distinct anomaly scenarios across multiple categories: individual behaviors, group behaviors, person--object interactions, cyclist interactions, vehicle & agent, each annotated with per-frame binary labels. A key design principle is the separation of social anomaly from motion-based or appearance-based anomaly: many scenarios involve motion of objects that appears unremarkable in isolation but is anomalous in relational context. We additionally provide real-world normal and anomalous videos as a sim-to-real transfer probe. We evaluate state-of-the-art video anomaly detection baselines and demonstrate that socially complex anomalies constitute a distinct and currently unsolved challenge. Our dataset, annotations, and generation code are publicly available.

arXiv:2607.00293v1cs.CV

Rosetta: 조합 가능한 네이티브 멀티모달 사전학습

Rosetta: Composable Native Multimodal Pretraining

새로운 모달리티를 통합할 때 발생하는 치명적 망각(catastrophic forgetting) 문제를 해결하기 위해, 전역 공유 전문가와 플러그 앤 플레이 모달리티별 전문가를 결합한 조합 가능한 사전학습 프레임워크 Rosetta를 제안한다. 핵심 기법인 MAOP(Momentum-Anchored Orthogonal Projection)는 옵티마이저의 모멘텀 상태를 암묵적 의미 앵커로 활용하여 추가 메모리 오버헤드 없이 충돌하는 그래디언트를 선택적으로 중화함으로써, 표준 MoE·MoT 대비 기존 언어·시각 이해 능력을 보존하면서 우수한 이미지 생성 성능과 크로스 모달 시너지를 동시에 달성한다.

arXiv:2608.20690v1cs.CV

정체성 인식 인간-객체 상호작용 동작 캡셔닝

Identity-Aware Human-Object Interaction Motion Captioning

기존 인간-객체 상호작용(HOI) 동작 캡셔닝 방법은 주체를 "a person"이나 "someone" 같은 일반적 표현으로 지칭하며 무엇이 일어나는지만 기술하는 경우가 많고, 캡션을 주체 정체성에 그라운딩(grounding)하지 않는다. 이 한계를 해결하기 위해 정체성 인식 인간-객체 상호작용 동작 캡셔닝(Identity-Aware Human-Object Interaction Motion Captioning) 과제를 제안한다. 이 과제는 생성되는 각 캡션이 주체 정체성과 해당 HOI 동작을 모두 명시하도록 요구한다. 예를 들어 모델은 "A person lifts the chair" 대신 "Sub_ID lifts the chair"를 생성한다. 이 과제를 위해 BEHAVE 및 InterCap 데이터셋을 기반으로 정체성 인식 HOI 동작 캡션을 설계한다. 또한 다중 시점(multi-view) 영상으로 학습하면서 단일 시점(single-view) 정체성 인식 HOI 동작 캡션 생성을 지원하는 ID-HOINet을 제안한다. ID-HOINet은 다중 시점 정체성-동작 학습 모듈(MVIML)과 2단계 캡션 재작성 전략(TSCR)이라는 두 핵심 구성 요소를 포함한다. MVIML은 시간 단계와 카메라 시점 간 의존성을 모델링하여 다중 시점 영상으로부터 학습하고, 정체성 및 상호작용 동작 특징을 포착한다. 추론 시 TSCR은 먼저 주체 정체성을 검색하고 정체성 무관(identity-agnostic) HOI 동작 캡션을 생성한 뒤, 예측된 정체성으로 해당 캡션을 재작성하여 최종 정체성 인식 HOI 동작 캡션을 만든다. 실험 결과 ID-HOINet이 최신(state-of-the-art) 성능을 달성함을 보여준다. 코드는 논문 채택 후 공개될 예정이다.

arXiv:2603.23149v2cs.AI

Describe-Then-Act: Proactive Agent Steering via Distilled Language-Action World Models

Deploying safety-critical agents requires anticipating the consequences of actions before they are executed. While world models offer a paradigm for this proactive foresight, current approaches relying on visual simulation incur prohibitive latencies, often exceeding several seconds per step. In this work, we challenge the assumption that visual processing is necessary for failure prevention. We show that a trained policy's latent state, combined with its planned actions, already encodes sufficient information to anticipate action outcomes, making visual simulation redundant for failure prevention. To this end, we introduce DILLO (DIstiLLed Language-ActiOn World Model), a fast steering layer that shifts the paradigm from "simulate-then-act" to "describe-then-act." DILLO is trained via cross-modal distillation, where a privileged Vision Language Model teacher annotates offline trajectories and a latent-conditioned Large Language Model student learns to predict semantic outcomes. This creates a text-only inference path, bypassing heavy visual generation entirely, achieving a 14x speedup over baselines. Experiments on MetaWorld and LIBERO demonstrate that DILLO produces high-fidelity descriptions of the next state and is able to steer the policy, improving episode success rate by up to 15 pp and 9.3 pp on average across tasks. Code is available at github.com/MaxPappa/DILLO.

arXiv:2609.00355v3cs.AI

비전은 오버헤드가 아니다: 비전-언어 모델에서의 손실 없는 추측 디코딩을 위한 원패스 블록 드래프팅

Vision Is Not Overhead: One-Pass Block Drafting for Lossless Speculative Decoding in Vision-Language Models

추측 디코딩(speculative decoding)은 출력을 바꾸지 않으면서 생성을 가속화하지만, 비전-언어 모델(VLM)에서는 자기강화적 순환 구조가 이를 가로막는다. 자기회귀(autoregressive) 드래프터는 드래프트한 토큰마다 순차적인 패스를 거쳐야 하므로 작은 크기를 유지해야 하며, 매 패스마다 이미지에 주의를 기울이기 어렵다. 이 때문에 기존 연구들은 이미지를 압축하거나 숨기는 방식을 택했고, 그 결과 드래프터는 이미지가 결정하는 텍스트에 대해 가장 취약해졌다. 우리는 수정되지 않은 VLM 타겟에서 이러한 순환을 깨는 원패스 블록 드래프터인 GLANCE를 제시한다. GLANCE의 블록-디퓨전(block-diffusion) 헤드는 타겟의 이미 융합된 비전-언어 상태에 대해 단 한 번의 순전파(forward pass)로 블록 전체를 드래프트하며, 드래프트가 얼마나 깊든 멀티모달 컨텍스트를 단 한 번만 읽는다. 타겟은 넓은 후보 트리를 한 번의 패스로 검증하고 자신의 그리디(greedy) 출력을 정확히 커밋한다. 고정된 라운드 예산을 사용하는 하나의 프로덕션 엔진에서, GLANCE는 자기회귀 디코딩보다 최대 3.05배 빠르게 디코딩하며, 평균적으로, 그리고 그라운딩(grounded) 작업에서는 약 11% 더 높은 성능으로 프로덕션 EAGLE3-VL 헤드를 앞선다. 엔트로피 법칙은 드래프팅이 언제 유리한지를 설명하며, 타겟의 다음 토큰 엔트로피가 가장 낮은 그라운딩 작업에서 가장 긴 수용 블록을 예측한다. 코드는 https://github.com/js-lee-AI/GLANCE 에서 확인할 수 있다.

arXiv:2609.01971v1cs.CL

NS-Copilot: 자율 신경과학 분석을 위한 LLM 기반 에이전트 시스템

NS-Copilot: An LLM-Driven Agent System for Autonomous Neuroscience Analysis

AI는 신경과학을 빠르게 발전시키고 있지만, 상당한 학제 간 장벽으로 인해 많은 연구실이 그 잠재력을 충분히 발휘하지 못하고 있다. 생리학적 데이터용 사전학습 신경 모델은 빠르게 진전하고 있으나, 이질적인 아키텍처와 모달리티별 제약이 체계적인 통합, 선택, 평가를 방해한다. 지능형 과학 응용을 위한 대규모 언어 모델(LLM) 기반 에이전트 시스템의 최근 발전에도 불구하고, 기존 접근법은 여전히 다양한 신경과학 사전학습 모델을 효과적으로 선택·조율하고 이 영역의 고유 데이터 유형을 다루는 데 필요한 도메인 전문성이 부족한 경우가 많다. 우리는 다양한 전문 작업에 대한 end-to-end 워크플로를 자율적으로 지원하는 신경과학 분석용 LLM 기반 멀티 에이전트 시스템 NS-Copilot을 제안한다. NS-Copilot은 도메인 특화 사전학습 모델을 통합하고, 자연어 인터페이스를 통해 EEG 및 세포외 스파이크(extracellular spike) 데이터를 포함한 핵심 신경과학 모달리티를 지원한다. 원시 데이터와 작업 설명이 주어지면 NS-Copilot은 계획, 적응형 제어, 코드 생성, 결과 합성을 담당하는 전문화된 역할의 에이전트들을 조율하여 데이터셋별 휴리스틱 없이 분석을 가능하게 한다. 우리는 알�heimer병, 파킨슨병, 작업 기억 스파이크 디코딩을 아우르는 신경과학 벤치마크에서 NS-Copilot을 평가했다. 작업당 8회 시행에서 시스템은 주요 지표에서 강력한 베이스라인을 꾸준히 능가했으며, 효과적이고 확장 가능한 신경과학 분석을 위한 NS-Copilot의 역량을 보여주었다.

arXiv:2604.02881v2cs.CL

하나의 모델로 모두를 번역할 수 있을까? 다국어 모델 병합을 향한 마운트 둠으로의 여정

One Model to Translate Them All? A Journey to Mount Doom for Multilingual Model Merging

가중치 공간 모델 병합(weight-space model merging)은 원본 학습 데이터에 접근하지 않고 독립적으로 미세조정된 체크포인트들을 결합한다. 병합은 멀티태스크 설정에서 유망한 결과를 보여 왔지만, 다국어 생성 시스템에서의 거동은 아직 충분히 탐구되지 않았다. 본 연구는 대규모 이중언어 코퍼스에서 언어 모델을 완전 미세조정하고, 공유 원어(shared-source), 공유 번역어(shared-target), 양방향 통합(bidirectional consolidation) 설정 전반에 대표적인 병합 전략을 평가함으로써 다국어 기계 번역에 대한 가중치 공간 병합을 체계적으로 조사한다. 실험 결과 강한 방향 비대칭이 드러난다. 모델들이 번역어를 공유할 때 병합이 상대적으로 더 효과적이며 기본 모델 대비 다국어 커버리지를 개선하지만, 언어별 체크포인트의 최고 성능은 여전히 보존하지 못한다. 반면 번역어가 다를 때, 특히 공유 원어 및 양방향 설정에서 성능이 급격히 저하된다. 이를 설명하기 위해 내부 표현을 분석한 결과, 미세조정이 언어별로 분리된 하위 네트워크를 만들지 않는다. 대신 독립적으로 미세조정된 모델들은 대체로 겹치는 뉴런을 활성화하면서, 상위 층의 번역어 생성 표현을 서로 호환되지 않는 기하 구조로 재형성한다. 이러한 발견은 다국어 병합 실패가 공유 연산 단위 내 번역어 측 기하학적 불일치에서 기인함을 시사하며, 다국어 번역을 위한 표준 가중치 공간 병합의 전제에 도전한다. 코드는 https://github.com/babangain/mt-model-merging 에서 공개한다.

arXiv:1905.13143공식 코드 · Papers with Code

Semantics-Aligned Representation Learning for Person Re-identification

Person re-identification (reID) aims to match person images to retrieve the ones with the same identity. This is a challenging task, as the images to be matched are generally semantically misaligned due to the diversity of human poses and capture viewpoints, incompleteness of the visible bodies (due to occlusion), etc. In this paper, we propose a framework that drives the reID network to learn semantics-aligned feature representation through delicate supervision designs. Specifically, we build a Semantics Aligning Network (SAN) which consists of a base network as encoder (SA-Enc) for re-ID, and a decoder (SA-Dec) for reconstructing/regressing the densely semantics aligned full texture image. We jointly train the SAN under the supervisions of person re-identification and aligned texture generation. Moreover, at the decoder, besides the reconstruction loss, we add Triplet ReID constraints over the feature maps as the perceptual losses. The decoder is discarded in the inference and thus our scheme is computationally efficient. Ablation studies demonstrate the effectiveness of our design. We achieve the state-of-the-art performances on the benchmark datasets CUHK03, Market1501, MSMT17, and the partial person reID dataset Partial REID. Code for our proposed method is available at: https://github.com/microsoft/Semantics-Aligned-Representation-Learning-for-Person-Re-identification.

arXiv:2004.12299공식 코드 · Papers with Code

Dual Learning for Semi-Supervised Natural Language Understanding

Natural language understanding (NLU) converts sentences into structured semantic forms. The paucity of annotated training samples is still a fundamental challenge of NLU. To solve this data sparsity problem, previous work based on semi-supervised learning mainly focuses on exploiting unlabeled sentences. In this work, we introduce a dual task of NLU, semantic-to-sentence generation (SSG), and propose a new framework for semi-supervised NLU with the corresponding dual model. The framework is composed of dual pseudo-labeling and dual learning method, which enables an NLU model to make full use of data (labeled and unlabeled) through a closed-loop of the primal and dual tasks. By incorporating the dual task, the framework can exploit pure semantic forms as well as unlabeled sentences, and further improve the NLU and SSG models iteratively in the closed-loop. The proposed approaches are evaluated on two public datasets (ATIS and SNIPS). Experiments in the semi-supervised setting show that our methods can outperform various baselines significantly, and extensive ablation studies are conducted to verify the effectiveness of our framework. Finally, our method can also achieve the state-of-the-art performance on the two datasets in the supervised setting. Our code is available at \url{https://github.com/rhythmcao/slu-dual-learning.git}.

arXiv:2010.12269공식 코드 · Papers with Code

Reducing Bias in Modeling Real-world Password Strength via Deep Learning and Dynamic Dictionaries

Password security hinges on an in-depth understanding of the techniques adopted by attackers. Unfortunately, real-world adversaries resort to pragmatic guessing strategies such as dictionary attacks that are inherently difficult to model in password security studies. In order to be representative of the actual threat, dictionary attacks must be thoughtfully configured and tuned. However, this process requires a domain-knowledge and expertise that cannot be easily replicated. The consequence of inaccurately calibrating dictionary attacks is the unreliability of password security analyses, impaired by a severe measurement bias. In the present work, we introduce a new generation of dictionary attacks that is consistently more resilient to inadequate configurations. Requiring no supervision or domain-knowledge, this technique automatically approximates the advanced guessing strategies adopted by real-world attackers. To achieve this: (1) We use deep neural networks to model the proficiency of adversaries in building attack configurations. (2) Then, we introduce dynamic guessing strategies within dictionary attacks. These mimic experts' ability to adapt their guessing strategies on the fly by incorporating knowledge on their targets. Our techniques enable more robust and sound password strength estimates within dictionary attacks, eventually reducing overestimation in modeling real-world threats in password security. Code available: https://github.com/TheAdamProject/adams

arXiv:2103.10764

DFS: A Diverse Feature Synthesis Model for Generalized Zero-Shot Learning

Generative based strategy has shown great potential in the Generalized Zero-Shot Learning task. However, it suffers severe generalization problem due to lacking of feature diversity for unseen classes to train a good classifier. In this paper, we propose to enhance the generalizability of GZSL models via improving feature diversity of unseen classes. For this purpose, we present a novel Diverse Feature Synthesis (DFS) model. Different from prior works that solely utilize semantic knowledge in the generation process, DFS leverages visual knowledge with semantic one in a unified way, thus deriving class-specific diverse feature samples and leading to robust classifier for recognizing both seen and unseen classes in the testing phase. To simplify the learning, DFS represents visual and semantic knowledge in the aligned space, making it able to produce good feature samples with a low-complexity implementation. Accordingly, DFS is composed of two consecutive generators: an aligned feature generator, transferring semantic and visual representations into aligned features; a synthesized feature generator, producing diverse feature samples of unseen classes in the aligned space. We conduct comprehensive experiments to verify the efficacy of DFS. Results demonstrate its effectiveness to generate diverse features for unseen classes, leading to superior performance on multiple benchmarks. Code will be released upon acceptance.

arXiv:2107.13170공식 코드 · Papers with Code

Accurate Grid Keypoint Learning for Efficient Video Prediction

Video prediction methods generally consume substantial computing resources in training and deployment, among which keypoint-based approaches show promising improvement in efficiency by simplifying dense image prediction to light keypoint prediction. However, keypoint locations are often modeled only as continuous coordinates, so noise from semantically insignificant deviations in videos easily disrupt learning stability, leading to inaccurate keypoint modeling. In this paper, we design a new grid keypoint learning framework, aiming at a robust and explainable intermediate keypoint representation for long-term efficient video prediction. We have two major technical contributions. First, we detect keypoints by jumping among candidate locations in our raised grid space and formulate a condensation loss to encourage meaningful keypoints with strong representative capability. Second, we introduce a 2D binary map to represent the detected grid keypoints and then suggest propagating keypoint locations with stochasticity by selecting entries in the discrete grid space, thus preserving the spatial structure of keypoints in the longterm horizon for better future frame generation. Extensive experiments verify that our method outperforms the state-ofthe-art stochastic video prediction methods while saves more than 98% of computing resources. We also demonstrate our method on a robotic-assisted surgery dataset with promising results. Our code is available at https://github.com/xjgaocs/Grid-Keypoint-Learning.

arXiv:2110.00375공식 코드 · Papers with Code

Fully Spiking Variational Autoencoder

Spiking neural networks (SNNs) can be run on neuromorphic devices with ultra-high speed and ultra-low energy consumption because of their binary and event-driven nature. Therefore, SNNs are expected to have various applications, including as generative models being running on edge devices to create high-quality images. In this study, we build a variational autoencoder (VAE) with SNN to enable image generation. VAE is known for its stability among generative models; recently, its quality advanced. In vanilla VAE, the latent space is represented as a normal distribution, and floating-point calculations are required in sampling. However, this is not possible in SNNs because all features must be binary time series data. Therefore, we constructed the latent space with an autoregressive SNN model, and randomly selected samples from its output to sample the latent variables. This allows the latent variables to follow the Bernoulli process and allows variational learning. Thus, we build the Fully Spiking Variational Autoencoder where all modules are constructed with SNN. To the best of our knowledge, we are the first to build a VAE only with SNN layers. We experimented with several datasets, and confirmed that it can generate images with the same or better quality compared to conventional ANNs. The code is available at https://github.com/kamata1729/FullySpikingVAE

arXiv:2211.02580공식 코드 · Papers with Code

Evaluating and Improving Factuality in Multimodal Abstractive Summarization

Current metrics for evaluating factuality for abstractive document summarization have achieved high correlations with human judgment, but they do not account for the vision modality and thus are not adequate for vision-and-language summarization. We propose CLIPBERTScore, a simple weighted combination of CLIPScore and BERTScore to leverage the robustness and strong factuality detection performance between image-summary and document-summary, respectively. Next, due to the lack of meta-evaluation benchmarks to evaluate the quality of multimodal factuality metrics, we collect human judgments of factuality with respect to documents and images. We show that this simple combination of two metrics in the zero-shot setting achieves higher correlations than existing factuality metrics for document summarization, outperforms an existing multimodal summarization metric, and performs competitively with strong multimodal factuality metrics specifically fine-tuned for the task. Our thorough analysis demonstrates the robustness and high correlation of CLIPBERTScore and its components on four factuality metric-evaluation benchmarks. Finally, we demonstrate two practical downstream applications of our CLIPBERTScore metric: for selecting important images to focus on during training, and as a reward for reinforcement learning to improve factuality of multimodal summary generation w.r.t automatic and human evaluation. Our data and code are publicly available at https://github.com/meetdavidwan/faithful-multimodal-summ

arXiv:2304.12432공식 코드 · Papers with Code

Generative Adversarial Neuroevolution for Control Behaviour Imitation

There is a recent surge in interest for imitation learning, with large human video-game and robotic manipulation datasets being used to train agents on very complex tasks. While deep neuroevolution has recently been shown to match the performance of gradient-based techniques on various reinforcement learning problems, the application of deep neuroevolution techniques to imitation learning remains relatively unexplored. In this work, we propose to explore whether deep neuroevolution can be used for behaviour imitation on popular simulation environments. We introduce a simple co-evolutionary adversarial generation framework, and evaluate its capabilities by evolving standard deep recurrent networks to imitate state-of-the-art pre-trained agents on 8 OpenAI Gym state-based control tasks. Across all tasks, we find the final elite actor agents capable of achieving scores as high as those obtained by the pre-trained agents, all the while closely following their score trajectories. Our results suggest that neuroevolution could be a valuable addition to deep learning techniques to produce accurate emulation of behavioural agents. We believe that the generality and simplicity of our approach opens avenues for imitating increasingly complex behaviours in increasingly complex settings, e.g. human behaviour in real-world settings. We provide our source code, model checkpoints and results at github.com/MaximilienLC/gane.

arXiv:2308.02205공식 코드 · Papers with Code

GEMRec: Towards Generative Model Recommendation

Recommender Systems are built to retrieve relevant items to satisfy users' information needs. The candidate corpus usually consists of a finite set of items that are ready to be served, such as videos, products, or articles. With recent advances in Generative AI such as GPT and Diffusion models, a new form of recommendation task is yet to be explored where items are to be created by generative models with personalized prompts. Taking image generation as an example, with a single prompt from the user and access to a generative model, it is possible to generate hundreds of new images in a few minutes. How shall we attain personalization in the presence of "infinite" items? In this preliminary study, we propose a two-stage framework, namely Prompt-Model Retrieval and Generated Item Ranking, to approach this new task formulation. We release GEMRec-18K, a prompt-model interaction dataset with 18K images generated by 200 publicly-available generative models paired with a diverse set of 90 textual prompts. Our findings demonstrate the promise of generative model recommendation as a novel personalization problem and the limitations of existing evaluation metrics. We highlight future directions for the RecSys community to advance towards generative recommender systems. Our code and dataset are available at https://github.com/MAPS-research/GEMRec.