Modern web services rely heavily on REST APIs, typically documented using the OpenAPI specification. The widespread adoption of this standard has resulted in the development of many black-box testing tools that generate tests based on OpenAPI specifications. Although Large Language Models (LLMs) have shown promising test-generation abilities, their application to REST API testing remains mostly unexplored. We present LlamaRestTest, a novel approach that employs two custom LLMs-created by fine-tuning and quantizing the Llama3-8B model using mined datasets of REST API example values and inter-parameter dependencies-to generate realistic test inputs and uncover inter-parameter dependencies during the testing process by analyzing server responses. We evaluated LlamaRestTest on 12 real-world services (including popular services such as Spotify), comparing it against RESTGPT, a GPT-powered specification-enhancement tool, as well as several state-of-the-art REST API testing tools, including RESTler, MoRest, EvoMaster, and ARAT-RL. Our results demonstrate that fine-tuning enables smaller models to outperform much larger models in detecting actionable parameter-dependency rules and generating valid inputs for REST API testing. We also evaluated different tool configurations, ranging from the base Llama3-8B model to fine-tuned versions, and explored multiple quantization techniques, including 2-bit, 4-bit, and 8-bit integer formats. Our study shows that small language models can perform as well as, or better than, large language models in REST API testing, balancing effectiveness and efficiency. Furthermore, LlamaRestTest outperforms state-of-the-art REST API testing tools in code coverage achieved and internal server errors identified, even when those tools use RESTGPT-enhanced specifications.
Tip-of-the-tongue (TOT) search occurs when a user struggles to recall a specific identifier, such as a document title. While common, existing search systems often fail to effectively support TOT scenarios. Research on TOT retrieval is further constrained by the challenge of collecting queries, as current approaches rely heavily on community question-answering (CQA) websites, leading to labor-intensive evaluation and domain bias. To overcome these limitations, we introduce two methods for eliciting TOT queries - leveraging large language models (LLMs) and human participants - to facilitate simulated evaluations of TOT retrieval systems. Our LLM-based TOT user simulator generates synthetic TOT queries at scale, achieving high correlations with how CQA-based TOT queries rank TOT retrieval systems when tested in the Movie domain. Additionally, these synthetic queries exhibit high linguistic similarity to CQA-derived queries. For human-elicited queries, we developed an interface that uses visual stimuli to place participants in a TOT state, enabling the collection of natural queries. In the Movie domain, system rank correlation and linguistic similarity analyses confirm that human-elicited queries are both effective and closely resemble CQA-based queries. These approaches reduce reliance on CQA-based data collection while expanding coverage to underrepresented domains, such as Landmark and Person. LLM-elicited queries for the Movie, Landmark, and Person domains have been released as test queries in the TREC 2024 TOT track, with human-elicited queries scheduled for inclusion in the TREC 2025 TOT track. Additionally, we provide source code for synthetic query generation and the human query collection interface, along with curated visual stimuli used for eliciting TOT queries.
We propose FairyGen, an automatic system for generating story-driven cartoon videos from a single child's drawing, while faithfully preserving its unique artistic style. Unlike previous storytelling methods that primarily focus on character consistency and basic motion, FairyGen explicitly disentangles character modeling from stylized background generation and incorporates cinematic shot design to support expressive and coherent storytelling. Given a single character sketch, we first employ an MLLM to generate a structured storyboard with shot-level descriptions that specify environment settings, character actions, and camera perspectives. To ensure visual consistency, we introduce a style propagation adapter that captures the character's visual style and applies it to the background, faithfully retaining the character's full visual identity while synthesizing style-consistent scenes. A shot design module further enhances visual diversity and cinematic quality through frame cropping and multi-view synthesis based on the storyboard. To animate the story, we reconstruct a 3D proxy of the character to derive physically plausible motion sequences, which are then used to fine-tune an MMDiT-based image-to-video diffusion model. We further propose a two-stage motion customization adapter: the first stage learns appearance features from temporally unordered frames, disentangling identity from motion; the second stage models temporal dynamics using a timestep-shift strategy with frozen identity weights. Once trained, FairyGen directly renders diverse and coherent video scenes aligned with the storyboard. Extensive experiments demonstrate that our system produces animations that are stylistically faithful, narratively structured natural motion, highlighting its potential for personalized and engaging story animation. The code will be available at https://github.com/GVCLab/FairyGen
뇌전도(EEG)는 비침습적이며 시간 해상도가 높아 뇌-컴퓨터 인터페이스 응용을 위한 뇌 활동 측정에 널리 사용된다. 그러나 시각적 의미에 대한 신경 반응 모델링의 진전은, 세밀한 시각적 특징의 변화와 그것이 지각과 맺는 관계를 분리하는 데 필요한 정밀한 통제가 부족한 고정된 이미지 모음에 의존함으로써 제한되어 왔다. 본 연구에서는 고품질 EEG와 생성 모델로 합성한 사실적인 얼굴 이미지를 짝지은 대규모 데이터셋 EEGFaceSem을 제시한다. 각 자극에 대한 생성 잠재 벡터를 제공하여 시각적 잠재 공간과 신경 표상 사이의 연속적 사상에 관한 연구를 가능하게 한다. 각 참가자의 뇌 반응은 빠른 순차 시각 제시 패러다임에서 특정 의미적 특징(예: 웃음, 머리 색)에 주의를 기울이는 동안 기록되었다. 이 데이터셋은 30명의 참가자로부터 얻은 EEG 신호를 포함하며, 시간 고정된 EEG 기록은 총 67,200개이다. 단일 피험자, 교차 피험자, 피험자 적응 설정에서의 단일 시행 의미 분류 벤치마크와, 참가자 간 자극별 예측을 집계하는 교차 피험자 투표 벤치마크를 제시한다. 또한 잠재 공간에서 의미 방향을 도출하여 뇌 지도 기반 이미지 편집을 수행하는 응용과, 뇌에서 해독한 레이블로 생성 모델을 조건화하여 뇌 지도 기반 이미지 생성을 수행하는 응용이라는 두 가지 상호 보완적 응용을 통해 이 데이터셋의 고유한 유용성을 보인다. 본 데이터셋은 의미 수준의 시각적 현저성 검출과 뇌 지도 기반 생성 모델링을 위한 뇌-컴퓨터 인터페이스(BCI)의 발전을 위한 길을 연다. 데이터셋과 코드는 OSF(https://osf.io/2guk3/)와 HuggingFace(https://huggingface.co/datasets/yefllower/EEGFaceSem)에 공개되어 있다.
멀티모달 파운데이션 모델은 대규모 광학 벤치마크 덕분에 빠르게 발전했지만, 합성 개구 레이더(SAR)를 위한 유사한 자원은 여전히 제한적입니다. 기존 SAR-광학 데이터셋은 대부분 저해상도이며 강도(intensity)만 포함하는 지상 거리 탐지(GRD) 제품에 의존하며, 복소수(complex-valued) SAR 측정값이나 원래의 취득 기하학(geometry)을 보존하지 않아 물리적으로 기반한(grounded) 멀티모달 학습을 제한합니다. 특히, 초고해상도(VHR) SAR SLC, 정렬된 광학 이미지, 자연어 설명을 결합한 대규모 공개 데이터셋은 여전히 부족합니다. 우리는 센서 독립적 복소 데이터(SICD)로 배포된 공개 접근 가능한 Umbra 스포트라이트 취득 데이터로 구축된 VHR SAR-광학-텍스트 데이터셋을 제시합니다. 약 2,500개의 전 세계 장면(VV/HH, 20cm~2m 원본 해상도)에서, 대역 제한 FFT 리샘플링을 통해 모든 SAR 데이터를 80cm 경사 거리(slant-range) 그리드로 표준화하고 이미지를 1024x1024 타일로 분할합니다. 각 SAR 패치에 대해 고해상도 광학 타일을 검색하고, 국부 픽셀 수준 정렬을 위해 국부 좌표 대응 관계를 사용하여 SAR 그리드로 워핑(warp)합니다. 또한 비전-언어 학습 및 평가를 지원하기 위해 샘플당 세 가지 캡션 변형(SHORT/MID/LONG)을 생성합니다. 우리 데이터셋은 72개국 257개 위치와 다양한 토지 유형 및 인프라를 포괄하는 119,566개의 삼중쌍(triplets)(복소수 및 진폭 경사 거리 SAR 패치, 정렬된 광학 패치, 자연어 설명)을 포함합니다. 우리는 원래의 SAR 기하학에서 교차 모달 검색 및 조건부 생성에 대한 멀티모달 정렬의 재현 가능한 벤치마크를 가능하게 하기 위해 고정된 학습/검증/테스트 분할과 전체 전처리 및 기준선(baseline) 코드를 공개합니다. 이 데이터셋은 https://huggingface.co/datasets/ONERA/SARLO-80의 Hugging Face Hub에서 공개적으로 이용할 수 있습니다.
비전-언어 모델(VLM)은 엔드-투-엔드 자율주행(E2E-AD) 계획을 위한 강력한 의미론적 이해와 상식 추론을 제공한다. 그러나 VLM이 직접 생성한 궤적은 대략적인 주행 의도만을 인코딩하는 경우가 많아, 기하학적으로 정확하고 미래를 인식하며 다중 시점에 기반한 계획에는 충분하지 않다. 이러한 한계를 해결하기 위해 우리는 층별 월드 모델 기반 주행 프레임워크(LWDrive)를 개발한다. LWDrive는 층별 월드 모델 안내를 통해 거친 궤적을 정제하는 VLM 계획 프레임워크이다. VLM 출력을 최종 궤적으로 처리하는 대신, LWDrive는 이를 의도 인식 거친 계획으로 활용하고, 그 주변에 다양한 후보 공간을 확장하며, Foresight Cascade Planner(FCP)를 통해 후보들을 점진적으로 정제한다. 구체적으로, 우리는 VLM이 미래 지향적인 장면 표현을 학습하도록 유도하기 위해 미래 프레임 생성 감독을 도입함으로써, 계획과 관련된 예측 동역학을 내부 은닉 상태에 주입한다. 이러한 월드 모델 감독 표현을 기반으로, FCP는 여러 층에 걸친 VLM 특징을 활용하고, 과거 시간 상태, Action-Query 표현, 현재 프레임의 다중 시점 조감도(BEV) 특징을 통합하여 후보 궤적을 거친 것에서 세밀한 방식으로 정제한다. 이 설계는 공간적 위치와 동작 추세를 점진적으로 수정하는 동시에, 다중 시점 장면 단서로 궤적 정제를 그라운딩하고 대형 모델이 생성한 고수준 주행 의도를 보존한다. 마지막으로, 점수 헤드가 정제된 후보들을 평가하여 최상의 궤적을 최종 계획 출력으로 선택한다. 실험 결과, LWDrive는 NAVSIM 벤치마크에서 92.0점, NAVSIM-v2에서 89.6점을 달성한다. 코드와 모델은 공개될 예정이다.
최근 멀티모달 대규모 언어 모델(MLLM)의 발전은 멀티모달 감정 인식(MER) 성능을 크게 향상시켰으며, 비디오, 오디오, 언어 등을 공동으로 모델링하여 해석 가능한 설명 생성을 가능하게 했다. 그러나 이러한 성능 향상은 종종 모델 매개변수 크기의 증가(예: 최소 7B)를 동반하며, 이는 동시에 높은 계산 비용을 초래하고 추론 효율을 저하시켜 로봇 및 모바일 기기와 같은 자원 제약 플랫폼에서의 실시간 배포를 어렵게 한다. 이는 근본적인 질문을 던진다: 고품질 MER를 위해 1B 매개변수보다 큰 멀티모달 MER 모델이 정말 필요한가? 본 논문에서는 더 큰 모델이 본질적으로 필요하다는 가정에 도전하고, 지식 증류를 통해 더 우수하고 빠른 멀티모달 감성 이해 및 인식을 달성하는 경량 MER 프레임워크(Light-MER)를 제안한다. 강력한 대규모 교사 모델에서 경량의 10억 미만 매개변수 학생 모델로 지식을 전달하여, 풍부한 멀티모달 감정 추론 및 인식을 유지하면서 배포 효율을 대폭 향상하는 것을 목표로 한다. 구체적으로, 지식 전달을 강화하기 위한 두 가지 새로운 최적화 전략을 도입한다: (1) Sliced Wasserstein Distance와 은닉 상태 정렬(hidden-state alignment)을 결합한 새로운 최적 전송(optimal transport) 손실, (2) MER 성능과 효율의 균형을 맞추어 학생 모델의 학습 능력을 더욱 향상시키는 GRPO 기반 새로운 다중 보상 최적화 전략. 9개 벤치마크 데이터셋에 대한 광범위한 실험은 Light-MER가 최첨단 성능을 달성하면서 추론 효율을 크게 향상시킴을 보여준다. 이는 향후 연구를 위한 소형 멀티모달 감정 언어 모델의 큰 잠재력을 부각한다. 코드는 https://github.com/GAIR-Lab/Light-MER 에서 이용 가능하다.
역변환(inversion) 기반 워터마킹은 확산(diffusion) 모델이 생성한 이미지를 인증하는 유망한 접근법이지만, 역변환 과정이 느리고 오류가 발생하기 쉽다는 점이 실용적 활용의 병목으로 작용합니다. 워터마킹 환경의 주된 과제는 외부 왜곡에 대한 강건성(robustness)이지만, 기존 방식들은 내부 절단 오차(truncation error)를 과도하게 최적화하며, 이 오차가 샘플러 스텝 크기에 비례하기 때문에 속도와 강건성이라는 두 가지 요구를 충족할 수 없는 고 NFE(함수 평가 횟수, number of function evaluations) 체제에 본질적으로 갇혀 있습니다. 본 연구에서는 두 가지 핵심 관찰을 도출했습니다. (i) 역변환 궤적은 순방향 생성 경로에 비해 곡률이 현저히 낮아 압축성이 높고 저 NFE 근사에 적합하며, (ii) 워터마크 검증을 위한 역변환에서는 외부 왜곡이 오차를 지배하므로 속도와 절단 오차 간의 트레이드오프가 덜 중요하다는 점입니다. 더 빠른 역변환기는 두 가지 이점을 제공합니다. 효율성이 높아질 뿐만 아니라, 기존의 긴 역변환 궤적에서는 계산 비용 문제로 불가능했던, 강건성을 직접 목표로 하는 엔드투엔드 적대적 학습(adversarial training)을 가능하게 합니다. 이를 바탕으로, 본 논문에서는 워터마크 추출을 위한 디노이저(denoiser)의 경량 적대적 LoRA 미세조정과 결합된 단일 단계 역변환 프레임워크인 FARI(Fast Asymmetric Robust Inversion)를 제안합니다. 통합 과정에서 내부 오차가 소폭 증가하지만, FARI는 속도와 강건성 모두에서 큰 향상을 이끌어냅니다. 단일 NVIDIA RTX A6000 GPU에서 약 20분의 미세조정만으로 추론 시간을 획기적으로 줄이는 동시에, 워터마크 검증 강건성 측면에서 50단계 DDIM 역변환을 능가합니다. 코드와 사전학습된 모델은 https://github.com/0xD009/FARI 에서 이용할 수 있습니다.
기업 환경에서 대규모 언어 모델(LLM)의 도입이 급증함에 따라 운영, 보안 및 거버넌스 위험이 대두되었습니다. 생성형 AI 애플리케이션이 파일럿 단계에서 프로덕션 환경으로 전환됨에 따라 위해 식별 및 완화를 수작업으로 확장하기가 점점 더 어려워지고 있습니다. 모델 평가, 적대적 테스팅(adversarial testing), 런타임 가드레일, 관측성(observability)을 지원하는 도구는 많지만, 도구 생태계는 여전히 파편화되어 있습니다. 이러한 도구들은 대개 특정 엔지니어링 작업을 위해 설계되었으며 거버넌스 프레임워크나 위험 분류체계와 일치하지 않는 기술 용어로 설명되므로, 어떤 도구가 어떤 위험을 해결하고 어디에 심각한 공백이 남아 있는지 파악하기 어렵습니다. 본 논문에서는 오픈소스 LLM 평가 및 보안 도구에 대한 분류체계 기반 분석을 통해 AI 위험 완화를 자동화하는 구조화된 프로토콜을 제안합니다. 저자들은 21개의 대표적인 오픈소스 도구의 기능을 확장된 MIT AI 위험 완화 및 대응 분류체계(MIT AI Risk Mitigation and Response Taxonomy)의 32개 하위 범주에 매핑합니다. LLM 지원 검색-증강 생성(RAG) 파이프라인이 소스 코드와 문서를 분석하여 각 분류 범주에 대한 기능을 추출합니다. 신뢰도 평가 결과 3명의 독립적인 검토자 간에 중간 수준의 일치도(Fleiss' Kappa = 0.509)를 보였습니다. 분석 결과, 도구들이 기술 및 운영 통제에 집중되어 있는 반면 거버넌스, 법률 및 규제, 금융 및 시장 통제는 거의 다루어지지 않는 매우 편중된 환경임이 드러났습니다. 이는 도구 기반 통제와 조직 및 규제 프로세스를 결합하는 계층화된 위험 완화 아키텍처의 필요성을 시사합니다. 매핑 프로토콜은 다수결 투표 후 75.5%의 F1 점수를 달성했습니다. 종합적으로 본 연구는 엔터프라이즈 AI 위험 범주와 오픈소스 완화 기능 간의 실질적인 매핑을 제공하고, 인간의 감독이 여전히 필요한 영역을 식별하며, 오픈소스 및 상용 솔루션 모두에 적용 가능한 분류체계 기반 프레임워크를 제시합니다.
전문가 혼합(MoE) 확산 언어 모델(DLM)은 유연한 병렬 디코딩과 향상된 모델 용량을 제공하지만, 방대한 수의 전문가 매개변수로 인해 상당한 계산 및 스토리지 비용이 발생합니다. 기존의 저랭크(low-rank) MoE 압축 방식은 주로 정적 인수분해와 고정된 랭크 할당에 의존하여 MoE DLM의 고유한 특성을 간과합니다. 구체적으로 본 논문에서는 두 가지 특성을 규명합니다. 첫째는 입력, 출력 및 전문가 모드 전반에 걸쳐 매개변수 중복성과 랭크 절단에 대한 민감도가 달라지는 '교차 모드 불균일 중복성'이며, 둘째는 핫(hot) 토큰과 콜드(cold) 토큰이 서로 다른 스펙트럼 특성과 전문가 활성화 패턴을 나타내는 '토큰별 활용 변동성'입니다. 이러한 문제를 해결하기 위해 본 논문에서는 MoE DLM을 위한 중요도 기반 토큰 인식 압축 프레임워크인 ITC-MoE를 제안합니다. ITC-MoE는 상호 보완적인 두 가지 구성 요소로 이루어져 있습니다. 첫째, 중요도 기반 적응형 터커 압축(IATC)은 활성화 및 그래디언트 중요도를 전문가 가중치 변환에 반영하고, 여러 모드에 걸쳐 전문가 가중치를 공동으로 인수분해하며, 고정된 매개변수 예산 내에서 적응적으로 랭크를 할당합니다. 둘째, 토큰 인식 보상 및 라우팅(TCR)은 압축에 민감한 핫 토큰에 경량 저랭크 보상을 적용하고, 집중된 라우팅 패턴을 보이는 콜드 토큰에 대해서는 후보 전문가 집합을 제한합니다. 매개변수 중복성과 토큰별 변동성에 맞춰 압축 용량과 추론 실행을 공동으로 적응시킴으로써, ITC-MoE는 생성 품질을 유지하면서도 MoE DLM의 계산 및 스토리지 비용을 크게 줄입니다. 예를 들어 SDAR-30B-A3B-Chat-b32에서 ITC-MoE는 30% 압축 예산 하에서도 MultiArith에서 96.33%의 정확도를 유지하는 동시에 최대 7.22배의 엔드투엔드 속도 향상을 달성합니다. 코드는 https://github.com/lianjunl13-sudo/ITC-MoE 에서 공개되어 있습니다.
The increasing use of Internet-of-Things (IoT) devices for monitoring a wide spectrum of applications, along with the challenges of "big data" streaming support they often require for data analysis, is nowadays pushing for an increased attention to the emerging edge computing paradigm. In particular, smart approaches to manage and analyze data directly on the network edge, are more and more investigated, and Artificial Intelligence (AI) powered edge computing is envisaged to be a promising direction. In this paper, we focus on Data Centers (DCs) and Supercomputers (SCs), where a new generation of high-resolution monitoring systems is being deployed, opening new opportunities for analysis like anomaly detection and security, but introducing new challenges for handling the vast amount of data it produces. In detail, we report on a novel lightweight and scalable approach to increase the security of DCs/SCs, that involves AI-powered edge computing on high-resolution power consumption. The method -- called pAElla -- targets real-time Malware Detection (MD), it runs on an out-of-band IoT-based monitoring system for DCs/SCs, and involves Power Spectral Density of power measurements, along with AutoEncoders. Results are promising, with an F1-score close to 1, and a False Alarm and Malware Miss rate close to 0%. We compare our method with State-of-the-Art MD techniques and show that, in the context of DCs/SCs, pAElla can cover a wider range of malware, significantly outperforming SoA approaches in terms of accuracy. Moreover, we propose a methodology for online training suitable for DCs/SCs in production, and release open dataset and code.
In recent years, Neural Machine Translation (NMT) has achieved notable results in various translation tasks. However, the word-by-word generation manner determined by the autoregressive mechanism leads to high translation latency of the NMT and restricts its low-latency applications. Non-Autoregressive Neural Machine Translation (NAT) removes the autoregressive mechanism and achieves significant decoding speedup through generating target words independently and simultaneously. Nevertheless, NAT still takes the word-level cross-entropy loss as the training objective, which is not optimal because the output of NAT cannot be properly evaluated due to the multimodality problem. In this article, we propose using sequence-level training objectives to train NAT models, which evaluate the NAT outputs as a whole and correlates well with the real translation quality. Firstly, we propose training NAT models to optimize sequence-level evaluation metrics (e.g., BLEU) based on several novel reinforcement algorithms customized for NAT, which outperforms the conventional method by reducing the variance of gradient estimation. Secondly, we introduce a novel training objective for NAT models, which aims to minimize the Bag-of-Ngrams (BoN) difference between the model output and the reference sentence. The BoN training objective is differentiable and can be calculated efficiently without doing any approximations. Finally, we apply a three-stage training strategy to combine these two methods to train the NAT model. We validate our approach on four translation tasks (WMT14 En$\leftrightarrow$De, WMT16 En$\leftrightarrow$Ro), which shows that our approach largely outperforms NAT baselines and achieves remarkable performance on all translation tasks. The source code is available at https://github.com/ictnlp/Seq-NAT.
Recently the use of self-attention has yielded to state-of-the-art results in vision-language tasks such as image captioning as well as natural language understanding and generation (NLU and NLG) tasks and computer vision tasks such as image classification. This is since self-attention maps the internal interactions among the elements of input source and target sequences. Although self-attention successfully calculates the attention values and maps the relationships among the elements of input source and target sequence, yet there is no mechanism to control the intensity of attention. In real world, when communicating with each other face to face or vocally, we tend to express different visual and linguistic context with various amounts of intensity. Some words might carry (be spoken with) more stress and weight indicating the importance of that word in the context of the whole sentence. Based on this intuition, we propose Zoneout Dropout Injection Attention Calculation (ZoDIAC) in which the intensities of attention values in the elements of the input sequence are calculated with respect to the context of the elements of input sequence. The results of our experiments reveal that employing ZoDIAC leads to better performance in comparison with the self-attention module in the Transformer model. The ultimate goal is to find out if we could modify self-attention module in the Transformer model with a method that is potentially extensible to other models that leverage on self-attention at their core. Our findings suggest that this particular goal deserves further attention and investigation by the research community. The code for ZoDIAC is available on www.github.com/zanyarz/zodiac .
The unlearning problem of deep learning models, once primarily an academic concern, has become a prevalent issue in the industry. The significant advances in text-to-image generation techniques have prompted global discussions on privacy, copyright, and safety, as numerous unauthorized personal IDs, content, artistic creations, and potentially harmful materials have been learned by these models and later utilized to generate and distribute uncontrolled content. To address this challenge, we propose \textbf{Forget-Me-Not}, an efficient and low-cost solution designed to safely remove specified IDs, objects, or styles from a well-configured text-to-image model in as little as 30 seconds, without impairing its ability to generate other content. Alongside our method, we introduce the \textbf{Memorization Score (M-Score)} and \textbf{ConceptBench} to measure the models' capacity to generate general concepts, grouped into three primary categories: ID, object, and style. Using M-Score and ConceptBench, we demonstrate that Forget-Me-Not can effectively eliminate targeted concepts while maintaining the model's performance on other concepts. Furthermore, Forget-Me-Not offers two practical extensions: a) removal of potentially harmful or NSFW content, and b) enhancement of model accuracy, inclusion and diversity through \textbf{concept correction and disentanglement}. It can also be adapted as a lightweight model patch for Stable Diffusion, allowing for concept manipulation and convenient distribution. To encourage future research in this critical area and promote the development of safe and inclusive generative models, we will open-source our code and ConceptBench at \href{https://github.com/SHI-Labs/Forget-Me-Not}{https://github.com/SHI-Labs/Forget-Me-Not}.
Recent advances in text-to-image diffusion models have enabled the generation of diverse and high-quality images. While impressive, the images often fall short of depicting subtle details and are susceptible to errors due to ambiguity in the input text. One way of alleviating these issues is to train diffusion models on class-labeled datasets. This approach has two disadvantages: (i) supervised datasets are generally small compared to large-scale scraped text-image datasets on which text-to-image models are trained, affecting the quality and diversity of the generated images, or (ii) the input is a hard-coded label, as opposed to free-form text, limiting the control over the generated images. In this work, we propose a non-invasive fine-tuning technique that capitalizes on the expressive potential of free-form text while achieving high accuracy through discriminative signals from a pretrained classifier. This is done by iteratively modifying the embedding of an added input token of a text-to-image diffusion model, by steering generated images toward a given target class according to a classifier. Our method is fast compared to prior fine-tuning methods and does not require a collection of in-class images or retraining of a noise-tolerant classifier. We evaluate our method extensively, showing that the generated images are: (i) more accurate and of higher quality than standard diffusion models, (ii) can be used to augment training data in a low-resource setting, and (iii) reveal information about the data used to train the guiding classifier. The code is available at \url{https://github.com/idansc/discriminative_class_tokens}.
Pedestrian attribute recognition (PAR) has received increasing attention because of its wide application in video surveillance and pedestrian analysis. Extracting robust feature representation is one of the key challenges in this task. The existing methods mainly use the convolutional neural network (CNN) as the backbone network to extract features. However, these methods mainly focus on small discriminative regions while ignoring the global perspective. To overcome these limitations, we propose a pure transformer-based multi-task PAR network named PARFormer, which includes four modules. In the feature extraction module, we build a transformer-based strong baseline for feature extraction, which achieves competitive results on several PAR benchmarks compared with the existing CNN-based baseline methods. In the feature processing module, we propose an effective data augmentation strategy named batch random mask (BRM) block to reinforce the attentive feature learning of random patches. Furthermore, we propose a multi-attribute center loss (MACL) to enhance the inter-attribute discriminability in the feature representations. In the viewpoint perception module, we explore the impact of viewpoints on pedestrian attributes, and propose a multi-view contrastive loss (MCVL) that enables the network to exploit the viewpoint information. In the attribute recognition module, we alleviate the negative-positive imbalance problem to generate the attribute predictions. The above modules interact and jointly learn a highly discriminative feature space, and supervise the generation of the final features. Extensive experimental results show that the proposed PARFormer network performs well compared to the state-of-the-art methods on several public datasets, including PETA, RAP, and PA100K. Code will be released at https://github.com/xwf199/PARFormer.
Visual grounding (VG) occupies a pivotal position in multi-modality vision-language models. In this study, we propose ViLaM, a large multi-modality model, that supports multi-tasks of VG using the cycle training strategy, with abundant interaction instructions. The cycle training between referring expression generation (REG) and referring expression comprehension (REC) is introduced. It enhances the consistency between visual location and referring expressions, and addresses the need for high-quality, multi-tasks VG datasets. Moreover, multi-tasks of VG are promoted in our model, contributed by the cycle training strategy. The multi-tasks in REC encompass a range of granularities, from region-level to pixel-level, which include referring bbox detection, referring keypoints detection, and referring image segmentation. In REG, referring region classification determines the fine-grained category of the target, while referring region captioning generates a comprehensive description. Meanwhile, all tasks participate in the joint training, synergistically enhancing one another and collectively improving the overall performance of the model. Furthermore, leveraging the capabilities of large language models, ViLaM extends a wide range of instructions, thereby significantly enhancing its generalization and interaction potentials. Extensive public datasets corroborate the superior capabilities of our model in VG with muti-tasks. Additionally, validating its robust generalization, ViLaM is validated under open-set and few-shot scenarios. Especially in the medical field, our model demonstrates cross-domain robust generalization capabilities. Furthermore, we contribute a VG dataset, especially with multi-tasks. To support and encourage the community focused on VG, we have made both the dataset and our code public: https://github.com/AnonymGiant/ViLaM.
As large language models (LLMs) advance, it becomes more challenging to reliably evaluate their output due to the high costs of human evaluation. To make progress towards better LLM autoraters, we introduce FLAMe, a family of Foundational Large Autorater Models. FLAMe is trained on our large and diverse collection of 100+ quality assessment tasks comprising 5M+ human judgments, curated and standardized using publicly released human evaluations from previous research. FLAMe significantly improves generalization to a wide variety of held-out tasks, outperforming LLMs trained on proprietary data like GPT-4 and Claude-3 on many tasks. We show that FLAMe can also serve as a powerful starting point for further downstream fine-tuning, using reward modeling evaluation as a case study (FLAMe-RM). Notably, on RewardBench, our FLAMe-RM-24B model (with an accuracy of 87.8%) is the top-performing generative model trained exclusively on permissively licensed data, outperforming both GPT-4-0125 (85.9%) and GPT-4o (84.7%). Additionally, we explore a more computationally efficient approach using a novel tail-patch fine-tuning strategy to optimize our FLAMe multitask mixture for reward modeling evaluation (FLAMe-Opt-RM), offering competitive RewardBench performance while requiring approximately 25x less training datapoints. Overall, our FLAMe variants outperform all popular proprietary LLM-as-a-Judge models we consider across 8 out of 12 autorater evaluation benchmarks, encompassing 53 quality assessment tasks, including RewardBench and LLM-AggreFact. Finally, our analysis reveals that FLAMe is significantly less biased than these LLM-as-a-Judge models on the CoBBLEr autorater bias benchmark, while effectively identifying high-quality responses for code generation.
Online High-Definition (HD) maps have emerged as the preferred option for autonomous driving, overshadowing the counterpart offline HD maps due to flexible update capability and lower maintenance costs. However, contemporary online HD map models embed parameters of visual sensors into training, resulting in a significant decrease in generalization performance when applied to visual sensors with different parameters. Inspired by the inherent potential of Inverse Perspective Mapping (IPM), where camera parameters are decoupled from the training process, we have designed a universal map generation framework, GenMapping. The framework is established with a triadic synergy architecture, including principal and dual auxiliary branches. When faced with a coarse road image with local distortion translated via IPM, the principal branch learns robust global features under the state space models. The two auxiliary branches are a dense perspective branch and a sparse prior branch. The former exploits the correlation information between static and moving objects, whereas the latter introduces the prior knowledge of OpenStreetMap (OSM). The triple-enhanced merging module is crafted to synergistically integrate the unique spatial features from all three branches. To further improve generalization capabilities, a Cross-View Map Learning (CVML) scheme is leveraged to realize joint learning within the common space. Additionally, a Bidirectional Data Augmentation (BiDA) module is introduced to mitigate reliance on datasets concurrently. A thorough array of experimental results shows that the proposed model surpasses current state-of-the-art methods in both semantic mapping and vectorized mapping, while also maintaining a rapid inference speed. The source code will be publicly available at https://github.com/lynn-yu/GenMapping.
Despite recent efforts in understanding the compression impact on large language models (LLMs) in terms of their downstream task performance and trustworthiness on relatively simpler uni-modal benchmarks (for example, question answering, common sense reasoning), their detailed study on multi-modal Large Vision-Language Models (LVLMs) is yet to be unveiled. Towards mitigating this gap, we present LVLM-Compress-Bench, a framework to first thoroughly study the broad impact of compression on the generative performance of LVLMs with multi-modal input driven tasks. In specific, we consider two major classes of compression for autoregressive models, namely KV cache and weight compression, for the dynamically growing intermediate cache and static weights, respectively. We use four LVLM variants of the popular LLaVA framework to present our analysis via integrating various state-of-the-art KV and weight compression methods including uniform, outlier-reduced, and group quantization for the KV cache and weights. With this framework we demonstrate on ten different multi-modal datasets with different capabilities including recognition, knowledge, language generation, spatial awareness, visual reasoning, hallucination and visual illusion identification, toxicity, stereotypes and bias. In specific, our framework demonstrates the compression impact on both general and ethically critical metrics leveraging a combination of real world and synthetic datasets to encompass diverse societal intersectional attributes. Extensive experimental evaluations yield diverse and intriguing observations on the behavior of LVLMs at different quantization budget of KV and weights, in both maintaining and losing performance as compared to the baseline model with FP16 data format. Code will be open-sourced at https://github.com/opengear-project/LVLM-compress-bench.