Low-Rank Adaptation (LoRA) is a popular parameter-efficient fine-tuning (PEFT) method for fine-tuning of large language models (LLMs), especially in federated learning, due to its strong performance and communication efficiency. In practice, LoRA is applied to the query and value projections of transformer layers without considering the distinct roles of these components. In this work, we analyze the impact of these projections in federated fine-tuning of LLMs using LoRA and observe that the value projection converges significantly faster than the query projection. Motivated by this finding, we propose FedLoVA, a federated fine-tuning framework that trains LoRA adapters on both query and value projections locally while sharing only the value projection with the server. This approach reduces communication overhead while improving convergence efficiency. Experiments on natural language understanding, multilingual sentiment analysis, and generation tasks show that FedLoVA reduces the communication cost by up to $7.8\times$ compared to existing federated LLM fine-tuning baselines while achieving comparable or better performance. The code is available at: https://github.com/EnsiehKhazaei/FedLoVA.
수화 생성(SLP)의 평균 회귀와 불연속 전환 문제를 희소 키프레임 학습으로 해결하기 위해, 수화 경계를 자동 추출하는 FAST와 조건부 흐름 매칭(CFM) 기반 SignSparK를 제안한다. 이산적 시간 앵커로 3D 수화 시퀀스를 합성하고 Keyframe-to-Pose 편집을 지원하며, 네 가지 수화 언어에서 다양한 SLP 벤치마크 최첨단 성능을 달성한다.
도구 사용 에이전트가 자체 경험으로부터 개선되도록 학습시키는 것은 여전히 어렵다. 지도 미세조정(supervised fine-tuning)은 고정된 교사 증류 궤적에 의존하고, 희소 보상 강화학습(reinforcement learning)은 장기 상호작용에 대해 약한 지도 신호를 제공하기 때문이다. 우리는 DeepSearch-Evolve를 제안한다. 이는 재현 가능한 검색 및 페이지 읽기 도구를 갖춘 결정론적이고 검증 가능한 환경인 DeepSearch-World 위에 구축된 웹 에이전트용 자기 증류 프레임워크다. DeepSearch-World는 엔티티 수준 랜덤 워크로 구성된 420K개의 멀티홉 QA 태스크를 포함하며, 진행 상황 검증, 근거 기반 반성, 실패 복구 등 자기 진화에 유용한 핵심 에이전트 인지 행동을 지원한다. DeepSearch-Evolve는 궤적 생성, 필터링, 데이터 혼합, 미세조정을 반복적으로 수행하여 더 강력한 에이전트를 학습시킨다. 더 강력한 모델로부터의 증류 없이도 DeepSearch-World-9B는 오픈소스 에이전트와 경쟁력 있는 성능을 달성하여, BrowseComp에서 31.2%, GAIA에서 61.5%, HotpotQA에서 93.4%를 기록하였다. 이는 검증 가능한 환경이 장기 웹 에이전트의 확장 가능한 자기 진화를 가능하게 함을 보여준다. 우리는 자기 개선 심층 검색 에이전트에 대한 향후 연구를 촉진하기 위해 환경, 420K 학습 풀, 검증 세트, 모델, 코드를 공개할 예정이다.
최근 대규모 언어 모델(LLM)의 발전과 광범위한 채택으로, 자연어가 소프트웨어 설계를 위한 프로그래밍 언어와 같은 형식 언어를 완전히 대체할 수 있다는 주장이 제기되고 있다. 본 포지션 논문에서 우리는 이러한 관점이 자연어의 근본적인 언어적 특성, 특히 개방형 맥락에서 과소 명시(underspecification)에 최적화되어 있다는 점을 간과한다고 주장한다. 우리는 작업 특이성(task specificity)을 중심으로 한 형식적 프레임워크를 제안하며, 이를 사용자의 구체적인 요구 사항이 주어졌을 때—가능한 모든 이미지와 같은—출력 공간에서의 불확실성을 정보 이론적으로 줄이는 것으로 정의한다. 우리는 특이성 교차 정리(specificity crossover theorem)를 증명하여, 형식적 요구 사항을 자연어로 표현하는 비용이 직접적인 형식 명세의 비용을 초과하는 임계값의 존재를 보인다. 이미지 생성, 코드 합성, 오디오 제작과 같은 다양한 모달리티에 걸친 사례 연구를 분석함으로써, 자연어는 낮은 특이성 작업에서 뛰어난 반면 형식 언어는 더 엄격한 요구 사항을 갖는 작업에서 유리함을 보여준다. 우리는 자연어와 형식 언어가 상호 보완적인 도구이며, 사용자가 특이성 스펙트럼을 넘나들 수 있게 하는 하이브리드 시스템의 개발을 주장한다.
대규모 언어 모델(LLM)이 연구 워크플로우와 소프트웨어 개발 파이프라인에서 점점 더 많이 활용되고 있지만, 이들의 출력은 여전히 입력 프롬프트 변형에 민감합니다. 본 논문에서는 코딩 작업 프롬프트의 서식화 및 LLM 기반 튜닝이 결과 코드의 성능, 효율성 및 안정성에 미치는 영향을 조사한 통제 연구인 «PromptResponse»를 제시합니다. 의미적으로는 동일하지만 구문적으로는 구별되는 HumanEval 데이터셋의 다섯 가지 변형(기준선(baseline), JSON, Markdown, YAML 및 LLM 튜닝 버전)을 사용하여, 8,200회 이상의 실행에 걸쳐 GPT-4o가 코딩 문제를 해결하도록 했습니다. 실험 결과, 일관된 서식화(특히 JSON)는 작업 성능의 소폭 향상과 함께 생성 효율성 및 구문 안정성을 개선하는 것으로 나타났습니다. 반대로, LLM으로 튜닝된 프롬프트는 다른 어떤 차원에서도 유의미한 개선 없이 작업 성능을 크게 저하시켰습니다. 이러한 결과는 적은 노력의 서식 재구성만으로도 측정 가능한 개선을 얻을 수 있는 반면, 튜닝은 모델 정렬(alignment)을 고려해야 함을 시사합니다. 우리는 본 연구의 결과를 바탕으로 도출된 일련의 실질적인 권장 사항을 제공하고, 향후 연구를 위해 데이터셋 변형 및 평가 파이프라인을 공개하며 논문을 마무리합니다.
대규모 언어 모델(LLM)은 추론, 계획, 도구 사용에서 강한 역량을 보여 왔지만, 많은 과학·공학 과제는 그럴듯한 텍스트와 코드 생성 이상을 요구한다. 이는 개입(intervention)에 시스템이 어떻게 반응하는지 이해하는 것이며, 실무에서는 통제 실험에 달려 있다. 본 연구에서는 제약 공정 설계를 위해 LLM 에이전트가 과학적 시뮬레이션 모델로 통제 실험을 수행할 수 있게 하는 다중 에이전트 프레임워크를 제안한다. 사용자 질의와 기준선 구성이 주어지면, 시스템은 구조화된 과제 표현을 구성하고, 실험을 설계하며, 비교 시뮬레이션을 실행하고, 그 결과를 해석한 뒤, 공정 매개변수 최적화를 위한 근거 기반 권고를 종합한다. 인터랙티브 에이전트 프레임워크에서 언어 모델과 고정밀(high-fidelity) 시뮬레이션 모델을 결합함으로써, 제안 시스템은 개입, 비교, 관찰을 통한 추론을 지원한다. 그 결과 언어만으로 추론하는 것보다 더 구체적이고 실행 가능한 출력을 생성한다. 산업 적용 환경에서 이 장점은 출력의 구체성 향상뿐 아니라 사용자 평가상의 정확성과 유용성 개선으로도 나타난다. Ablation 연구와 시각화된 사례 분석은 시뮬레이션 통합 실험적 추론의 효과와 실용적 가치를 추가로 보여준다.
범주형 그래프를 생성하려면 노드 순서에 의존하지 않으면서 일관된 구조를 이루는 노드 및 간선 유형을 선택해야 한다. 많은 그래프 생성기는 범주를 고정된 원-핫(one-hot) 벡터로 인코딩하는데, 이는 범주들이 등거리에 놓인 인위적인 기하를 부과할 수 있다. 우리는 Embedded Graph Flows(EGF)를 제안한다. 이는 노드 및 순서 없는(unordered) 간선 범주에 대한 연속 임베딩(embedding)을 학습하고, 순열 등변(permutation-equivariant) 그래프 트랜스포머를 사용하여 가우시안 노이즈를 학습된 종점 쪽으로 수송(transport)하는 생성 모델이다. 최종 readout은 임베딩을 이산 그래프 범주로 다시 매핑한다. 분자 벤치마크 전반에서 EGF는 경쟁력 있는 성능을 달성했다. QM9에서 EGF는 세 방법 중 보고된 네 가지 지표 모두에서 최고 결과를 내며, Fréchet ChemNet Distance(FCD)는 0.150으로, 범주형 확산(categorical-diffusion) 기준선 DiGress의 0.717, bridge 기반 기준선 GruM의 0.812와 비교된다. ZINC250k의 더 큰 분자에 적용할 때, EGF는 neighbourhood subgraph pairwise distance 커널(NSPDK)을 사용한 maximum mean discrepancy(MMD)가 가장 낮게 유지되어, 참조 분자의 국소 부분구조와 밀접히 일치함을 나타낸다. 코드는 https://github.com/Trusted-System-Lab/EGF 에서 이용할 수 있다.
Pre-trained discrete-token VLMs generate images by sampling sequences of codebook indices, but their visual fidelity is bounded by the VQ-VAE decoding stage. Prior work addresses this bottleneck by replacing the discrete tokenizer with continuous or hybrid visual token representations, or by integrating diffusion into the generation pipeline; both require foundation-scale VLM retraining or realignment. We show this is unnecessary: before each token is sampled, discrete-token VLMs already compute a full logit distribution over the codebook, but standard decoding discards it after selecting a single index. Under Otsu's threshold, the pre-sampling logits of capable VLMs assign statistically meaningful probability to at least two codebook entries per token, and recovering this distribution can substantially close the fidelity gap. We propose DCDD, a post-hoc framework that conditions a diffusion decoder on this discarded distribution while leaving the VLM frozen. At inference, DCDD converts the VLM's pre-sampling logits into distribution-weighted code vectors; during training, it uses VQ-VAE encoder-derived proxy logits calibrated to match the distributional statistics of the VLM's inference-time logits. The decoder maps these representations to high-fidelity images as an alternative to the native VQ-VAE decoder. Trained on ImageNet-1K for 50K steps, DCDD reduces reconstruction FID by 74% and Janus-Pro 7B's generation FID on MJHQ-30K by 34%, while semantic alignment across GenEval, DPG-Bench, and WISE remain consistent. Controlled comparisons with diffusion models from the same backbone family, applied either as an image-to-image refiner or as an identically trained decoder on the sampled codes, confirm that the gain stems from conditioning on the VLM's pre-sampling logit distribution rather than from the diffusion model itself.
Modern computing systems heavily rely on hardware as the root of trust. However, their increasing complexity has given rise to security-critical vulnerabilities that cross-layer at-tacks can exploit. Traditional hardware vulnerability detection methods, such as random regression and formal verification, have limitations. Random regression, while scalable, is slow in exploring hardware, and formal verification techniques are often concerned with manual effort and state explosions. Hardware fuzzing has emerged as an effective approach to exploring and detecting security vulnerabilities in large-scale designs like modern processors. They outperform traditional methods regarding coverage, scalability, and efficiency. However, state-of-the-art fuzzers struggle to achieve comprehensive coverage of intricate hardware designs within a practical timeframe, often falling short of a 70% coverage threshold. We propose a novel ML-based hardware fuzzer, ChatFuzz, to address this challenge. Ourapproach leverages LLMs like ChatGPT to understand processor language, focusing on machine codes and generating assembly code sequences. RL is integrated to guide the input generation process by rewarding the inputs using code coverage metrics. We use the open-source RISCV-based RocketCore processor as our testbed. ChatFuzz achieves condition coverage rate of 75% in just 52 minutes compared to a state-of-the-art fuzzer, which requires a lengthy 30-hour window to reach a similar condition coverage. Furthermore, our fuzzer can attain 80% coverage when provided with a limited pool of 10 simulation instances/licenses within a 130-hour window. During this time, it conducted a total of 199K test cases, of which 6K produced discrepancies with the processor's golden model. Our analysis identified more than 10 unique mismatches, including two new bugs in the RocketCore and discrepancies from the RISC-V ISA Simulator.
Domain specific accelerators present new challenges and opportunities for code generation onto novel instruction sets, communication fabrics, and memory architectures. In this paper we introduce an intermediate representation (IR) which enables both deep learning computational kernels and hardware capabilities to be described in the same IR. We then formulate and apply instruction mapping to determine the possible ways a computation can be performed on a hardware system. Next, our scheduler chooses a specific mapping and determines the data movement and computation order. In order to manage the large search space of mappings and schedules, we developed a flexible framework that allows heuristics, cost models, and potentially machine learning to facilitate this search problem. With this system, we demonstrate the automated extraction of matrix multiplication kernels out of recent deep learning kernels such as depthwise-separable convolution. In addition, we demonstrate two to five times better performance on DeepBench sized GEMMs and GRU RNN execution when compared to state-of-the-art (SOTA) implementations on new hardware and up to 85% of the performance for SOTA implementations on existing hardware.
Current state-of-the-art approaches for Semi-supervised Video Object Segmentation (Semi-VOS) propagates information from previous frames to generate segmentation mask for the current frame. This results in high-quality segmentation across challenging scenarios such as changes in appearance and occlusion. But it also leads to unnecessary computations for stationary or slow-moving objects where the change across frames is minimal. In this work, we exploit this observation by using temporal information to quickly identify frames with minimal change and skip the heavyweight mask generation step. To realize this efficiency, we propose a novel dynamic network that estimates change across frames and decides which path -- computing a full network or reusing previous frame's feature -- to choose depending on the expected similarity. Experimental results show that our approach significantly improves inference speed without much accuracy degradation on challenging Semi-VOS datasets -- DAVIS 16, DAVIS 17, and YouTube-VOS. Furthermore, our approach can be applied to multiple Semi-VOS methods demonstrating its generality. The code is available in https://github.com/HYOJINPARK/Reuse_VOS.
Data engineering is one of the fastest-growing fields within machine learning (ML). As ML becomes more common, the appetite for data grows more ravenous. But ML requires more data than individual teams of data engineers can readily produce, which presents a severe challenge to ML deployment at scale. Much like the software-engineering revolution, where mass adoption of open-source software replaced the closed, in-house development model for infrastructure code, there is a growing need to enable rapid development and open contribution to massive machine learning data sets. This article shows that open-source data sets are the rocket fuel for research and innovation at even some of the largest AI organizations. Our analysis of nearly 2000 research publications from Facebook, Google and Microsoft over the past five years shows the widespread use and adoption of open data sets. Open data sets that are easily accessible to the public are vital to accelerating ML innovation for everyone. But such open resources are scarce in the wild. So, what if we are able to accelerate data-set creation via automatic data set generation tools?
Generative Adversarial Networks (GANs) are able to generate high-quality images, but it remains difficult to explicitly specify the semantics of synthesized images. In this work, we aim to better understand the semantic representation of GANs, and thereby enable semantic control in GAN's generation process. Interestingly, we find that a well-trained GAN encodes image semantics in its internal feature maps in a surprisingly simple way: a linear transformation of feature maps suffices to extract the generated image semantics. To verify this simplicity, we conduct extensive experiments on various GANs and datasets; and thanks to this simplicity, we are able to learn a semantic segmentation model for a trained GAN from a small number (e.g., 8) of labeled images. Last but not least, leveraging our findings, we propose two few-shot image editing approaches, namely Semantic-Conditional Sampling and Semantic Image Editing. Given a trained GAN and as few as eight semantic annotations, the user is able to generate diverse images subject to a user-provided semantic layout, and control the synthesized image semantics. We have made the code publicly available.
Generative Adversarial Networks (GANs) can generate near photo realistic images in narrow domains such as human faces. Yet, modeling complex distributions of datasets such as ImageNet and COCO-Stuff remains challenging in unconditional settings. In this paper, we take inspiration from kernel density estimation techniques and introduce a non-parametric approach to modeling distributions of complex datasets. We partition the data manifold into a mixture of overlapping neighborhoods described by a datapoint and its nearest neighbors, and introduce a model, called instance-conditioned GAN (IC-GAN), which learns the distribution around each datapoint. Experimental results on ImageNet and COCO-Stuff show that IC-GAN significantly improves over unconditional models and unsupervised data partitioning baselines. Moreover, we show that IC-GAN can effortlessly transfer to datasets not seen during training by simply changing the conditioning instances, and still generate realistic images. Finally, we extend IC-GAN to the class-conditional case and show semantically controllable generation and competitive quantitative results on ImageNet; while improving over BigGAN on ImageNet-LT. Code and trained models to reproduce the reported results are available at https://github.com/facebookresearch/ic_gan.
Data-driven design and innovation is a process to reuse and provide valuable and useful information. However, existing semantic networks for design innovation is built on data source restricted to technological and scientific information. Besides, existing studies build the edges of a semantic network only on either statistical or semantic relationships, which is less likely to make full use of the benefits from both types of relationships and discover implicit knowledge for design innovation. Therefore, we constructed WikiLink, a semantic network based on Wikipedia. Combined weight which fuses both the statistic and semantic weights between concepts is introduced in WikiLink, and four algorithms are developed for inspiring new ideas. Evaluation experiments are undertaken and results show that the network is characterised by high coverage of terms, relationships and disciplines, which proves the network's effectiveness and usefulness. Then a demonstration and case study results indicate that WikiLink can serve as an idea generation tool for innovation in conceptual design. The source code of WikiLink and the backend data are provided open-source for more users to explore and build on.
Recent large-scale generative models learned on big data are capable of synthesizing incredible images yet suffer from limited controllability. This work offers a new generation paradigm that allows flexible control of the output image, such as spatial layout and palette, while maintaining the synthesis quality and model creativity. With compositionality as the core idea, we first decompose an image into representative factors, and then train a diffusion model with all these factors as the conditions to recompose the input. At the inference stage, the rich intermediate representations work as composable elements, leading to a huge design space (i.e., exponentially proportional to the number of decomposed factors) for customizable content creation. It is noteworthy that our approach, which we call Composer, supports various levels of conditions, such as text description as the global information, depth map and sketch as the local guidance, color histogram for low-level details, etc. Besides improving controllability, we confirm that Composer serves as a general framework and facilitates a wide range of classical generative tasks without retraining. Code and models will be made available.
3D object detection from visual sensors is a cornerstone capability of robotic systems. State-of-the-art methods focus on reasoning and decoding object bounding boxes from multi-view camera input. In this work we gain intuition from the integral role of multi-view consistency in 3D scene understanding and geometric learning. To this end, we introduce VEDet, a novel 3D object detection framework that exploits 3D multi-view geometry to improve localization through viewpoint awareness and equivariance. VEDet leverages a query-based transformer architecture and encodes the 3D scene by augmenting image features with positional encodings from their 3D perspective geometry. We design view-conditioned queries at the output level, which enables the generation of multiple virtual frames during training to learn viewpoint equivariance by enforcing multi-view consistency. The multi-view geometry injected at the input level as positional encodings and regularized at the loss level provides rich geometric cues for 3D object detection, leading to state-of-the-art performance on the nuScenes benchmark. The code and model are made available at https://github.com/TRI-ML/VEDet.
The Segment Anything Model (SAM) has established itself as a powerful zero-shot image segmentation model, enabled by efficient point-centric annotation and prompt-based models. While click and brush interactions are both well explored in interactive image segmentation, the existing methods on videos focus on mask annotation and propagation. This paper presents SAM-PT, a novel method for point-centric interactive video segmentation, empowered by SAM and long-term point tracking. SAM-PT leverages robust and sparse point selection and propagation techniques for mask generation. Compared to traditional object-centric mask propagation strategies, we uniquely use point propagation to exploit local structure information agnostic to object semantics. We highlight the merits of point-based tracking through direct evaluation on the zero-shot open-world Unidentified Video Objects (UVO) benchmark. Our experiments on popular video object segmentation and multi-object segmentation tracking benchmarks, including DAVIS, YouTube-VOS, and BDD100K, suggest that a point-based segmentation tracker yields better zero-shot performance and efficient interactions. We release our code that integrates different point trackers and video segmentation benchmarks at https://github.com/SysCV/sam-pt.
Assessing the robustness of perception models to covariate shifts and their ability to detect out-of-distribution (OOD) inputs is crucial for safety-critical applications such as autonomous vehicles. By nature of such applications, however, the relevant data is difficult to collect and annotate. In this paper, we show for the first time how synthetic data can be specifically generated to assess comprehensively the real-world reliability of semantic segmentation models. By fine-tuning Stable Diffusion with only in-domain data, we perform zero-shot generation of visual scenes in OOD domains or inpainted with OOD objects. This synthetic data is employed to evaluate the robustness of pretrained segmenters, thereby offering insights into their performance when confronted with real edge cases. Through extensive experiments, we demonstrate a high correlation between the performance of models when evaluated on our synthetic OOD data and when evaluated on real OOD inputs, showing the relevance of such virtual testing. Furthermore, we demonstrate how our approach can be utilized to enhance the calibration and OOD detection capabilities of segmenters. Code and data are made public.
Indoor imaging is a critical task for robotics and internet-of-things. WiFi as an omnipresent signal is a promising candidate for carrying out passive imaging and synchronizing the up-to-date information to all connected devices. This is the first research work to consider WiFi indoor imaging as a multi-modal image generation task that converts the measured WiFi power into a high-resolution indoor image. Our proposed WiFi-GEN network achieves a shape reconstruction accuracy that is 275% of that achieved by physical model-based inversion methods. Additionally, the Frechet Inception Distance score has been significantly reduced by 82%. To examine the effectiveness of models for this task, the first large-scale dataset is released containing 80,000 pairs of WiFi signal and imaging target. Our model absorbs challenges for the model-based methods including the non-linearity, ill-posedness and non-certainty into massive parameters of our generative AI network. The network is also designed to best fit measured WiFi signals and the desired imaging output. For reproducibility, we will release the data and code upon acceptance.