The scientific scale-up of large language models (LLMs) necessitates a comprehensive understanding of their scaling properties. However, the existing literature on the scaling properties only yields an incomplete answer: optimization loss decreases predictably as the model size increases, in line with established scaling law; yet no scaling law for task has been established and the task performances are far from predictable during scaling. Task performances typically show minor gains on small models until they improve dramatically once models exceed a size threshold, exemplifying the ``emergent abilities''. In this study, we discover that small models, although they exhibit minor performance, demonstrate critical and consistent task performance improvements that are not captured by conventional evaluation strategies due to insufficient measurement resolution. To measure such improvements, we introduce PassUntil, an evaluation strategy with theoretically infinite resolution, through massive sampling in the decoding phase. With PassUntil, we conduct a quantitative investigation into the scaling law of task performance. The investigation contains two parts. Firstly, a strict task scaling law that is not conventionally known to exist, is identified, enhancing the predictability of task performances. Remarkably, we are able to predict the performance of the 2.4B model on code generation with merely 0.05\% deviation before training starts, which is the first systematic attempt to verify predictable scaling proposed by GPT-4's report. Secondly, we are able to study emergent abilities quantitatively. We identify a kind of accelerated emergence whose scaling curve cannot be fitted by standard scaling law function and has a increasing speed. We then examine two hypothesis and imply that the ``multiple circuits hypothesis'' might be responsible for the accelerated emergence.
Complicated image registration is a key issue in medical image analysis, and deep learning-based methods have achieved better results than traditional methods. The methods include ConvNet-based and Transformer-based methods. Although ConvNets can effectively utilize local information to reduce redundancy via small neighborhood convolution, the limited receptive field results in the inability to capture global dependencies. Transformers can establish long-distance dependencies via a self-attention mechanism; however, the intense calculation of the relationships among all tokens leads to high redundancy. We propose a novel unsupervised image registration method named the unified Transformer and superresolution (UTSRMorph) network, which can enhance feature representation learning in the encoder and generate detailed displacement fields in the decoder to overcome these problems. We first propose a fusion attention block to integrate the advantages of ConvNets and Transformers, which inserts a ConvNet-based channel attention module into a multihead self-attention module. The overlapping attention block, a novel cross-attention method, uses overlapping windows to obtain abundant correlations with match information of a pair of images. Then, the blocks are flexibly stacked into a new powerful encoder. The decoder generation process of a high-resolution deformation displacement field from low-resolution features is considered as a superresolution process. Specifically, the superresolution module was employed to replace interpolation upsampling, which can overcome feature degradation. UTSRMorph was compared to state-of-the-art registration methods in the 3D brain MR (OASIS, IXI) and MR-CT datasets. The qualitative and quantitative results indicate that UTSRMorph achieves relatively better performance. The code and datasets are publicly available at https://github.com/Runshi-Zhang/UTSRMorph.
High-quality video-text preference data is crucial for Multimodal Large Language Models (MLLMs) alignment. However, existing preference data is very scarce. Obtaining VQA preference data for preference training is costly, and manually annotating responses is highly unreliable, which could result in low-quality pairs. Meanwhile, AI-generated responses controlled by temperature adjustment lack diversity. To address these issues, we propose a high-quality VQA preference dataset, called \textit{\textbf{M}ultiple \textbf{M}ultimodal \textbf{A}rtificial \textbf{I}ntelligence \textbf{P}reference Datasets in \textbf{V}QA} (\textbf{MMAIP-V}), which is constructed by sampling from the response distribution set and using an external scoring function for response evaluation. Furthermore, to fully leverage the preference knowledge in MMAIP-V and ensure sufficient optimization, we propose \textit{\textbf{Iter}ative \textbf{W}eak-to-\textbf{S}trong \textbf{R}einforcement \textbf{L}earning from \textbf{AI} \textbf{F}eedback for video MLLMs} (\textbf{Iter-W2S-RLAIF}), a framework that gradually enhances MLLMs' alignment capabilities by iteratively updating the reference model and performing parameter extrapolation. Finally, we propose an unbiased and information-complete evaluation scheme in VQA evaluation. Experiments demonstrate that MMAIP-V is beneficial for MLLMs in preference learning and Iter-W2S-RLAIF fully exploits the alignment information in MMAIP-V. We believe that the proposed automatic VQA preference data generation pipeline based on AI feedback can greatly promote future work in the MLLMs alignment. \textbf{Code and dataset are available} \href{https://anonymous.4open.science/r/MMAIP-V_Iter-W2S-RLAIF-702F}{MMAIP-V\_Iter-W2S-RLAIF-702F}.
LLMs have revolutionized the field of artificial intelligence and have emerged as the de-facto tool for many tasks. The current established technology of LLMs is to process input and generate output at the token level. This is in sharp contrast to humans who operate at multiple levels of abstraction, well beyond single words, to analyze information and to generate creative content. In this paper, we present an attempt at an architecture which operates on an explicit higher-level semantic representation, which we name a concept. Concepts are language- and modality-agnostic and represent a higher level idea or action in a flow. Hence, we build a "Large Concept Model". In this study, as proof of feasibility, we assume that a concept corresponds to a sentence, and use an existing sentence embedding space, SONAR, which supports up to 200 languages in both text and speech modalities. The Large Concept Model is trained to perform autoregressive sentence prediction in an embedding space. We explore multiple approaches, namely MSE regression, variants of diffusion-based generation, and models operating in a quantized SONAR space. These explorations are performed using 1.6B parameter models and training data in the order of 1.3T tokens. We then scale one architecture to a model size of 7B parameters and training data of about 2.7T tokens. We perform an experimental evaluation on several generative tasks, namely summarization and a new task of summary expansion. Finally, we show that our model exhibits impressive zero-shot generalization performance to many languages, outperforming existing LLMs of the same size. The training code of our models is freely available.
In order to be widely applicable, speech-driven 3D head avatars must articulate their lips in accordance with speech, while also conveying the appropriate emotions with dynamically changing facial expressions. The key problem is that deterministic models produce high-quality lip-sync but without rich expressions, whereas stochastic models generate diverse expressions but with lower lip-sync quality. To get the best of both, we seek a stochastic model with accurate lip-sync. To that end, we develop a new approach based on the following observation: if a method generates realistic 3D lip motions, it should be possible to infer the spoken audio from the lip motion. The inferred speech should match the original input audio, and erroneous predictions create a novel supervision signal for training 3D talking head avatars with accurate lip-sync. To demonstrate this effect, we propose THUNDER (Talking Heads Under Neural Differentiable Elocution Reconstruction), a 3D talking head avatar framework that introduces a novel supervision mechanism via differentiable sound production. First, we train a novel mesh-to-speech model that regresses audio from facial animation. Then, we incorporate this model into a diffusion-based talking avatar framework. During training, the mesh-to-speech model takes the generated animation and produces a sound that is compared to the input speech, creating a differentiable analysis-by-audio-synthesis supervision loop. Our extensive qualitative and quantitative experiments demonstrate that THUNDER significantly improves the quality of the lip-sync of talking head avatars while still allowing for generation of diverse, high-quality, expressive facial animations. The code and models will be available at https://thunder.is.tue.mpg.de/
Vascular diseases pose a significant threat to human health, with X-ray angiography established as the gold standard for diagnosis, allowing for detailed observation of blood vessels. However, angiographic X-rays expose personnel and patients to higher radiation levels than non-angiographic X-rays, which are unwanted. Thus, modality translation from non-angiographic to angiographic X-rays is desirable. Data-driven deep approaches are hindered by the lack of paired large-scale X-ray angiography datasets. While making high-quality vascular angiography synthesis crucial, it remains challenging. We find that current medical image synthesis primarily operates at pixel level and struggles to adapt to the complex geometric structure of blood vessels, resulting in unsatisfactory quality of blood vessel image synthesis, such as disconnections or unnatural curvatures. To overcome this issue, we propose a self-supervised method via diffusion models to transform non-angiographic X-rays into angiographic X-rays, mitigating data shortages for data-driven approaches. Our model comprises a diffusion model that learns the distribution of vascular data from diffusion latent, a generator for vessel synthesis, and a mask-based adversarial module. To enhance geometric accuracy, we propose a parametric vascular model to fit the shape and distribution of blood vessels. The proposed method contributes a pipeline and a synthetic dataset for X-ray angiography. We conducted extensive comparative and ablation experiments to evaluate the Angio-Diff. The results demonstrate that our method achieves state-of-the-art performance in synthetic angiography image quality and more accurately synthesizes the geometric structure of blood vessels. The code is available at https://github.com/zfw-cv/AngioDiff.
대규모 언어 모델(LLM)은 검색 증강 생성(RAG)을 통해 대화형 추천 시스템(RS)에서 추천 생성기로 점점 더 많이 사용되고 있다. 이러한 유연한 설계는 새로운 구조적 편향의 원인을 도입한다. 본 논문은 top-1 및 단일 턴 LLM 기반 RS에서의 위치 편향, 즉 추천되는 항목이 프롬프트에서 후보가 제시되는 순서에 체계적으로 의존하는 현상을 연구한다. 우리는 서로 보완적인 두 가지 탐지 방법론을 제안한다. (i) 분할표 행렬 분석과 카이제곱 독립성 검정에 기반한 POB-ICM, 그리고 (ii) 순열 기반 평가에서 균등성 귀무 모형으로부터 도출된 표본 효율적 접근법으로, 형식적인 통계적 보장과 함께 확률적 디코딩을 가능하게 하는 POB-UNBP이다. 실제 제품 카탈로그를 기반으로 구축한 제품 추천용 통제된 RAG 테스트베드, 고정된 프롬프트 템플릿, 그리고 템플릿으로 생성한 2000개의 사용자 질의(각각 20개의 순열, 총 40,000개 인스턴스)를 사용한 결과, 두 방법론 모두 평가된 모든 최신 모델에서 위치 독립성과 균등성이라는 귀무가설을 기각한다. 우리는 이러한 효과를 정량화하기 위해 순열 민감도, 위치 편향, 강한 위치 편향이라는 세 가지 상호 보완적인 지표를 도입하며, 마지막 지표는 위치가 단지 비슷하게 관련된 항목들 사이에서 선택하는 데 그치지 않고 추천을 명백히 덜 관련된 항목으로 옮기는 경우를 겨냥한다. 후보 집합 크기와 디코딩 온도에 걸친 강건성 분석은 이 편향이 구조적이며 모델에 따라 달라지고, 결정적 디코딩이나 후보 집합 크기의 산물이 아님을 보여 준다. 이러한 결과는 평가된 환경에서 위치 편향이 프롬프트 매개 추천 파이프라인의 측정 가능하고 재현 가능한 속성임을 입증하며, 배포된 LLM 기반 RS에 대해 순서 무작위화, 표준화된 프롬프팅 등 정기적인 감사와 거버넌스 통제의 필요성을 뒷받침한다. 소스 코드는 공개되어 있다.
온폴리시 증류(OPD)는 고정된 교사 모델의 안내를 받아 학생 언어 모델을 자신의 응답으로 학습시킨다. 이러한 응답을 생성하는 비용이 크기 때문에 시스템은 양자화 추론과 비동기 롤아웃을 사용하여 비용을 줄인다. 그러나 두 방식 모두 롤아웃 엔진과 학습 중인 모델 사이에 학습-추론 불일치를 유발한다. 이러한 불일치는 강화학습(RL)을 불안정하게 만들 수 있으나, OPD에 미치는 영향은 아직 명확하지 않다. 본 논문은 OPD가 불일치를 얼마나 허용하는지, 그 영향이 어디에서 발생하는지, 그리고 학습 시스템이 이러한 허용 범위를 어떻게 활용할 수 있는지를 조사한다. 모든 불일치 원인을 하나의 척도로 측정하여, 세 가지 학생-교사 쌍에서 양자화를 검토하고 한 쌍에서는 가중치 노후화와 주입된 잡음에 대한 추가 실험을 수행한다. OPD는 학습 품질의 유의미한 손실 없이 상당한 불일치를 허용하지만 더 높은 수준에서는 성능이 저하되며, 이는 경험적인 불일치 예산이 존재함을 보여 준다. 불일치로 인한 품질 비용과 양자화가 유발하는 불일치는 모두 모델에 따라 달라지며, 일부 층은 정밀도 감소에 특히 민감하다. 통제된 비교에서 OPD는 RL 방법을 불안정하게 만드는 롤아웃 구성에서도 안정적으로 유지된다. 가중치 잡음이 있는 경우 불일치는 생성 중에 도달하는 접두사도 변화시켜, 토큰별 보정으로는 해결할 수 없는 오류를 일으킨다. 이러한 결과를 실제에 적용하기 위해, 후보마다 전체 학습을 수행하지 않고도 보정된 예산에 비추어 롤아웃 구성을 선별하는 경량 허용 검사를 제안한다. 모델별 FP8 레시피는 학습 품질을 유지하면서 BF16보다 최소 $1.93\times$ 많은 KV 캐시 토큰을 수용할 수 있다. FP8과 비동기 롤아웃을 결합하면 동기식 BF16 학습보다 $1.62\times$ 빠르게 동일한 학습 품질에 도달한다. 코드는 https://github.com/benyucong/train-inference-mismatch-opd 에서 확인할 수 있다.
MLLM 기반 GUI 그라운딩 방법은 일반적으로 목표 위치 파악을 자기회귀적 좌표 생성으로 정식화하여, 모델이 MLLM의 강력한 인스트럭션 수행 및 의미 이해 능력을 활용할 수 있도록 한다. 그러나 이 정식화는 GUI 클릭에서 요구되는 공간적 정밀도로 좌표 토큰을 디코딩하는 동시에, 모델이 영역 수준의 목표 증거를 유지해야 한다는 과제를 안고 있다. 진단 분석을 통해, 목표 영역 인식이 디코더 중간 층에서 나타나지만 최종 좌표 예측에 유지되거나 반영되지 않음을 확인하였다. 기존의 ZoomIn 방식 방법들은 외부 크롭 후 재실행(crop-and-rerun) 순전파를 통해 이 문제를 해결하는데, 이는 위치 파악 성능을 향상시키지만 엔드-투-엔드 지연 시간과 계산 비용을 증가시킨다. 이러한 추가 비용 없이 2회 줌인의 정확도 이점을 유지하기 위해, 우리는 층 간 증거 브리징을 위한 단일 순전파 프레임워크인 InnerZoom을 제안한다. InnerZoom은 원래 순전파에서 목표 관련 단서를 압축된 층 간 증거 상태로 변환한 후, 이 상태를 이후 디코딩 층 전반에 걸쳐 보존·정제·재주입하여 좌표 예측을 안내한다. 광범위한 실험 결과, InnerZoom-4B는 6개의 GUI 그라운딩 벤치마크 전체에서 최고 성능을 달성하였으며, OSWorld-G에서 64.7, UI-Vision에서 40.2, OSWorld-GR에서 73.1, MMBench-GUI에서 87.6을 기록하여 각각 이전 최고 결과를 4.1, 3.2, 2.9, 2.3포인트 상회하였다. 통제된 4B 설정에서 InnerZoom은 동일한 SFT+RL 기준선 대비 평균 5.3포인트 향상을 달성하고, 2회 순전파 ZoomIn보다 평균 1.3포인트 높은 성능을 보이면서, 엔드-투-엔드 지연 시간을 최대 31.8%, TFLOPs를 약 29% 감소시킨다. 코드와 모델은 공개될 예정이다.
마이크로초 단위의 시간 해상도와 매우 높은 동적 범위(HDR)를 특징으로 하는 이벤트 카메라는 인식 태스크를 위해 고속 이벤트 스트림을 발생시킵니다. 최근 발전에서 그래프 신경망(GNN) 기반 방법론들은 이벤트 인식 분야에서 큰 잠재력을 보여주고 있습니다. 그러나 기존 방법들은 일반적으로 유클리드 공간에서의 단순한 쌍별(pairwise) 노드 연결성에 의존하므로, 장거리 의존성을 포착하고 이벤트 스트림 본연의 계층적 구조를 충실히 특징화하는 데 어려움을 겪습니다. 이를 위해 본 논문에서는 우리가 아는 한 유클리드 및 쌍곡 공간에서 이벤트 스트림을 공동으로 모델링하는 최초의 이중 공간 이벤트 인식 접근 방식인 EHGCN을 제안합니다. 이벤트 스트림 인식에 쌍곡 기하학을 도입함으로써, EHGCN은 불균일하고 모션에 의해 발생하는 이벤트 스트림의 이방성(anisotropic) 및 계층적 구조를 자연스럽게 포착할 수 있습니다. 구체적으로, 먼저 다중 스케일 복셀 그리드와 가우시안 분포 모델링을 기반으로 한 분포 인지 이벤트 선별 기법을 도입하여 무질서한 노이즈를 감쇠시키면서 판별력 있는 이벤트를 유지합니다. 그런 다음 모션 일관성 에너지 함수를 최소화하여 짧은 시간 간격 내의 일관된 전역 모션 패턴을 명시적으로 포착하는 마르코프 무작위 장(MRF) 최적화 기반 모션 인지 하이퍼에지(hyperedge) 생성 방식을 제시합니다. 이를 통해 타깃 간 가짜 연관성을 제거하고 이벤트 간의 장거리 의존성을 포착하면서 중요한 위상학적 사전 정보를 제공합니다. 마지막으로, 국소 유클리드 공간 및 전역 쌍곡 공간에서 각각 조밀하게 집계되고 계층적으로 모델링된 망막 이벤트를 융합하여 하이브리드 이벤트 인식을 달성하는 유클리드-쌍곡 GCN을 제안합니다. 객체 검출(object detection) 및 인식과 같은 이벤트 인식 태스크에 대한 광범위한 실험 결과는 제안하는 접근 방식의 유효성을 입증합니다. 소스 코드는 https://github.com/ev-lluo/EHGCN 에서 공개될 예정입니다.
Self-supervision is recently surging at its new frontier of graph learning. It facilitates graph representations beneficial to downstream tasks; but its success could hinge on domain knowledge for handcraft or the often expensive trials and errors. Even its state-of-the-art representative, graph contrastive learning (GraphCL), is not completely free of those needs as GraphCL uses a prefabricated prior reflected by the ad-hoc manual selection of graph data augmentations. Our work aims at advancing GraphCL by answering the following questions: How to represent the space of graph augmented views? What principle can be relied upon to learn a prior in that space? And what framework can be constructed to learn the prior in tandem with contrastive learning? Accordingly, we have extended the prefabricated discrete prior in the augmentation set, to a learnable continuous prior in the parameter space of graph generators, assuming that graph priors per se, similar to the concept of image manifolds, can be learned by data generation. Furthermore, to form contrastive views without collapsing to trivial solutions due to the prior learnability, we have leveraged both principles of information minimization (InfoMin) and information bottleneck (InfoBN) to regularize the learned priors. Eventually, contrastive learning, InfoMin, and InfoBN are incorporated organically into one framework of bi-level optimization. Our principled and automated approach has proven to be competitive against the state-of-the-art graph self-supervision methods, including GraphCL, on benchmarks of small graphs; and shown even better generalizability on large-scale graphs, without resorting to human expertise or downstream validation. Our code is publicly released at https://github.com/Shen-Lab/GraphCL_Automated.
Estimating the number of distinct values (NDV) in a column is useful for many tasks in database systems, such as columnstore compression and data profiling. In this work, we focus on how to derive accurate NDV estimations from random (online/offline) samples. Such efficient estimation is critical for tasks where it is prohibitive to scan the data even once. Existing sample-based estimators typically rely on heuristics or assumptions and do not have robust performance across different datasets as the assumptions on data can easily break. On the other hand, deriving an estimator from a principled formulation such as maximum likelihood estimation is very challenging due to the complex structure of the formulation. We propose to formulate the NDV estimation task in a supervised learning framework, and aim to learn a model as the estimator. To this end, we need to answer several questions: i) how to make the learned model workload agnostic; ii) how to obtain training data; iii) how to perform model training. We derive conditions of the learning framework under which the learned model is workload agnostic, in the sense that the model/estimator can be trained with synthetically generated training data, and then deployed into any data warehouse simply as, e.g., user-defined functions (UDFs), to offer efficient (within microseconds on CPU) and accurate NDV estimations for unseen tables and workloads. We compare the learned estimator with the state-of-the-art sample-based estimators on nine real-world datasets to demonstrate its superior estimation accuracy. We publish our code for training data generation, model training, and the learned estimator online for reproducibility.
This work addresses the problem of discovering non-linear interpretable paths in the latent space of pre-trained GANs in a model-agnostic manner. In the proposed method, the discovery is driven by a set of pairs of natural language sentences with contrasting semantics, named semantic dipoles, that serve as the limits of the interpretation that we require by the trainable latent paths to encode. By using the pre-trained CLIP encoder, the sentences are projected into the vision-language space, where they serve as dipoles, and where RBF-based warping functions define a set of non-linear directional paths, one for each semantic dipole, allowing in this way traversals from one semantic pole to the other. By defining an objective that discovers paths in the latent space of GANs that generate changes along the desired paths in the vision-language embedding space, we provide an intuitive way of controlling the underlying generative factors and address some of the limitations of the state-of-the-art works, namely, that a) they are typically tailored to specific GAN architectures (i.e., StyleGAN), b) they disregard the relative position of the manipulated and the original image in the image embedding and the relative position of the image and the text embeddings, and c) they lead to abrupt image manipulations and quickly arrive at regions of low density and, thus, low image quality, providing limited control of the generative factors. We provide extensive qualitative and quantitative results that demonstrate our claims with two pre-trained GANs, and make the code and the pre-trained models publicly available at: https://github.com/chi0tzp/ContraCLIP
Recent years have witnessed success in AIGC (AI Generated Content). People can make use of a pre-trained diffusion model to generate images of high quality or freely modify existing pictures with only prompts in nature language. More excitingly, the emerging personalization techniques make it feasible to create specific-desired images with only a few images as references. However, this induces severe threats if such advanced techniques are misused by malicious users, such as spreading fake news or defaming individual reputations. Thus, it is necessary to regulate personalization models (i.e., concept censorship) for their development and advancement. In this paper, we focus on the personalization technique dubbed Textual Inversion (TI), which is becoming prevailing for its lightweight nature and excellent performance. TI crafts the word embedding that contains detailed information about a specific object. Users can easily download the word embedding from public websites like Civitai and add it to their own stable diffusion model without fine-tuning for personalization. To achieve the concept censorship of a TI model, we propose leveraging the backdoor technique for good by injecting backdoors into the Textual Inversion embeddings. Briefly, we select some sensitive words as triggers during the training of TI, which will be censored for normal use. In the subsequent generation stage, if the triggers are combined with personalized embeddings as final prompts, the model will output a pre-defined target image rather than images including the desired malicious concept. To demonstrate the effectiveness of our approach, we conduct extensive experiments on Stable Diffusion, a prevailing open-sourced text-to-image model. Our code, data, and results are available at https://concept-censorship.github.io.
Large language models (LLMs) have shown remarkable achievements in natural language processing tasks, producing high-quality outputs. However, LLMs still exhibit limitations, including the generation of factually incorrect information. In safety-critical applications, it is important to assess the confidence of LLM-generated content to make informed decisions. Retrieval Augmented Language Models (RALMs) is relatively a new area of research in NLP. RALMs offer potential benefits for scientific NLP tasks, as retrieved documents, can serve as evidence to support model-generated content. This inclusion of evidence enhances trustworthiness, as users can verify and explore the retrieved documents to validate model outputs. Quantifying uncertainty in RALM generations further improves trustworthiness, with retrieved text and confidence scores contributing to a comprehensive and reliable model for scientific applications. However, there is limited to no research on UQ for RALMs, particularly in scientific contexts. This study aims to address this gap by conducting a comprehensive evaluation of UQ in RALMs, focusing on scientific tasks. This research investigates how uncertainty scores vary when scientific knowledge is incorporated as pretraining and retrieval data and explores the relationship between uncertainty scores and the accuracy of model-generated outputs. We observe that an existing RALM finetuned with scientific knowledge as the retrieval data tends to be more confident in generating predictions compared to the model pretrained only with scientific knowledge. We also found that RALMs are overconfident in their predictions, making inaccurate predictions more confidently than accurate ones. Scientific knowledge provided either as pretraining or retrieval corpus does not help alleviate this issue. We released our code, data and dashboards at https://github.com/pnnl/EXPERT2.
Automated machine learning (AutoML) accelerates AI development by automating tasks in the development pipeline, such as optimal model search and hyperparameter tuning. Existing AutoML systems often require technical expertise to set up complex tools, which is in general time-consuming and requires a large amount of human effort. Therefore, recent works have started exploiting large language models (LLM) to lessen such burden and increase the usability of AutoML frameworks via a natural language interface, allowing non-expert users to build their data-driven solutions. These methods, however, are usually designed only for a particular process in the AI development pipeline and do not efficiently use the inherent capacity of the LLMs. This paper proposes AutoML-Agent, a novel multi-agent framework tailored for full-pipeline AutoML, i.e., from data retrieval to model deployment. AutoML-Agent takes user's task descriptions, facilitates collaboration between specialized LLM agents, and delivers deployment-ready models. Unlike existing work, instead of devising a single plan, we introduce a retrieval-augmented planning strategy to enhance exploration to search for more optimal plans. We also decompose each plan into sub-tasks (e.g., data preprocessing and neural network design) each of which is solved by a specialized agent we build via prompting executing in parallel, making the search process more efficient. Moreover, we propose a multi-stage verification to verify executed results and guide the code generation LLM in implementing successful solutions. Extensive experiments on seven downstream tasks using fourteen datasets show that AutoML-Agent achieves a higher success rate in automating the full AutoML process, yielding systems with good performance throughout the diverse domains.
Multimodal large language models (MLLMs) are critical for developing general-purpose AI assistants, yet they face growing safety risks. How can we ensure that MLLMs are safely aligned to prevent undesired behaviors such as discrimination, misinformation, or violations of ethical standards? In a further step, we need to explore how to fine-tune MLLMs to enhance reasoning performance while ensuring they satisfy safety constraints. Fundamentally, this can be formulated as a min-max optimization problem. In this study, we propose Safe RLHF-V, the first multimodal safety alignment framework that jointly optimizes helpfulness and safety using separate multimodal reward and cost models within a Lagrangian-based constrained optimization framework. Given that there is a lack of preference datasets that separate helpfulness and safety in multimodal scenarios, we introduce BeaverTails-V, the first open-source dataset with dual preference annotations for helpfulness and safety, along with multi-level safety labels (minor, moderate, severe). Additionally, we design a Multi-level Guardrail System to proactively defend against unsafe queries and adversarial attacks. By applying the Beaver-Guard-V moderation for 5 rounds of filtering and re-generation on the precursor model, the overall safety of the upstream model is significantly improved by an average of 40.9%. Experimental results demonstrate that fine-tuning different MLLMs with Safe RLHF can effectively enhance model helpfulness while ensuring improved safety. Specifically, Safe RLHF-V improves model safety by 34.2% and helpfulness by 34.3%. All of datasets, models, and code can be found at https://github.com/SafeRLHF-V to support the safety development of MLLMs and reduce potential societal risks.
Birds' Eye View (BEV) semantic segmentation is an indispensable perception task in end-to-end autonomous driving systems. Unsupervised and semi-supervised learning for BEV tasks, as pivotal for real-world applications, underperform due to the homogeneous distribution of the labeled data. In this work, we explore the potential of synthetic data from driving world models to enhance the diversity of labeled data for robustifying BEV segmentation. Yet, our preliminary findings reveal that generation noise in synthetic data compromises efficient BEV model learning. To fully harness the potential of synthetic data from world models, this paper proposes NRSeg, a noise-resilient learning framework for BEV semantic segmentation. Specifically, a Perspective-Geometry Consistency Metric (PGCM) is proposed to quantitatively evaluate the guidance capability of generated data for model learning. This metric originates from the alignment measure between the perspective road mask of generated data and the mask projected from the BEV labels. Moreover, a Bi-Distribution Parallel Prediction (BiDPP) is designed to enhance the inherent robustness of the model, where the learning process is constrained through parallel prediction of multinomial and Dirichlet distributions. The former efficiently predicts semantic probabilities, whereas the latter adopts evidential deep learning to realize uncertainty quantification. Furthermore, a Hierarchical Local Semantic Exclusion (HLSE) module is designed to address the non-mutual exclusivity inherent in BEV semantic segmentation tasks. Experimental results demonstrate that NRSeg achieves state-of-the-art performance, yielding the highest improvements in mIoU of 13.8% and 11.4% in unsupervised and semi-supervised BEV segmentation tasks, respectively. The source code will be made publicly available at https://github.com/lynn-yu/NRSeg.
대규모 언어 모델(LLM)은 표준, 네트워크 구성, 수학적 모델, 소스 코드 및 운영 로그를 기반으로 추론하여 광범위한 통신 엔지니어링 작업을 자동화할 수 있는 큰 잠재력을 제공합니다. 그러나 기존 통신 LLM은 이러한 다양한 작업과 데이터 유형 전반에서 안정적으로 추론하는 데 어려움을 겪고 있습니다. 범용 LLM은 통신 특화 지식에 대한 신뢰할 수 있는 그라운딩(grounding)이 부족한 경우가 많으며, 통신 특화 모델은 일반적으로 더 좁은 작업군을 대상으로 개발되어 제한적인 다중 작업 성능을 보입니다. 이러한 격차를 해소하기 위해, 우리는 프로토콜, 지식, 모델링, 장애라는 상호 보완적인 네 가지 축을 중심으로 구조화된 오픈 소스 통합 통신 추론 모델 제품군인 TelecomGPT-R1을 소개합니다. 먼저, 가공되지 않은 공개 통신 산출물을 검증된 질문-답변 쌍과 고품질의 사고의 연쇄(Chain-of-Thought, CoT) 추론 궤적으로 정제하는 축 인식(axis-aware) 데이터 생성 프레임워크를 개발하여 104,880개의 예제를 포함하는 학습 말뭉치를 구축합니다. 이 말뭉치를 기반으로 지도 미세조정(SFT)을 통해 통신 지식과 증거 기반 추론 패턴을 주입함으로써 강화학습(RL)의 콜드 스타트 장벽을 극복합니다. 그런 다음 작업별로 라우팅되는 루브릭 보상을 갖춘 동적 샘플링 정책 최적화(DAPO)를 적용하여 이종 통신 추론 작업 전반에서 RL 업데이트가 유익하고 안정적으로 유지되도록 합니다. 이러한 보상은 축 특화 CoT 추적을 검증 가능한 추론 단위로 분해하고 그라운딩된 조밀한 과정 크레딧(dense process credit)과 결과의 정확성을 결합하여, RL이 검증 가능한 통신 증거로부터 일반화 가능한 문제 해결 행동을 학습할 수 있도록 합니다. 우리는 추가적인 커뮤니티 발전을 지원하기 위해 TelecomGPT-R1 모델과 재현 가능한 학습 레시피를 공개합니다. GSMA Open Telco Leaderboard의 7개 벤치마크 평가 결과, 오픈 소스 TelecomGPT-R1-27B는 89.64%의 평균 점수를 달성하여 GPT-5, Claude, Gemini를 포함한 주요 상용 모델을 능가하는 성능을 보였습니다.
주파수 붕괴 어텐션(Frequency-collapse attention) [Zeris, 2026e]은 Q/K 내적을 학습된 주파수에서의 대역통과 필터링된 내적으로 대체함으로써 표준 점곱 어텐션(dot-product attention) 대비 큰 성능 향상을 달성한다. 자연스럽게 이어지는 후속 질문은 어떤 필터 형태가 가장 우수하며, 그 이유는 무엇인가이다. 본 논문에서는 문자 수준 언어 모델링(TinyShakespeare, 6층 GPT)에 대한 통제된 소거 연구(ablation study)를 통해 직류(DC) 억제, 나이퀴스트(Nyquist) 억제, 대역폭, 중심 주파수, 다중 스케일 커버리지라는 필터 특성에 관한 5가지 가설을 검증한다. 주요 발견은 다음과 같다: (1) DC 및 나이퀴스트 성분은 성능에 명백히 유해하며(val ~= 2.0, 위상 무작위화와 동등), 이는 단순한 저차원 스펙트럼 요약이 아닌 진동성 대역통과 구조가 필수적임을 확인해 준다; (2) 최적의 단일 스케일 대역폭은 문단 스케일(~70개 토큰)을 중심으로 한 sigma ~= 2 빈(bin)으로, BASE-DOT 대비 Delta = +1.15 nats의 뚜렷한 성능 향상을 제공한다; (3) 허용 가능한 필터(zero-mean, 멕시칸 햇 DOG m = 2)는 동일 스케일의 비허용 가우시안 필터보다 성능이 뛰어나며 양방향 FFT 누설에 대해 부분적인 보호를 제공한다; (4) 양방향 FFT 누설은 스펙트럼 커버리지에 따라 단조적으로 스케일링된다(협대역 필터(gap > +4)는 누설이 적고, 광대역 필터(gap < +2)는 누설이 많음); (5) 문자 스케일에서 인과적 시간 도메인 몰렛(Morlet)은 BASE-DOT을 넘어서지 못하며(K=128 탭은 T=256 컨텍스트의 50%를 커버), 이는 자매 논문인 MorletQK [Zeris, 2026f]에서 단어 수준 실험을 진행하는 동기가 된다. 종합적으로, 결과 (1)-(5)는 FourierQK가 학습(training)과 추론(inference) 시점 모두에서 전체 시퀀스 컨텍스트를 활용할 수 있는 양방향 어텐션 설정(인코더 스타일, 예: BERT)에서 효과적임을 보여준다. 자기회귀적 생성에는 MorletQK [Zeris, 2026f](디코더 스타일, 예: GPT)와 같은 인과적 스펙트럼 변형이 필요하다. 코드는 https://github.com/AthanasiosZeris/energy-gated-attention 에서 제공된다.