대규모 언어 모델(LLM)은 점점 더 중요한 의사결정에 활용되고 있습니다. 모델 오류에 따른 파급력이 커짐에 따라, 개별 답변을 얼마나 신뢰할 수 있는가라는 핵심 질문을 더 이상 간과하기 어려워졌습니다. 하지만 언어 모델은 유창한 설명과 권위 있는 어조로 잘못된 결론을 제시할 수 있기 때문에, 언제 답변을 보류해야 하는지 판단하기는 여전히 어렵습니다. 불확실성 정량화(Uncertainty quantification)는 개별 예측의 신뢰도를 추정함으로써 이러한 간극을 해소하고자 합니다. 그러나 기존의 많은 방식은 반복적인 생성이나 별도로 학습된 구성 요소를 필요로 하며, 스칼라 형태의 추정치로는 추론 과정에서 불확실성이 어디서 발생하고 어떻게 전개되는지 드러내지 못합니다. 최근 연구에서는 생성 길이가 불확실성 추정치 및 정답 여부와 강하게 연관될 수 있음이 밝혀져, 추정기의 예측 성능 중 얼마만큼이 출력 길이 자체가 아닌 불확실성 고유의 정보에서 비롯되는가라는 의문이 제기되고 있습니다. 기계론적 해석가능성(Mechanistic interpretability)은 인간이 해석할 수 있는 개념을 중간 모델 상태와 연결함으로써 이러한 한계를 해결할 수 있는 방법을 제시합니다. 이러한 기능을 바탕으로, 본 논문에서는 모델의 변화하는 불확실성을 측정하고 해석할 수 있게 해주는 저차원 부분공간인 U-Space를 소개합니다. 의심과 확신을 나타내는 의미적 앵커를 식별하고, 이들의 언임베딩(unembedding) 방향을 잔차 공간(residual space)으로 다시 매핑한 뒤, 이들의 대조를 직교 기저로 결합합니다. U-Lens는 각 토큰 상태를 이러한 기저 벡터에 투영하여, 직접 검사하거나 단일 스칼라 불확실성 점수로 집계할 수 있는 해석 가능한 토큰 수준 불확실성 맵을 생성합니다. 본 연구의 접근 방식은 정답 라벨, 반복적인 생성 또는 별도의 학습을 필요로 하지 않습니다. 다양한 추론 벤치마크 전반에서 본 방식의 신뢰도 점수는 표준 평가와 길이 제어 평가 모두에서 기존 베이스라인보다 뛰어난 성능을 보였으며, 지도 학습 기반 추정기보다 더 안정적으로 전이됩니다. 코드: https://github.com/s2labres/U-Space.
단계 주석이 달린 대화형 데이터는 미성년자 대상 온라인 그루밍(online grooming)의 진행 과정을 연구하고 조기 경보 도구를 개발하는 데 필수적입니다. 그러나 공개적으로 접근 가능한 유일한 출처인 Perverted-Justice 아카이브(ChatCoder를 사용해 코딩된 부분집합 포함)는 규모가 작고, 오래되었으며, 추가 업데이트가 중단된 상태입니다. 본 연구에서는 기관 내 하드웨어에서 실행되는 검색-증강 파이프라인을 사용해 생성한 그루머-피해자 대화 합성 코퍼스를 제시합니다. 이 파이프라인은 공개적으로 사용 가능한 80억 개 매개변수(parameter) 규모의 거부 완화(refusal-ablated) 언어 모델(language model)을 활용합니다. 해당 코퍼스는 합성 데이터셋을 위한 책임 있는 생성 프로토콜의 5단계를 따라 구축되었습니다: ChatCoder 유인 소통 체계 기반의 온톨로지 주도 설계(4개 그루밍 단계에 매핑된 8개 소통 태그), 다중 모델 프롬프트(prompt) 테스트, 인간 참여형(human-in-the-loop) 감독, 의미 및 특징(feature) 수준의 검증(validation), 모달리티 인식 문서화. 공개된 코퍼스는 7개 태그와 4개 단계에 걸친 2,121개의 고유한 대화로 구성되며, 각각 플루치크(Plutchik) 감정 필드가 함께 제공됩니다. 검증을 통해 코퍼스의 무결성과 어휘 프로파일뿐만 아니라 실제 시드 대화의 재현 여부, 도메인 내 레이블 복원력(태그 정확도: 0.85, 단계 정확도: 0.93), 그리고 8개 태그 수준에서 자동 코딩된 대화(31,640행) 및 3개 구성 개념 수준에서 인간 코더가 코딩한 대화(10,876개 게시글)인 실제 ChatCoder 가해자 대화로의 전이 능력을 정량화했습니다. 어휘 및 문장 임베딩(embedding) 특징에 대해 정확도를 측정했으며, 실제-합성 판별기(discriminator)는 두 도메인을 91%의 정확도로 구분할 수 있었습니다. 본 코퍼스의 활용 범위에는 통제된 단계 조건부 실험, 파이프라인 개발, 실제 데이터 및 인식 제고 자료와 병행한 데이터 증강이 포함됩니다. 코퍼스와 데이터시트, 감사 코드는 심사를 거친 접근 권한 하에 공개됩니다.
The ability to segment unknown objects in depth images has potential to enhance robot skills in grasping and object tracking. Recent computer vision research has demonstrated that Mask R-CNN can be trained to segment specific categories of objects in RGB images when massive hand-labeled datasets are available. As generating these datasets is time consuming, we instead train with synthetic depth images. Many robots now use depth sensors, and recent results suggest training on synthetic depth data can transfer successfully to the real world. We present a method for automated dataset generation and rapidly generate a synthetic training dataset of 50,000 depth images and 320,000 object masks using simulated heaps of 3D CAD models. We train a variant of Mask R-CNN with domain randomization on the generated dataset to perform category-agnostic instance segmentation without any hand-labeled data and we evaluate the trained network, which we refer to as Synthetic Depth (SD) Mask R-CNN, on a set of real, high-resolution depth images of challenging, densely-cluttered bins containing objects with highly-varied geometry. SD Mask R-CNN outperforms point cloud clustering baselines by an absolute 15% in Average Precision and 20% in Average Recall on COCO benchmarks, and achieves performance levels similar to a Mask R-CNN trained on a massive, hand-labeled RGB dataset and fine-tuned on real images from the experimental setup. We deploy the model in an instance-specific grasping pipeline to demonstrate its usefulness in a robotics application. Code, the synthetic training dataset, and supplementary material are available at https://bit.ly/2letCuE.
Spiking Neural Networks (SNNs), the third generation NNs, have come under the spotlight for machine learning based applications due to their biological plausibility and reduced complexity compared to traditional artificial Deep Neural Networks (DNNs). These SNNs can be implemented with extreme energy efficiency on neuromorphic processors like the Intel Loihi research chip, and fed by event-based sensors, such as DVS cameras. However, DNNs with many layers can achieve relatively high accuracy on image classification and recognition tasks, as the research on learning rules for SNNs for real-world applications is still not mature. The accuracy results for SNNs are typically obtained either by converting the trained DNNs into SNNs, or by directly designing and training SNNs in the spiking domain. Towards the conversion from a DNN to an SNN, we perform a comprehensive analysis of such process, specifically designed for Intel Loihi, showing our methodology for the design of an SNN that achieves nearly the same accuracy results as its corresponding DNN. Towards the usage of the event-based sensors, we design a pre-processing method, evaluated for the DvsGesture dataset, which makes it possible to be used in the DNN domain. Hence, based on the outcome of the first analysis, we train a DNN for the pre-processed DvsGesture dataset, and convert it into the spike domain for its deployment on Intel Loihi, which enables real-time gesture recognition. The results show that our SNN achieves 89.64% classification accuracy and occupies only 37 Loihi cores. The source code for generating our experiments is available online at https://github.com/albertomarchisio/EfficientSNN.
Open-set object detection (OSOD) aims to detect the known categories and reject unknown objects in a dynamic world, which has achieved significant attention. However, previous approaches only consider this problem in data-abundant conditions, while neglecting the few-shot scenes. In this paper, we seek a solution for the generalized few-shot open-set object detection (G-FOOD), which aims to avoid detecting unknown classes as known classes with a high confidence score while maintaining the performance of few-shot detection. The main challenge for this task is that few training samples induce the model to overfit on the known classes, resulting in a poor open-set performance. We propose a new G-FOOD algorithm to tackle this issue, named \underline{F}ew-sh\underline{O}t \underline{O}pen-set \underline{D}etector (FOOD), which contains a novel class weight sparsification classifier (CWSC) and a novel unknown decoupling learner (UDL). To prevent over-fitting, CWSC randomly sparses parts of the normalized weights for the logit prediction of all classes, and then decreases the co-adaptability between the class and its neighbors. Alongside, UDL decouples training the unknown class and enables the model to form a compact unknown decision boundary. Thus, the unknown objects can be identified with a confidence probability without any threshold, prototype, or generation. We compare our method with several state-of-the-art OSOD methods in few-shot scenes and observe that our method improves the F-score of unknown classes by 4.80\%-9.08\% across all shots in VOC-COCO dataset settings \footnote[1]{The source code is available at \url{https://github.com/binyisu/food}}.
Objective: To improve performance of medical entity normalization across many languages, especially when fewer language resources are available compared to English. Materials and Methods: We introduce xMEN, a modular system for cross-lingual medical entity normalization, which performs well in both low- and high-resource scenarios. When synonyms in the target language are scarce for a given terminology, we leverage English aliases via cross-lingual candidate generation. For candidate ranking, we incorporate a trainable cross-encoder model if annotations for the target task are available. We also evaluate cross-encoders trained in a weakly supervised manner based on machine-translated datasets from a high resource domain. Our system is publicly available as an extensible Python toolkit. Results: xMEN improves the state-of-the-art performance across a wide range of multilingual benchmark datasets. Weakly supervised cross-encoders are effective when no training data is available for the target task. Through the compatibility of xMEN with the BigBIO framework, it can be easily used with existing and prospective datasets. Discussion: Our experiments show the importance of balancing the output of general-purpose candidate generators with subsequent trainable re-rankers, which we achieve through a rank regularization term in the loss function of the cross-encoder. However, error analysis reveals that multi-word expressions and other complex entities are still challenging. Conclusion: xMEN exhibits strong performance for medical entity normalization in multiple languages, even when no labeled data and few terminology aliases for the target language are available. Its configuration system and evaluation modules enable reproducible benchmarks. Models and code are available online at the following URL: https://github.com/hpi-dhc/xmen
Dynamic analysis methods effectively identify shelled, wrapped, or obfuscated malware, thereby preventing them from invading computers. As a significant representation of dynamic malware behavior, the API (Application Programming Interface) sequence, comprised of consecutive API calls, has progressively become the dominant feature of dynamic analysis methods. Though there have been numerous deep learning models for malware detection based on API sequences, the quality of API call representations produced by those models is limited. These models cannot generate representations for unknown API calls, which weakens both the detection performance and the generalization. Further, the concept drift phenomenon of API calls is prominent. To tackle these issues, we introduce a prompt engineering-assisted malware dynamic analysis using GPT-4. In this method, GPT-4 is employed to create explanatory text for each API call within the API sequence. Afterward, the pre-trained language model BERT is used to obtain the representation of the text, from which we derive the representation of the API sequence. Theoretically, this proposed method is capable of generating representations for all API calls, excluding the necessity for dataset training during the generation process. Utilizing the representation, a CNN-based detection model is designed to extract the feature. We adopt five benchmark datasets to validate the performance of the proposed model. The experimental results reveal that the proposed detection algorithm performs better than the state-of-the-art method (TextCNN). Specifically, in cross-database experiments and few-shot learning experiments, the proposed model achieves excellent detection performance and almost a 100% recall rate for malware, verifying its superior generalization performance. The code is available at: github.com/yan-scnu/Prompted_Dynamic_Detection.
We introduce Bonito, an open-source model for conditional task generation that converts unannotated text into task-specific training datasets for instruction tuning. We aim to enable zero-shot task adaptation of large language models on users' specialized, private data. We train Bonito by fine-tuning a pretrained large language model on a new large-scale dataset with 1.65M examples created by remixing existing instruction tuning datasets into meta-templates. The meta-templates for a dataset produce training examples where the input is the unannotated text and the task attribute and the output consists of the instruction and the response. We use Bonito to generate synthetic tasks for seven datasets from specialized domains with unannotated text across three task types -- yes-no question answering, extractive question answering, and natural language inference -- and adapt language models. We show that Bonito significantly improves the average performance of pretrained and instruction tuned models over the de facto self supervised baseline. For example, adapting Mistral-Instruct-v2 and instruction tuned variants of Mistral and Llama2 with Bonito improves the strong zero-shot performance by 22.1 F1 points whereas the next word prediction objective undoes some of the benefits of instruction tuning and reduces the average performance by 0.8 F1 points. We conduct additional experiments with Bonito to understand the effects of the domain, the size of the training set, and the choice of alternative synthetic task generators. Overall, we show that learning with synthetic instruction tuning datasets is an effective way to adapt language models to new domains. The model, dataset, and code are available at https://github.com/BatsResearch/bonito.
Recent advancements in large language models (LLMs) have significantly advanced the automation of software development tasks, including code synthesis, program repair, and test generation. More recently, researchers and industry practitioners have developed various autonomous LLM agents to perform end-to-end software development tasks. These agents are equipped with the ability to use tools, run commands, observe feedback from the environment, and plan for future actions. However, the complexity of these agent-based approaches, together with the limited abilities of current LLMs, raises the following question: Do we really have to employ complex autonomous software agents? To attempt to answer this question, we build Agentless -- an agentless approach to automatically solve software development problems. Compared to the verbose and complex setup of agent-based approaches, Agentless employs a simplistic three-phase process of localization, repair, and patch validation, without letting the LLM decide future actions or operate with complex tools. Our results on the popular SWE-bench Lite benchmark show that surprisingly the simplistic Agentless is able to achieve both the highest performance (32.00%, 96 correct fixes) and low cost ($0.70) compared with all existing open-source software agents! Furthermore, we manually classified the problems in SWE-bench Lite and found problems with exact ground truth patch or insufficient/misleading issue descriptions. As such, we construct SWE-bench Lite-S by excluding such problematic issues to perform more rigorous evaluation and comparison. Our work highlights the current overlooked potential of a simple, interpretable technique in autonomous software development. We hope Agentless will help reset the baseline, starting point, and horizon for autonomous software agents, and inspire future work along this crucial direction.
In practice, domain shifts are likely to occur between training and test data, necessitating domain adaptation (DA) to adjust the pre-trained source model to the target domain. Recently, universal domain adaptation (UniDA) has gained attention for addressing the possibility of an additional category (label) shift between the source and target domain. This means new classes can appear in the target data, some source classes may no longer be present, or both at the same time. For practical applicability, UniDA methods must handle both source-free and online scenarios, enabling adaptation without access to the source data and performing batch-wise updates in parallel with prediction. In an online setting, preserving knowledge across batches is crucial. However, existing methods often require substantial memory, which is impractical because memory is limited and valuable, in particular on embedded systems. Therefore, we consider memory-efficiency as an additional constraint. To achieve memory-efficient online source-free universal domain adaptation (SF-UniDA), we propose a novel method that continuously captures the distribution of known classes in the feature space using a Gaussian mixture model (GMM). This approach, combined with entropy-based out-of-distribution detection, allows for the generation of reliable pseudo-labels. Finally, we combine a contrastive loss with a KL divergence loss to perform the adaptation. Our approach not only achieves state-of-the-art results in all experiments on the DomainNet and Office-Home datasets but also significantly outperforms the existing methods on the challenging VisDA-C dataset, setting a new benchmark for online SF-UniDA. Our code is available at https://github.com/pascalschlachter/GMM.
Recent years have witnessed the rapid development of Neuro-Symbolic (NeSy) AI systems, which integrate symbolic reasoning into deep neural networks. However, most of the existing benchmarks for NeSy AI fail to provide long-horizon reasoning tasks with complex multi-agent interactions. Furthermore, they are usually constrained by fixed and simplistic logical rules over limited entities, making them far from real-world complexities. To address these crucial gaps, we introduce LogiCity, the first simulator based on customizable first-order logic (FOL) for an urban-like environment with multiple dynamic agents. LogiCity models diverse urban elements using semantic and spatial concepts, such as IsAmbulance(X) and IsClose(X, Y). These concepts are used to define FOL rules that govern the behavior of various agents. Since the concepts and rules are abstractions, they can be universally applied to cities with any agent compositions, facilitating the instantiation of diverse scenarios. Besides, a key feature of LogiCity is its support for user-configurable abstractions, enabling customizable simulation complexities for logical reasoning. To explore various aspects of NeSy AI, LogiCity introduces two tasks, one features long-horizon sequential decision-making, and the other focuses on one-step visual reasoning, varying in difficulty and agent behaviors. Our extensive evaluation reveals the advantage of NeSy frameworks in abstract reasoning. Moreover, we highlight the significant challenges of handling more complex abstractions in long-horizon multi-agent scenarios or under high-dimensional, imbalanced data. With its flexible design, various features, and newly raised challenges, we believe LogiCity represents a pivotal step forward in advancing the next generation of NeSy AI. All the code and data are open-sourced at our website.
Recently, text-guided scalable vector graphics (SVG) synthesis has demonstrated significant potential in domains such as iconography and sketching. However, SVGs generated from existing Text-to-SVG methods often lack editability and exhibit deficiencies in visual quality and diversity. In this paper, we propose a novel text-guided vector graphics synthesis method to address these limitations. To enhance the editability of output SVGs, we introduce a Hierarchical Image VEctorization (HIVE) framework that operates at the semantic object level and supervises the optimization of components within the vector object. This approach facilitates the decoupling of vector graphics into distinct objects and component levels. Our proposed HIVE algorithm, informed by image segmentation priors, not only ensures a more precise representation of vector graphics but also enables fine-grained editing capabilities within vector objects. To improve the diversity of output SVGs, we present a Vectorized Particle-based Score Distillation (VPSD) approach. VPSD addresses over-saturation issues in existing methods and enhances sample diversity. A pre-trained reward model is incorporated to re-weight vector particles, improving aesthetic appeal and enabling faster convergence. Additionally, we design a novel adaptive vector primitives control strategy, which allows for the dynamic adjustment of the number of primitives, thereby enhancing the presentation of graphic details. Extensive experiments validate the effectiveness of the proposed method, demonstrating its superiority over baseline methods in terms of editability, visual quality, and diversity. We also show that our new method supports up to six distinct vector styles, capable of generating high-quality vector assets suitable for stylized vector design and poster design. Code and demo will be released at: http://ximinng.github.io/SVGDreamerV2Project/
Training robust deep video representations has proven to be computationally challenging due to substantial decoding overheads, the enormous size of raw video streams, and their inherent high temporal redundancy. Different from existing schemes, operating exclusively in the compressed video domain and exploiting all freely available modalities, i.e., I-frames, and P-frames (motion vectors and residuals) offers a compute-efficient alternative. Existing methods approach this task as a naive multi-modality problem, ignoring the temporal correlation and implicit sparsity across P-frames for modeling stronger shared representations for videos of the same action, making training and generalization easier. By revisiting the high-level design of dominant video understanding backbones, we increase inference speed by a factor of $56$ while retaining similar performance. For this, we propose a hybrid end-to-end framework that factorizes learning across three key concepts to reduce inference cost by $330\times$ versus prior art: First, a specially designed dual-encoder scheme with efficient Spiking Temporal Modulators to minimize latency while retaining cross-domain feature aggregation. Second, a unified transformer model to capture inter-modal dependencies using global self-attention to enhance I-frame -- P-frame contextual interactions. Third, a Multi-Modal Mixer Block to model rich representations from the joint spatiotemporal token embeddings. Experiments show that our method results in a lightweight architecture achieving state-of-the-art video recognition performance on UCF-101, HMDB-51, K-400, K-600 and SS-v2 datasets with favorable costs ($0.73$J/V) and fast inference ($16$V/s). Our observations bring new insights into practical design choices for efficient next-generation spatiotemporal learners. Code is available.
The memorization of sensitive and personally identifiable information (PII) by large language models (LLMs) poses growing privacy risks as models scale and are increasingly deployed in real-world applications. Existing efforts to study sensitive and PII data memorization and develop mitigation strategies are hampered by the absence of comprehensive, realistic, and ethically sourced datasets reflecting the diversity of sensitive information found on the web. We introduce PANORAMA - Profile-based Assemblage for Naturalistic Online Representation and Attribute Memorization Analysis, a large-scale synthetic corpus of 384,789 samples derived from 9,674 synthetic profiles designed to closely emulate the distribution, variety, and context of PII and sensitive data as it naturally occurs in online environments. Our data generation pipeline begins with the construction of internally consistent, multi-attribute human profiles using constrained selection to reflect real-world demographics such as education, health attributes, financial status, etc. Using a combination of zero-shot prompting and OpenAI o3-mini, we generate diverse content types - including wiki-style articles, social media posts, forum discussions, online reviews, comments, and marketplace listings - each embedding realistic, contextually appropriate PII and other sensitive information. We validate the utility of PANORAMA by fine-tuning the Mistral-7B model on 1x, 5x, 10x, and 25x data replication rates with a subset of data and measure PII memorization rates - revealing not only consistent increases with repetition but also variation across content types, highlighting PANORAMA's ability to model how memorization risks differ by context. Our dataset and code are publicly available, providing a much-needed resource for privacy risk assessment, model auditing, and the development of privacy-preserving LLMs.
Large Reasoning Models (LRMs) have shown promising accuracy improvements on complex problem-solving tasks. While these models have attained high accuracy by leveraging additional computation at test time, they need to generate long chain-of-thought reasoning in order to think before answering, which requires generating thousands of tokens. While sparse attention methods can help reduce the KV cache pressure induced by this long autoregressive reasoning, these methods can introduce errors which disrupt the reasoning process. Additionally, prior methods often pre-process the input to make it easier to identify the important prompt tokens when computing attention during generation, and this pre-processing is challenging to perform online for newly generated reasoning tokens. Our work addresses these challenges by introducing Multipole Attention, which accelerates autoregressive reasoning by only computing exact attention for the most important tokens, while maintaining approximate representations for the remaining tokens. Our method first performs clustering to group together semantically similar key vectors, and then uses the cluster centroids both to identify important key vectors and to approximate the remaining key vectors in order to retain high accuracy. We design a fast cluster update process to quickly re-cluster the input and previously generated tokens, thereby allowing for accelerating attention to the previous output tokens. We evaluate our method using emerging LRMs such as Qwen-8B, demonstrating that our approach can maintain accuracy on complex reasoning tasks even with aggressive attention sparsity settings. We also provide kernel implementations to demonstrate the practical efficiency gains from our method, achieving up to 4.5$\times$ speedup for attention in long-context reasoning applications. Our code is available at https://github.com/SqueezeAILab/MultipoleAttention.
검색 증강 생성(RAG)은 대규모 언어 모델(LLM)이 학습 데이터 범위를 넘어선 사실적 질문에 답하도록 돕지만, 저장 비용과 추론 비용이 추가된다. 이전에 검색된 정보를 파라미터 메모리로 증류하면 온라인 검색 없이 이를 재사용할 수 있다. 그러나 메모리가 검색기를 모방하도록 학습시키면 독립적으로 정확한 예측에 보상이 주어지는데, 이러한 예측이 LLM의 기존 예측과 결합될 때 최종 출력 품질을 반드시 향상시키지는 않는다. 이를 해결하기 위해 본 논문은 LLM이 검색된 텍스트를 오프라인으로 읽어 교사 타깃을 생성하는 ResMem을 제안한다. 이 설계는 k-최근접 이웃(kNN) 데이터스토어에 의존하는 MLP Memory와 같은 기존 파라미터 메모리 접근법과 달리 방대한 토큰 표현을 저장하지 않아도 된다. 더 나아가 LLM을 입력과 함께 특권적 검색 증거를 읽는 오프라인 교사로 활용하여, 파라미터 메모리가 가산적 로짓 잔차를 학습하도록 훈련한다. 이 잔차는 동일한 백본을 공유하는 학생의 예측을 교사 분포에 정렬하며, 자기 증류 갱신은 파라미터 메모리에만 국한된다. Wikipedia 부분집합에서 ResMem은 평가된 kNN 파이프라인 대비 교사 구성을 위해 유지되는 저장 공간을 94.7% 줄인다. 다섯 개의 질의응답 데이터셋에 걸쳐 ResMem은 평균 토큰 F1을 고정된 LLM의 32.83%에서 34.48%로 높인다. 또한 kNN으로 학습한 메모리보다 LLM의 정답을 훼손하는 경우가 21% 적다. 이러한 결과는 검색을 학습 자원으로 간주하고, 검색된 지식이 모델의 향후 예측에 활용되는 방식에 따라 파라미터 메모리를 형성한다는 본 연구의 동기를 뒷받침한다. 코드와 모델 가중치는 https://github.com/SSRand/From-Lookups-to-Lessons 및 https://huggingface.co/RandHan/ResMem-Mistral-7B-v0.3 에서 확인할 수 있다.
투기적 디코딩(speculative decoding)은 타깃 분포를 변경하지 않고 대규모 언어 모델(LLM)의 추론을 가속하지만, 단일 초안 작성(drafting) 구조가 모든 실제 워크로드에서 최적의 성능을 내지는 못합니다. 자기회귀 멀티 토큰 예측(MTP)은 가볍고 안정적인 제안 메커니즘인 반면, 블록 병렬 확산(block-parallel diffusion)은 훨씬 더 긴 후보 시퀀스에 걸쳐 초안 작성 지연 시간을 분산시킵니다. 따라서 더 나은 선택은 출력 분포에 크게 좌우됩니다. 본 논문에서는 세 가지 수준에서 이러한 이종성을 해결하는 MTP 및 블록 병렬 투기적 디코딩 통합 학습 프레임워크인 AngelSpec을 제시합니다. 학습 수준에서는 획일적인 데이터 혼합에 하나의 범용 초안 모델을 맞추기보다 구조와 데이터를 함께 특화(co-specialize)합니다. 즉, MTP 초안 모델은 엔트로피가 높은 개방형 대화를 위한 다양한 대화 데이터로 학습시키고, 블록 확산 초안 모델은 더 길고 예측 가능한 연속 생성을 위한 코드 및 수학 데이터로 학습시킵니다. 아키텍처 수준에서는 하이브리드 타깃 조건화 백본과 이전 토큰 조건화 자기회귀 헤드를 결합한 블록 확산 프레임워크인 DFly를 제안하여, 생성을 병렬로 유지하면서도 타깃 특징 활용도와 블록 내 의존성 모델링을 향상합니다. 추론 수준에서는 수용 길이와 검증 비용이 도메인, 요청, 온라인 부하, 하드웨어에 따라 달라지므로, DFly는 검증을 공유 배치 수준 자원으로 취급합니다. 즉, 요청 전반에 걸쳐 신뢰도가 높은 접두사로 연산을 재할당하고, 기대 효용과 프로파일링된 비용 모델을 결합하여 검증 깊이를 온라인에서 적응적으로 조절합니다. Hy3 시리즈 전반에 걸쳐 DFly는 Hy3-A21B의 평균 수용 길이를 약 30% 증가시켰으며, 4에서 64까지 테스트된 모든 동시성 수준에서 가장 높은 평균 처리량을 달성하여 자기회귀 디코딩 대비 1.98~2.40배의 속도 향상과 DFlash 대비 10.5~11.8% 더 높은 처리량을 기록했습니다. 이러한 방법들의 학습 및 확장을 지원하기 위해 AngelSpec을 공개합니다.
본 연구는 복잡한 과수원 환경에서의 세밀한 로봇 인식을 위해 Ultralytics YOLO27(YOLOv27), YOLO26(YOLOv26), YOLO11(YOLOv11), YOLOv8에 대한 세대 간 아키텍처 및 실험 분석을 제시한다. 초기 단계 과실 해부학 구조에 대한 세밀한 객체 검출 및 인스턴스 분할(instance segmentation)은 제한된 픽셀 점유, green-on-green 유사성, 가림(occlusion), 큰 스케일 변화로 인해 복잡한 과수원 환경에서 여전히 어려운 과제이다. YOLO27은 발표되었으나 공개 구현체와 학습 가능한 분할 모델은 아직 제공되지 않았으므로, 본 연구는 YOLO27에 대해서는 아키텍처 분석을 제공하고, 통제된 실험에서는 YOLOv8, YOLO11, YOLO26을 벤치마크한다. YOLO27 실험은 공개 모델이 제공된 뒤 추가될 예정이다. nano(n), small(s), medium(m), large(l), extra-large(x)의 다섯 가지 모델 스케일을 기존 640 x 640 구성과 소형 객체 중심 960 x 960 구성으로 과실(fruitlet), 꽃받침(calyx), 꽃자루(peduncle)의 검출 및 분할에 대해 평가하여 총 30개의 실험을 수행하였다. YOLO11s-960이 관찰된 최고 mask mAP@50:95(0.402)와 box mAP@50:95(0.426)를 달성했으며, YOLO26s-960은 10.37M 파라미터와 34.1 GFLOPs만으로 각각 0.397과 0.425의 유사한 수치를 달성했다. 꽃자루는 여전히 가장 어려운 클래스였으며, 모델 용량을 늘리는 것이 정확도를 일관되게 향상시키지는 않았다. 전반적으로 소형~중간 규모 YOLO 모델과 소형 객체 중심 학습을 결합하면 정확도와 효율성 사이에 유리한 절충안을 제공하였으며, 세밀한 농업용 로봇 인식을 위한 재현 가능한 벤치마크를 확립한다. 코드, 학습된 모델, 실험 구성은 공개되어 있으며, 다음 Github 링크를 통해 업데이트될 예정이다: https://github.com/rnjnspkt/Optimizing-and-Comparing-Ultralytics-YOLOv26-YOLOv11-and-YOLOv8-for-Small-Object-Detection-and-Seg
GPU 공간 공유는 연산 집약적인 프리필과 메모리 집약적인 디코드처럼 상호보완적인 요구를 갖는 인스턴스를 같은 GPU에 배치하여 비용 효율적인 다중 LLM 서빙을 가능하게 할 것으로 기대된다. 기존 시스템은 연산 자원을 분할하고 공유 메모리 용량을 조율하지만, 이질적인 LLM의 동시 배치된 인스턴스 간 대역폭 경합은 모델링하지 않는다. 이러한 경합 때문에 각 인스턴스의 자체 자원 할당이 변하지 않더라도 그 지연 시간은 함께 실행되는 인스턴스의 메모리 접근 패턴에 좌우된다. 광범위한 측정을 바탕으로, 대역폭 경합이 지연 시간에 미치는 영향은 경합 구간과 비경합 구간을 갖는 시분할 메모리 접근 모델로 포착할 수 있으며, 따라서 전체 지연 시간은 각 구간의 지연 시간 추정치의 시간 가중 평균으로 해석적으로 추정할 수 있음을 확인하였다. 본 논문은 메모리 자원을 관리하고 각 모델의 프리필 및 디코드 단계를 별도 인스턴스로 스케줄링하며 이들의 동시 배치를 배치 결정으로 다루는 경합 인지 다중 LLM 서빙 시스템 JigSLO를 제안한다. 경합 인지 예측기는 단독 프로파일만으로 임의의 동시 배치에서의 인스턴스별 지연 시간을 추정한다. 이러한 예측에 기반하여 배치 알고리즘은 각 인스턴스의 텐서 병렬화, 연산 비율, 배치 구성을 함께 선택하고, 상호보완적인 메모리 접근 패턴을 짝지어 각 모델의 지연 시간 목표 내에서 처리량을 최대화한다. JigSLO의 공유 자원 관리자는 가중치 공유, KV 캐시 풀링, 프리필 인스턴스와 디코드 인스턴스 간 KV 캐시 전송을 통해 용량 측면에서 이를 보완한다. 채팅, 코드 생성, 장문맥 요약을 아우르는 세 데이터셋에서 JigSLO는 최신 기준 시스템보다 SLO 달성률에서 평균 $2.3\times$, 처리량에서 평균 $1.2\times$ 우수하며, 장문맥 워크로드에서는 절반의 GPU로 이들의 최고 달성률에 필적한다.
Many applications today, such as NLP, network analysis, and code analysis, rely on semantically embedding objects into low-dimensional fixed-length vectors. Such embeddings naturally provide a way to perform useful downstream tasks, such as identifying relations among objects or predicting objects for a given context, etc. Unfortunately, the training necessary for accurate embeddings is usually computationally intensive and requires processing large amounts of data. Furthermore, distributing this training is challenging. Most embedding training uses stochastic gradient descent (SGD), an "inherently" sequential algorithm. Prior approaches to parallelizing SGD do not honor these dependencies and thus potentially suffer poor convergence. This paper presents a distributed training framework for a class of applications that use Skip-gram-like models to generate embeddings. We call this class Any2Vec and it includes Word2Vec, DeepWalk, and Node2Vec among others. We first formulate Any2Vec training algorithm as a graph application and leverage the state-of-the-art distributed graph analytics framework, D-Galois. We adapt D-Galois to support dynamic graph generation and repartitioning, and incorporate novel communication optimizations. Finally, we introduce a novel way to combine gradients during distributed training to prevent accuracy loss. We show that our framework, called GraphAny2Vec, matches on a cluster of 32 hosts the accuracy of the state-of-the-art shared-memory implementations of Word2Vec and Vertex2Vec on 1 host, and gives a geo-mean speedup of 12x and 5x respectively. Furthermore, GraphAny2Vec is on average 2x faster than the state-of-the-art distributed Word2Vec implementation, DMTK, on 32 hosts. We also show the superiority of our Gradient Combiner independent of GraphAny2Vec by incorporating it in DMTK, which raises its accuracy by > 30%.