PyTorchKR Papers

“Code Generation”

총 10,223편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2312.08317공식 코드 · Papers with Code

Prompt Engineering-assisted Malware Dynamic Analysis Using GPT-4

Dynamic analysis methods effectively identify shelled, wrapped, or obfuscated malware, thereby preventing them from invading computers. As a significant representation of dynamic malware behavior, the API (Application Programming Interface) sequence, comprised of consecutive API calls, has progressively become the dominant feature of dynamic analysis methods. Though there have been numerous deep learning models for malware detection based on API sequences, the quality of API call representations produced by those models is limited. These models cannot generate representations for unknown API calls, which weakens both the detection performance and the generalization. Further, the concept drift phenomenon of API calls is prominent. To tackle these issues, we introduce a prompt engineering-assisted malware dynamic analysis using GPT-4. In this method, GPT-4 is employed to create explanatory text for each API call within the API sequence. Afterward, the pre-trained language model BERT is used to obtain the representation of the text, from which we derive the representation of the API sequence. Theoretically, this proposed method is capable of generating representations for all API calls, excluding the necessity for dataset training during the generation process. Utilizing the representation, a CNN-based detection model is designed to extract the feature. We adopt five benchmark datasets to validate the performance of the proposed model. The experimental results reveal that the proposed detection algorithm performs better than the state-of-the-art method (TextCNN). Specifically, in cross-database experiments and few-shot learning experiments, the proposed model achieves excellent detection performance and almost a 100% recall rate for malware, verifying its superior generalization performance. The code is available at: github.com/yan-scnu/Prompted_Dynamic_Detection.

arXiv:2402.18334공식 코드 · Papers with Code

Learning to Generate Instruction Tuning Datasets for Zero-Shot Task Adaptation

We introduce Bonito, an open-source model for conditional task generation that converts unannotated text into task-specific training datasets for instruction tuning. We aim to enable zero-shot task adaptation of large language models on users' specialized, private data. We train Bonito by fine-tuning a pretrained large language model on a new large-scale dataset with 1.65M examples created by remixing existing instruction tuning datasets into meta-templates. The meta-templates for a dataset produce training examples where the input is the unannotated text and the task attribute and the output consists of the instruction and the response. We use Bonito to generate synthetic tasks for seven datasets from specialized domains with unannotated text across three task types -- yes-no question answering, extractive question answering, and natural language inference -- and adapt language models. We show that Bonito significantly improves the average performance of pretrained and instruction tuned models over the de facto self supervised baseline. For example, adapting Mistral-Instruct-v2 and instruction tuned variants of Mistral and Llama2 with Bonito improves the strong zero-shot performance by 22.1 F1 points whereas the next word prediction objective undoes some of the benefits of instruction tuning and reduces the average performance by 0.8 F1 points. We conduct additional experiments with Bonito to understand the effects of the domain, the size of the training set, and the choice of alternative synthetic task generators. Overall, we show that learning with synthetic instruction tuning datasets is an effective way to adapt language models to new domains. The model, dataset, and code are available at https://github.com/BatsResearch/bonito.

arXiv:2407.01489공식 코드 · Papers with Code

Agentless: Demystifying LLM-based Software Engineering Agents

Recent advancements in large language models (LLMs) have significantly advanced the automation of software development tasks, including code synthesis, program repair, and test generation. More recently, researchers and industry practitioners have developed various autonomous LLM agents to perform end-to-end software development tasks. These agents are equipped with the ability to use tools, run commands, observe feedback from the environment, and plan for future actions. However, the complexity of these agent-based approaches, together with the limited abilities of current LLMs, raises the following question: Do we really have to employ complex autonomous software agents? To attempt to answer this question, we build Agentless -- an agentless approach to automatically solve software development problems. Compared to the verbose and complex setup of agent-based approaches, Agentless employs a simplistic three-phase process of localization, repair, and patch validation, without letting the LLM decide future actions or operate with complex tools. Our results on the popular SWE-bench Lite benchmark show that surprisingly the simplistic Agentless is able to achieve both the highest performance (32.00%, 96 correct fixes) and low cost ($0.70) compared with all existing open-source software agents! Furthermore, we manually classified the problems in SWE-bench Lite and found problems with exact ground truth patch or insufficient/misleading issue descriptions. As such, we construct SWE-bench Lite-S by excluding such problematic issues to perform more rigorous evaluation and comparison. Our work highlights the current overlooked potential of a simple, interpretable technique in autonomous software development. We hope Agentless will help reset the baseline, starting point, and horizon for autonomous software agents, and inspire future work along this crucial direction.

arXiv:2407.14208공식 코드 · Papers with Code

Memory-Efficient Pseudo-Labeling for Online Source-Free Universal Domain Adaptation using a Gaussian Mixture Model

In practice, domain shifts are likely to occur between training and test data, necessitating domain adaptation (DA) to adjust the pre-trained source model to the target domain. Recently, universal domain adaptation (UniDA) has gained attention for addressing the possibility of an additional category (label) shift between the source and target domain. This means new classes can appear in the target data, some source classes may no longer be present, or both at the same time. For practical applicability, UniDA methods must handle both source-free and online scenarios, enabling adaptation without access to the source data and performing batch-wise updates in parallel with prediction. In an online setting, preserving knowledge across batches is crucial. However, existing methods often require substantial memory, which is impractical because memory is limited and valuable, in particular on embedded systems. Therefore, we consider memory-efficiency as an additional constraint. To achieve memory-efficient online source-free universal domain adaptation (SF-UniDA), we propose a novel method that continuously captures the distribution of known classes in the feature space using a Gaussian mixture model (GMM). This approach, combined with entropy-based out-of-distribution detection, allows for the generation of reliable pseudo-labels. Finally, we combine a contrastive loss with a KL divergence loss to perform the adaptation. Our approach not only achieves state-of-the-art results in all experiments on the DomainNet and Office-Home datasets but also significantly outperforms the existing methods on the challenging VisDA-C dataset, setting a new benchmark for online SF-UniDA. Our code is available at https://github.com/pascalschlachter/GMM.

arXiv:2411.00773공식 코드 · Papers with Code

LogiCity: Advancing Neuro-Symbolic AI with Abstract Urban Simulation

Recent years have witnessed the rapid development of Neuro-Symbolic (NeSy) AI systems, which integrate symbolic reasoning into deep neural networks. However, most of the existing benchmarks for NeSy AI fail to provide long-horizon reasoning tasks with complex multi-agent interactions. Furthermore, they are usually constrained by fixed and simplistic logical rules over limited entities, making them far from real-world complexities. To address these crucial gaps, we introduce LogiCity, the first simulator based on customizable first-order logic (FOL) for an urban-like environment with multiple dynamic agents. LogiCity models diverse urban elements using semantic and spatial concepts, such as IsAmbulance(X) and IsClose(X, Y). These concepts are used to define FOL rules that govern the behavior of various agents. Since the concepts and rules are abstractions, they can be universally applied to cities with any agent compositions, facilitating the instantiation of diverse scenarios. Besides, a key feature of LogiCity is its support for user-configurable abstractions, enabling customizable simulation complexities for logical reasoning. To explore various aspects of NeSy AI, LogiCity introduces two tasks, one features long-horizon sequential decision-making, and the other focuses on one-step visual reasoning, varying in difficulty and agent behaviors. Our extensive evaluation reveals the advantage of NeSy frameworks in abstract reasoning. Moreover, we highlight the significant challenges of handling more complex abstractions in long-horizon multi-agent scenarios or under high-dimensional, imbalanced data. With its flexible design, various features, and newly raised challenges, we believe LogiCity represents a pivotal step forward in advancing the next generation of NeSy AI. All the code and data are open-sourced at our website.

arXiv:2411.17832

SVGDreamer++: Advancing Editability and Diversity in Text-Guided SVG Generation

Recently, text-guided scalable vector graphics (SVG) synthesis has demonstrated significant potential in domains such as iconography and sketching. However, SVGs generated from existing Text-to-SVG methods often lack editability and exhibit deficiencies in visual quality and diversity. In this paper, we propose a novel text-guided vector graphics synthesis method to address these limitations. To enhance the editability of output SVGs, we introduce a Hierarchical Image VEctorization (HIVE) framework that operates at the semantic object level and supervises the optimization of components within the vector object. This approach facilitates the decoupling of vector graphics into distinct objects and component levels. Our proposed HIVE algorithm, informed by image segmentation priors, not only ensures a more precise representation of vector graphics but also enables fine-grained editing capabilities within vector objects. To improve the diversity of output SVGs, we present a Vectorized Particle-based Score Distillation (VPSD) approach. VPSD addresses over-saturation issues in existing methods and enhances sample diversity. A pre-trained reward model is incorporated to re-weight vector particles, improving aesthetic appeal and enabling faster convergence. Additionally, we design a novel adaptive vector primitives control strategy, which allows for the dynamic adjustment of the number of primitives, thereby enhancing the presentation of graphic details. Extensive experiments validate the effectiveness of the proposed method, demonstrating its superiority over baseline methods in terms of editability, visual quality, and diversity. We also show that our new method supports up to six distinct vector styles, capable of generating high-quality vector assets suitable for stylized vector design and poster design. Code and demo will be released at: http://ximinng.github.io/SVGDreamerV2Project/

arXiv:2503.13724

Towards Scalable Modeling of Compressed Videos for Efficient Action Recognition

Training robust deep video representations has proven to be computationally challenging due to substantial decoding overheads, the enormous size of raw video streams, and their inherent high temporal redundancy. Different from existing schemes, operating exclusively in the compressed video domain and exploiting all freely available modalities, i.e., I-frames, and P-frames (motion vectors and residuals) offers a compute-efficient alternative. Existing methods approach this task as a naive multi-modality problem, ignoring the temporal correlation and implicit sparsity across P-frames for modeling stronger shared representations for videos of the same action, making training and generalization easier. By revisiting the high-level design of dominant video understanding backbones, we increase inference speed by a factor of $56$ while retaining similar performance. For this, we propose a hybrid end-to-end framework that factorizes learning across three key concepts to reduce inference cost by $330\times$ versus prior art: First, a specially designed dual-encoder scheme with efficient Spiking Temporal Modulators to minimize latency while retaining cross-domain feature aggregation. Second, a unified transformer model to capture inter-modal dependencies using global self-attention to enhance I-frame -- P-frame contextual interactions. Third, a Multi-Modal Mixer Block to model rich representations from the joint spatiotemporal token embeddings. Experiments show that our method results in a lightweight architecture achieving state-of-the-art video recognition performance on UCF-101, HMDB-51, K-400, K-600 and SS-v2 datasets with favorable costs ($0.73$J/V) and fast inference ($16$V/s). Our observations bring new insights into practical design choices for efficient next-generation spatiotemporal learners. Code is available.

arXiv:2505.12238공식 코드 · Papers with Code

PANORAMA: A synthetic PII-laced dataset for studying sensitive data memorization in LLMs

The memorization of sensitive and personally identifiable information (PII) by large language models (LLMs) poses growing privacy risks as models scale and are increasingly deployed in real-world applications. Existing efforts to study sensitive and PII data memorization and develop mitigation strategies are hampered by the absence of comprehensive, realistic, and ethically sourced datasets reflecting the diversity of sensitive information found on the web. We introduce PANORAMA - Profile-based Assemblage for Naturalistic Online Representation and Attribute Memorization Analysis, a large-scale synthetic corpus of 384,789 samples derived from 9,674 synthetic profiles designed to closely emulate the distribution, variety, and context of PII and sensitive data as it naturally occurs in online environments. Our data generation pipeline begins with the construction of internally consistent, multi-attribute human profiles using constrained selection to reflect real-world demographics such as education, health attributes, financial status, etc. Using a combination of zero-shot prompting and OpenAI o3-mini, we generate diverse content types - including wiki-style articles, social media posts, forum discussions, online reviews, comments, and marketplace listings - each embedding realistic, contextually appropriate PII and other sensitive information. We validate the utility of PANORAMA by fine-tuning the Mistral-7B model on 1x, 5x, 10x, and 25x data replication rates with a subset of data and measure PII memorization rates - revealing not only consistent increases with repetition but also variation across content types, highlighting PANORAMA's ability to model how memorization risks differ by context. Our dataset and code are publicly available, providing a much-needed resource for privacy risk assessment, model auditing, and the development of privacy-preserving LLMs.

arXiv:2506.13059

Multipole Attention for Efficient Long Context Reasoning

Large Reasoning Models (LRMs) have shown promising accuracy improvements on complex problem-solving tasks. While these models have attained high accuracy by leveraging additional computation at test time, they need to generate long chain-of-thought reasoning in order to think before answering, which requires generating thousands of tokens. While sparse attention methods can help reduce the KV cache pressure induced by this long autoregressive reasoning, these methods can introduce errors which disrupt the reasoning process. Additionally, prior methods often pre-process the input to make it easier to identify the important prompt tokens when computing attention during generation, and this pre-processing is challenging to perform online for newly generated reasoning tokens. Our work addresses these challenges by introducing Multipole Attention, which accelerates autoregressive reasoning by only computing exact attention for the most important tokens, while maintaining approximate representations for the remaining tokens. Our method first performs clustering to group together semantically similar key vectors, and then uses the cluster centroids both to identify important key vectors and to approximate the remaining key vectors in order to retain high accuracy. We design a fast cluster update process to quickly re-cluster the input and previously generated tokens, thereby allowing for accelerating attention to the previous output tokens. We evaluate our method using emerging LRMs such as Qwen-8B, demonstrating that our approach can maintain accuracy on complex reasoning tasks even with aggressive attention sparsity settings. We also provide kernel implementations to demonstrate the practical efficiency gains from our method, achieving up to 4.5$\times$ speedup for attention in long-context reasoning applications. Our code is available at https://github.com/SqueezeAILab/MultipoleAttention.

arXiv:2611.00450v1cs.CL

조회에서 교훈으로: 검색된 지식을 파라미터 메모리로 전환하기

From Lookups to Lessons: Turning Retrieved Knowledge into Parametric Memory

검색 증강 생성(RAG)은 대규모 언어 모델(LLM)이 학습 데이터 범위를 넘어선 사실적 질문에 답하도록 돕지만, 저장 비용과 추론 비용이 추가된다. 이전에 검색된 정보를 파라미터 메모리로 증류하면 온라인 검색 없이 이를 재사용할 수 있다. 그러나 메모리가 검색기를 모방하도록 학습시키면 독립적으로 정확한 예측에 보상이 주어지는데, 이러한 예측이 LLM의 기존 예측과 결합될 때 최종 출력 품질을 반드시 향상시키지는 않는다. 이를 해결하기 위해 본 논문은 LLM이 검색된 텍스트를 오프라인으로 읽어 교사 타깃을 생성하는 ResMem을 제안한다. 이 설계는 k-최근접 이웃(kNN) 데이터스토어에 의존하는 MLP Memory와 같은 기존 파라미터 메모리 접근법과 달리 방대한 토큰 표현을 저장하지 않아도 된다. 더 나아가 LLM을 입력과 함께 특권적 검색 증거를 읽는 오프라인 교사로 활용하여, 파라미터 메모리가 가산적 로짓 잔차를 학습하도록 훈련한다. 이 잔차는 동일한 백본을 공유하는 학생의 예측을 교사 분포에 정렬하며, 자기 증류 갱신은 파라미터 메모리에만 국한된다. Wikipedia 부분집합에서 ResMem은 평가된 kNN 파이프라인 대비 교사 구성을 위해 유지되는 저장 공간을 94.7% 줄인다. 다섯 개의 질의응답 데이터셋에 걸쳐 ResMem은 평균 토큰 F1을 고정된 LLM의 32.83%에서 34.48%로 높인다. 또한 kNN으로 학습한 메모리보다 LLM의 정답을 훼손하는 경우가 21% 적다. 이러한 결과는 검색을 학습 자원으로 간주하고, 검색된 지식이 모델의 향후 예측에 활용되는 방식에 따라 파라미터 메모리를 형성한다는 본 연구의 동기를 뒷받침한다. 코드와 모델 가중치는 https://github.com/SSRand/From-Lookups-to-Lessons 및 https://huggingface.co/RandHan/ResMem-Mistral-7B-v0.3 에서 확인할 수 있다.

arXiv:2607.25852v2cs.CL

AngelSpec: 투기적 디코딩을 통한 실환경 고성능 추론을 향하여

AngelSpec: Towards Real-World High Performance Inference with Speculative Decoding

투기적 디코딩(speculative decoding)은 타깃 분포를 변경하지 않고 대규모 언어 모델(LLM)의 추론을 가속하지만, 단일 초안 작성(drafting) 구조가 모든 실제 워크로드에서 최적의 성능을 내지는 못합니다. 자기회귀 멀티 토큰 예측(MTP)은 가볍고 안정적인 제안 메커니즘인 반면, 블록 병렬 확산(block-parallel diffusion)은 훨씬 더 긴 후보 시퀀스에 걸쳐 초안 작성 지연 시간을 분산시킵니다. 따라서 더 나은 선택은 출력 분포에 크게 좌우됩니다. 본 논문에서는 세 가지 수준에서 이러한 이종성을 해결하는 MTP 및 블록 병렬 투기적 디코딩 통합 학습 프레임워크인 AngelSpec을 제시합니다. 학습 수준에서는 획일적인 데이터 혼합에 하나의 범용 초안 모델을 맞추기보다 구조와 데이터를 함께 특화(co-specialize)합니다. 즉, MTP 초안 모델은 엔트로피가 높은 개방형 대화를 위한 다양한 대화 데이터로 학습시키고, 블록 확산 초안 모델은 더 길고 예측 가능한 연속 생성을 위한 코드 및 수학 데이터로 학습시킵니다. 아키텍처 수준에서는 하이브리드 타깃 조건화 백본과 이전 토큰 조건화 자기회귀 헤드를 결합한 블록 확산 프레임워크인 DFly를 제안하여, 생성을 병렬로 유지하면서도 타깃 특징 활용도와 블록 내 의존성 모델링을 향상합니다. 추론 수준에서는 수용 길이와 검증 비용이 도메인, 요청, 온라인 부하, 하드웨어에 따라 달라지므로, DFly는 검증을 공유 배치 수준 자원으로 취급합니다. 즉, 요청 전반에 걸쳐 신뢰도가 높은 접두사로 연산을 재할당하고, 기대 효용과 프로파일링된 비용 모델을 결합하여 검증 깊이를 온라인에서 적응적으로 조절합니다. Hy3 시리즈 전반에 걸쳐 DFly는 Hy3-A21B의 평균 수용 길이를 약 30% 증가시켰으며, 4에서 64까지 테스트된 모든 동시성 수준에서 가장 높은 평균 처리량을 달성하여 자기회귀 디코딩 대비 1.98~2.40배의 속도 향상과 DFlash 대비 10.5~11.8% 더 높은 처리량을 기록했습니다. 이러한 방법들의 학습 및 확장을 지원하기 위해 AngelSpec을 공개합니다.

arXiv:2608.23636v2cs.CV

복잡한 과수원 환경에서 세밀한 소형 객체 검출 및 분할을 위한 YOLO27, YOLO26, YOLO11, YOLOv8 최적화

Optimizing YOLO27, YOLO26, YOLO11, and YOLOv8 for Fine-Grained Small-Object Detection and Segmentation in Complex Orchard Environments

본 연구는 복잡한 과수원 환경에서의 세밀한 로봇 인식을 위해 Ultralytics YOLO27(YOLOv27), YOLO26(YOLOv26), YOLO11(YOLOv11), YOLOv8에 대한 세대 간 아키텍처 및 실험 분석을 제시한다. 초기 단계 과실 해부학 구조에 대한 세밀한 객체 검출 및 인스턴스 분할(instance segmentation)은 제한된 픽셀 점유, green-on-green 유사성, 가림(occlusion), 큰 스케일 변화로 인해 복잡한 과수원 환경에서 여전히 어려운 과제이다. YOLO27은 발표되었으나 공개 구현체와 학습 가능한 분할 모델은 아직 제공되지 않았으므로, 본 연구는 YOLO27에 대해서는 아키텍처 분석을 제공하고, 통제된 실험에서는 YOLOv8, YOLO11, YOLO26을 벤치마크한다. YOLO27 실험은 공개 모델이 제공된 뒤 추가될 예정이다. nano(n), small(s), medium(m), large(l), extra-large(x)의 다섯 가지 모델 스케일을 기존 640 x 640 구성과 소형 객체 중심 960 x 960 구성으로 과실(fruitlet), 꽃받침(calyx), 꽃자루(peduncle)의 검출 및 분할에 대해 평가하여 총 30개의 실험을 수행하였다. YOLO11s-960이 관찰된 최고 mask mAP@50:95(0.402)와 box mAP@50:95(0.426)를 달성했으며, YOLO26s-960은 10.37M 파라미터와 34.1 GFLOPs만으로 각각 0.397과 0.425의 유사한 수치를 달성했다. 꽃자루는 여전히 가장 어려운 클래스였으며, 모델 용량을 늘리는 것이 정확도를 일관되게 향상시키지는 않았다. 전반적으로 소형~중간 규모 YOLO 모델과 소형 객체 중심 학습을 결합하면 정확도와 효율성 사이에 유리한 절충안을 제공하였으며, 세밀한 농업용 로봇 인식을 위한 재현 가능한 벤치마크를 확립한다. 코드, 학습된 모델, 실험 구성은 공개되어 있으며, 다음 Github 링크를 통해 업데이트될 예정이다: https://github.com/rnjnspkt/Optimizing-and-Comparing-Ultralytics-YOLOv26-YOLOv11-and-YOLOv8-for-Small-Object-Detection-and-Seg

arXiv:2509.06261v3cs.DC

JigSLO: 효율적인 다중 LLM 서빙을 위한 자원 퍼즐 해결

JigSLO: Solving the Resource Puzzle for Efficient Multi-LLM Serving

GPU 공간 공유는 연산 집약적인 프리필과 메모리 집약적인 디코드처럼 상호보완적인 요구를 갖는 인스턴스를 같은 GPU에 배치하여 비용 효율적인 다중 LLM 서빙을 가능하게 할 것으로 기대된다. 기존 시스템은 연산 자원을 분할하고 공유 메모리 용량을 조율하지만, 이질적인 LLM의 동시 배치된 인스턴스 간 대역폭 경합은 모델링하지 않는다. 이러한 경합 때문에 각 인스턴스의 자체 자원 할당이 변하지 않더라도 그 지연 시간은 함께 실행되는 인스턴스의 메모리 접근 패턴에 좌우된다. 광범위한 측정을 바탕으로, 대역폭 경합이 지연 시간에 미치는 영향은 경합 구간과 비경합 구간을 갖는 시분할 메모리 접근 모델로 포착할 수 있으며, 따라서 전체 지연 시간은 각 구간의 지연 시간 추정치의 시간 가중 평균으로 해석적으로 추정할 수 있음을 확인하였다. 본 논문은 메모리 자원을 관리하고 각 모델의 프리필 및 디코드 단계를 별도 인스턴스로 스케줄링하며 이들의 동시 배치를 배치 결정으로 다루는 경합 인지 다중 LLM 서빙 시스템 JigSLO를 제안한다. 경합 인지 예측기는 단독 프로파일만으로 임의의 동시 배치에서의 인스턴스별 지연 시간을 추정한다. 이러한 예측에 기반하여 배치 알고리즘은 각 인스턴스의 텐서 병렬화, 연산 비율, 배치 구성을 함께 선택하고, 상호보완적인 메모리 접근 패턴을 짝지어 각 모델의 지연 시간 목표 내에서 처리량을 최대화한다. JigSLO의 공유 자원 관리자는 가중치 공유, KV 캐시 풀링, 프리필 인스턴스와 디코드 인스턴스 간 KV 캐시 전송을 통해 용량 측면에서 이를 보완한다. 채팅, 코드 생성, 장문맥 요약을 아우르는 세 데이터셋에서 JigSLO는 최신 기준 시스템보다 SLO 달성률에서 평균 $2.3\times$, 처리량에서 평균 $1.2\times$ 우수하며, 장문맥 워크로드에서는 절반의 GPU로 이들의 최고 달성률에 필적한다.

arXiv:2611.25873v1cs.CV

WHU-Road3D: A dataset and benchmark towards fine-grained road surface mapping

Accurately extracting and mapping road surface elements is crucial for applications such as high-definition maps, road asset management, and intelligent transportation systems. This study first provides a comprehensive review of the benchmark datasets for extracting and mapping road elements, encompassing overhead-to-perspective data, large-to-local scales, and coarse-to-fine details. Subsequently, we thoroughly review the relevant approaches for extracting road surface elements. However, we argue that existing datasets still require enhancements in semantic richness, data modality diversity, and scenario complexity. To bridge these gaps, this paper presents WHU-Road3D, a large-scale multimodal dataset for fine-grained road surface mapping to address these shortcomings. WHU-Road3D provides approximately 100 km of diverse Chinese highways and urban streets. It features a rich combination of MLS point clouds, associated panoramic images, vehicle trajectories, detailed vectorized annotations, and pointwise instance annotations for road surface elements. The dataset carefully annotates 15 distinct categories of road elements, including common markings like road boundaries, zebra crossings, lane dividers, stop lines, as well as 10 types of direction symbols. To our knowledge, WHU-Road3D stands as the first multimodal dataset for mapping such a comprehensive range of road surface elements, designed to facilitate cutting-edge research in instance segmentation, full-element extraction and mapping on road surfaces. Beyond these immediate applications, WHU-Road3D also provides a valuable source for hierarchical road topology reasoning, map updating, and street scene generation. The dataset, along with the code for our benchmark methods, will be made publicly available at \href{https://github.com/WHU-USI3DV/WHU-Road3D.git}{https://github.com/WHU-USI3DV/WHU-Road3D.git}.

arXiv:1909.03359

Distributed Training of Embeddings using Graph Analytics

Many applications today, such as NLP, network analysis, and code analysis, rely on semantically embedding objects into low-dimensional fixed-length vectors. Such embeddings naturally provide a way to perform useful downstream tasks, such as identifying relations among objects or predicting objects for a given context, etc. Unfortunately, the training necessary for accurate embeddings is usually computationally intensive and requires processing large amounts of data. Furthermore, distributing this training is challenging. Most embedding training uses stochastic gradient descent (SGD), an "inherently" sequential algorithm. Prior approaches to parallelizing SGD do not honor these dependencies and thus potentially suffer poor convergence. This paper presents a distributed training framework for a class of applications that use Skip-gram-like models to generate embeddings. We call this class Any2Vec and it includes Word2Vec, DeepWalk, and Node2Vec among others. We first formulate Any2Vec training algorithm as a graph application and leverage the state-of-the-art distributed graph analytics framework, D-Galois. We adapt D-Galois to support dynamic graph generation and repartitioning, and incorporate novel communication optimizations. Finally, we introduce a novel way to combine gradients during distributed training to prevent accuracy loss. We show that our framework, called GraphAny2Vec, matches on a cluster of 32 hosts the accuracy of the state-of-the-art shared-memory implementations of Word2Vec and Vertex2Vec on 1 host, and gives a geo-mean speedup of 12x and 5x respectively. Furthermore, GraphAny2Vec is on average 2x faster than the state-of-the-art distributed Word2Vec implementation, DMTK, on 32 hosts. We also show the superiority of our Gradient Combiner independent of GraphAny2Vec by incorporating it in DMTK, which raises its accuracy by > 30%.

arXiv:2006.07991공식 코드 · Papers with Code

Emergent Properties of Foveated Perceptual Systems

The goal of this work is to characterize the representational impact that foveation operations have for machine vision systems, inspired by the foveated human visual system, which has higher acuity at the center of gaze and texture-like encoding in the periphery. To do so, we introduce models consisting of a first-stage \textit{fixed} image transform followed by a second-stage \textit{learnable} convolutional neural network, and we varied the first stage component. The primary model has a foveated-textural input stage, which we compare to a model with foveated-blurred input and a model with spatially-uniform blurred input (both matched for perceptual compression), and a final reference model with minimal input-based compression. We find that: 1) the foveated-texture model shows similar scene classification accuracy as the reference model despite its compressed input, with greater i.i.d. generalization than the other models; 2) the foveated-texture model has greater sensitivity to high-spatial frequency information and greater robustness to occlusion, w.r.t the comparison models; 3) both the foveated systems, show a stronger center image-bias relative to the spatially-uniform systems even with a weight sharing constraint. Critically, these results are preserved over different classical CNN architectures throughout their learning dynamics. Altogether, this suggests that foveation with peripheral texture-based computations yields an efficient, distinct, and robust representational format of scene information, and provides symbiotic computational insight into the representational consequences that texture-based peripheral encoding may have for processing in the human visual system, while also potentially inspiring the next generation of computer vision models via spatially-adaptive computation. Code + Data available here: https://github.com/ArturoDeza/EmergentProperties

arXiv:2308.09804공식 코드 · Papers with Code

VL-PET: Vision-and-Language Parameter-Efficient Tuning via Granularity Control

As the model size of pre-trained language models (PLMs) grows rapidly, full fine-tuning becomes prohibitively expensive for model training and storage. In vision-and-language (VL), parameter-efficient tuning (PET) techniques are proposed to integrate modular modifications (e.g., Adapter and LoRA) into encoder-decoder PLMs. By tuning a small set of trainable parameters, these techniques perform on par with full fine-tuning. However, excessive modular modifications and neglecting the functionality gap between the encoders and decoders can lead to performance degradation, while existing PET techniques (e.g., VL-Adapter) overlook these critical issues. In this paper, we propose a Vision-and-Language Parameter-Efficient Tuning (VL-PET) framework to impose effective control over modular modifications via a novel granularity-controlled mechanism. Considering different granularity-controlled matrices generated by this mechanism, a variety of model-agnostic VL-PET modules can be instantiated from our framework for better efficiency and effectiveness trade-offs. We further propose lightweight PET module designs to enhance VL alignment and modeling for the encoders and maintain text generation for the decoders. Extensive experiments conducted on four image-text tasks and four video-text tasks demonstrate the efficiency, effectiveness and transferability of our VL-PET framework. In particular, our VL-PET-large with lightweight PET module designs significantly outperforms VL-Adapter by 2.92% (3.41%) and LoRA by 3.37% (7.03%) with BART-base (T5-base) on image-text tasks. Furthermore, we validate the enhanced effect of employing our VL-PET designs on existing PET techniques, enabling them to achieve significant performance improvements. Our code is available at https://github.com/HenryHZY/VL-PET.

arXiv:2404.17136공식 코드 · Papers with Code

Automated Data Visualization from Natural Language via Large Language Models: An Exploratory Study

The Natural Language to Visualization (NL2Vis) task aims to transform natural-language descriptions into visual representations for a grounded table, enabling users to gain insights from vast amounts of data. Recently, many deep learning-based approaches have been developed for NL2Vis. Despite the considerable efforts made by these approaches, challenges persist in visualizing data sourced from unseen databases or spanning multiple tables. Taking inspiration from the remarkable generation capabilities of Large Language Models (LLMs), this paper conducts an empirical study to evaluate their potential in generating visualizations, and explore the effectiveness of in-context learning prompts for enhancing this task. In particular, we first explore the ways of transforming structured tabular data into sequential text prompts, as to feed them into LLMs and analyze which table content contributes most to the NL2Vis. Our findings suggest that transforming structured tabular data into programs is effective, and it is essential to consider the table schema when formulating prompts. Furthermore, we evaluate two types of LLMs: finetuned models (e.g., T5-Small) and inference-only models (e.g., GPT-3.5), against state-of-the-art methods, using the NL2Vis benchmarks (i.e., nvBench). The experimental results reveal that LLMs outperform baselines, with inference-only models consistently exhibiting performance improvements, at times even surpassing fine-tuned models when provided with certain few-shot demonstrations through in-context learning. Finally, we analyze when the LLMs fail in NL2Vis, and propose to iteratively update the results using strategies such as chain-of-thought, role-playing, and code-interpreter. The experimental results confirm the efficacy of iterative updates and hold great potential for future study.

arXiv:2409.08264공식 코드 · Papers with Code

Windows Agent Arena: Evaluating Multi-Modal OS Agents at Scale

Large language models (LLMs) show remarkable potential to act as computer agents, enhancing human productivity and software accessibility in multi-modal tasks that require planning and reasoning. However, measuring agent performance in realistic environments remains a challenge since: (i) most benchmarks are limited to specific modalities or domains (e.g. text-only, web navigation, Q&A, coding) and (ii) full benchmark evaluations are slow (on order of magnitude of days) given the multi-step sequential nature of tasks. To address these challenges, we introduce the Windows Agent Arena: a reproducible, general environment focusing exclusively on the Windows operating system (OS) where agents can operate freely within a real Windows OS and use the same wide range of applications, tools, and web browsers available to human users when solving tasks. We adapt the OSWorld framework (Xie et al., 2024) to create 150+ diverse Windows tasks across representative domains that require agent abilities in planning, screen understanding, and tool usage. Our benchmark is scalable and can be seamlessly parallelized in Azure for a full benchmark evaluation in as little as 20 minutes. To demonstrate Windows Agent Arena's capabilities, we also introduce a new multi-modal agent, Navi. Our agent achieves a success rate of 19.5% in the Windows domain, compared to 74.5% performance of an unassisted human. Navi also demonstrates strong performance on another popular web-based benchmark, Mind2Web. We offer extensive quantitative and qualitative analysis of Navi's performance, and provide insights into the opportunities for future research in agent development and data generation using Windows Agent Arena. Webpage: https://microsoft.github.io/WindowsAgentArena Code: https://github.com/microsoft/WindowsAgentArena

arXiv:2411.07040

FlexiGen: Stochastic Dataset Generator for Electric Vehicle Charging Energy Flexibility

Electric vehicles (EVs) and renewable energy sources (RES) are vital components of sustainable energy systems, yet their uncoordinated integration can pose substantial challenges to grid stability, such as unmanaged peak loads and energy balance issues. Vehicle-to-Grid (V2G), offer a promising solution to address these challenges by enabling bidirectional energy flow between EVs and the grid. As such, EVs can be used in advances Demand Response (DR) strategies to optimize energy use and mitigate the intermittency of renewable generation. To reach such advantages, optimization algorithms need data on EV energy flexibility, such as charging patterns and usage preferences. However, data collection remains constrained by challenges such as high costs, user engagement, data privacy concerns, and limited access to open-source datasets on EV energy flexibility. This paper presents FlexiGen an open-source stochastic dataset generator tool designed to overcome the data limitations in EV flexibility for V2G and V1G DR applications. FlexiGen generates synthetic datasets encompassing realist EV usage patterns, behaviours and flexibility scenarios for household and office routines. To generate these datasets, FlexiGen uses a series of configurable probabilistic variables, such as stochastic user routines, traffic conditions, charger types, car average electricity consumption and State of Charge (SoC). The generated datasets include an hourly routine with the EV State of connection, Destination Charger, Estimated Departure Time, Required SOC at Departure, Estimated Arrival Time, and Estimated SOC at Arrival. Accompanying this publication an example dataset is generated for 3 households with 1 EV each, and 1 office building with 3 EVs. The generated dataset is analyzed and discussed on the paper and published alongside the open-source code for FlexiGen tool.