PyTorchKR Papers

최신 논문

총 3,273편
태그·기간 등 상세 필터

카테고리 탐색

로그인 후 이 태그 구독

arXiv:2502.07115

Online Scheduling for LLM Inference with KV Cache Constraints

Large Language Model (LLM) inference, where a trained model generates text one word at a time in response to user prompts, is a computationally intensive process requiring efficient scheduling to optimize latency and resource utilization. A key challenge in LLM inference is the management of the Key-Value (KV) cache, which reduces redundant computations but introduces memory constraints. In this work, we model LLM inference with KV cache constraints theoretically and propose a novel batching and scheduling algorithm that minimizes inference latency while effectively managing the KV cache's memory. More specifically, we make the following contributions. First, to evaluate the performance of online algorithms for scheduling in LLM inference, we introduce a hindsight optimal benchmark, formulated as an integer program that computes the minimum total inference latency under full future information. Second, we prove that no deterministic online algorithm can achieve a constant competitive ratio when the arrival process is arbitrary. Third, motivated by the computational intractability of solving the integer program at scale, we propose a polynomial-time online scheduling algorithm and show that under certain conditions it can achieve a constant competitive ratio. We also demonstrate our algorithm's strong empirical performance by comparing it to the hindsight optimal in a synthetic dataset. Finally, we conduct empirical evaluations on a real-world public LLM inference dataset, simulating the Llama2-70B model on A100 GPUs, and show that our algorithm significantly outperforms the benchmark algorithms. Overall, our results offer a path toward more sustainable and cost-effective LLM deployment.

arXiv:2502.06052

A Comprehensive Energy Management Application Method considering Smart Home Occupant Behavior using IoT and Real Big Data

One of the most far-reaching use cases of the internet of things is in smart grid and smart home operation. The smart home concept allows residents to control, monitor, and manage their energy consumption with minimum loss and self-involvement. Since each household's lifestyle and energy consumption is unique, the management system needs background knowledge about residents' energy consumption behavioral patterns for more accurate planning. To obtain this information, data related to residents' consumption records must be processed. This research has attempted to provide an optimal decentralized management system consisting of interoperable sections to forecast, optimize, schedule, and implement load management in a smart home. Comparing different prediction models using 4 years of 1-min interval real data of a smart home with photovoltaic generation (PV) and electric vehicle (EV), forecasting non-controllable loads and taking a deterministic approach in different scenarios, the system uses mixed integer linear programming (MILP) to provide load scheduling with the objective of an optimal total energy cost reduction with minimum changes in the household's desired consumption compared to the initial state. The results have shown that the proposed system has reliable performance due to the high precision of the forecast and has led to increased energy efficiency, reduced energy cost (up to 62. 05\%), reduced peak-to-average ratio (PAR) (up to 44. 19\%) and reduced standard deviation (SD) (up to 19. 70\%) in net consumption.

arXiv:2502.06581

A Survey on Video Analytics in Cloud-Edge-Terminal Collaborative Systems

The explosive growth of video data has driven the development of distributed video analytics in cloud-edge-terminal collaborative (CETC) systems, enabling efficient video processing, real-time inference, and privacy-preserving analysis. Among multiple advantages, CETC systems can distribute video processing tasks and enable adaptive analytics across cloud, edge, and terminal devices, leading to breakthroughs in video surveillance, autonomous driving, and smart cities. In this survey, we first analyze fundamental architectural components, including hierarchical, distributed, and hybrid frameworks, alongside edge computing platforms and resource management mechanisms. Building upon these foundations, edge-centric approaches emphasize on-device processing, edge-assisted offloading, and edge intelligence, while cloud-centric methods leverage powerful computational capabilities for complex video understanding and model training. Our investigation also covers hybrid video analytics incorporating adaptive task offloading and resource-aware scheduling techniques that optimize performance across the entire system. Beyond conventional approaches, recent advances in large language models and multimodal integration reveal both opportunities and challenges in platform scalability, data protection, and system reliability. Future directions also encompass explainable systems, efficient processing mechanisms, and advanced video analytics, offering valuable insights for researchers and practitioners in this dynamic field.

arXiv:2502.06798

Prompt-Aware Scheduling for Efficient Text-to-Image Inferencing System

Traditional ML models utilize controlled approximations during high loads, employing faster, but less accurate models in a process called accuracy scaling. However, this method is less effective for generative text-to-image models due to their sensitivity to input prompts and performance degradation caused by large model loading overheads. This work introduces a novel text-to-image inference system that optimally matches prompts across multiple instances of the same model operating at various approximation levels to deliver high-quality images under high loads and fixed budgets.

arXiv:2502.05221공식 코드 · Papers with Code

Blackout DIFUSCO

This study explores the integration of Blackout Diffusion into the DIFUSCO framework for combinatorial optimization, specifically targeting the Traveling Salesman Problem (TSP). Inspired by the success of discrete-time diffusion models (D3PM) in maintaining structural integrity, we extend the paradigm to a continuous-time framework, leveraging the unique properties of Blackout Diffusion. Continuous-time modeling introduces smoother transitions and refined control, hypothesizing enhanced solution quality over traditional discrete methods. We propose three key improvements to enhance the diffusion process. First, we transition from a discrete-time-based model to a continuous-time framework, providing a more refined and flexible formulation. Second, we refine the observation time scheduling to ensure a smooth and linear transformation throughout the diffusion process, allowing for a more natural progression of states. Finally, building upon the second improvement, we further enhance the reverse process by introducing finer time slices in regions that are particularly challenging for the model, thereby improving accuracy and stability in the reconstruction phase. Although the experimental results did not exceed the baseline performance, they demonstrate the effectiveness of these methods in balancing simplicity and complexity, offering new insights into diffusion-based combinatorial optimization. This work represents the first application of Blackout Diffusion to combinatorial optimization, providing a foundation for further advancements in this domain. * The code is available for review at https://github.com/Giventicket/BlackoutDIFUSCO.

arXiv:2502.04376

MEETING DELEGATE: Benchmarking LLMs on Attending Meetings on Our Behalf

In contemporary workplaces, meetings are essential for exchanging ideas and ensuring team alignment but often face challenges such as time consumption, scheduling conflicts, and inefficient participation. Recent advancements in Large Language Models (LLMs) have demonstrated their strong capabilities in natural language generation and reasoning, prompting the question: can LLMs effectively delegate participants in meetings? To explore this, we develop a prototype LLM-powered meeting delegate system and create a comprehensive benchmark using real meeting transcripts. Our evaluation reveals that GPT-4/4o maintain balanced performance between active and cautious engagement strategies. In contrast, Gemini 1.5 Pro tends to be more cautious, while Gemini 1.5 Flash and Llama3-8B/70B display more active tendencies. Overall, about 60\% of responses address at least one key point from the ground-truth. However, improvements are needed to reduce irrelevant or repetitive content and enhance tolerance for transcription errors commonly found in real-world settings. Additionally, we implement the system in practical settings and collect real-world feedback from demos. Our findings underscore the potential and challenges of utilizing LLMs as meeting delegates, offering valuable insights into their practical application for alleviating the burden of meetings.

arXiv:2502.03859

Stabilizing scheduling logic for networked control systems under limited capacity and lossy communication networks

In this paper we address the problem of designing scheduling logic for stabilizing Networked Control Systems (NCSs) with plants and controllers remotely-located over a limited capacity communication network subject to data losses. Our specific contributions include characterization of stability under worst case data loss using an inequality associated with a cycle on a graph. This is eventually formulated as a feasibility problem to solve for certain parameters (\(T\)-factors) used to design a periodic scheduling logic. We show that given a solution to the feasibility problem, the designed scheduling logic guarantees \emph{global asymptotic stability} for all plants of the network under all admissible data losses. We also derive sufficient conditions on the number of plants and the capacity of the network for the existence of a solution to the feasibility problem. Given that a sufficient condition is satisfied, we discuss the procedure to obtain the feasible \(T\)-factors. We use tools from switched systems theory and graph theory in this work. A numerical experiment is provided to verify our results.

arXiv:2502.03092공식 코드 · Papers with Code

E-3SFC: Communication-Efficient Federated Learning with Double-way Features Synthesizing

The exponential growth in model sizes has significantly increased the communication burden in Federated Learning (FL). Existing methods to alleviate this burden by transmitting compressed gradients often face high compression errors, which slow down the model's convergence. To simultaneously achieve high compression effectiveness and lower compression errors, we study the gradient compression problem from a novel perspective. Specifically, we propose a systematical algorithm termed Extended Single-Step Synthetic Features Compressing (E-3SFC), which consists of three sub-components, i.e., the Single-Step Synthetic Features Compressor (3SFC), a double-way compression algorithm, and a communication budget scheduler. First, we regard the process of gradient computation of a model as decompressing gradients from corresponding inputs, while the inverse process is considered as compressing the gradients. Based on this, we introduce a novel gradient compression method termed 3SFC, which utilizes the model itself as a decompressor, leveraging training priors such as model weights and objective functions. 3SFC compresses raw gradients into tiny synthetic features in a single-step simulation, incorporating error feedback to minimize overall compression errors. To further reduce communication overhead, 3SFC is extended to E-3SFC, allowing double-way compression and dynamic communication budget scheduling. Our theoretical analysis under both strongly convex and non-convex conditions demonstrates that 3SFC achieves linear and sub-linear convergence rates with aggregation noise. Extensive experiments across six datasets and six models reveal that 3SFC outperforms state-of-the-art methods by up to 13.4% while reducing communication costs by 111.6 times. These findings suggest that 3SFC can significantly enhance communication efficiency in FL without compromising model performance.

arXiv:2502.02861

Algorithms with Calibrated Machine Learning Predictions

The field of algorithms with predictions incorporates machine learning advice in the design of online algorithms to improve real-world performance. A central consideration is the extent to which predictions can be trusted -- while existing approaches often require users to specify an aggregate trust level, modern machine learning models can provide estimates of prediction-level uncertainty. In this paper, we propose calibration as a principled and practical tool to bridge this gap, demonstrating the benefits of calibrated advice through two case studies: the ski rental and online job scheduling problems. For ski rental, we design an algorithm that achieves near-optimal prediction-dependent performance and prove that, in high-variance settings, calibrated advice offers more effective guidance than alternative methods for uncertainty quantification. For job scheduling, we demonstrate that using a calibrated predictor leads to significant performance improvements over existing methods. Evaluations on real-world data validate our theoretical findings, highlighting the practical impact of calibration for algorithms with predictions.

arXiv:2502.05124

Fixed-Throughput GRAND with FIFO Scheduling

Guessing random additive noise decoding (GRAND) is a code-agnostic decoding method that iteratively guesses the noise pattern affecting the received codeword. The number of noise sequences to test depends on the noise realization. Thus, GRAND exhibits random runtime which results in nondeterministic throughput. However, real-time systems must process the incoming data at a fixed rate, necessitating a fixed-throughput decoder in order to avoid losing data. We propose a first-in first-out (FIFO) scheduling architecture that enables a fixed throughput while improving the block error rate (BLER) compared to the common approach of imposing a maximum runtime constraint per received codeword. Moreover, we demonstrate that the average throughput metric of GRAND-based hardware implementations typically provided in the literature can be misleading as one needs to operate at approximately one order of magnitude lower throughput to achieve the BLER of an unconstrained decoder.

arXiv:2502.02430

A Scalable Crawling Algorithm Utilizing Noisy Change-Indicating Signals

Web refresh crawling is the problem of keeping a cache of web pages fresh, that is, having the most recent copy available when a page is requested, given a limited bandwidth available to the crawler. Under the assumption that the change and request events, resp., to each web page follow independent Poisson processes, the optimal scheduling policy was derived by Azar et al. 2018. In this paper, we study an extension of this problem where side information indicating content changes, such as various types of web pings, for example, signals from sitemaps, content delivery networks, etc., is available. Incorporating such side information into the crawling policy is challenging, because (i) the signals can be noisy with false positive events and with missing change events; and (ii) the crawler should achieve a fair performance over web pages regardless of the quality of the side information, which might differ from web page to web page. We propose a scalable crawling algorithm which (i) uses the noisy side information in an optimal way under mild assumptions; (ii) can be deployed without heavy centralized computation; (iii) is able to crawl web pages at a constant total rate without spikes in the total bandwidth usage over any time interval, and automatically adapt to the new optimal solution when the total bandwidth changes without centralized computation. Experiments clearly demonstrate the versatility of our approach.

arXiv:2502.00350

OrcaLoca: An LLM Agent Framework for Software Issue Localization

Recent developments in Large Language Model (LLM) agents are revolutionizing Autonomous Software Engineering (ASE), enabling automated coding, problem fixes, and feature improvements. However, localization -- precisely identifying software problems by navigating to relevant code sections -- remains a significant challenge. Current approaches often yield suboptimal results due to a lack of effective integration between LLM agents and precise code search mechanisms. This paper introduces OrcaLoca, an LLM agent framework that improves accuracy for software issue localization by integrating priority-based scheduling for LLM-guided action, action decomposition with relevance scoring, and distance-aware context pruning. Experimental results demonstrate that OrcaLoca becomes the new open-source state-of-the-art (SOTA) in function match rate (65.33%) on SWE-bench Lite. It also improves the final resolved rate of an open-source framework by 6.33 percentage points through its patch generation integration.

arXiv:2502.01268

Resilient UAV Trajectory Planning via Few-Shot Meta-Offline Reinforcement Learning

Reinforcement learning (RL) has been a promising essence in future 5G-beyond and 6G systems. Its main advantage lies in its robust model-free decision-making in complex and large-dimension wireless environments. However, most existing RL frameworks rely on online interaction with the environment, which might not be feasible due to safety and cost concerns. Another problem with online RL is the lack of scalability of the designed algorithm with dynamic or new environments. This work proposes a novel, resilient, few-shot meta-offline RL algorithm combining offline RL using conservative Q-learning (CQL) and meta-learning using model-agnostic meta-learning (MAML). The proposed algorithm can train RL models using static offline datasets without any online interaction with the environments. In addition, with the aid of MAML, the proposed model can be scaled up to new unseen environments. We showcase the proposed algorithm for optimizing an unmanned aerial vehicle (UAV) 's trajectory and scheduling policy to minimize the age-of-information (AoI) and transmission power of limited-power devices. Numerical results show that the proposed few-shot meta-offline RL algorithm converges faster than baseline schemes, such as deep Q-networks and CQL. In addition, it is the only algorithm that can achieve optimal joint AoI and transmission power using an offline dataset with few shots of data points and is resilient to network failures due to unprecedented environmental changes.

arXiv:2502.01129

Deep Reinforcement Learning for Dynamic Resource Allocation in Wireless Networks

This report investigates the application of deep reinforcement learning (DRL) algorithms for dynamic resource allocation in wireless communication systems. An environment that includes a base station, multiple antennas, and user equipment is created. Using the RLlib library, various DRL algorithms such as Deep Q-Network (DQN) and Proximal Policy Optimization (PPO) are then applied. These algorithms are compared based on their ability to optimize resource allocation, focusing on the impact of different learning rates and scheduling policies. The findings demonstrate that the choice of algorithm and learning rate significantly influences system performance, with DRL providing more efficient resource allocation compared to traditional methods.

arXiv:2502.00937

ModServe: Scalable and Resource-Efficient Large Multimodal Model Serving

Large multimodal models (LMMs) demonstrate impressive capabilities in understanding images, videos, and audio beyond text. However, efficiently serving LMMs in production environments poses significant challenges due to their complex architectures and heterogeneous characteristics across their multi-stage inference pipelines. We present the first comprehensive systems analysis of two prominent LMM architectures, decoder-only and cross-attention, across six representative open-source models, revealing key systems design implications. We also present an in-depth analysis of production LMM inference traces, uncovering unique workload characteristics, including variable, heavy-tailed request distributions and bursty traffic patterns. Based on these insights, we propose ModServe, a modular LMM serving system that decouples stages for independent optimization and adaptive scaling. ModServe dynamically reconfigures stages and handles bursty traffic with modality-aware scheduling and autoscaling to meet tail latency SLOs while minimizing costs. ModServe achieves 3.3-5.5x higher throughput (leading to 25-41.3% cost saving) while meeting SLOs on a 128-GPU cluster with production traces.

arXiv:2502.00045

Restless Multi-armed Bandits under Frequency and Window Constraints for Public Service Inspections

Municipal inspections are an important part of maintaining the quality of goods and services. In this paper, we approach the problem of intelligently scheduling service inspections to maximize their impact, using the case of food establishment inspections in Chicago as a case study. The Chicago Department of Public Health (CDPH) inspects thousands of establishments each year, with a substantial fail rate (over 3,000 failed inspection reports in 2023). To balance the objectives of ensuring adherence to guidelines, minimizing disruption to establishments, and minimizing inspection costs, CDPH assigns each establishment an inspection window every year and guarantees that they will be inspected exactly once during that window. These constraints create a challenge for a restless multi-armed bandit (RMAB) approach, for which there are no existing methods. We develop an extension to Whittle index-based systems for RMABs that can guarantee action window constraints and frequencies, and furthermore can be leveraged to optimize action window assignments themselves. Briefly, we combine MDP reformulation and integer programming-based lookahead to maximize the impact of inspections subject to constraints. A neural network-based supervised learning model is developed to model state transitions of real Chicago establishments using public CDPH inspection records, which demonstrates 10\% AUC improvements compared with directly predicting establishments' failures. Our experiments not only show up to 24\% (in simulation) or 33\% (on real data) reward improvements resulting from our approach but also give insight into the impact of scheduling constraints.

논문 ID: document/1083685

Dynamics-incorporated Modeling Framework for Stability Constrained Scheduling Under High-penetration of Renewable Energy

In this paper, a modularized modeling framework is designed to enable a dynamics-incorporated power system scheduling under high-penetration of renewable energy. This unique framework incorporates an adapted hybrid symbolic-numeric approach to scheduling models, effectively bridging the gap between device- and system-level optimization models and streamlining the scheduling modeling effort. The adaptability of the proposed framework stems from four key aspects: extensible scheduling formulations through modeling blocks, scalable performance via effective vectorization and sparsity-aware techniques, compatible data structure aligned with dynamic simulators by common power flow data, and interoperable dynamic interface for bi-direction data exchange between steady-state generation scheduling and time-domain dynamic simulation. Through extensive benchmarks with various usage scenarios, the framework's accuracy and scalability are validated. The case studies also demonstrate the efficient interoperation of generation scheduling and dynamics, significantly reducing the modeling conversion work in stability-constrained grid operation towards high-penetration of renewable energy.

arXiv:2501.19063

Optimizing Job Allocation using Reinforcement Learning with Graph Neural Networks

Efficient job allocation in complex scheduling problems poses significant challenges in real-world applications. In this report, we propose a novel approach that leverages the power of Reinforcement Learning (RL) and Graph Neural Networks (GNNs) to tackle the Job Allocation Problem (JAP). The JAP involves allocating a maximum set of jobs to available resources while considering several constraints. Our approach enables learning of adaptive policies through trial-and-error interactions with the environment while exploiting the graph-structured data of the problem. By leveraging RL, we eliminate the need for manual annotation, a major bottleneck in supervised learning approaches. Experimental evaluations on synthetic and real-world data demonstrate the effectiveness and generalizability of our proposed approach, outperforming baseline algorithms and showcasing its potential for optimizing job allocation in complex scheduling problems.

arXiv:2501.18965공식 코드 · Papers with Code

The Surprising Agreement Between Convex Optimization Theory and Learning-Rate Scheduling for Large Model Training

We show that learning-rate schedules for large model training behave surprisingly similar to a performance bound from non-smooth convex optimization theory. We provide a bound for the constant schedule with linear cooldown; in particular, the practical benefit of cooldown is reflected in the bound due to the absence of logarithmic terms. Further, we show that this surprisingly close match between optimization theory and practice can be exploited for learning-rate tuning: we achieve noticeable improvements for training 124M and 210M Llama-type models by (i) extending the schedule for continued training with optimal learning-rate, and (ii) transferring the optimal learning-rate across schedules.

arXiv:2501.17873

Split-Aperture Phased Array Radar Resource Management for Tracking Tasks

The next generation of radar systems will include advanced digital front-end technology in the apertures allowing for spatially subdividing radar tasks over the array, the so-called split-aperture phased array (SAPA) concept. The goal of this paper is to introduce radar resource management for the SAPA concept and to demonstrate the added benefit of the SAPA concept for active tracking tasks. To do so, the radar resource management problem is formulated and solved by employing the quality of service based resource allocation model (Q-RAM) framework. As active tracking tasks may be scheduled simultaneously, the resource allocation of tasks becomes dependent on the other tasks. The solution to the resource allocation problem is obtained by introducing the adaptive fast traversal algorithm combined with a three dimensional strip packing algorithm to handle task dependencies. It will be demonstrated by a simulation example that the SAPA concept can significantly increase the number of active tracks of a multifunction radar system compared to scheduling tasks sequentially.