스트리밍 비디오 대규모 언어 모델(VLM)은 지속적인 비디오 이해를 가능하게 하지만, 기존 벤치마크는 저역동적 시나리오에 치중되어 있다. 제한된 컨텍스트 예산 하에서 모델은 시간적 이력, 공간 해상도, 시간적 세분도(temporal granularity) 사이의 균형을 유지해야 하며, 1~2 FPS의 희소 샘플링(sparse sampling)은 빠르게 지나가는 사건을 놓치게 된다. 본 연구에서는 실제 비디오 스트림에서 고역동적 인지 능력을 평가하기 위한 FastBench를 도입한다. 궤적 기반(trajectory-grounded) 파이프라인은 높은 FPS 클립으로부터의 질의응답(QA) 생성, 2 FPS에서 답변 가능한 질문 필터링, SAM3 및 CoTracker3 궤적을 사용한 답변 검증, 그리고 세 차례의 인간 검수를 결합한다. FastBench는 8개 도메인, 6개 역량, 그리고 전방(forward), 즉각(instant), 후방(backward) 시간 범위에 걸쳐 인간이 주석을 단 증거 구간을 포함하는 306개의 QA 쌍을 포함한다. 또한 텍스트 토큰을 통해 입력 프레임 속도를 조절하는 학습(training) 불필요 기준선(baseline)인 ProactiveFrame을 제시한다. 2계층 슬라이딩 윈도우는 최근의 높은 FPS 관측값을 유지하면서 이전 관측값은 희소한 이력으로 다운샘플링한다. 실험 결과 상당한 한계점이 드러났다. 가장 강력한 모델(model)인 Gemini-3.5-Flash조차 50.7%에 불과한 점수를 기록했다. 더 조밀한 샘플링은 Qwen3-VL-8B의 성능을 2 FPS에서의 32.9%에서 24 FPS에서의 44.6%로 향상시키지만, 이력이 압축됨에 따라 이득은 포화된다. ProactiveFrame은 희소 균일 샘플링보다 5.4 및 1.5 퍼센트포인트 높은 성능을 보이지만, 오라클 유도 집중(oracle-guided focusing)보다는 크게 뒤처져 있어 현재 VLM이 스트림만으로 더 미세한 시간적 인지가 필요한 시점을 판단하는 데 어려움을 겪고 있음을 보여준다. FastBench는 고역동적 스트리밍 비디오 이해를 위한 테스트베드를 제공한다. 코드 및 데이터: https://github.com/Ashone3/FastBench.
Recent methods (e.g. MaterialGAN) have used unconditional GANs to generate per-pixel material maps, or as a prior to reconstruct materials from input photographs. These models can generate varied random material appearance, but do not have any mechanism to constrain the generated material to a specific category or to control the coarse structure of the generated material, such as the exact brick layout on a brick wall. Furthermore, materials reconstructed from a single input photo commonly have artifacts and are generally not tileable, which limits their use in practical content creation pipelines. We propose TileGen, a generative model for SVBRDFs that is specific to a material category, always tileable, and optionally conditional on a provided input structure pattern. TileGen is a variant of StyleGAN whose architecture is modified to always produce tileable (periodic) material maps. In addition to the standard "style" latent code, TileGen can optionally take a condition image, giving a user direct control over the dominant spatial (and optionally color) features of the material. For example, in brick materials, the user can specify a brick layout and the brick color, or in leather materials, the locations of wrinkles and folds. Our inverse rendering approach can find a material perceptually matching a single target photograph by optimization. This reconstruction can also be conditional on a user-provided pattern. The resulting materials are tileable, can be larger than the target image, and are editable by varying the condition.
Forecasts of regional electricity net-demand, consumption minus embedded generation, are an essential input for reliable and economic power system operation, and energy trading. While such forecasts are typically performed region by region, operations such as managing power flows require spatially coherent joint forecasts, which account for cross-regional dependencies. Here, we forecast the joint distribution of net-demand across the 14 regions constituting Great Britain's electricity network. Joint modelling is complicated by the fact that the net-demand variability within each region, and the dependencies between regions, vary with temporal, socio-economical and weather-related factors. We accommodate for these characteristics by proposing a multivariate Gaussian model based on a modified Cholesky parametrisation, which allows us to model each unconstrained parameter via an additive model. Given that the number of model parameters and covariates is large, we adopt a semi-automated approach to model selection, based on gradient boosting. In addition to comparing the forecasting performance of several versions of the proposed model with that of two non-Gaussian copula-based models, we visually explore the model output to interpret how the covariates affect net-demand variability and dependencies. The code for reproducing the results in this paper is available at https://doi.org/10.5281/zenodo.7315105, while methods for building and fitting multivariate Gaussian additive models are provided by the SCM R package, available at https://github.com/VinGioia90/SCM.
In this study, we delve into the task of few-shot Generative Domain Adaptation (GDA), which involves transferring a pre-trained generator from one domain to a new domain using only a few reference images. Inspired by the way human brains acquire knowledge in new domains, we present an innovative generator structure called Domain Re-Modulation (DoRM). DoRM not only meets the criteria of high quality, large synthesis diversity, and cross-domain consistency, which were achieved by previous research in GDA, but also incorporates memory and domain association, akin to how human brains operate. Specifically, DoRM freezes the source generator and introduces new mapping and affine modules (M&A modules) to capture the attributes of the target domain during GDA. This process resembles the formation of new synapses in human brains. Consequently, a linearly combinable domain shift occurs in the style space. By incorporating multiple new M&A modules, the generator gains the capability to perform high-fidelity multi-domain and hybrid-domain generation. Moreover, to maintain cross-domain consistency more effectively, we introduce a similarity-based structure loss. This loss aligns the auto-correlation map of the target image with its corresponding auto-correlation map of the source image during training. Through extensive experiments, we demonstrate the superior performance of our DoRM and similarity-based structure loss in few-shot GDA, both quantitatively and qualitatively. The code will be available at https://github.com/wuyi2020/DoRM.
Digital twin is a problem of augmenting real objects with their digital counterparts. It can underpin a wide range of applications in augmented reality (AR), autonomy, and UI/UX. A critical component in a good digital-twin system is real-time, accurate 3D object tracking. Most existing works solve 3D object tracking through the lens of robotic grasping, employ older generations of depth sensors, and measure performance metrics that may not apply to other digital-twin applications such as in AR. In this work, we create a novel RGB-D dataset, called Digital Twin Tracking Dataset (DTTD), to enable further research of the problem and extend potential solutions towards longer ranges and mm localization accuracy. To reduce point cloud noise from the input source, we select the latest Microsoft Azure Kinect as the state-of-the-art time-of-flight (ToF) camera. In total, 103 scenes of 10 common off-the-shelf objects with rich textures are recorded, with each frame annotated with a per-pixel semantic segmentation and ground-truth object poses provided by a commercial motion capturing system. Through extensive experiments with model-level and dataset-level analysis, we demonstrate that DTTD can help researchers develop future object tracking methods and analyze new challenges. The dataset, data generation, annotation, and model evaluation pipeline are made publicly available as open source code at: https://github.com/augcog/DTTDv1.
Complex Named Entity Recognition (NER) is the task of detecting linguistically complex named entities in low-context text. In this paper, we present ACLM Attention-map aware keyword selection for Conditional Language Model fine-tuning), a novel data augmentation approach based on conditional generation to address the data scarcity problem in low-resource complex NER. ACLM alleviates the context-entity mismatch issue, a problem existing NER data augmentation techniques suffer from and often generates incoherent augmentations by placing complex named entities in the wrong context. ACLM builds on BART and is optimized on a novel text reconstruction or denoising task - we use selective masking (aided by attention maps) to retain the named entities and certain keywords in the input sentence that provide contextually relevant additional knowledge or hints about the named entities. Compared with other data augmentation strategies, ACLM can generate more diverse and coherent augmentations preserving the true word sense of complex entities in the sentence. We demonstrate the effectiveness of ACLM both qualitatively and quantitatively on monolingual, cross-lingual, and multilingual complex NER across various low-resource settings. ACLM outperforms all our neural baselines by a significant margin (1%-36%). In addition, we demonstrate the application of ACLM to other domains that suffer from data scarcity (e.g., biomedical). In practice, ACLM generates more effective and factual augmentations for these domains than prior methods. Code: https://github.com/Sreyan88/ACLM
Query auto-completion (QAC) aims to suggest plausible completions for a given query prefix. Traditionally, QAC systems have leveraged tries curated from historical query logs to suggest most popular completions. In this context, there are two specific scenarios that are difficult to handle for any QAC system: short prefixes (which are inherently ambiguous) and unseen prefixes. Recently, personalized Natural Language Generation (NLG) models have been proposed to leverage previous session queries as context for addressing these two challenges. However, such NLG models suffer from two drawbacks: (1) some of the previous session queries could be noisy and irrelevant to the user intent for the current prefix, and (2) NLG models cannot directly incorporate historical query popularity. This motivates us to propose a novel NLG model for QAC, Trie-NLG, which jointly leverages popularity signals from trie and personalization signals from previous session queries. We train the Trie-NLG model by augmenting the prefix with rich context comprising of recent session queries and top trie completions. This simple modeling approach overcomes the limitations of trie-based and NLG-based approaches and leads to state-of-the-art performance. We evaluate the Trie-NLG model using two large QAC datasets. On average, our model achieves huge ~57% and ~14% boost in MRR over the popular trie-based lookup and the strong BART-based baseline methods, respectively. We make our code publicly available.
As large language models (LLMs) take on complex tasks, their inputs are supplemented with longer contexts that incorporate domain knowledge. Yet using long contexts is challenging, as nothing can be generated until the whole context is processed by the LLM. While the context-processing delay can be reduced by reusing the KV cache of a context across different inputs, fetching the KV cache, which contains large tensors, over the network can cause high extra network delays. CacheGen is a fast context-loading module for LLM systems. First, CacheGen uses a custom tensor encoder, leveraging KV cache's distributional properties to encode a KV cache into more compact bitstream representations with negligible decoding overhead, to save bandwidth usage. Second, CacheGen adapts the compression level of different parts of a KV cache to cope with changes in available bandwidth, in order to maintain low context-loading delay and high generation quality. % When available bandwidth drops, CacheGen may raise the compression level for a part of the context or recompute its KV cache on the fly. We test CacheGen on popular LLMs and datasets. Compared to the recent systems that reuse the KV cache, CacheGen reduces the KV cache size by 3.5-4.3x and the total delay in fetching and processing contexts by 3.2-3.7x with negligible impact on the LLM response quality. Our code is at: https://github.com/UChi-JCL/CacheGen.
Chest X-rays (CXRs) are commonly utilized as a low-dose modality for lung screening. Nonetheless, the efficacy of CXRs is somewhat impeded, given that approximately 75% of the lung area overlaps with bone, which in turn hampers the detection and diagnosis of diseases. As a remedial measure, bone suppression techniques have been introduced. The current dual-energy subtraction imaging technique in the clinic requires costly equipment and subjects being exposed to high radiation. To circumvent these issues, deep learning-based image generation algorithms have been proposed. However, existing methods fall short in terms of producing high-quality images and capturing texture details, particularly with pulmonary vessels. To address these issues, this paper proposes a new bone suppression framework, termed BS-Diff, that comprises a conditional diffusion model equipped with a U-Net architecture and a simple enhancement module to incorporate an autoencoder. Our proposed network cannot only generate soft tissue images with a high bone suppression rate but also possesses the capability to capture fine image details. Additionally, we compiled the largest dataset since 2010, including data from 120 patients with high-definition, high-resolution paired CXRs and soft tissue images collected by our affiliated hospital. Extensive experiments, comparative analyses, ablation studies, and clinical evaluations indicate that the proposed BS-Diff outperforms several bone-suppression models across multiple metrics. Our code can be accessed at https://github.com/Benny0323/BS-Diff.
Diffusion probabilistic models (DPMs) have exhibited excellent performance for high-fidelity image generation while suffering from inefficient sampling. Recent works accelerate the sampling procedure by proposing fast ODE solvers that leverage the specific ODE form of DPMs. However, they highly rely on specific parameterization during inference (such as noise/data prediction), which might not be the optimal choice. In this work, we propose a novel formulation towards the optimal parameterization during sampling that minimizes the first-order discretization error of the ODE solution. Based on such formulation, we propose DPM-Solver-v3, a new fast ODE solver for DPMs by introducing several coefficients efficiently computed on the pretrained model, which we call empirical model statistics. We further incorporate multistep methods and a predictor-corrector framework, and propose some techniques for improving sample quality at small numbers of function evaluations (NFE) or large guidance scales. Experiments show that DPM-Solver-v3 achieves consistently better or comparable performance in both unconditional and conditional sampling with both pixel-space and latent-space DPMs, especially in 5$\sim$10 NFEs. We achieve FIDs of 12.21 (5 NFE), 2.51 (10 NFE) on unconditional CIFAR10, and MSE of 0.55 (5 NFE, 7.5 guidance scale) on Stable Diffusion, bringing a speed-up of 15%$\sim$30% compared to previous state-of-the-art training-free methods. Code is available at https://github.com/thu-ml/DPM-Solver-v3.
Recently, program synthesis driven by large language models (LLMs) has become increasingly popular. However, program synthesis for machine learning (ML) tasks still poses significant challenges. This paper explores a novel form of program synthesis, targeting ML programs, by combining LLMs and automated machine learning (autoML). Specifically, our goal is to fully automate the generation and optimization of the code of the entire ML workflow, from data preparation to modeling and post-processing, utilizing only textual descriptions of the ML tasks. To manage the length and diversity of ML programs, we propose to break each ML program into smaller, manageable parts. Each part is generated separately by the LLM, with careful consideration of their compatibilities. To ensure compatibilities, we design a testing technique for ML programs. Unlike traditional program synthesis, which typically relies on binary evaluations (i.e., correct or incorrect), evaluating ML programs necessitates more than just binary judgments. Our approach automates the numerical evaluation and optimization of these programs, selecting the best candidates through autoML techniques. In experiments across various ML tasks, our method outperforms existing methods in 10 out of 12 tasks for generating ML programs. In addition, autoML significantly improves the performance of the generated ML programs. In experiments, given the textual task description, our method, Text-to-ML, generates the complete and optimized ML program in a fully autonomous process. The implementation of our method is available at https://github.com/JLX0/llm-automl.
Multi-document (MD) processing is crucial for LLMs to handle real-world tasks such as summarization and question-answering across large sets of documents. While LLMs have improved at processing long inputs, MD contexts still present unique difficulties, including management of inter-document dependencies, redundancy, and incoherent structures. To address this challenge, we introduce MDCure, a scalable and effective instruction data generation framework to enhance the MD capabilities of LLMs without the computational cost of pre-training or reliance on human-annotated data. MDCure generates high-quality synthetic MD instruction data over sets of articles via targeted prompts. We also introduce MDCureRM, a cost-effective, MD-specific reward model to score and filter generated data based on their training utility for MD settings. MDCure is compatible with open- and closed-source models in addition to policy optimization methods such as PPO, enabling even small open-source models to surpass proprietary LLMs as strong generators of high-quality MD instruction data without further data filtering. With MDCure, we fine-tune a wide variety of LLMs up to 70B parameters in size from the FlanT5, Qwen2, and LLAMA3.1 model families. Extensive evaluations on a wide range of MD and long-context benchmarks spanning various tasks and domains show MDCure consistently improves performance over pre-trained baselines and base models by up to 75.1%. Our code, datasets, and models are available at https://github.com/yale-nlp/MDCure.
Recent advances in Large Vision-Language Models (LVLMs) have showcased strong reasoning abilities across multiple modalities, achieving significant breakthroughs in various real-world applications. Despite this great success, the safety guardrail of LVLMs may not cover the unforeseen domains introduced by the visual modality. Existing studies primarily focus on eliciting LVLMs to generate harmful responses via carefully crafted image-based jailbreaks designed to bypass alignment defenses. In this study, we reveal that a safe image can be exploited to achieve the same jailbreak consequence when combined with additional safe images and prompts. This stems from two fundamental properties of LVLMs: universal reasoning capabilities and safety snowball effect. Building on these insights, we propose Safety Snowball Agent (SSA), a novel agent-based framework leveraging agents' autonomous and tool-using abilities to jailbreak LVLMs. SSA operates through two principal stages: (1) initial response generation, where tools generate or retrieve jailbreak images based on potential harmful intents, and (2) harmful snowballing, where refined subsequent prompts induce progressively harmful outputs. Our experiments demonstrate that \ours can use nearly any image to induce LVLMs to produce unsafe content, achieving high success jailbreaking rates against the latest LVLMs. Unlike prior works that exploit alignment flaws, \ours leverages the inherent properties of LVLMs, presenting a profound challenge for enforcing safety in generative multimodal systems. Our code is avaliable at \url{https://github.com/gzcch/Safety_Snowball_Agent}.
Large Language Models (LLMs) have achieved remarkable success across a wide range of natural language tasks, and recent efforts have sought to extend their capabilities to multimodal domains and resource-constrained environments. However, existing approaches often rely on costly supervised fine-tuning or assume fixed training conditions, limiting their generalization when facing unseen modalities, limited data, or restricted compute resources. This dissertation presents a systematic study toward generalizing LLM usability under real-world constraints. First, it introduces a robust text-centric alignment framework that enables LLMs to seamlessly integrate diverse modalities-including text, images, tables, and any modalities - via natural language interfaces. This approach supports in-context adaptation to unseen or dynamically changing modalities without requiring retraining. To enhance robustness against noisy and missing modalities, an adversarial prompting technique is proposed, generating semantically challenging perturbations at the prompt level to stress-test model reliability. Beyond multimodal setting, the dissertation investigates inference-time optimization strategies for LLMs, leveraging prompt search and uncertainty quantification to improve performance without additional model training. This perspective offers an efficient alternative to scaling model parameters or retraining from scratch. Additionally, the work addresses low-resource domains such as Verilog code generation by designing correct-by-construction synthetic data pipelines and logic-enhanced reasoning models, achieving state-of-the-art performance with minimal data. Together, these contributions form a unified effort to enhance the adaptability, scalability, and efficiency of large language models under practical constraints.
Diffusion language models (dLLMs), such as LLaDA and Dream, have become competitive with autoregressive (AR) LLMs in generation quality while supporting native parallel decoding. A standard acceleration strategy is block-wise decoding, where each forward pass predicts a block of length B and commits high-confidence tokens. Confidence-threshold commitment adapts which tokens are accepted, but a fixed B still constrains the prediction scope. Existing horizon policies use manually chosen sizes, volatility statistics, or candidate-boundary scores. We instead use the model's own per-step confidence: in-window confidence typically follows a context-dependent cliff, providing a direct signal for adapting the look-ahead horizon. In this work, we propose PACE-dLLM, which refits this parametric cliff periodically in closed form, sets the next horizon at a shared confidence anchor, and retains confidence-threshold token commitment. Under a saturated-yield abstraction with iid, policy-independent monotone profiles, the oracle horizon is the smallest attaining maximal per-pass yield and improves the asymptotic forward-pass rate when fixed blocks undershoot it with positive probability. On four reasoning and code benchmarks across two open-source dLLM backbones, PACE-dLLM achieves the best average accuracy among the evaluated decoders on both backbones, with average throughput speedups of 5.23x on LLaDA and 3.06x on Dream (up to 8.52x on LLaDA MATH) over the unaccelerated semi-AR baseline, advancing the quality-throughput Pareto frontier.
전문가 혼합(MoE)이 조건부 연산을 통해 용량을 확장하는 반면, 트랜스포머는 지식 검색을 위한 기본 프리미티브가 없어 연산을 통해 비효율적으로 검색을 시뮬레이션해야 합니다. 이를 해결하기 위해, 우리는 보완적인 희소성 축으로서 조건부 메모리를 도입합니다. 이는 O(1) 검색을 위해 고전적인 N-gram 임베딩을 현대화한 모듈인 Engram을 통해 구현됩니다. 희소성 할당 문제를 공식화함으로써, 신경망 연산(MoE)과 정적 메모리(Engram) 간의 트레이드오프를 최적화하는 U자형 스케일링 법칙을 발견합니다. 이 법칙에 따라 Engram을 270억 매개변수까지 확장하여, 동일 매개변수 및 동일 FLOPs의 MoE 기준선 대비 우수한 성능을 달성합니다. 특히, 메모리 모듈이 지식 검색에 기여할 것으로 예상되었으나(예: MMLU +3.4; CMMLU +4.0), 일반 추론(예: BBH +5.0; ARC-Challenge +3.7) 및 코드/수학 영역(HumanEval +3.0; MATH +2.4)에서 더욱 큰 성능 향상이 관찰되었습니다. 메커니즘 분석을 통해 Engram이 백본의 초기 레이어를 정적 재구성 부담에서 해방시켜 복잡한 추론을 위해 네트워크를 효과적으로 심화시키는 것이 밝혀졌습니다. 또한, 로컬 의존성을 검색에 위임함으로써 글로벌 컨텍스트를 위한 어텐션 용량을 확보하여 장문 컨텍스트 검색 성능을 크게 향상시킵니다(예: Multi-Query NIAH: 84.2 → 97.0). 마지막으로, Engram은 인프라 인식 효율성을 확립합니다: 결정론적 주소 지정 방식을 통해 호스트 메모리에서의 런타임 프리페칭이 가능하며 무시할 수 있는 수준의 오버헤드만 발생합니다. 우리는 조건부 메모리를 차세대 희소 모델을 위한 필수적인 모델링 프리미티브로 전망합니다.
음성 녹음에는 전체 발화를 다시 합성하지 않고 재구성하거나 수정해야 하는 누락, 손상 또는 잘못된 영역이 포함되는 경우가 많습니다. 음성 인페인팅(speech inpainting)은 누락된 구간을 복원하는 반면, 음성 편집(speech editing)은 편집된 전사 텍스트에 따라 발화 내용을 대체합니다. 두 작업 모두 생성된 음성이 의도한 단어를 표현하는 동시에 주변 화자의 정체성, 운율, 타이밍 및 녹음 환경과 일관성을 유지해야 합니다. 이산 확산(discrete diffusion)은 좌우 음향적 문맥(context) 모두를 공동으로 조건화하면서 마스킹된 토큰을 반복적으로 개선(refine)할 수 있어 이러한 작업에 특히 적합합니다. 본 논문에서는 계층적 코덱 토큰 기반의 텍스트 안내 음성 인페인팅 및 편집을 위한 이산 확산 프레임워크인 SIEDD를 소개합니다. 핵심 아키텍처인 HiCoDD는 이전에 생성된 코드북을 깨끗하고 확정된 음향적 문맥으로 표현하고 현재 개선 대상 코드북에만 확산을 적용함으로써 RVQ 생성 순서를 따릅니다. 이러한 분리를 통해 순차적인 coarse-to-fine 추론(inference)과 일치하면서도 정보 누출이 없는 결합 학습(joint training)이 가능해집니다. 또한 본 모델은 음소 수준의 조건화, 구간 국소화된 분류기 없는 유도(span-localized classifier-free guidance), 그리고 길이 예측을 결합하여 고정 길이 인페인팅과 가변 길이 텍스트 편집을 모두 지원합니다. RealEdit 벤치마크에서 SIEDD는 평가된 방법 중 전반적으로 가장 뛰어난 음성 편집 성능을 달성했습니다. 또한 단일 및 다중 결측 구간 모두의 모든 음성 인페인팅 설정에서 평가된 자기회귀(autoregressive) 베이스라인들을 능가합니다. 이러한 결과는 코덱 계층 구조를 명시적으로 모델링하는 것이 문맥을 보존하는 음성 재구성 및 편집을 실질적으로 향상시킴을 보여줍니다. 전체 코드는 https://github.com/iftachShoham/SIEDD 에서 확인할 수 있습니다.
투기적 디코딩(speculative decoding)은 비용이 적게 드는 초안(draft) 모델을 사용해 토큰을 제안하고 전체 크기의 타깃 모델이 이를 병렬로 검증함으로써 대규모 언어 모델(LLM) 추론을 가속화합니다. 병렬 및 준자기회귀(semi-autoregressive, semi-AR) 초안 모델은 단일 순전파에서 전체 블록을 제안하여 초안 작성 효율성을 높이지만, 이를 학습시키는 과정에서 새로운 어려움이 발생합니다. 특정 위치에 대한 초안 분포는 디코딩 라운드가 어디서 시작하는지에 따라 달라지며, 라운드의 시작 위치는 이전 라운드에서 얼마나 많은 토큰을 수락했는지에 좌우되기 때문입니다. 기존 학습 목표는 일반적으로 이러한 라운드 간 결합을 무시하는 블록 단위 국소 대리(surrogate) 손실에 의존하므로, 전역적인 디코딩 효율성을 직접 최적화하지 못합니다. 본 연구에서는 투기적 디코딩을 마르코프 보상 과정(Markov reward process)으로 표현하여 이러한 초안 모델을 학습 및 평가하기 위한 이론적 프레임워크를 개발합니다. 이 정식화를 통해 상태 점유율로 국소 기각 비용에 가중치를 부여하고 기대 디코딩 라운드 수와 정확히 일치하는 EDR(Expected Decoding Rounds) 목표를 도출합니다. 기존의 대리 목표와 달리 EDR은 보조 하이퍼파라미터를 도입하지 않습니다. 이어 타깃 모델 롤아웃으로부터 편향 없는 확률적 최적화를 지원하는 정확한 시간차(temporal-difference) 그래디언트를 유도합니다. 동일한 프레임워크는 라운드 수에 대한 정확한 오프라인 평가기도 제공하여, 투기적 디코딩을 직접 실행하지 않고도 공유된 타깃 롤아웃 상에서 쌍을 이루는 초안 모델 간의 비교를 가능하게 합니다. 최신 초안 모델인 DSpark와 DFly를 EDR로 파인튜닝한 결과, 평균 수락 길이가 지속적으로 향상되었으며 수학적 추론, 코드 생성, 대화에 걸친 9개 벤치마크 전반에서 기존 학습 목표들을 능가했습니다.
조밀 예측(dense prediction)을 위한 합성 열화(synthetic degradation)를 선택하려면 유한한 학습 예산 하에서 가져오는 일반화 이득인 학습 효용성을 추정해야 합니다. 원본(clean)과 열화된 쌍은 내용과 라벨을 공유하므로, 짧은 학습이 열화로 인해 발생한 초과 오류를 얼마나 줄이는지에 기반한 점수를 고려해 볼 수 있습니다. 그러나 이 격차는 원본 성능이 저하될 때도 줄어들 수 있습니다. 열화된 이미지에서의 개선만을 측정하면 이러한 혼란 요인은 피할 수 있지만, 동일한 양의 원본 학습으로도 얻을 수 있었을 진전을 여전히 열화의 공으로 돌리게 됩니다. 본 연구에서는 열화 유형과 심각도로 정의되는 영역에 대해 \textbf{제어된 감소 가능 열화 격차}(controlled Reducible Degradation Gap, cRDG)를 제안합니다. 공통 체크포인트에서 시작하여, cRDG는 합성 열화 또는 원본 증강(clean augmentation) 중 하나만을 담는 하나의 증강 슬롯에서만 차이가 나는 예산 일치 프로브 두 개를 실행합니다. 점수는 원본 대조 프로브 대비 미학습 열화 이미지에서 얻은 이득입니다. 원본 성능 저하는 별도의 실행 가능성 제약 조건으로 다룹니다. cRDG는 사용 가능한 예산 내에서 열화에 대한 학습이 높은 제어된 이득을 가져오는 교정 가능한 심각도 구간을 밝혀내며, 이 구간은 예측기(predictor), 시작 체크포인트, 학습 예산에 따라 이동합니다. \textbf{감소 가능 구간 선별}(Curation of Reducible Bands, \method)은 예측기를 변경하지 않고 cRDG를 사용하여 합성 데이터를 선택합니다. 시맨틱 세그멘테이션과 현저한 객체 검출(salient object detection)에서, \method{}는 일치된 합성 데이터 예산 및 학습 스케줄 하에서 대표적인 예측기들의 성능을 개선하고, 기존 데이터 생성 파이프라인으로 확장 가능하며, 원본 성능을 보존합니다. 코드 및 지원 자료는 공개될 예정입니다.
Instance Search (INS) is a fundamental problem for many applications, while it is more challenging comparing to traditional image search since the relevancy is defined at the instance level. Existing works have demonstrated the success of many complex ensemble systems that are typically conducted by firstly generating object proposals, and then extracting handcrafted and/or CNN features of each proposal for matching. However, object bounding box proposals and feature extraction are often conducted in two separated steps, thus the effectiveness of these methods collapses. Also, due to the large amount of generated proposals, matching speed becomes the bottleneck that limits its application to large-scale datasets. To tackle these issues, in this paper we propose an effective and efficient Deep Region Hashing (DRH) approach for large-scale INS using an image patch as the query. Specifically, DRH is an end-to-end deep neural network which consists of object proposal, feature extraction, and hash code generation. DRH shares full-image convolutional feature map with the region proposal network, thus enabling nearly cost-free region proposals. Also, each high-dimensional, real-valued region features are mapped onto a low-dimensional, compact binary codes for the efficient object region level matching on large-scale dataset. Experimental results on four datasets show that our DRH can achieve even better performance than the state-of-the-arts in terms of MAP, while the efficiency is improved by nearly 100 times.