PyTorchKR Papers

“Code Generation”

총 10,163편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2407.16944

Adaptive Gradient Regularization: A Faster and Generalizable Optimization Technique for Deep Neural Networks

Stochastic optimization plays a crucial role in the advancement of deep learning technologies. Over the decades, significant effort has been dedicated to improving the training efficiency and robustness of deep neural networks, via various strategies including gradient normalization (GN) and gradient centralization (GC). Nevertheless, to the best of our knowledge, no one has considered to capture the optimal gradient descent trajectory, by adaptively controlling gradient descent direction. To address this concern, this paper is the first attempt to study a new optimization technique for deep neural networks, using the sum normalization of a gradient vector as coefficients, to dynamically regularize gradients and thus to effectively control optimization direction. The proposed technique is hence named as the adaptive gradient regularization (AGR). It can be viewed as an adaptive gradient clipping method. The theoretical analysis reveals that the AGR can effectively smooth the loss landscape, and hence can significantly improve the training efficiency and model generalization performance. We note that AGR can greatly improve the training efficiency of vanilla optimizers' including Adan and AdamW, by adding only three lines of code. The final experiments conducted on image generation, image classification, and language representation, demonstrate that the AGR method can not only improve the training efficiency but also enhance the model generalization performance.

arXiv:2407.16704공식 코드 · Papers with Code

Applying generative neural networks for fast simulations of the ALICE (CERN) experiment

This thesis investigates the application of state-of-the-art advances in generative neural networks for fast simulation of the Zero Degree Calorimeter (ZDC) neutron detector in the ALICE experiment at CERN. Traditional simulation methods using the GEANT Monte Carlo toolkit, while accurate, are computationally demanding. With increasing computational needs at CERN, efficient simulation techniques are essential. The thesis provides a comprehensive literature review on the application of neural networks in computer vision, fast simulations using machine learning, and generative neural networks in high-energy physics. The theory of the analyzed models is also discussed, along with technical aspects and the challenges associated with a practical implementation. The experiments evaluate various neural network architectures, including convolutional neural networks, vision transformers, and MLP-Mixers, as well as generative frameworks such as autoencoders, generative adversarial networks, vector quantization models, and diffusion models. Key contributions include the implementation and evaluation of these models, a significant improvement in the Wasserstein metric compared to existing methods with a low generation time of 5 milliseconds per sample, and the formulation of a list of recommendations for developing models for fast ZDC simulation. Open-source code and detailed hyperparameter settings are provided for reproducibility. Additionally, the thesis outlines future research directions to further enhance simulation fidelity and efficiency.

arXiv:2408.09647공식 코드 · Papers with Code

C2P-CLIP: Injecting Category Common Prompt in CLIP to Enhance Generalization in Deepfake Detection

This work focuses on AIGC detection to develop universal detectors capable of identifying various types of forgery images. Recent studies have found large pre-trained models, such as CLIP, are effective for generalizable deepfake detection along with linear classifiers. However, two critical issues remain unresolved: 1) understanding why CLIP features are effective on deepfake detection through a linear classifier; and 2) exploring the detection potential of CLIP. In this study, we delve into the underlying mechanisms of CLIP's detection capabilities by decoding its detection features into text and performing word frequency analysis. Our finding indicates that CLIP detects deepfakes by recognizing similar concepts (Fig. \ref{fig:fig1} a). Building on this insight, we introduce Category Common Prompt CLIP, called C2P-CLIP, which integrates the category common prompt into the text encoder to inject category-related concepts into the image encoder, thereby enhancing detection performance (Fig. \ref{fig:fig1} b). Our method achieves a 12.41\% improvement in detection accuracy compared to the original CLIP, without introducing additional parameters during testing. Comprehensive experiments conducted on two widely-used datasets, encompassing 20 generation models, validate the efficacy of the proposed method, demonstrating state-of-the-art performance. The code is available at \url{https://github.com/chuangchuangtan/C2P-CLIP-DeepfakeDetection}

arXiv:2502.18219

Synthesizing Consistent Novel Views via 3D Epipolar Attention without Re-Training

Large diffusion models demonstrate remarkable zero-shot capabilities in novel view synthesis from a single image. However, these models often face challenges in maintaining consistency across novel and reference views. A crucial factor leading to this issue is the limited utilization of contextual information from reference views. Specifically, when there is an overlap in the viewing frustum between two views, it is essential to ensure that the corresponding regions maintain consistency in both geometry and appearance. This observation leads to a simple yet effective approach, where we propose to use epipolar geometry to locate and retrieve overlapping information from the input view. This information is then incorporated into the generation of target views, eliminating the need for training or fine-tuning, as the process requires no learnable parameters. Furthermore, to enhance the overall consistency of generated views, we extend the utilization of epipolar attention to a multi-view setting, allowing retrieval of overlapping information from the input view and other target views. Qualitative and quantitative experimental results demonstrate the effectiveness of our method in significantly improving the consistency of synthesized views without the need for any fine-tuning. Moreover, This enhancement also boosts the performance of downstream applications such as 3D reconstruction. The code is available at https://github.com/botaoye/ConsisSyn.

arXiv:2502.20307공식 코드 · Papers with Code

Mobius: Text to Seamless Looping Video Generation via Latent Shift

We present Mobius, a novel method to generate seamlessly looping videos from text descriptions directly without any user annotations, thereby creating new visual materials for the multi-media presentation. Our method repurposes the pre-trained video latent diffusion model for generating looping videos from text prompts without any training. During inference, we first construct a latent cycle by connecting the starting and ending noise of the videos. Given that the temporal consistency can be maintained by the context of the video diffusion model, we perform multi-frame latent denoising by gradually shifting the first-frame latent to the end in each step. As a result, the denoising context varies in each step while maintaining consistency throughout the inference process. Moreover, the latent cycle in our method can be of any length. This extends our latent-shifting approach to generate seamless looping videos beyond the scope of the video diffusion model's context. Unlike previous cinemagraphs, the proposed method does not require an image as appearance, which will restrict the motions of the generated results. Instead, our method can produce more dynamic motion and better visual quality. We conduct multiple experiments and comparisons to verify the effectiveness of the proposed method, demonstrating its efficacy in different scenarios. All the code will be made available.

arXiv:2503.10637

Distilling Diversity and Control in Diffusion Models

Distilled diffusion models suffer from a critical limitation: reduced sample diversity compared to their base counterparts. In this work, we uncover that despite this diversity loss, distilled models retain the fundamental concept representations of base models. We demonstrate control distillation - where control mechanisms like Concept Sliders and LoRAs trained on base models can be seamlessly transferred to distilled models and vice-versa, effectively distilling control without any retraining. This preservation of representational structure prompted our investigation into the mechanisms of diversity collapse during distillation. To understand how distillation affects diversity, we introduce Diffusion Target (DT) Visualization, an analysis and debugging tool that reveals how models predict final outputs at intermediate steps. Through DT-Visualization, we identify generation artifacts, inconsistencies, and demonstrate that initial diffusion timesteps disproportionately determine output diversity, while later steps primarily refine details. Based on these insights, we introduce diversity distillation - a hybrid inference approach that strategically employs the base model for only the first critical timestep before transitioning to the efficient distilled model. Our experiments demonstrate that this simple modification not only restores the diversity capabilities from base to distilled models but surprisingly exceeds it, while maintaining nearly the computational efficiency of distilled inference, all without requiring additional training or model modifications. Our code and data are available at https://distillation.baulab.info

arXiv:2505.23786공식 코드 · Papers with Code

Mind the Gap: A Practical Attack on GGUF Quantization

With the increasing size of frontier LLMs, post-training quantization has become the standard for memory-efficient deployment. Recent work has shown that basic rounding-based quantization schemes pose security risks, as they can be exploited to inject malicious behaviors into quantized models that remain hidden in full precision. However, existing attacks cannot be applied to more complex quantization methods, such as the GGUF family used in the popular ollama and llama$.$cpp frameworks. In this work, we address this gap by introducing the first attack on GGUF. Our key insight is that the quantization error -- the difference between the full-precision weights and their (de-)quantized version -- provides sufficient flexibility to construct malicious quantized models that appear benign in full precision. Leveraging this, we develop an attack that trains the target malicious LLM while constraining its weights based on quantization errors. We demonstrate the effectiveness of our attack on three popular LLMs across nine GGUF quantization data types on three diverse attack scenarios: insecure code generation ($\Delta$=$88.7\%$), targeted content injection ($\Delta$=$85.0\%$), and benign instruction refusal ($\Delta$=$30.1\%$). Our attack highlights that (1) the most widely used post-training quantization method is susceptible to adversarial interferences, and (2) the complexity of quantization schemes alone is insufficient as a defense.

arXiv:2610.38382v1cs.AI

Compute Has Shape: Self-Morphing the Internal Computation of Frozen LLMs

Test-time scaling improves large language models through additional computation, but how to adapt the topology of internal computation to individual inputs remains underexplored. Existing approaches largely fix recurrent or parallel architectures during training, or reconfigure frozen models along a single structural axis. We introduce SelfMorph, which makes computational shape an input-dependent inference-time decision. It unifies the original forward path (BASE), serial reuse of a layer span (REPEAT), and local branching and merging of perturbed hidden states (BRANCH) in a unified space parameterized by type, location, and intensity. Since they offer complementary benefits rather than uniform dominance, we use a lightweight selector that reads the backbone's depth-wise hidden state trajectory on the prompt to choose a morphology before response generation. We train the selector by distilling utility-based distribution from offline morphing that jointly account for task performance and computational cost. Evaluated on three LLMs spanning 4B-27B, SelfMorph achieves a 8.06% mean relative improvement on five in-distribution benchmarks and 4.45% on four unseen benchmarks. It achieves the strongest performance in 24 of 27 settings. These findings identify input-adaptive computational topology as a complementary axis of test-time scaling beyond generating more tokens. Our code is available at https://github.com/creator-xi/SelfMorph.

arXiv:2603.08255v2cs.RO

FlowTouch: View-Invariant Visuo-Tactile Prediction

Tactile sensation is essential for contact-rich manipulation tasks. It provides direct feedback on object geometry, surface properties, and interaction forces, enhancing perception and enabling fine-grained control. An inherent limitation of tactile sensors is that readings are available only when an object is touched. This precludes their use during planning and the initial execution phase of a task. Predicting tactile information from visual information can bridge this gap. A common approach is to learn a direct mapping from camera images to the output of vision-based tactile sensors. However, the resulting model will depend strongly on the specific setup and on how well the camera can capture the area where an object is touched. In this work, we introduce FlowTouch, a novel model for view-invariant visuo-tactile prediction. Our key idea is to use an object's local 3D mesh to encode rich information for predicting tactile patterns while abstracting away from scene-dependent details. FlowTouch integrates scene reconstruction and Flow Matching-based models for image generation. Our results show that FlowTouch is able to bridge the sim-to-real gap and generalize to new sensor instances. We further show that the resulting tactile images can be used for downstream grasp stability prediction. Our code, datasets and videos are available at https://flowtouch.github.io/

arXiv:2611.04472v1cs.CV

Beyond Token Representations: Explicit Visual Object Grounding for Video Reasoning Segmentation

Video Reasoning Segmentation (VRS) aims to infer and segment target objects specified by reasoning queries in videos. Existing methods typically first infer the target and generate a few segmentation tokens for it, which are then fed into a mask decoder for mask prediction. However, such segmentation tokens tend to be both semantically monotonous and spatially ambiguous, making them largely insufficient to represent the target to guide mask prediction. In this paper, we propose an explicit visual object grounding framework for VRS. Specifically, we first propose a spatial-aware prompt generation and refinement scheme, in which we query a multimodal large language model to generate explicit spatial prompts (i.e., bounding box and point set) for the target and then refine them through multi-step dialogue. Furthermore, we introduce a query diversification and alignment module to generate auxiliary queries that describe the same target from different perspectives. We then enforce cross-query consistency between their predicted spatial prompts to alleviate the training bias caused by query limitations. Finally, we select several reliable frames using a prompt-based keyframe selection strategy to support mask decoding and propagation. Extensive experiments on eight datasets demonstrate that our method significantly outperforms state-of-the-art methods. The code is available at https://github.com/viscom-tongji/EVOG.

arXiv:2606.24168v1eess.IV

A Dual Edge Spatial Jacobian Image Graph for Interpretable Diabetic Retinopathy Grading

당뇨망막병증(DR) 등급화에서 병변 증거가 망막 혈관 주변에 어떻게 분포하는지와 혈관 바이오마커와의 관계를 해석하기 위해, 혈관–병변 공간 엣지와 임베딩–바이오마커 Jacobian 엣지를 2토큰 어텐션으로 융합한 이중 엣지 공간-Jacobian 이미지 그래프를 제안한다. APTOS 2,910장에서 5등급 분류 정확도 0.8076·QWK 0.8312, 전환 필요 DR 이진 분류 AUROC 0.9711을 달성했으며, 임상 배포용 분류기가 아니라 병변–바이오마커 가설 생성을 위한 설명 가능한 표현학습 도구로 제시된다.

arXiv:2607.29302v1cs.RO

BWM: 로봇 학습을 위한 저비용 고충실도 월드 시뮬레이터

BWM: A Low-Cost High-Fidelity World Simulator for Robot Learning

신뢰할 수 있는 로봇 학습을 위해서는 물리적 하드웨어에서 실행하기 전, 위험하고 실패하기 쉬운 결과를 포함하여 행동의 영향을 예측할 수 있는 월드 시뮬레이터(world simulator)가 필요합니다. 기존의 물리 시뮬레이터는 상당한 에셋 구축 및 보정이 필요하고 여전히 심투리얼(sim-to-real) 격차에 직면해 있는 반면, 비디오 생성기는 세분화된 로봇 행동에 대한 반응을 정밀하게 제어하기 어려운 경우가 많습니다. 본 논문에서는 로봇 조작을 위한 오픈소스 저비용 고충실도(high-fidelity) 월드 시뮬레이터인 BWM(Boundless World Model)을 제시합니다. BWM은 미래 관측치를 상태 기반(stateful) 자기회귀(autoregressive) 방식으로 예측하기 위해 초기 환경 안내(initial-environment guidance), 동적 시각적 이력, 시간적으로 정렬된 로봇 행동 조건화를 결합한 행동 조건부 월드 모델입니다. 궤적 재생(trajectory replay), 중첩 클립 샘플링, 초기 관측치 강화를 통해 행동과 정렬된 학습 클립을 구축합니다. BWM은 행동 정렬 롤아웃(rollout)으로 모방 학습(imitation learning) 데이터를 증강하는 데이터 엔진의 역할과, 폐루프(closed-loop) 평가, 위험 예측, 정책 순위 매김을 수행하는 정책 평가기(policy evaluator) 역할을 합니다. WorldArena 벤치마크 및 실제 로봇을 통한 실험은 데이터 엔진과 정책 평가기 설정 전반에서 향상된 시뮬레이터 충실도와 기능적 유용성을 입증합니다. BWM은 WorldArena 챌린지의 트랙 1과 두 개의 트랙 2 응용 분야 전반에서 종합 1위를 차지했습니다. 모델 체크포인트, 학습 및 추론 코드, 데이터 생성 및 정책 평가용 인터페이스를 포함한 BWM 오픈소스 생태계를 공개합니다.

arXiv:2608.08727v1cs.CV

TomaMMU: 토마토 잎 질병을 위한 종합적인 멀티모달 이해 벤치마크

TomaMMU: A Comprehensive Multimodal Understanding Benchmark for Tomato Leaf Diseases

이러한 격차를 해결하기 위해, 우리는 토마토 질병 이해에 대한 비전-언어 모델(VLM)을 평가하기 위한 벤치마크인 TomaBench와 함께 대규모 토마토 잎 질병 멀티모달 이해 데이터셋인 TomaMMU를 소개한다. TomaMMU는 15개 범주에 걸친 28,808개의 고품질 이미지와 데이터 수집(Data Collection), 인적 주석 작업(Human Annotation), 질의응답 생성(Question-Answer Generation)으로 구성된 3단계 파이프라인을 통해 생성된 213,119개의 사람이 주석을 단 시각적 질의응답 쌍으로 구성된다. 이러한 기반을 바탕으로 TomaBench는 기본 인지(Basic Perception), 병리학적 이해(Pathology Understanding), 전문가 진단(Expert Diagnosis)에 걸친 계층적 3단계 분류 체계로 7개의 농업 과업을 구성하며, 이는 저수준 시각 인식부터 고수준 진단 추론까지 체계적인 평가를 가능하게 한다. 이 과업들은 시각적 증상 인식, 분류학적 관계, 진단 추론을 평가하여 모델이 식물 병리학을 얼마나 잘 파악하는지에 대한 종합적인 관점을 제공한다. 14개의 최첨단 VLM을 대상으로 한 실험 결과, 미세 인식(fine-grained recognition) 및 사실 기반 추론에서 뚜렷한 격차가 드러났으며, 고난도 객관식 질문(MCQ)과 개방형 질문 모두에서 일관되게 저조한 성능을 보였다. 이러한 결과는 현재의 VLM이 시각적 인식을 신뢰할 수 있는 진단 지식으로 변환하는 데 어려움을 겪고 있음을 시사하며, 특정 도메인 적응의 필요성을 촉구한다. TomaMMU를 사용한 단순한 미세조정만으로도 이러한 격차를 크게 좁혀 난도 높은 MCQ의 정확도를 96.09%로 높였으며, 최근 VLM들을 능가하고 향후 연구를 위한 유망한 방향을 제시한다. 모든 데이터와 코드는 https://huggingface.co/datasets/enalis/TomaMMU 에서 제공된다.

arXiv:2610.00949v1cs.AI

PG-SFT: 오프라인 에이전트 미세조정에서 능력 획득과 유지 간의 균형 잡기

PG-SFT: Balancing Capability Acquisition and Retention in Offline Agent Fine-Tuning

오프라인 에이전트(agent) 궤적에 대한 지도 미세조정(SFT)은 특화된 도구 사용 에이전트를 학습(training)하기 위한 표준적인 접근법이지만, 모델이 추론과 행동을 토큰(token) 단위로 모방하도록 강제하는 것은 베이스 모델의 다른 능력(예: 일반 추론, 도구 호출, 코드 생성)을 손상시킬 수 있습니다. 본 연구에서는 에이전트 궤적 SFT 과정에서 새로운 능력을 획득하는 것과 기존 능력을 보존하는 것 사이의 트레이드오프(trade-off)를 어떻게 더 잘 균형 잡을 수 있는지에 대해 중점적으로 연구합니다. 본 실험 환경에서 여러 베이스라인을 비교한 결과, 표준 SFT는 목표 벤치마크 점수를 향상시키는 반면 목표 외 여러 벤치마크 점수를 낮추었습니다. 한편, KL 페널티를 사용해 분포 드리프트를 단순히 억제하거나 업데이트 크기를 제한하는 방식으로는 이러한 성능 저하 추세를 피할 수 없었습니다. 최근의 토큰별 적응형 학습 목표에서 영감을 받아, 본 연구에서는 에이전트 궤적의 턴(turn) 수준 정보 이득을 지표로 활용하여 지도(supervision) 강도를 조절하는 Privilege-Guided SFT(PG-SFT)를 제안합니다. PG-SFT는 평가된 벤치마크에서 목표 작업 성능의 미미한 저하를 대가로 분포 드리프트와 광범위한 능력 저하를 대폭 줄여, 관찰된 트레이드오프를 훨씬 더 유리하게 만듭니다. 본 연구의 결과는 획득-유지 간의 트레이드오프 균형이 모델이 기본 동작에 고정되어 있는지 여부뿐만 아니라, 지도가 해당 동작에서 어디서 얼마나 강하게 벗어나야 하는지에도 달려 있음을 시사합니다.

arXiv:1910.07224공식 코드 · Papers with Code

Teacher algorithms for curriculum learning of Deep RL in continuously parameterized environments

We consider the problem of how a teacher algorithm can enable an unknown Deep Reinforcement Learning (DRL) student to become good at a skill over a wide range of diverse environments. To do so, we study how a teacher algorithm can learn to generate a learning curriculum, whereby it sequentially samples parameters controlling a stochastic procedural generation of environments. Because it does not initially know the capacities of its student, a key challenge for the teacher is to discover which environments are easy, difficult or unlearnable, and in what order to propose them to maximize the efficiency of learning over the learnable ones. To achieve this, this problem is transformed into a surrogate continuous bandit problem where the teacher samples environments in order to maximize absolute learning progress of its student. We present a new algorithm modeling absolute learning progress with Gaussian mixture models (ALP-GMM). We also adapt existing algorithms and provide a complete study in the context of DRL. Using parameterized variants of the BipedalWalker environment, we study their efficiency to personalize a learning curriculum for different learners (embodiments), their robustness to the ratio of learnable/unlearnable environments, and their scalability to non-linear and high-dimensional parameter spaces. Videos and code are available at https://github.com/flowersteam/teachDeepRL.

arXiv:2007.03310공식 코드 · Papers with Code

DAM: Deliberation, Abandon and Memory Networks for Generating Detailed and Non-repetitive Responses in Visual Dialogue

Visual Dialogue task requires an agent to be engaged in a conversation with human about an image. The ability of generating detailed and non-repetitive responses is crucial for the agent to achieve human-like conversation. In this paper, we propose a novel generative decoding architecture to generate high-quality responses, which moves away from decoding the whole encoded semantics towards the design that advocates both transparency and flexibility. In this architecture, word generation is decomposed into a series of attention-based information selection steps, performed by the novel recurrent Deliberation, Abandon and Memory (DAM) module. Each DAM module performs an adaptive combination of the response-level semantics captured from the encoder and the word-level semantics specifically selected for generating each word. Therefore, the responses contain more detailed and non-repetitive descriptions while maintaining the semantic accuracy. Furthermore, DAM is flexible to cooperate with existing visual dialogue encoders and adaptive to the encoder structures by constraining the information selection mode in DAM. We apply DAM to three typical encoders and verify the performance on the VisDial v1.0 dataset. Experimental results show that the proposed models achieve new state-of-the-art performance with high-quality responses. The code is available at https://github.com/JXZe/DAM.

arXiv:2211.16799공식 코드 · Papers with Code

NOPE-SAC: Neural One-Plane RANSAC for Sparse-View Planar 3D Reconstruction

This paper studies the challenging two-view 3D reconstruction in a rigorous sparse-view configuration, which is suffering from insufficient correspondences in the input image pairs for camera pose estimation. We present a novel Neural One-PlanE RANSAC framework (termed NOPE-SAC in short) that exerts excellent capability to learn one-plane pose hypotheses from 3D plane correspondences. Building on the top of a siamese plane detection network, our NOPE-SAC first generates putative plane correspondences with a coarse initial pose. It then feeds the learned 3D plane parameters of correspondences into shared MLPs to estimate the one-plane camera pose hypotheses, which are subsequently reweighed in a RANSAC manner to obtain the final camera pose. Because the neural one-plane pose minimizes the number of plane correspondences for adaptive pose hypotheses generation, it enables stable pose voting and reliable pose refinement in a few plane correspondences for the sparse-view inputs. In the experiments, we demonstrate that our NOPE-SAC significantly improves the camera pose estimation for the two-view inputs with severe viewpoint changes, setting several new state-of-the-art performances on two challenging benchmarks, i.e., MatterPort3D and ScanNet, for sparse-view 3D reconstruction. The source code is released at https://github.com/IceTTTb/NopeSAC for reproducible research.

arXiv:2303.14742공식 코드 · Papers with Code

Exploring the Impact of Instruction Data Scaling on Large Language Models: An Empirical Study on Real-World Use Cases

The success of ChatGPT has recently attracted numerous efforts to replicate it, with instruction-tuning strategies being a key factor in achieving remarkable results. Instruction-tuning not only significantly enhances the model's performance and generalization but also makes the model's generated results more consistent with human speech patterns. However current research rarely studies the impact of different amounts of instruction data on model performance, especially in the real-world use cases. In this paper we explore the performance of large language models based on instruction tuning across different scales of instruction data. An evaluation dataset consisting of 12 major online use cases is constructed in the experiment. With Bloomz-7B1-mt as the base model, the results show that 1) merely increasing the amount of instruction data leads to continuous improvement in tasks such as open-ended generation, 2) in tasks such as math and code, the model performance curve remains quite flat while increasing data size. We further analyze the possible causes of these phenomena and propose potential future research directions such as effectively selecting high-quality training data, scaling base models and training methods specialized for hard tasks. We will release our training and evaluation datasets, as well as model checkpoints.

arXiv:2310.12746공식 코드 · Papers with Code

TabuLa: Harnessing Language Models for Tabular Data Synthesis

Tabular data synthesis is crucial for addressing privacy and security concerns in industries reliant on tabular data. While recent advancements adopt large language models (LLMs) for realistic tabular data generation, their long training times and limited reusability hinder practical applications. In this paper, we propose Tabula, a tabular data synthesizer that leverages the structure of LLM. Unlike state-of-the-art (SOTA) LLM-based tabular data synthesizers that rely on pre-trained LLMs, Tabula discards the pre-trained weights originally designed for natural language tasks, focusing instead on a tailored approach for tabular data. In addition, Tabula introduces a token sequence compression strategy that significantly reduces training time while maintaining data quality, alongside a novel token padding method that improves sequence alignment across training batches. Experiments on six datasets show that Tabula achieves superior synthetic data utility compared to current SOTA methods. Additionally, the results demonstrate that Tabula model trained on tabular datasets serves effectively as a foundational model for synthesizing new tabular datasets. Furthermore, the proposed padding method outperforms the conventional left and right padding strategies. Finally, the results highlight that Tabula averagely reduces training time per epoch by 46.2% compared to state-of-the-art LLM approaches while achieving higher data utility. Our code is available at https://github.com/zhao-zilong/Tabula

arXiv:2310.13120공식 코드 · Papers with Code

RSAdapter: Adapting Multimodal Models for Remote Sensing Visual Question Answering

In recent years, with the rapid advancement of transformer models, transformer-based multimodal architectures have found wide application in various downstream tasks, including but not limited to Image Captioning, Visual Question Answering (VQA), and Image-Text Generation. However, contemporary approaches to Remote Sensing (RS) VQA often involve resource-intensive techniques, such as full fine-tuning of large models or the extraction of image-text features from pre-trained multimodal models, followed by modality fusion using decoders. These approaches demand significant computational resources and time, and a considerable number of trainable parameters are introduced. To address these challenges, we introduce a novel method known as RSAdapter, which prioritizes runtime and parameter efficiency. RSAdapter comprises two key components: the Parallel Adapter and an additional linear transformation layer inserted after each fully connected (FC) layer within the Adapter. This approach not only improves adaptation to pre-trained multimodal models but also allows the parameters of the linear transformation layer to be integrated into the preceding FC layers during inference, reducing inference costs. To demonstrate the effectiveness of RSAdapter, we conduct an extensive series of experiments using three distinct RS-VQA datasets and achieve state-of-the-art results on all three datasets. The code for RSAdapter is available online at https://github.com/Y-D-Wang/RSAdapter.