PyTorchKR Papers

“Code Generation”

총 10,163편
태그·기간 등 상세 필터

카테고리 탐색

arXiv:2408.14765

CrossViewDiff: A Cross-View Diffusion Model for Satellite-to-Street View Synthesis

Satellite-to-street view synthesis aims at generating a realistic street-view image from its corresponding satellite-view image. Although stable diffusion models have exhibit remarkable performance in a variety of image generation applications, their reliance on similar-view inputs to control the generated structure or texture restricts their application to the challenging cross-view synthesis task. In this work, we propose CrossViewDiff, a cross-view diffusion model for satellite-to-street view synthesis. To address the challenges posed by the large discrepancy across views, we design the satellite scene structure estimation and cross-view texture mapping modules to construct the structural and textural controls for street-view image synthesis. We further design a cross-view control guided denoising process that incorporates the above controls via an enhanced cross-view attention module. To achieve a more comprehensive evaluation of the synthesis results, we additionally design a GPT-based scoring method as a supplement to standard evaluation metrics. We also explore the effect of different data sources (e.g., text, maps, building heights, and multi-temporal satellite imagery) on this task. Results on three public cross-view datasets show that CrossViewDiff outperforms current state-of-the-art on both standard and GPT-based evaluation metrics, generating high-quality street-view panoramas with more realistic structures and textures across rural, suburban, and urban scenes. The code and models of this work will be released at https://opendatalab.github.io/CrossViewDiff/.

arXiv:2409.01691

When 3D Partial Points Meets SAM: Tooth Point Cloud Segmentation with Sparse Labels

Tooth point cloud segmentation is a fundamental task in many orthodontic applications. Current research mainly focuses on fully supervised learning which demands expensive and tedious manual point-wise annotation. Although recent weakly-supervised alternatives are proposed to use weak labels for 3D segmentation and achieve promising results, they tend to fail when the labels are extremely sparse. Inspired by the powerful promptable segmentation capability of the Segment Anything Model (SAM), we propose a framework named SAMTooth that leverages such capacity to complement the extremely sparse supervision. To automatically generate appropriate point prompts for SAM, we propose a novel Confidence-aware Prompt Generation strategy, where coarse category predictions are aggregated with confidence-aware filtering. Furthermore, to fully exploit the structural and shape clues in SAM's outputs for assisting the 3D feature learning, we advance a Mask-guided Representation Learning that re-projects the generated tooth masks of SAM into 3D space and constrains these points of different teeth to possess distinguished representations. To demonstrate the effectiveness of the framework, we conduct experiments on the public dataset and surprisingly find with only 0.1\% annotations (one point per tooth), our method can surpass recent weakly supervised methods by a large margin, and the performance is even comparable to the recent fully-supervised methods, showcasing the significant potential of applying SAM to 3D perception tasks with sparse labels. Code is available at https://github.com/CUHK-AIM-Group/SAMTooth.

arXiv:2410.03026공식 코드 · Papers with Code

Characterizing Context Influence and Hallucination in Summarization

Although Large Language Models (LLMs) have achieved remarkable performance in numerous downstream tasks, their ubiquity has raised two significant concerns. One is that LLMs can hallucinate by generating content that contradicts relevant contextual information; the other is that LLMs can inadvertently leak private information due to input regurgitation. Many prior works have extensively studied each concern independently, but none have investigated them simultaneously. Furthermore, auditing the influence of provided context during open-ended generation with a privacy emphasis is understudied. To this end, we comprehensively characterize the influence and hallucination of contextual information during summarization. We introduce a definition for context influence and Context-Influence Decoding (CID), and then we show that amplifying the context (by factoring out prior knowledge) and the context being out of distribution with respect to prior knowledge increases the context's influence on an LLM. Moreover, we show that context influence gives a lower bound of the private information leakage of CID. We corroborate our analytical findings with experimental evaluations that show improving the F1 ROGUE-L score on CNN-DM for LLaMA 3 by $\textbf{10}$% over regular decoding also leads to $\textbf{1.5x}$ more influence by the context. Moreover, we empirically evaluate how context influence and hallucination are affected by (1) model capacity, (2) context size, (3) the length of the current response, and (4) different token $n$-grams of the context. Our code can be accessed here: https://github.com/james-flemings/context_influence.

arXiv:2503.10529

PiSA: A Self-Augmented Data Engine and Training Strategy for 3D Understanding with Large Models

3D Multimodal Large Language Models (MLLMs) have recently made substantial advancements. However, their potential remains untapped, primarily due to the limited quantity and suboptimal quality of 3D datasets. Current approaches attempt to transfer knowledge from 2D MLLMs to expand 3D instruction data, but still face modality and domain gaps. To this end, we introduce PiSA-Engine (Point-Self-Augmented-Engine), a new framework for generating instruction point-language datasets enriched with 3D spatial semantics. We observe that existing 3D MLLMs offer a comprehensive understanding of point clouds for annotation, while 2D MLLMs excel at cross-validation by providing complementary information. By integrating holistic 2D and 3D insights from off-the-shelf MLLMs, PiSA-Engine enables a continuous cycle of high-quality data generation. We select PointLLM as the baseline and adopt this co-evolution training framework to develop an enhanced 3D MLLM, termed PointLLM-PiSA. Additionally, we identify limitations in previous 3D benchmarks, which often feature coarse language captions and insufficient category diversity, resulting in inaccurate evaluations. To address this gap, we further introduce PiSA-Bench, a comprehensive 3D benchmark covering six key aspects with detailed and diverse labels. Experimental results demonstrate PointLLM-PiSA's state-of-the-art performance in zero-shot 3D object captioning and generative classification on our PiSA-Bench, achieving significant improvements of 46.45% (+8.33%) and 63.75% (+16.25%), respectively. We will release the code, datasets, and benchmark.

arXiv:2505.05031공식 코드 · Papers with Code

LSRP: A Leader-Subordinate Retrieval Framework for Privacy-Preserving Cloud-Device Collaboration

Cloud-device collaboration leverages on-cloud Large Language Models (LLMs) for handling public user queries and on-device Small Language Models (SLMs) for processing private user data, collectively forming a powerful and privacy-preserving solution. However, existing approaches often fail to fully leverage the scalable problem-solving capabilities of on-cloud LLMs while underutilizing the advantage of on-device SLMs in accessing and processing personalized data. This leads to two interconnected issues: 1) Limited utilization of the problem-solving capabilities of on-cloud LLMs, which fail to align with personalized user-task needs, and 2) Inadequate integration of user data into on-device SLM responses, resulting in mismatches in contextual user information. In this paper, we propose a Leader-Subordinate Retrieval framework for Privacy-preserving cloud-device collaboration (LSRP), a novel solution that bridges these gaps by: 1) enhancing on-cloud LLM guidance to on-device SLM through a dynamic selection of task-specific leader strategies named as user-to-user retrieval-augmented generation (U-U-RAG), and 2) integrating the data advantages of on-device SLMs through small model feedback Direct Preference Optimization (SMFB-DPO) for aligning the on-cloud LLM with the on-device SLM. Experiments on two datasets demonstrate that LSRP consistently outperforms state-of-the-art baselines, significantly improving question-answer relevance and personalization, while preserving user privacy through efficient on-device retrieval. Our code is available at: https://github.com/Zhang-Yingyi/LSRP.

arXiv:2505.19849

HIT Model: A Hierarchical Interaction-Enhanced Two-Tower Model for Pre-Ranking Systems

Online display advertising platforms rely on pre-ranking systems to efficiently filter and prioritize candidate ads from large corpora, balancing relevance to users with strict computational constraints. The prevailing two-tower architecture, though highly efficient due to its decoupled design and pre-caching, suffers from cross-domain interaction and coarse similarity metrics, undermining its capacity to model complex user-ad relationships. In this study, we propose the Hierarchical Interaction-Enhanced Two-Tower (HIT) model, a new architecture that augments the two-tower paradigm with two key components: $\textit{generators}$ that pre-generate holistic vectors incorporating coarse-grained user-ad interactions through a dual-generator framework with a cosine-similarity-based generation loss as the training objective, and $\textit{multi-head representers}$ that project embeddings into multiple latent subspaces to capture fine-grained, multi-faceted user interests and multi-dimensional ad attributes. This design enhances modeling effectiveness without compromising inference efficiency. Extensive experiments on public datasets and large-scale online A/B testing on Tencent's advertising platform demonstrate that HIT significantly outperforms several baselines in relevance metrics, yielding a $1.66\%$ increase in Gross Merchandise Volume and a $1.55\%$ improvement in Return on Investment, alongside similar serving latency to the vanilla two-tower models. The HIT model has been successfully deployed in Tencent's online display advertising system, serving billions of impressions daily. The code is available at https://anonymous.4open.science/r/HIT_model-5C23.

arXiv:2611.14571v1cs.CE

MechFoundry: An agentic loop that poses and performs engineering analyses and optimization of real-world objects

AI agents can automate software engineering because a compiler or a test suite checks that the code does what the user or the agent intended. In engineering analysis, however, the steps that precede the computation have few easily verifiable checks: modeling a design or an object, deciding how it is used, where supports and loads are applied, and what material each part is made of. A human expert therefore makes these decisions, carries out the steps, and judges the result, one object at a time. We present MechFoundry, an agentic loop coupled with bespoke software that automates structural analyses of real-world objects without a human in the loop. From an object's lexical description, it generates a shape, poses a loading case relevant to how the object is used, assigns a material to each part, solves the nonlinear structural response, and improves the design against the loads it carries. A differentiable GPU solver built on an adaptive octree that need not conform to the surface enables the transition from shape generation to simulation and optimization. Across 50 everyday and engineering objects, the agent posed and solved a loading case in 6.5 minutes and for US$1.3 per object on average. Without a test suite to check against, we had language-model reviewers grade each session, as a first indication of where the agent's engineering judgment holds and where it fails.

arXiv:2606.21940v1cs.LG

DevoTG: C. elegans 발달 커넥토믹스 모델링을 위한 시간적 그래프 신경망

DevoTG: Temporal Graph Neural Networks for Modeling C. elegans Developmental Connectomics

신경계가 출생부터 성체까지 스스로 배선되는 방식을 이해하는 것은 발달 신경과학의 근본적인 과제이다. 우리는 시간적 그래프 신경망(TGN)을 C. elegans 신경 발달의 두 가지 상호 보완적 표현에 적용하는 시간적 그래프 프레임워크 DevoTG를 제시한다: 세포 계보 데이터에서 도출된 세포 분열 이벤트의 연속 시간 동적 그래프(CTDG)와, 8개의 재구성된 전자현미경 데이터셋에 걸쳐 발달 중인 시냅스 커넥톰의 이산 시간 동적 그래프(DTDG)이다. 계보 예측 과제에서 우리의 TGN은 평균 테스트 AUC 0.839 ± 0.007(5개 시드; 검증 AUC 0.937 ± 0.001)을 달성하며, 동일한 아키텍처의 정적 GNN을 26 AUC 포인트(0.577 ± 0.080) 차이로 능가함으로써 시간적 메모리가 결정적 요인임을 입증한다. 커넥톰 DTDG에 적용된 DevoTG는 발달 과정(L1 출생~성체)에 걸쳐 225개 뉴런과 858~2,496개 연결에서 세 가지 연결 안정성 분류(안정적, 발달적, 가변적)를 식별하며, Witvliet et al.의 개체 변이성 분류에 대한 시간적 그래프 이론적 보완을 제공한다. 허브 명령 개재뉴런 AVA, AVB, AVE에 대한 분석은 이들의 지속적인 중심성과 유충 단계 전반에 걸쳐 통합 역할이 점진적으로 강화되는 방식을 밝혀낸다. 대화형 시각화(3D 애니메이션 네트워크, 중심성 히트맵, 시공간 계보 그래프)를 통해 발달 동역학을 생물학적 가설 생성에 활용하기 쉽게 제공한다. DevoTG는 오픈 소스이며 다른 발달 중인 신경계로의 확장을 위해 설계되었다. 코드는 https://github.com/DevoLearn/DevoGraph/tree/main/DevoTG에서 공개적으로 이용 가능하다.

arXiv:2607.00060v1cs.CV

시너지적 인식-추론 거버넌스: 검증 가능한 해부학적 근거로 의료 MLLM 그라운딩

Synergistic Perception-Reasoning Governance: Grounding Medical MLLMs with Verifiable Anatomical Evidence

MedSAM으로 추출한 해부학적 관심 영역(ROI) 사전 정보를 활용해 시각적 활성화를 재보정하고 해부학적 좌표를 이산 의미 토큰으로 프롬프트에 주입하는 학습 불필요(training-free) 이중 근거 주입 프레임워크 SPRG를 제안한다. LLaVA, Qwen3-VL, InternVL 등 6개 의료 MLLM에 대한 실험에서 폐쇄형 정확도를 최대 약 6% 향상시키고 개방형 환각을 최대 약 35% 감소시켜 기존 기준 모델을 일관되게 능가했다.

arXiv:2608.23955v2cs.NE

신뢰하되 검증하라: 디지털 진화를 위한 최선 노력 고성능 컴퓨팅의 엄밀한 프로파일링

Trust, but Verify: Rigorously Profiling Best-Effort High-Performance Computing for Digital Evolution

고성능 컴퓨팅(HPC) 기술의 발전은 가용 연산 능력을 급격히 지속해서 증가시키고 있습니다. 디지털 진화(digital evolution)의 맥락에서 이러한 폭발적인 성장은 다중 스케일 생물학적 현상에 대한 가설 기반 탐구와 난해한 문제 영역을 겨냥한 응용 주도형 진화적 최적화(evolutionary optimization) 모두를 발전시킬 기회를 제공합니다. 특히 880,000개 프로세서를 탑재한 Cerebras Wafer-Scale Engine(WSE)과 같은 차세대 AI/ML 하드웨어 가속기 플랫폼의 등장으로 특별한 기회가 생겨나고 있습니다. 그러나 이러한 하드웨어는 온디바이스 데이터 저장 및 이동을 제약하며, 수많은 장치 구성 요소 전반에서 발생하는 장애 취약성으로 인해 이 문제는 더욱 가중됩니다. 전통적인 결정론적(deterministic) 컴퓨팅 패러다임에서 벗어난 최선 노력(best-effort) 기반의 완화 방식은 이러한 제약을 수용하는 데 도움이 될 수 있지만, 재현성을 복잡하게 만들고 인위적인 편향(artifactual bias)을 도입할 위험이 있습니다. 본 연구에서는 이러한 우려 사항을 탐구하여 최선 노력 코드의 런타임 동작을 측정하는 프레임워크를 개발하고, 디지털 진화 프로젝트에서의 최선 노력 컴퓨팅 사례 연구를 검토합니다. 첫 번째 사례 연구는 다세포성(multicellularity) 진화 모델에 최선 노력 CPU 클러스터 다중 처리를 적용한 것으로, 64개 프로세스에서 92%의 스케일링 효율($2.1\times$ 가속)을 제공하고 하드웨어 이상 상황에서도 견고한 서비스 품질 중앙값을 보였습니다. 두 번째 사례 연구는 WSE 기반 시뮬레이션을 검토하여, 하드웨어 결함을 허용하는 희소하고 비동기적인 디바이스-투-호스트(device-to-host) 샘플링을 통해 시공간적 개체군 이력을 추적하는 최선 노력 전략을 입증합니다. 결론적으로, 잠재적인 최선 노력 완화의 다양한 형태와 범위 전반에서 디지털 진화가 탈결정론적(post-deterministic) HPC 패러다임을 개발하는 데 기여할 수 있는 독보적인 위치에 있다고 주장합니다.

arXiv:2610.02799v1cs.CV

FUSEye: 중첩 뷰와 제로 초기화 어댑터를 활용한 경량 학습 어안 객체 탐지

FUSEye: Training-Light Fisheye Detection with Overlapping Views and Zero-Initialized Adapters

어안 렌즈(fisheye) 카메라는 이동 로봇에게 단일 센서 기반의 저비용 주변 시야를 제공하지만, 실무에서 흔히 재사용되는 COCO 사전학습 탐지기는 이러한 카메라에서 제대로 작동하지 않는다. 강한 방사 왜곡(radial distortion)이 국소적 이미지 구조를 뒤틀리게 하고 경계부 압축으로 인해 객체가 거의 보이지 않는 크기로 줄어들기 때문이다. 전체 미세조정(fine-tuning)을 수행하면 이러한 격차를 크게 줄일 수 있지만 대량의 어안 라벨과 연산 자원이 필요하다. 본 논문에서는 백본(backbone)이 동결된 COCO 사전학습 초대형 YOLO26 탐지기(YOLO26-x)를 어안 탐지기로 전환하는 경량 학습 프레임워크인 FUSEye를 제안한다. FUSEye는 삽입된 모듈과 사전학습된 탐지 헤드를 업데이트하면서 약 22만 7천 개의 새로운 파라미터만을 추가한다. 또한 인과적으로 연결된 세 가지 수준에서 전이 격차를 해결한다. 입력 수준에서는 중첩 그리드 뷰 생성 및 박스 재매핑(GridViews)을 통해 압축된 경계 영역을 확대한다. 특징(feature) 수준에서는 제로 초기화 잔차 어댑터(Z-Adapters)가 왜곡으로 인해 유발된 특징 불일치를 교정한다. 결정 수준에서는 학습된 교차 투영 일치도 융합(AgreeFusion)을 통해 여러 뷰에 걸쳐 일관된 근거로 뒷받침되는 경우에만 낮은 신뢰도의 탐지 결과를 승격시킨다. WoodScape 서라운드 뷰 어안 벤치마크에서 FUSEye는 YOLO26-x의 mAP50을 0.148에서 0.266으로 끌어올렸으며 전체 미세조정 정확도의 84.3%를 유지했다. 나아가 라벨링된 학습 이미지의 25%만 무작위로 사용했을 때도 0.2597 mAP50을 달성하여 전체 라벨 사용 성능의 97.6%를 보존했다. 또한 FUSEye는 YOLOv8부터 YOLO11까지의 탐지기 성능을 일관되게 향상시켜 이 기법이 아키텍처에 구애받지 않음을 보였다. 소스 코드는 https://github.com/Su-wenya/FUSEye 에서 공개될 예정이다.

arXiv:2004.06816

Bounding boxes for weakly supervised segmentation: Global constraints get close to full supervision

We propose a novel weakly supervised learning segmentation based on several global constraints derived from box annotations. Particularly, we leverage a classical tightness prior to a deep learning setting via imposing a set of constraints on the network outputs. Such a powerful topological prior prevents solutions from excessive shrinking by enforcing any horizontal or vertical line within the bounding box to contain, at least, one pixel of the foreground region. Furthermore, we integrate our deep tightness prior with a global background emptiness constraint, guiding training with information outside the bounding box. We demonstrate experimentally that such a global constraint is much more powerful than standard cross-entropy for the background class. Our optimization problem is challenging as it takes the form of a large set of inequality constraints on the outputs of deep networks. We solve it with sequence of unconstrained losses based on a recent powerful extension of the log-barrier method, which is well-known in the context of interior-point methods. This accommodates standard stochastic gradient descent (SGD) for training deep networks, while avoiding computationally expensive and unstable Lagrangian dual steps and projections. Extensive experiments over two different public data sets and applications (prostate and brain lesions) demonstrate that the synergy between our global tightness and emptiness priors yield very competitive performances, approaching full supervision and outperforming significantly DeepCut. Furthermore, our approach removes the need for computationally expensive proposal generation. Our code is shared anonymously.

arXiv:2103.14024공식 코드 · Papers with Code

PlenOctrees for Real-time Rendering of Neural Radiance Fields

We introduce a method to render Neural Radiance Fields (NeRFs) in real time using PlenOctrees, an octree-based 3D representation which supports view-dependent effects. Our method can render 800x800 images at more than 150 FPS, which is over 3000 times faster than conventional NeRFs. We do so without sacrificing quality while preserving the ability of NeRFs to perform free-viewpoint rendering of scenes with arbitrary geometry and view-dependent effects. Real-time performance is achieved by pre-tabulating the NeRF into a PlenOctree. In order to preserve view-dependent effects such as specularities, we factorize the appearance via closed-form spherical basis functions. Specifically, we show that it is possible to train NeRFs to predict a spherical harmonic representation of radiance, removing the viewing direction as an input to the neural network. Furthermore, we show that PlenOctrees can be directly optimized to further minimize the reconstruction loss, which leads to equal or better quality compared to competing methods. Moreover, this octree optimization step can be used to reduce the training time, as we no longer need to wait for the NeRF training to converge fully. Our real-time neural rendering approach may potentially enable new applications such as 6-DOF industrial and product visualizations, as well as next generation AR/VR systems. PlenOctrees are amenable to in-browser rendering as well; please visit the project page for the interactive online demo, as well as video and code: https://alexyu.net/plenoctrees

arXiv:2209.15000

REST: REtrieve & Self-Train for generative action recognition

This work is on training a generative action/video recognition model whose output is a free-form action-specific caption describing the video (rather than an action class label). A generative approach has practical advantages like producing more fine-grained and human-readable output, and being naturally open-world. To this end, we propose to adapt a pre-trained generative Vision & Language (V&L) Foundation Model for video/action recognition. While recently there have been a few attempts to adapt V&L models trained with contrastive learning (e.g. CLIP) for video/action, to the best of our knowledge, we propose the very first method that sets outs to accomplish this goal for a generative model. We firstly show that direct fine-tuning of a generative model to produce action classes suffers from severe overfitting. To alleviate this, we introduce REST, a training framework consisting of two key components: an unsupervised method for adapting the generative model to action/video by means of pseudo-caption generation and Self-training, i.e. without using any action-specific labels; (b) a Retrieval approach based on CLIP for discovering a diverse set of pseudo-captions for each video to train the model. Importantly, we show that both components are necessary to obtain high accuracy. We evaluate REST on the problem of zero-shot action recognition where we show that our approach is very competitive when compared to contrastive learning-based methods. Code will be made available.

arXiv:2212.04489공식 코드 · Papers with Code

SINE: SINgle Image Editing with Text-to-Image Diffusion Models

Recent works on diffusion models have demonstrated a strong capability for conditioning image generation, e.g., text-guided image synthesis. Such success inspires many efforts trying to use large-scale pre-trained diffusion models for tackling a challenging problem--real image editing. Works conducted in this area learn a unique textual token corresponding to several images containing the same object. However, under many circumstances, only one image is available, such as the painting of the Girl with a Pearl Earring. Using existing works on fine-tuning the pre-trained diffusion models with a single image causes severe overfitting issues. The information leakage from the pre-trained diffusion models makes editing can not keep the same content as the given image while creating new features depicted by the language guidance. This work aims to address the problem of single-image editing. We propose a novel model-based guidance built upon the classifier-free guidance so that the knowledge from the model trained on a single image can be distilled into the pre-trained diffusion model, enabling content creation even with one given image. Additionally, we propose a patch-based fine-tuning that can effectively help the model generate images of arbitrary resolution. We provide extensive experiments to validate the design choices of our approach and show promising editing capabilities, including changing style, content addition, and object manipulation. The code is available for research purposes at https://github.com/zhang-zx/SINE.git .

arXiv:2303.14042공식 코드 · Papers with Code

Class-Incremental Exemplar Compression for Class-Incremental Learning

Exemplar-based class-incremental learning (CIL) finetunes the model with all samples of new classes but few-shot exemplars of old classes in each incremental phase, where the "few-shot" abides by the limited memory budget. In this paper, we break this "few-shot" limit based on a simple yet surprisingly effective idea: compressing exemplars by downsampling non-discriminative pixels and saving "many-shot" compressed exemplars in the memory. Without needing any manual annotation, we achieve this compression by generating 0-1 masks on discriminative pixels from class activation maps (CAM). We propose an adaptive mask generation model called class-incremental masking (CIM) to explicitly resolve two difficulties of using CAM: 1) transforming the heatmaps of CAM to 0-1 masks with an arbitrary threshold leads to a trade-off between the coverage on discriminative pixels and the quantity of exemplars, as the total memory is fixed; and 2) optimal thresholds vary for different object classes, which is particularly obvious in the dynamic environment of CIL. We optimize the CIM model alternatively with the conventional CIL model through a bilevel optimization problem. We conduct extensive experiments on high-resolution CIL benchmarks including Food-101, ImageNet-100, and ImageNet-1000, and show that using the compressed exemplars by CIM can achieve a new state-of-the-art CIL accuracy, e.g., 4.8 percentage points higher than FOSTER on 10-Phase ImageNet-1000. Our code is available at https://github.com/xfflzl/CIM-CIL.

arXiv:2308.04699공식 코드 · Papers with Code

GIFD: A Generative Gradient Inversion Method with Feature Domain Optimization

Federated Learning (FL) has recently emerged as a promising distributed machine learning framework to preserve clients' privacy, by allowing multiple clients to upload the gradients calculated from their local data to a central server. Recent studies find that the exchanged gradients also take the risk of privacy leakage, e.g., an attacker can invert the shared gradients and recover sensitive data against an FL system by leveraging pre-trained generative adversarial networks (GAN) as prior knowledge. However, performing gradient inversion attacks in the latent space of the GAN model limits their expression ability and generalizability. To tackle these challenges, we propose \textbf{G}radient \textbf{I}nversion over \textbf{F}eature \textbf{D}omains (GIFD), which disassembles the GAN model and searches the feature domains of the intermediate layers. Instead of optimizing only over the initial latent code, we progressively change the optimized layer, from the initial latent space to intermediate layers closer to the output images. In addition, we design a regularizer to avoid unreal image generation by adding a small ${l_1}$ ball constraint to the searching range. We also extend GIFD to the out-of-distribution (OOD) setting, which weakens the assumption that the training sets of GANs and FL tasks obey the same data distribution. Extensive experiments demonstrate that our method can achieve pixel-level reconstruction and is superior to the existing methods. Notably, GIFD also shows great generalizability under different defense strategy settings and batch sizes.

arXiv:2308.13259공식 코드 · Papers with Code

Knowledge-Driven CoT: Exploring Faithful Reasoning in LLMs for Knowledge-intensive Question Answering

Equipped with Chain-of-Thought (CoT), Large language models (LLMs) have shown impressive reasoning ability in various downstream tasks. Even so, suffering from hallucinations and the inability to access external knowledge, LLMs often come with incorrect or unfaithful intermediate reasoning steps, especially in the context of answering knowledge-intensive tasks such as KBQA. To alleviate this issue, we propose a framework called Knowledge-Driven Chain-of-Thought (KD-CoT) to verify and modify reasoning traces in CoT via interaction with external knowledge, and thus overcome the hallucinations and error propagation. Concretely, we formulate the CoT rationale process of LLMs into a structured multi-round QA format. In each round, LLMs interact with a QA system that retrieves external knowledge and produce faithful reasoning traces based on retrieved precise answers. The structured CoT reasoning of LLMs is facilitated by our developed KBQA CoT collection, which serves as in-context learning demonstrations and can also be utilized as feedback augmentation to train a robust retriever. Extensive experiments on WebQSP and ComplexWebQuestion datasets demonstrate the effectiveness of proposed KD-CoT in task-solving reasoning generation, which outperforms the vanilla CoT ICL with an absolute success rate of 8.0% and 5.1%. Furthermore, our proposed feedback-augmented retriever outperforms the state-of-the-art baselines for retrieving knowledge, achieving significant improvement in Hit and recall performance. Our code and data are released on https://github.com/AdelWang/KD-CoT/tree/main.

arXiv:2310.01152공식 코드 · Papers with Code

Large Language Model-Powered Smart Contract Vulnerability Detection: New Perspectives

This paper provides a systematic analysis of the opportunities, challenges, and potential solutions of harnessing Large Language Models (LLMs) such as GPT-4 to dig out vulnerabilities within smart contracts based on our ongoing research. For the task of smart contract vulnerability detection, achieving practical usability hinges on identifying as many true vulnerabilities as possible while minimizing the number of false positives. Nonetheless, our empirical study reveals contradictory yet interesting findings: generating more answers with higher randomness largely boosts the likelihood of producing a correct answer but inevitably leads to a higher number of false positives. To mitigate this tension, we propose an adversarial framework dubbed GPTLens that breaks the conventional one-stage detection into two synergistic stages $-$ generation and discrimination, for progressive detection and refinement, wherein the LLM plays dual roles, i.e., auditor and critic, respectively. The goal of auditor is to yield a broad spectrum of vulnerabilities with the hope of encompassing the correct answer, whereas the goal of critic that evaluates the validity of identified vulnerabilities is to minimize the number of false positives. Experimental results and illustrative examples demonstrate that auditor and critic work together harmoniously to yield pronounced improvements over the conventional one-stage detection. GPTLens is intuitive, strategic, and entirely LLM-driven without relying on specialist expertise in smart contracts, showcasing its methodical generality and potential to detect a broad spectrum of vulnerabilities. Our code is available at: https://github.com/git-disl/GPTLens.

arXiv:2405.12528공식 코드 · Papers with Code

SirLLM: Streaming Infinite Retentive LLM

As Large Language Models (LLMs) become increasingly prevalent in various domains, their ability to process inputs of any length and maintain a degree of memory becomes essential. However, the one-off input of overly long texts is limited, as studies have shown that when input lengths exceed the LLMs' pre-trained text length, there is a dramatic decline in text generation capabilities. Moreover, simply extending the length of pre-training texts is impractical due to the difficulty in obtaining long text data and the substantial memory consumption costs this would entail for LLMs. Recent efforts have employed streaming inputs to alleviate the pressure of excessively long text inputs, but this approach can significantly impair the model's long-term memory capabilities. Motivated by this challenge, we introduce Streaming Infinite Retentive LLM (SirLLM), which allows LLMs to maintain longer memory during infinite-length dialogues without the need for fine-tuning. SirLLM utilizes the Token Entropy metric and a memory decay mechanism to filter key phrases, endowing LLMs with both long-lasting and flexible memory. We designed three distinct tasks and constructed three datasets to measure the effectiveness of SirLLM from various angles: (1) DailyDialog; (2) Grocery Shopping; (3) Rock-Paper-Scissors. Our experimental results robustly demonstrate that SirLLM can achieve stable and significant improvements across different LLMs and tasks, compellingly proving its effectiveness. When having a coversation, "A sir could forget himself," but SirLLM never does! Our code is publicly available at https://github.com/Zoeyyao27/SirLLM