Context: Generative Artificial Intelligence (GenAI) is transforming much of software development, yet its application in software architecture is still in its infancy, and no prior study has systematically addressed the topic. Aim: We aim to systematically synthesize the use, rationale, contexts, usability, and future challenges of GenAI in software architecture. Method: We performed a multivocal literature review (MLR), analyzing peer-reviewed and gray literature, identifying current practices, models, adoption contexts, and reported challenges, extracting themes via open coding. Results: Our review identified significant adoption of GenAI for architectural decision support and architectural reconstruction. OpenAI GPT models are predominantly applied, and there is consistent use of techniques such as few-shot prompting and retrieved-augmented generation (RAG). GenAI has been applied mostly to initial stages of the Software Development Life Cycle (SDLC), such as Requirements-to-Architecture and Architecture-to-Code. Monolithic and microservice architectures were the dominant targets. However, rigorous testing of GenAI outputs was typically missing from the studies. Among the most frequent challenges are model precision, hallucinations, ethical aspects, privacy issues, lack of architecture-specific datasets, and the absence of sound evaluation frameworks. Conclusions: GenAI shows significant potential in software design, but several challenges remain on its path to greater adoption. Research efforts should target designing general evaluation methodologies, handling ethics and precision, increasing transparency and explainability, and promoting architecture-specific datasets and benchmarks to bridge the gap between theoretical possibilities and practical use.
In many real-world settings, machine learning models and interactive systems have access to both structured knowledge, e.g., knowledge graphs or tables, and unstructured content, e.g., natural language documents. However, most rely on either. Semi-Structured Knowledge Bases (SKBs) bridge this gap by linking unstructured content to nodes within structured data, thereby enabling new strategies for knowledge access and use. In this work, we present FocusedRetriever, a modular SKB-based framework for multi-hop question answering. It integrates components (VSS-based entity search, LLM-based generation of Cypher queries and pairwise re-ranking) in a way that enables it to outperform state-of-the-art methods across all three STaRK benchmark test sets, covering diverse domains and multiple performance metrics. The average first-hit rate exceeds that of the second-best method by 25.7%. FocusedRetriever leverages (1) the capacity of Large Language Models (LLMs) to extract relational facts and entity attributes from unstructured text, (2) node set joins to filter answer candidates based on these extracted triplets and constraints, (3) vector similarity search to retrieve and rank relevant unstructured content, and (4) the contextual capabilities of LLMs to finally rank the top-k answers. For generality, we only incorporate base LLMs in FocusedRetriever in our evaluation. However, our analysis of intermediate results highlights several opportunities for further upgrades including finetuning. The source code is publicly available at https://github.com/kramerlab/FocusedRetriever .
Direct preference optimization (DPO) is widely used as a simple and stable method for aligning large language models (LLMs) with human preferences. This paper investigates a generalized DPO loss that enables a policy model to match the target policy from a likelihood ratio estimation perspective. The ratio of the target policy provides a unique identification of the policy distribution without relying on reward models or partition functions. This allows the generalized loss to retain both simplicity and theoretical guarantees, which prior work such as $f$-PO fails to achieve simultaneously. We propose Bregman preference optimization (BPO), a generalized framework for ratio matching that provides a family of objective functions achieving target policy optimality. BPO subsumes DPO as a special case and offers tractable forms for all instances, allowing implementation with a few lines of code. We further develop scaled Basu's power divergence (SBA), a gradient scaling method that can be used for BPO instances. The BPO framework complements other DPO variants and is applicable to target policies defined by these variants. In experiments, unlike other probabilistic loss extensions such as $f$-DPO or $f$-PO, which exhibit a trade-off between generation fidelity and diversity, instances of BPO improve both win rate and entropy compared with DPO. When applied to Llama-3-Instruct-8B, BPO achieves state-of-the-art performance among Llama-3-8B backbones, with a 55.9\% length-controlled win rate on AlpacaEval2.
Vision-Language Models (VLMs) now generate discourse-level, multi-sentence visual descriptions, challenging text scene graph parsers originally designed for single-sentence caption-to-graph mapping. Current approaches typically merge sentence-level parsing outputs for discourse input, often missing phenomena like cross-sentence coreference, resulting in fragmented graphs and degraded downstream VLM task performance. To address this, we introduce a new task, Discourse-level text Scene Graph parsing (DiscoSG), supported by our dataset DiscoSG-DS, which comprises 400 expert-annotated and 8,430 synthesised multi-sentence caption-graph pairs for images. Each caption averages 9 sentences, and each graph contains at least 3 times more triples than those in existing datasets. While fine-tuning large PLMs (i.e., GPT-4) on DiscoSG-DS improves SPICE by approximately 48% over the best sentence-merging baseline, high inference cost and restrictive licensing hinder its open-source use, and smaller fine-tuned PLMs struggle with complex graphs. We propose DiscoSG-Refiner, which drafts a base graph using one small PLM, then employs a second PLM to iteratively propose graph edits, reducing full-graph generation overhead. Using two Flan-T5-Base models, DiscoSG-Refiner still improves SPICE by approximately 30% over the best baseline while achieving 86 times faster inference than GPT-4. It also consistently improves downstream VLM tasks like discourse-level caption evaluation and hallucination detection. Code and data are available at: https://github.com/ShaoqLin/DiscoSG
Supervised fine-tuning (SFT) is a key component of language model post-training, enabling effective task adaptation and serving as an important foundation for subsequent reinforcement learning. Recent methods seek to improve SFT by introducing token-level weights into the training objective, where these weights are typically derived from a model's output distribution at each token position. To better understand how such weighting schemes affect SFT, we analyze their optimization objectives and theoretically characterize the target distributions they induce. Our analysis shows that dynamic weighting methods, such as Dynamic Fine-Tuning (DFT), induce a heavily sharpened distribution, whereas static weighting methods fit a target distribution proportional to the product of the data distribution and a frozen prior. These findings suggest that static weighting provides a more stable training target, while its effectiveness critically depends on the quality of the prior distribution. Building on this observation, we propose Self-Privileged Fine-Tuning (SPFT), which uses a frozen copy of the initial model conditioned on privileged information (such as ground truth for mathematical questions) as a prior to provide guidance for token-level weighting during fine-tuning. Through a Bayesian decomposition, we further show that SPFT reweights the standard static prior according to how informative each target token is for predicting the ground-truth outcome, thereby assigning larger weights to more predictive tokens. Experiments on mathematical reasoning and code generation show that SPFT improves over standard SFT and other baselines across the main settings.
Recent advances in large language models have enabled agentic systems that automate machine learning pipeline construction from minimal user input. However, existing agentic AutoML methods either depend on rigid templates that limit adaptability, target narrow task domains, or rely on free-form code generation that lacks reliability guarantees, leaving intermediate errors to propagate silently and degrade the final model. We present EvoML, an agentic framework for end-to-end ML pipeline synthesis built on expert-grounded skills that evolve across runs, executed through a validation-gated harness. The framework decomposes pipeline construction into four phases coordinated by three agents: an Orchestrator that enforces validation gates, advancing each phase only when its outputs pass structural, execution, and semantic checks, which precludes error propagation and keeps every artifact auditable; an Executor that runs expert-grounded skills, grounded in the nnU-Net workflow and pairing reusable templates with online adaptation; and a Curator that performs post-run evolution, consolidating the scripts and guidance recovered during a run into the skill library so it improves across datasets. We evaluate EvoML on eight datasets spanning six sophisticated, real-world ML task types, against agentic AutoML systems and command-line coding tools. EvoML achieves full automation and the best final model quality on all eight, spanning segmentation (up to 0.94 Dice), classification (0.99 accuracy), reconstruction (0.95 SSIM), and denoising (0.79 normalized PSNR), while transferring to broader ML tasks.
Aerial imagery captured by unmanned aerial vehicles (UAVs) plays a vital role in applications such as surveillance, disaster response, and large-scale environmental monitoring. Automatic generation of textual descriptions for such imagery is essential for reducing manual inspection overhead and enabling better downstream decision-making. However, Multimodal Large Language Models (MLLMs), despite their impressive captioning capabilities, are prone to hallucination, producing descriptions that are inconsistent with the visual input. This problem is significantly amplified in aerial imagery due to its unique challenges such as extreme viewpoint variation, small objects, class imbalance, and ambiguous visual entities. Existing hallucination detection approaches either rely on computationally expensive large external models such as GPT-4, which perform poorly on aerial domains, or treat the problem as a rigid binary classification task, ignoring the inherent uncertainty in ambiguous cases. To address these limitations, we propose a lightweight, reference-free framework for hallucination detection and correction in free-form aerial image captioning. Our method leverages state-of-the-art MLLMs, specifically InternVL3 and Qwen2.5-VL, to detect hallucinated content in generated captions with minimal computational overhead. Detected hallucinations are corrected through a dedicated refinement module that improves the factual consistency of the original captions. Extensive experiments on the Labelled Illusion Dataset (LID) demonstrate that our framework significantly reduces hallucination rates while remaining practical for real-time deployment in resource-constrained environments. Code is available at: https://github.com/Debolena7/Hal_Detect.
멀티모달 대규모 언어 모델(MLLM)은 시각 이해 작업에서 괄목할 만한 진전을 이루었습니다. 그러나 대부분의 기존 MLLM은 자기회귀 생성에 의존하기 때문에, 여러 영역에 대한 캡션 생성이 필요한 인지 작업에서 효율성이 제한됩니다. 본 연구에서는 효율적인 병렬 영역 인지에 최적화된 멀티모달 디퓨전 언어 모델인 PerceptionDLM을 제안합니다. 오픈 소스 디퓨전 MLLM 중 최첨단 성능을 달성한 강력한 기초 기준선인 PerceptionDLM-Base를 기반으로 구축된 본 아키텍처는 DLM의 병렬 디코딩 특성을 완전히 활용합니다. 구체적으로, 효율적인 프롬프팅과 구조화된 어텐션 마스킹을 도입하여 여러 마스크된 영역을 동시에 인지할 수 있도록 하며, 이를 통해 모델이 시퀀스 및 토큰 수준에서 영역 설명을 병렬로 생성할 수 있게 합니다. 이러한 설계는 영역을 순차적으로 처리하는 기존 접근 방식에 비해 추론 효율성을 크게 향상시킵니다. DLM의 시각 인지 능력에 대한 병렬성 속성을 체계적으로 평가하기 위해, DLC-Bench를 확장하여 이미지당 여러 영역 마스크를 포함하도록 한 새로운 병렬 상세 지역화 캡션 벤치마크(ParaDLC-Bench)를 구축하여 캡션 품질과 추론 효율성을 공동으로 평가할 수 있게 했습니다. 실험 결과 PerceptionDLM은 영역 캡션 생성에서 경쟁력 있는 성능을 유지하면서 다중 영역 인지 작업에서 상당한 속도 향상을 달성함을 보여줍니다. 본 연구 결과는 효율적이고 병렬적인 시각 인지를 위한 멀티모달 디퓨전 언어 모델의 잠재력을 강조합니다. 본 연구진의 지식 범위 내에서, 디퓨전 언어 모델의 장점을 활용하여 병렬 영역 캡션 및 인지를 달성한 것은 이번이 처음입니다. 코드, 모델, 데이터셋은 공개되었습니다.
저자원 환경에서 TTS를 배포할 때는 일반적으로 추론 비용이 큰 대규모 음성 복제(voice-cloning) 모델과, 화자별 말뭉치가 필요한 소형 고정 화자(fixed-voice) 시스템 중 하나를 선택해야 한다. 본 연구는 세 번째 경로를 다룬다. 대규모 음성 복제 모델을 프로그래밍 가능한 데이터 소스로 사용해 짧은 음성 참조(예: 15초)만으로, 합성 음성만으로 학습한 소형 고정 화자 학생(student) 모델을 만드는 방법이다. 이 설정에서는 파이프라인 설계가 결과에 큰 영향을 미친다. 교사(teacher) 모델의 오류가 학습 목표가 되고, 실패한 생성물을 걸러내면 어려운 텍스트에 대한 커버리지가 줄어들 수 있다. 태국어는 모호한 단어 경계, 어휘 성조(lexical tone), 고유명·외래어, 숫자의 구어화(verbalization), 태국어-영어 코드 스위칭 등으로 추가적인 난제를 만든다. 텍스트 준비, 합성 생성, 품질 필터링, 거부 샘플링, 프론트엔드 선택이 최종 학생 모델에 어떻게 영향을 주는지, 교사 모델의 한계는 어디에 남는지를 분석한다. CER, Challenge-Set Keyword Accuracy, Prosody Pause Accuracy, 화자 유사도, 발화 속도를 평가한다. 그 결과 8200만 매개변수 모델 Wayu-Paxa-TTS-Edge는 참조 오디오 없이 온디바이스 태국어 TTS를 가능하게 한다. Challenge-Set Keyword Accuracy 68.2%(Gemini 3.1의 85.5%), pause 정밀도 91.4%를 달성하며, OmniVoice 교사 모델의 pause 정밀도(89.9%)를 넘고 Gemini 3.1의 94.8%에 이른다. 또한 세 시스템 중 pause 배치 오류와 단어 내 pause 비율이 가장 낮고, 태국어·영어 CER은 각각 3.7%, 1.1%이다. 태국어 TTS 개발을 위해 모델과 평가 프레임워크를 오픈소스로 공개한다.
다수준 중간 표현(MLIR, Multi-Level Intermediate Representation)은 도메인 특화 연산을 위한 재사용 가능한 컴파일러 인프라를 실용화했습니다. 그러나 MLIR의 엄격한 컴파일 시점 타입 요구사항과 저수준(C++) 확장 모델은 Julia와 같은 고수준의 동적 특화 언어에는 잘 맞지 않을 수 있습니다. MLIR은 타입 시스템 및 추상화 수준 측면에서 동적 프로그래밍 언어에 몇 가지 단점이 있습니다. 이로 인해 컴파일러 비전문가나 과학 컴퓨팅 사용자가 새로운 프로그래밍 추상화를 도입하고 자연스러우면서도 최적화 가능한 형태로 알고리즘 구현을 표현하기가 극도로 어렵습니다. 그 결과 선형대수, 메시 처리, 편미분 방정식 및 관련 도메인을 위한 라이브러리 인터페이스는 컴파일러 최적화 경로의 외부에 머무는 경우가 많습니다. 본 논문에서는 일반적인 Julia 코드로 사용 가능한 상태를 유지하면서 MLIR 스타일의 다수준 방언(dialect) 중심 컴파일의 주요 이점을 Julia 생태계에 제공하는 Julia 네이티브 중간 표현 프레임워크인 JLIR(Julia-native Level Intermediate Representation)을 소개합니다. JLIR은 저수준화(lowering) 이전의 Julia 프로그램을 표현하고, Julia의 언어 메커니즘을 통해 확장 가능한 연산 및 변환 패스(pass)를 지원하며, 구체적인 타입이 알려질 때까지 부분적으로 타입이 지정된 프로그램이 변환 가능한 상태를 유지하도록 합니다. 이 프레임워크에는 산술, 제어 흐름, 함수, 구조화된 루프, 메모리 연산을 위한 내장 방언이 포함되어 있으며, 핵심 시스템을 수정하지 않고도 새로운 도메인 연산을 추가할 수 있는 경량 메커니즘도 포함되어 있습니다. JLIR의 기능을 입증하기 위해, 이를 자동 가속기용 Julia(JACC, Julia for Accelerators) 커널 생성에 적용했습니다.
시스템 엔지니어는 시스템의 구조와 동작을 기술하기 위해 여러 다이어그램을 사용합니다. 엔지니어들은 이 다이어그램들이 동일한 요소를 사용하고 상호 일관성을 유지하도록 함께 작성합니다. 대규모 언어 모델(LLM)은 다이어그램을 텍스트나 코드로 생성할 수 있어 시스템 다이어그램의 자동 생성을 가능하게 합니다. 그러나 일관된 다이어그램 세트를 생성하는 LLM의 능력은 아직 잘 알려져 있지 않으며, 기존 데이터셋과 벤치마크는 이러한 능력을 대규모로 직접 측정하지 못합니다. 본 논문에서는 3,000개의 엔지니어링 컨텍스트와 15,000개의 다이어그램으로 구성된 데이터셋인 SEMAADB(Systems Engineering Modeling Assistant with AI Dataset and Benchmark)를 소개합니다. 각 컨텍스트는 요구사항(Requirement), 블록 정의(Block Definition), 액티비티(Activity), 상태 머신(State Machine), 시퀀스(Sequence)라는 5개의 연결된 SysML 뷰를 포함합니다. 여기서 뷰는 시스템의 한 측면을 나타내는 다이어그램입니다. 연구진은 다이어그램 세트의 일관성과 유효한 렌더링 여부를 검증했습니다. 또한 100개의 컨텍스트 세트는 사람이 직접 검증하여 벤치마크 테스트 세트를 구성했습니다. 연구진은 두 가지 태스크에서 세 가지 언어 모델을 평가했습니다. 다이어그램 수정(diagram repair) 태스크에서 가장 우수한 모델은 의미적 오류의 64.3%를 수정했습니다. 다이어그램 간 업데이트(cross-diagram update)에서는 모델이 관련된 여러 다이어그램에 걸쳐 단일 변경 사항을 전파할 때 최고 전파 F1 점수 80.7%를 기록했습니다. 실험 결과 구문 수정은 거의 해결되었으나, 의미적 수정과 다이어그램 간 일관성 유지는 여전히 모델에게 어려운 과제임을 보여줍니다. 따라서 SEMAADB는 대규모 다이어그램 리소스와 일관된 다중 다이어그램 SysML 생성을 측정하기 위한 벤치마크 세트를 모두 제공합니다.
We infer and generate three-dimensional (3D) scene information from a single input image and without supervision. This problem is under-explored, with most prior work relying on supervision from, e.g., 3D ground-truth, multiple images of a scene, image silhouettes or key-points. We propose Pix2Shape, an approach to solve this problem with four components: (i) an encoder that infers the latent 3D representation from an image, (ii) a decoder that generates an explicit 2.5D surfel-based reconstruction of a scene from the latent code (iii) a differentiable renderer that synthesizes a 2D image from the surfel representation, and (iv) a critic network trained to discriminate between images generated by the decoder-renderer and those from a training distribution. Pix2Shape can generate complex 3D scenes that scale with the view-dependent on-screen resolution, unlike representations that capture world-space resolution, i.e., voxels or meshes. We show that Pix2Shape learns a consistent scene representation in its encoded latent space and that the decoder can then be applied to this latent representation in order to synthesize the scene from a novel viewpoint. We evaluate Pix2Shape with experiments on the ShapeNet dataset as well as on a novel benchmark we developed, called 3D-IQTT, to evaluate models based on their ability to enable 3d spatial reasoning. Qualitative and quantitative evaluation demonstrate Pix2Shape's ability to solve scene reconstruction, generation, and understanding tasks.
Generative adversarial networks (GANs) have been a popular deep generative model for real-world applications. Despite many recent efforts on GANs that have been contributed, mode collapse and instability of GANs are still open problems caused by their adversarial optimization difficulties. In this paper, motivated by the cooperative co-evolutionary algorithm, we propose a Cooperative Dual Evolution based Generative Adversarial Network (CDE-GAN) to circumvent these drawbacks. In essence, CDE-GAN incorporates dual evolution with respect to the generator(s) and discriminators into a unified evolutionary adversarial framework to conduct effective adversarial multi-objective optimization. Thus it exploits the complementary properties and injects dual mutation diversity into training to steadily diversify the estimated density in capturing multi-modes and improve generative performance. Specifically, CDE-GAN decomposes the complex adversarial optimization problem into two subproblems (generation and discrimination), and each subproblem is solved with a separated subpopulation (E-Generator} and E-Discriminators), evolved by its own evolutionary algorithm. Additionally, we further propose a Soft Mechanism to balance the trade-off between E-Generators and E-Discriminators to conduct steady training for CDE-GAN. Extensive experiments on one synthetic dataset and three real-world benchmark image datasets demonstrate that the proposed CDE-GAN achieves a competitive and superior performance in generating good quality and diverse samples over baselines. The code and more generated results are available at our project homepage: https://shiming-chen.github.io/CDE-GAN-website/CDE-GAN.html.
AI creation, such as poem or lyrics generation, has attracted increasing attention from both industry and academic communities, with many promising models proposed in the past few years. Existing methods usually estimate the outputs based on single and independent visual or textual information. However, in reality, humans usually make creations according to their experiences, which may involve different modalities and be sequentially correlated. To model such human capabilities, in this paper, we define and solve a novel AI creation problem based on human experiences. More specifically, we study how to generate texts based on sequential multi-modal information. Compared with the previous works, this task is much more difficult because the designed model has to well understand and adapt the semantics among different modalities and effectively convert them into the output in a sequential manner. To alleviate these difficulties, we firstly design a multi-channel sequence-to-sequence architecture equipped with a multi-modal attention network. For more effective optimization, we then propose a curriculum negative sampling strategy tailored for the sequential inputs. To benchmark this problem and demonstrate the effectiveness of our model, we manually labeled a new multi-modal experience dataset. With this dataset, we conduct extensive experiments by comparing our model with a series of representative baselines, where we can demonstrate significant improvements in our model based on both automatic and human-centered metrics. The code and data are available at: \url{https://github.com/Aman-4-Real/MMTG}.
Unsupervised generation of 3D-aware clothed humans with various appearances and controllable geometries is important for creating virtual human avatars and other AR/VR applications. Existing methods are either limited to rigid object modeling, or not generative and thus unable to generate high-quality virtual humans and animate them. In this work, we propose AvatarGen, the first method that enables not only geometry-aware clothed human synthesis with high-fidelity appearances but also disentangled human animation controllability, while only requiring 2D images for training. Specifically, we decompose the generative 3D human synthesis into pose-guided mapping and canonical representation with predefined human pose and shape, such that the canonical representation can be explicitly driven to different poses and shapes with the guidance of a 3D parametric human model SMPL. AvatarGen further introduces a deformation network to learn non-rigid deformations for modeling fine-grained geometric details and pose-dependent dynamics. To improve the geometry quality of the generated human avatars, it leverages the signed distance field as geometric proxy, which allows more direct regularization from the 3D geometric priors of SMPL. Benefiting from these designs, our method can generate animatable 3D human avatars with high-quality appearance and geometry modeling, significantly outperforming previous 3D GANs. Furthermore, it is competent for many applications, e.g., single-view reconstruction, re-animation, and text-guided synthesis/editing. Code and pre-trained model will be available at http://jeff95.me/projects/avatargen.html.
Large-scale vision-language pre-training has achieved significant performance in multi-modal understanding and generation tasks. However, existing methods often perform poorly on image-text matching tasks that require structured representations, i.e., representations of objects, attributes, and relations. As illustrated in Fig.~reffig:case (a), the models cannot make a distinction between ``An astronaut rides a horse" and ``A horse rides an astronaut". This is because they fail to fully leverage structured knowledge when learning representations in multi-modal scenarios. In this paper, we present an end-to-end framework Structure-CLIP, which integrates Scene Graph Knowledge (SGK) to enhance multi-modal structured representations. Firstly, we use scene graphs to guide the construction of semantic negative examples, which results in an increased emphasis on learning structured representations. Moreover, a Knowledge-Enhance Encoder (KEE) is proposed to leverage SGK as input to further enhance structured representations. To verify the effectiveness of the proposed framework, we pre-train our model with the aforementioned approaches and conduct experiments on downstream tasks. Experimental results demonstrate that Structure-CLIP achieves state-of-the-art (SOTA) performance on VG-Attribution and VG-Relation datasets, with 12.5% and 4.1% ahead of the multi-modal SOTA model respectively. Meanwhile, the results on MSCOCO indicate that Structure-CLIP significantly enhances the structured representations while maintaining the ability of general representations. Our code is available at https://github.com/zjukg/Structure-CLIP.
Deep learning-based medical image segmentation models often suffer from domain shift, where the models trained on a source domain do not generalize well to other unseen domains. As a prompt-driven foundation model with powerful generalization capabilities, the Segment Anything Model (SAM) shows potential for improving the cross-domain robustness of medical image segmentation. However, SAM performs significantly worse in automatic segmentation scenarios than when manually prompted, hindering its direct application to domain generalization. Upon further investigation, we discovered that the degradation in performance was related to the coupling effect of inevitable poor prompts and mask generation. To address the coupling effect, we propose the Decoupled SAM (DeSAM). DeSAM modifies SAM's mask decoder by introducing two new modules: a prompt-relevant IoU module (PRIM) and a prompt-decoupled mask module (PDMM). PRIM predicts the IoU score and generates mask embeddings, while PDMM extracts multi-scale features from the intermediate layers of the image encoder and fuses them with the mask embeddings from PRIM to generate the final segmentation mask. This decoupled design allows DeSAM to leverage the pre-trained weights while minimizing the performance degradation caused by poor prompts. We conducted experiments on publicly available cross-site prostate and cross-modality abdominal image segmentation datasets. The results show that our DeSAM leads to a substantial performance improvement over previous state-of-theart domain generalization methods. The code is publicly available at https://github.com/yifangao112/DeSAM.
In this paper, we present our method for neural face reenactment, called HyperReenact, that aims to generate realistic talking head images of a source identity, driven by a target facial pose. Existing state-of-the-art face reenactment methods train controllable generative models that learn to synthesize realistic facial images, yet producing reenacted faces that are prone to significant visual artifacts, especially under the challenging condition of extreme head pose changes, or requiring expensive few-shot fine-tuning to better preserve the source identity characteristics. We propose to address these limitations by leveraging the photorealistic generation ability and the disentangled properties of a pretrained StyleGAN2 generator, by first inverting the real images into its latent space and then using a hypernetwork to perform: (i) refinement of the source identity characteristics and (ii) facial pose re-targeting, eliminating this way the dependence on external editing methods that typically produce artifacts. Our method operates under the one-shot setting (i.e., using a single source frame) and allows for cross-subject reenactment, without requiring any subject-specific fine-tuning. We compare our method both quantitatively and qualitatively against several state-of-the-art techniques on the standard benchmarks of VoxCeleb1 and VoxCeleb2, demonstrating the superiority of our approach in producing artifact-free images, exhibiting remarkable robustness even under extreme head pose changes. We make the code and the pretrained models publicly available at: https://github.com/StelaBou/HyperReenact .
In this work, we present a learning method for lateral and longitudinal motion control of an ego-vehicle for vehicle pursuit. The car being controlled does not have a pre-defined route, rather it reactively adapts to follow a target vehicle while maintaining a safety distance. To train our model, we do not rely on steering labels recorded from an expert driver but effectively leverage a classical controller as an offline label generation tool. In addition, we account for the errors in the predicted control values, which can lead to a loss of tracking and catastrophic crashes of the controlled vehicle. To this end, we propose an effective data augmentation approach, which allows to train a network capable of handling different views of the target vehicle. During the pursuit, the target vehicle is firstly localized using a Convolutional Neural Network. The network takes a single RGB image along with cars' velocities and estimates the target vehicle's pose with respect to the ego-vehicle. This information is then fed to a Multi-Layer Perceptron, which regresses the control commands for the ego-vehicle, namely throttle and steering angle. We extensively validate our approach using the CARLA simulator on a wide range of terrains. Our method demonstrates real-time performance and robustness to different scenarios including unseen trajectories and high route completion. The project page containing code and multimedia can be publicly accessed here: https://changyaozhou.github.io/Autonomous-Vehicle-Pursuit/.
We propose an instruction-based process for trustworthy data curation in materials science (MatSci-Instruct), which we then apply to finetune a LLaMa-based language model targeted for materials science (HoneyBee). MatSci-Instruct helps alleviate the scarcity of relevant, high-quality materials science textual data available in the open literature, and HoneyBee is the first billion-parameter language model specialized to materials science. In MatSci-Instruct we improve the trustworthiness of generated data by prompting multiple commercially available large language models for generation with an Instructor module (e.g. Chat-GPT) and verification from an independent Verifier module (e.g. Claude). Using MatSci-Instruct, we construct a dataset of multiple tasks and measure the quality of our dataset along multiple dimensions, including accuracy against known facts, relevance to materials science, as well as completeness and reasonableness of the data. Moreover, we iteratively generate more targeted instructions and instruction-data in a finetuning-evaluation-feedback loop leading to progressively better performance for our finetuned HoneyBee models. Our evaluation on the MatSci-NLP benchmark shows HoneyBee's outperformance of existing language models on materials science tasks and iterative improvement in successive stages of instruction-data refinement. We study the quality of HoneyBee's language modeling through automatic evaluation and analyze case studies to further understand the model's capabilities and limitations. Our code and relevant datasets are publicly available at \url{https://github.com/BangLab-UdeM-Mila/NLP4MatSci-HoneyBee}.