Multi-modality image fusion aims to integrate the merits of images from different sources and render high-quality fusion images. However, existing feature extraction and fusion methods are either constrained by inherent local reduction bias and static parameters during inference (CNN) or limited by quadratic computational complexity (Transformers), and cannot effectively extract and fuse features. To solve this problem, we propose a dual-branch image fusion network called Tmamba. It consists of linear Transformer and Mamba, which has global modeling capabilities while maintaining linear complexity. Due to the difference between the Transformer and Mamba structures, the features extracted by the two branches carry channel and position information respectively. T-M interaction structure is designed between the two branches, using global learnable parameters and convolutional layers to transfer position and channel information respectively. We further propose cross-modal interaction at the attention level to obtain cross-modal attention. Experiments show that our Tmamba achieves promising results in multiple fusion tasks, including infrared-visible image fusion and medical image fusion. Code with checkpoints will be available after the peer-review process.
Multimodal Large Language Models (MLLMs) have significantly improved performance across various image-language applications. Recently, there has been a growing interest in adapting image pre-trained MLLMs for video-related tasks. However, most efforts concentrate on enhancing the vision encoder and projector components, while the core part, Large Language Models (LLMs), remains comparatively under-explored. In this paper, we propose two strategies to enhance the model's capability in video understanding tasks by improving inter-layer attention computation in LLMs. Specifically, the first approach focuses on the enhancement of Rotary Position Embedding (RoPE) with Temporal-Aware Dual RoPE, which introduces temporal position information to strengthen the MLLM's temporal modeling capabilities while preserving the relative position relationships of both visual and text tokens. The second approach involves enhancing the Attention Mask with the Frame-wise Block Causal Attention Mask, a simple yet effective method that broadens visual token interactions within and across video frames while maintaining the causal inference mechanism. Based on these proposed methods, we adapt LLaVA for video understanding tasks, naming it Temporal-Considered LLaVA (TC-LLaVA). Our TC-LLaVA achieves new state-of-the-art performance across various video understanding benchmarks with only supervised fine-tuning (SFT) on video-related datasets.
Stance detection (SD) identifies the text position towards a target, typically labeled as favor, against, or none. We introduce Open-Target Stance Detection (OTSD), the most realistic task where targets are neither seen during training nor provided as input. We evaluate Large Language Models (LLMs) from GPT, Gemini, Llama, and Mistral families, comparing their performance to the only existing work, Target-Stance Extraction (TSE), which benefits from predefined targets. Unlike TSE, OTSD removes the dependency of a predefined list, making target generation and evaluation more challenging. We also provide a metric for evaluating target quality that correlates well with human judgment. Our experiments reveal that LLMs outperform TSE in target generation, both when the real target is explicitly and not explicitly mentioned in the text. Similarly, LLMs overall surpass TSE in stance detection for both explicit and non-explicit cases. However, LLMs struggle in both target generation and stance detection when the target is not explicit.
The problem of cleaning a messy table using Deep Neural Networks is a very interesting problem in both social and industrial robotics. This project focuses on the social application of this technology. A neural network model that is capable of detecting and recognizing common tabletop objects, such as a mug, mouse, or stapler is developed. The model also predicts the angle at which these objects are placed on a table,with respect to some reference. Assuming each object has a fixed intended position and orientation on the tabletop, the orientation of a particular object predicted by the deep learning model can be used to compute the transformation matrix to move the object from its initial position to the intended position. This can be fed to a pick and place robot to carry out the transfer.This paper talks about the deep learning approaches used in this project for object detection and orientation estimation.
Temporal motion modeling has always been a key component in multiple object tracking (MOT) which can ensure smooth trajectory movement and provide accurate positional information to enhance association precision. However, current motion models struggle to be both efficient and effective across different application scenarios. To this end, we propose TrackSSM inspired by the recently popular state space models (SSM), a unified encoder-decoder motion framework that uses data-dependent state space model to perform temporal motion of trajectories. Specifically, we propose Flow-SSM, a module that utilizes the position and motion information from historical trajectories to guide the temporal state transition of object bounding boxes. Based on Flow-SSM, we design a flow decoder. It is composed of a cascaded motion decoding module employing Flow-SSM, which can use the encoded flow information to complete the temporal position prediction of trajectories. Additionally, we propose a Step-by-Step Linear (S$^2$L) training strategy. By performing linear interpolation between the positions of the object in the previous frame and the current frame, we construct the pseudo labels of step-by-step linear training, ensuring that the trajectory flow information can better guide the object bounding box in completing temporal transitions. TrackSSM utilizes a simple Mamba-Block to build a motion encoder for historical trajectories, forming a temporal motion model with an encoder-decoder structure in conjunction with the flow decoder. TrackSSM is applicable to various tracking scenarios and achieves excellent tracking performance across multiple benchmarks, further extending the potential of SSM-like temporal motion models in multi-object tracking tasks. Code and models are publicly available at \url{https://github.com/Xavier-Lin/TrackSSM}.
Abstract—In solving many-objective optimization problems (MaOPs), existing nondominated sorting-based multi-objective evolutionary algorithms suffer from the fast loss of selection pres- sure. Most candidate solutions become nondominated during the evolutionary process, thus leading to the failure of producing off- spring toward Pareto-optimal front with diversity. Can we find a more effective way to select nondominated solutions and resolve this issue? To answer this critical question, this work proposes to evolve solutions through line complex rather than solution points in Euclidean space. First, Plücker coordinates are used to project solution points to line complex composed of position vectors and momentum ones. Besides position vectors of the solution points, momentum vectors are used to extend the comparability of non- dominated solutions and enhance selection pressure. Then, a new distance function designed for high-dimensional space is pro- posed to replace Euclidean distance as a more effective distance- based estimator. Based on them, a novel many-objective evolu- tionary algorithm (MaOEA) is proposed by integrating a line complex-based environmental selection strategy into the NSGA- III framework. The proposed algorithm is compared with the state of the art on widely used benchmark problems with up to 15 objectives. Experimental results demonstrate its superior compet- itiveness in solving MaOPs.
This paper leverages Fisher information to examine the interaction between low-Earth orbit (LEO) satellites and 5G base stations (BSs) in enabling 9D receiver localization and refining LEO ephemeris. First, we propose a channel model that incorporates all relevant links: LEO-receiver, LEO-BS, and BS-receiver.Then, we utilize the Fisher information matrix (FIM) to quantify the information available about the channel parameters in these links. By transforming these FIMs, we derive the FIM for 9D receiver localization parameters-comprising 3D position, 3D orientation, and 3D velocity-along with LEO position and velocity offsets. We present closed-form expressions for the FIM entries corresponding to these localization parameters. Our identifiability analysis based on the FIM reveals that: i) With a single LEO, three BSs, and three time slots are required to estimate the 9D localization parameters and correct the LEO position and velocity. ii) With two LEOs, the same configuration (three BSs and three time slots) suffices for both tasks. iii) With three LEOs, three BSs and four time slots are needed to achieve the same goal. A key insight from the Cramer-Rao lower bound (CRLB) analysis is that, under the configuration of one LEO, three BSs, and three time slots, the estimated errors for receiver positioning, velocity, and orientation, as well as LEO position and velocity offsets, are 0.1 cm, 1 mm/s, 0.001 rad, 0.01 m, and 1 m/s, respectively. The receiver localization parameters are estimated after 1 s while the LEO offset parameters are estimated after 20 s. Additionally, our CRLB analysis indicates that operating frequency has minimal impact on receiver orientation estimation accuracy, and the number of receive antennas has a negligible effect on LEO velocity estimation accuracy.
In this paper, we derive the fundamental limits of low earth orbit (LEO) enabled localization by analyzing the available information in signals from multiple LEOs during different transmission time slots received on a multiple antennas and evaluate the utility of these signals for $9$D localization ($3$D position, $3$D orientation, and $3$D velocity estimation). We start by deriving the Fisher Information Matrix (FIM) for the channel parameters that are present in the signals received from LEOs in the same or multiple constellations during multiple transmission time slots. To accomplish this, we define a system model that captures i) time offset between LEOs caused by having relatively cheap clocks, ii) frequency offset between LEOs, iii) the unknown Doppler rate caused by high mobility LEOs, and iv) multiple transmission time slots from a particular LEO. We transform the FIM for the channel parameters to the FIM for the location parameters and determine the required conditions for localization. To do this, we start with the $3$D localization cases: i) $3$D positioning with known velocity and orientation, ii) $3$D orientation estimation with known position and velocity, and iii) $3$D velocity estimation with known position and orientation. Subsequently, we derive the FIM for the full $9$D localization case ($3$D position, $3$D orientation, and $3$D velocity estimation) in terms of the FIM for the $3$D localization. Using these results, we determine the number of LEOs, the operating frequency, the number of transmission time slots, and the number of receive antennas that allow for different levels of location estimation.
With line-of-sight mode deployment and fast response, unmanned aerial vehicle (UAV), equipped with the cutting-edge integrated sensing and communication (ISAC) technique, is poised to deliver high-quality communication and sensing services in maritime emergency scenarios. In practice, however, the real-time transmission of ISAC signals at the UAV side cannot be realized unless the reliable wireless fronthaul link between the terrestrial base station and UAV are available. This paper proposes a multicarrier-division duplex based joint fronthaul-access scheme, where mutually orthogonal subcarrier sets are leveraged to simultaneously support four types of fronthaul/access transmissions. In order to maximize the end-to-end communication rate while maintaining an adequate sensing quality-of-service (QoS) in such a complex scheme, the UAV trajectory, subcarrier assignment and power allocation are jointly optimized. The overall optimization process is designed in two stages. As the emergency area is usually far away from the coast, the optimal initial operating position for the UAV is first found. Once the UAV passes the initial operating position, the UAV's trajectory and resource allocation are optimized during the mission period to maximize the end-to-end communication rate under the constraint of minimum sensing QoS. Simulation results demonstrate the effectiveness of the proposed scheme in dealing with the joint fronthaul-access optimization problem in maritime ISAC networks, offering the advantages over benchmark schemes.
The problem of career trajectory prediction (CTP) aims to predict one's future employer or job position. While several CTP methods have been developed for this problem, we posit that none of these methods (1) jointly considers the mutual ternary dependency between three key units (i.e., user, position, and company) of a career and (2) captures the characteristic shifts of key units in career over time, leading to an inaccurate understanding of the job movement patterns in the labor market. To address the above challenges, we propose a novel solution, named as CAPER, that solves the challenges via sophisticated temporal knowledge graph (TKG) modeling. It enables the utilization of a graph-structured knowledge base with rich expressiveness, effectively preserving the changes in job movement patterns. Furthermore, we devise an extrapolated career reasoning task on TKG for a realistic evaluation. The experiments on a real-world career trajectory dataset demonstrate that CAPER consistently and significantly outperforms four baselines, two recent TKG reasoning methods, and five state-of-the-art CTP methods in predicting one's future companies and positions--i.e., on average, yielding 6.80% and 34.58% more accurate predictions, respectively. The codebase of CAPER is available at https://github.com/Bigdasgit/CAPER.
Despite the success of vision-based dynamics prediction models, which predict object states by utilizing RGB images and simple object descriptions, they were challenged by environment misalignments. Although the literature has demonstrated that unifying visual domains with both environment context and object abstract, such as semantic segmentation and bounding boxes, can effectively mitigate the visual domain misalignment challenge, discussions were focused on the abstract of environment context, and the insight of using bounding box as the object abstract is under-explored. Furthermore, we notice that, as empirical results shown in the literature, even when the visual appearance of objects is removed, object bounding boxes alone, instead of being directly fed into the network, can indirectly provide sufficient position information via the Region of Interest Pooling operation for dynamics prediction. However, previous literature overlooked discussions regarding how such position information is implicitly encoded in the dynamics prediction model. Thus, in this paper, we provide detailed studies to investigate the process and necessary conditions for encoding position information via using the bounding box as the object abstract into output features. Furthermore, we study the limitation of solely using object abstracts, such that the dynamics prediction performance will be jeopardized when the environment context varies.
This paper constructs question answering system for bridge design specification based on large language model. Three implementation schemes are tried: full fine-tuning of the Bert pretrained model, parameter-efficient fine-tuning of the Bert pretrained model, and self-built language model from scratch. Through the self-built question and answer task dataset, based on the tensorflow and keras deep learning platform framework, the model is constructed and trained to predict the start position and end position of the answer in the bridge design specification given by the user. The experimental results show that full fine-tuning of the Bert pretrained model achieves 100% accuracy in the training-dataset, validation-dataset and test-dataset, and the system can extract the answers from the bridge design specification given by the user to answer various questions of the user; While parameter-efficient fine-tuning of the Bert pretrained model and self-built language model from scratch perform well in the training-dataset, their generalization ability in the test-dataset needs to be improved. The research of this paper provides a useful reference for the development of question answering system in professional field.
Movable antennas (MAs) have received increasing attention in wireless communications due to their capability of antenna position adjustment to reconfigure wireless channels. However, moving MAs results in non-negligible delay, which may decrease the effective data transmission time. To reduce the movement delay, we study in this paper a new MA trajectory optimization problem. In particular, given the desired destination positions of multiple MAs, we aim to jointly optimize their associations with the initial MA positions and the trajectories for moving them from their respective initial to destination positions within a given two-dimensional (2D) region, such that the delay of antenna movement is minimized, subject to the inter-MA minimum distance constraints in the movement. However, this problem is a continuous-time mixed-integer linear programming (MILP) problem that is challenging to solve. To tackle this challenge, we propose a two-stage optimization framework that sequentially optimizes the MAs' position associations and trajectories, respectively. First, we relax the inter-MA distance constraints and optimally solve the resulted delay minimization problem. Next, we check if the obtained MA association and trajectory solutions satisfy the inter-MA distance constraints. If not satisfied, we then employ a successive convex approximation (SCA) algorithm to adjust the MAs' trajectories until they satisfy the given constraints. Simulation results are provided to show the effectiveness of our proposed trajectory optimization method in reducing the movement delay as well as draw useful insights.
Fluid antennas (FAs) and mobile antennas (MAs) are innovative technologies in wireless communications that are able to proactively improve channel conditions by dynamically adjusting the transmit/receive antenna positions within a given spatial region. In this paper, we investigate an MA-enhanced multiple-input single-output (MISO) secure communication system, aiming to maximize the secrecy rate by jointly optimizing the positions of multiple MAs. Instead of continuously searching for the optimal MA positions as in prior works, we propose to discretize the transmit region into multiple sampling points, thereby converting the continuous antenna position optimization into a discrete sampling point selection problem. However, this point selection problem is combinatory and thus difficult to be optimally solved. To tackle this challenge, we ingeniously transform this combinatory problem into a recursive path selection problem in graph theory and propose a partial enumeration algorithm to obtain its optimal solution without the need for high-complexity exhaustive search. To further reduce the complexity, a linear-time sequential update algorithm is also proposed to obtain a high-quality suboptimal solution. Numerical results show that our proposed algorithms yield much higher secrecy rates as compared to the conventional FPA and other baseline schemes.
This paper presents a radar target tracking framework for addressing main-beam range deception jamming attacks using random finite sets (RFSs). Our system handles false alarms and detections with false range information through multiple hypothesis tracking (MHT) to resolve data association uncertainties. We focus on range gate pull-off (RGPO) attacks, where the attacker adds positive delays to the radar pulse, thereby mimicking the target trajectory while appearing at a larger distance from the radar. The proposed framework incorporates knowledge about the spatial behavior of the attack into the assumed RFS clutter model and uses only position information without relying on additional signal features. We present an adaptive solution that estimates the jammer-induced biases to improve tracking accuracy as well as a simpler non-adaptive version that performs well when accurate priors on the jamming range are available. Furthermore, an expression for RGPO attack detection is derived, where the adaptive solution offers superior performance. The presented strategies provide tracking resilience against multiple RGPO attacks in terms of position estimation accuracy and jamming detection without degrading tracking performance in the absence of jamming.
This letter investigates movable antenna (MA)-aided downlink (DL) multiuser communication systems under the near-field channel condition, where both the base station (BS) and the users are equipped with MAs to fully exploit the degrees of freedom (DoFs) in antenna position optimization. We develop a general channel model to accurately describe the channel characteristics in the near-field region and formulate an MA-position optimization problem to minimize the BS's transmit power subject to users' individual rate constraints. To solve this problem, we propose a two-loop dynamic neighborhood pruning particle swarm optimization (DNPPSO) algorithm that significantly reduces the computational complexity as compared to the standard particle swarm optimization (PSO) algorithm while achieving similar performance. Simulation results validate the effectiveness and advantages of the proposed scheme in power-saving for near-field multiuser communications.
The nasopharyngeal (NP) swab test is a method for collecting cultures to diagnose for different types of respiratory illnesses, including COVID-19. Delegating this task to robots would be beneficial in terms of reducing infection risks and bolstering the healthcare system, but a critical component of the NP swab test is having the swab aligned properly with the nasal cavity so that it does not cause excessive discomfort or injury by traveling down the wrong passage. Existing research towards robotic NP swabbing typically assumes the patient's head is held within a fixture. This simplifies the alignment problem, but is also dissimilar to clinical scenarios where patients are typically free-standing. Consequently, our work creates a vision-guided pipeline to allow an instrumented robot arm to properly position and orient NP swabs with respect to the nostrils of free-standing patients. The first component of the pipeline is a precomputed joint lookup table to allow the arm to meet the patient's arbitrary position in the designated workspace, while avoiding joint limits. Our pipeline leverages semantic face models from computer vision to estimate the Euclidean pose of the face with respect to a monocular RGB-D camera placed on the end-effector. These estimates are passed into an unscented Kalman filter on manifolds state estimator and a pose based visual servo control loop to move the swab to the designated pose in front of the nostril. Our pipeline was validated with human trials, featuring a cohort of 25 participants. The system is effective, reaching the nostril for 84% of participants, and our statistical analysis did not find significant demographic biases within the cohort.
Several methodologies have recently been proposed to evaluate the ability of Pretrained Language Models (PLMs) to interpret negation. In this article, we build on Gubelmann and Handschuh (2022), which studies the modification of PLMs' predictions as a function of the polarity of inputs, in English. Crucially, this test uses ``self-contained'' inputs ending with a masked position: depending on the polarity of a verb in the input, a particular token is either semantically ruled out or allowed at the masked position. By replicating Gubelmann and Handschuh (2022) experiments, we have uncovered flaws that weaken the conclusions that can be drawn from this test. We thus propose an improved version, the Self-Contained Neg Test, which is more controlled, more systematic, and entirely based on examples forming minimal pairs varying only in the presence or absence of verbal negation in English. When applying our test to the roberta and bert base and large models, we show that only roberta-large shows trends that match the expectations, while bert-base is mostly insensitive to negation. For all the tested models though, in a significant number of test instances the top-1 prediction remains the token that is semantically forbidden by the context, which shows how much room for improvement remains for a proper treatment of the negation phenomenon.
Point cloud analysis has achieved significant development and is well-performed in multiple downstream tasks like point cloud classification and segmentation, etc. Being conscious of the simplicity of the position encoding structure in Transformer-based architectures, we attach importance to the position encoding as a high-dimensional part and the patch encoder to offer multi-scale information. Together with the sequential Transformer, the whole module with position encoding comprehensively constructs a multi-scale feature abstraction module that considers both the local parts from the patch and the global parts from center points as position encoding. With only a few parameters, the position embedding module fits the setting of PEFT (Parameter-Efficient Fine-Tuning) tasks pretty well. Thus we unfreeze these parameters as a fine-tuning part. At the same time, we review the existing prompt and adapter tuning methods, proposing a fresh way of prompts and synthesizing them with adapters as dynamic adjustments. Our Proposed method of PEFT tasks, namely PPT, with only 1.05% of parameters for training, gets state-of-the-art results in several mainstream datasets, such as 95.01% accuracy in the ScanObjectNN OBJ_BG dataset. Codes will be released at https://github.com/zsc000722/PPT.
In high-energy physics, accurately estimating the kinematic parameters (position and momentum) of anti-neutrons ($\bar{n}$) is essential for exploring the fundamental governing principles. However, this process is particularly challenging when using an electromagnetic calorimeter (EMC) as the energy detector, due to their limited accuracy and efficiency in interacting with $\bar{n}$. To address this issue, we propose Vision Calorimeter (ViC), a data-driven framework which migrates visual object detection techniques to high-energy particle images. To accommodate the unique characteristics of particle images, we introduce the heat-conduction operator (HCO) into both the backbone and the head of the conventional object detector and conduct significant structural improvements. HCO enjoys the advantage of both radial prior and global attention, as it is inspired by physical heat conduction which naturally aligns with the pattern of particle incidence. Implemented via the Discrete Cosine Transform (DCT), HCO extracts frequency-domain features, bridging the distribution gap between the particle images and the natural images on which visual object detectors are pre-trained. Experimental results demonstrate that ViC significantly outperforms traditional approaches, reducing the incident position prediction error by 46.16% (from 17.31$^{\circ}$ to 9.32$^{\circ}$) and providing the first baseline result with an incident momentum regression error of 21.48%. This study underscores ViC's great potential as a general-purpose particle parameter estimator in high-energy physics. Code is available at https://github.com/yuhongtian17/ViC.