Analysis of high-resolution satellite images has been an important research topic for traffic management, city planning, and road monitoring. One of the problems here is automatic and precise road extraction. From an original image, it is difficult and computationally expensive to extract roads due to presences of other road-like features with straight edges. In this paper, we propose an approach for automatic road extraction based on a fully convolutional neural network of U-net family. This network consists of ResNet-34 pre-trained on ImageNet and decoder adapted from vanilla U-Net. Based on validation results, leaderboard and our own experience this network shows superior results for the DEEPGLOBE - CVPR 2018 road extraction sub-challenge. Moreover, this network uses moderate memory that allows using just one GTX 1080 or 1080ti video cards to perform whole training and makes pretty fast predictions.
We introduce Generative Neural Machine Translation (GNMT), a latent variable architecture which is designed to model the semantics of the source and target sentences. We modify an encoder-decoder translation model by adding a latent variable as a language agnostic representation which is encouraged to learn the meaning of the sentence. GNMT achieves competitive BLEU scores on pure translation tasks, and is superior when there are missing words in the source sentence. We augment the model to facilitate multilingual translation and semi-supervised learning without adding parameters. This framework significantly reduces overfitting when there is limited paired data available, and is effective for translating between pairs of languages not seen during training.
While the major white matter tracts are of great interest to numerous studies in neuroscience and medicine, their manual dissection in larger cohorts from diffusion MRI tractograms is time-consuming, requires expert knowledge and is hard to reproduce. Tract orientation mapping (TOM) is a novel concept that facilitates bundle-specific tractography based on a learned mapping from the original fiber orientation distribution function (fODF) peaks to a list of tract orientation maps (also abbr. TOM). Each TOM represents one of the known tracts with each voxel containing no more than one orientation vector. TOMs can act as a prior or even as direct input for tractography. We use an encoder-decoder fully-convolutional neural network architecture to learn the required mapping. In comparison to previous concepts for the reconstruction of specific bundles, the presented one avoids various cumbersome processing steps like whole brain tractography, atlas registration or clustering. We compare it to four state of the art bundle recognition methods on 20 different bundles in a total of 105 subjects from the Human Connectome Project. Results are anatomically convincing even for difficult tracts, while reaching low angular errors, unprecedented runtimes and top accuracy values (Dice). Our code and our data are openly available.
A major proportion of a text summary includes important entities found in the original text. These entities build up the topic of the summary. Moreover, they hold commonsense information once they are linked to a knowledge base. Based on these observations, this paper investigates the usage of linked entities to guide the decoder of a neural text summarizer to generate concise and better summaries. To this end, we leverage on an off-the-shelf entity linking system (ELS) to extract linked entities and propose Entity2Topic (E2T), a module easily attachable to a sequence-to-sequence model that transforms a list of entities into a vector representation of the topic of the summary. Current available ELS's are still not sufficiently effective, possibly introducing unresolved ambiguities and irrelevant entities. We resolve the imperfections of the ELS by (a) encoding entities with selective disambiguation, and (b) pooling entity vectors using firm attention. By applying E2T to a simple sequence-to-sequence model with attention mechanism as base model, we see significant improvements of the performance in the Gigaword (sentence to title) and CNN (long document to multi-sentence highlights) summarization datasets by at least 2 ROUGE points.
Multi-label classification is an important yet challenging task in natural language processing. It is more complex than single-label classification in that the labels tend to be correlated. Existing methods tend to ignore the correlations between labels. Besides, different parts of the text can contribute differently for predicting different labels, which is not considered by existing models. In this paper, we propose to view the multi-label classification task as a sequence generation problem, and apply a sequence generation model with a novel decoder structure to solve it. Extensive experimental results show that our proposed methods outperform previous work by a substantial margin. Further analysis of experimental results demonstrates that the proposed methods not only capture the correlations between labels, but also select the most informative words automatically when predicting different labels.
In this paper, we propose a method for obtaining sentence-level embeddings. While the problem of securing word-level embeddings is very well studied, we propose a novel method for obtaining sentence-level embeddings. This is obtained by a simple method in the context of solving the paraphrase generation task. If we use a sequential encoder-decoder model for generating paraphrase, we would like the generated paraphrase to be semantically close to the original sentence. One way to ensure this is by adding constraints for true paraphrase embeddings to be close and unrelated paraphrase candidate sentence embeddings to be far. This is ensured by using a sequential pair-wise discriminator that shares weights with the encoder that is trained with a suitable loss function. Our loss function penalizes paraphrase sentence embedding distances from being too large. This loss is used in combination with a sequential encoder-decoder network. We also validated our method by evaluating the obtained embeddings for a sentiment analysis task. The proposed method results in semantic embeddings and outperforms the state-of-the-art on the paraphrase generation and sentiment analysis task on standard datasets. These results are also shown to be statistically significant.
The variational encoder-decoder (VED) encodes source information as a set of random variables using a neural network, which in turn is decoded into target data using another neural network. In natural language processing, sequence-to-sequence (Seq2Seq) models typically serve as encoder-decoder networks. When combined with a traditional (deterministic) attention mechanism, the variational latent space may be bypassed by the attention model, and thus becomes ineffective. In this paper, we propose a variational attention mechanism for VED, where the attention vector is also modeled as Gaussian distributed random variables. Results on two experiments show that, without loss of quality, our proposed method alleviates the bypassing phenomenon as it increases the diversity of generated sentences.
Image-to-image translation is considered a new frontier in the field of medical image analysis, with numerous potential applications. However, a large portion of recent approaches offers individualized solutions based on specialized task-specific architectures or require refinement through non-end-to-end training. In this paper, we propose a new framework, named MedGAN, for medical image-to-image translation which operates on the image level in an end-to-end manner. MedGAN builds upon recent advances in the field of generative adversarial networks (GANs) by merging the adversarial framework with a new combination of non-adversarial losses. We utilize a discriminator network as a trainable feature extractor which penalizes the discrepancy between the translated medical images and the desired modalities. Moreover, style-transfer losses are utilized to match the textures and fine-structures of the desired target images to the translated images. Additionally, we present a new generator architecture, titled CasNet, which enhances the sharpness of the translated medical outputs through progressive refinement via encoder-decoder pairs. Without any application-specific modifications, we apply MedGAN on three different tasks: PET-CT translation, correction of MR motion artefacts and PET image denoising. Perceptual analysis by radiologists and quantitative evaluations illustrate that the MedGAN outperforms other existing translation approaches.
End-to-end models have been showing superiority in Automatic Speech Recognition (ASR). At the same time, the capacity of streaming recognition has become a growing requirement for end-to-end models. Following these trends, an encoder-decoder recurrent neural network called Recurrent Neural Aligner (RNA) has been freshly proposed and shown its competitiveness on two English ASR tasks. However, it is not clear if RNA can be further improved and applied to other spoken language. In this work, we explore the applicability of RNA in Mandarin Chinese and present four effective extensions: In the encoder, we redesign the temporal down-sampling and introduce a powerful convolutional structure. In the decoder, we utilize a regularizer to smooth the output distribution and conduct joint training with a language model. On two Mandarin Chinese conversational telephone speech recognition (MTS) datasets, our Extended-RNA obtains promising performance. Particularly, it achieves 27.7% character error rate (CER), which is superior to current state-of-the-art result on the popular HKUST task.
Brain extraction is a fundamental step for most brain imaging studies. In this paper, we investigate the problem of skull stripping and propose complementary segmentation networks (CompNets) to accurately extract the brain from T1-weighted MRI scans, for both normal and pathological brain images. The proposed networks are designed in the framework of encoder-decoder networks and have two pathways to learn features from both the brain tissue and its complementary part located outside of the brain. The complementary pathway extracts the features in the non-brain region and leads to a robust solution to brain extraction from MRIs with pathologies, which do not exist in our training dataset. We demonstrate the effectiveness of our networks by evaluating them on the OASIS dataset, resulting in the state of the art performance under the two-fold cross-validation setting. Moreover, the robustness of our networks is verified by testing on images with introduced pathologies and by showing its invariance to unseen brain pathologies. In addition, our complementary network design is general and can be extended to address other image segmentation problems with better generalization.
Close human-robot cooperation is a key enabler for new developments in advanced manufacturing and assistive applications. Close cooperation require robots that can predict human actions and intent, and understand human non-verbal cues. Recent approaches based on neural networks have led to encouraging results in the human action prediction problem both in continuous and discrete spaces. Our approach extends the research in this direction. Our contributions are three-fold. First, we validate the use of gaze and body pose cues as a means of predicting human action through a feature selection method. Next, we address two shortcomings of existing literature: predicting multiple and variable-length action sequences. This is achieved by introducing an encoder-decoder recurrent neural network topology in the discrete action prediction problem. In addition, we theoretically demonstrate the importance of predicting multiple action sequences as a means of estimating the stochastic reward in a human robot cooperation scenario. Finally, we show the ability to effectively train the prediction model on a action prediction dataset, involving human motion data, and explore the influence of the model's parameters on its performance. Source code repository: https://github.com/pschydlo/ActionAnticipation
Image generation has been successfully cast as an autoregressive sequence generation or transformation problem. Recent work has shown that self-attention is an effective way of modeling textual sequences. In this work, we generalize a recently proposed model architecture based on self-attention, the Transformer, to a sequence modeling formulation of image generation with a tractable likelihood. By restricting the self-attention mechanism to attend to local neighborhoods we significantly increase the size of images the model can process in practice, despite maintaining significantly larger receptive fields per layer than typical convolutional neural networks. While conceptually simple, our generative models significantly outperform the current state of the art in image generation on ImageNet, improving the best published negative log-likelihood on ImageNet from 3.83 to 3.77. We also present results on image super-resolution with a large magnification ratio, applying an encoder-decoder configuration of our architecture. In a human evaluation study, we find that images generated by our super-resolution model fool human observers three times more often than the previous state of the art.
We present a new end-to-end architecture for automatic speech recognition (ASR) that can be trained using \emph{symbolic} input in addition to the traditional acoustic input. This architecture utilizes two separate encoders: one for acoustic input and another for symbolic input, both sharing the attention and decoder parameters. We call this architecture a multi-modal data augmentation network (MMDA), as it can support multi-modal (acoustic and symbolic) input and enables seamless mixing of large text datasets with significantly smaller transcribed speech corpora during training. We study different ways of transforming large text corpora into a symbolic form suitable for training our MMDA network. Our best MMDA setup obtains small improvements on character error rate (CER), and as much as 7-10\% relative word error rate (WER) improvement over a baseline both with and without an external language model.
Speech-to-text translation has many potential applications for low-resource languages, but the typical approach of cascading speech recognition with machine translation is often impossible, since the transcripts needed to train a speech recognizer are usually not available for low-resource languages. Recent work has found that neural encoder-decoder models can learn to directly translate foreign speech in high-resource scenarios, without the need for intermediate transcription. We investigate whether this approach also works in settings where both data and computation are limited. To make the approach efficient, we make several architectural changes, including a change from character-level to word-level decoding. We find that this choice yields crucial speed improvements that allow us to train with fewer computational resources, yet still performs well on frequent words. We explore models trained on between 20 and 160 hours of data, and find that although models trained on less data have considerably lower BLEU scores, they can still predict words with relatively high precision and recall---around 50% for a model trained on 50 hours of data, versus around 60% for the full 160 hour model. Thus, they may still be useful for some low-resource scenarios.
We address the task of 6D pose estimation of known rigid objects from single input images in scenarios where the objects are partly occluded. Recent RGB-D-based methods are robust to moderate degrees of occlusion. For RGB inputs, no previous method works well for partly occluded objects. Our main contribution is to present the first deep learning-based system that estimates accurate poses for partly occluded objects from RGB-D and RGB input. We achieve this with a new instance-aware pipeline that decomposes 6D object pose estimation into a sequence of simpler steps, where each step removes specific aspects of the problem. The first step localizes all known objects in the image using an instance segmentation network, and hence eliminates surrounding clutter and occluders. The second step densely maps pixels to 3D object surface positions, so called object coordinates, using an encoder-decoder network, and hence eliminates object appearance. The third, and final, step predicts the 6D pose using geometric optimization. We demonstrate that we significantly outperform the state-of-the-art for pose estimation of partly occluded objects for both RGB and RGB-D input.
신경망 의미 파서(neural semantic parser)는 토큰 단위로 의미 표상을 생성하지만, 적형성(well-formedness)은 생성 과정에서 전개되는 참조, 스코프 및 그래프 관계에 따라 결정됩니다. 제약 디코딩(constrained decoding)은 국소적으로 유효하지 않은 토큰을 방지하지만, 유효한 접두 문맥(prefix)이 고정된 출력 길이 제한 내에서 완성될 수 있음을 보장하지는 못합니다. 본 연구에서는 DRS 파싱의 세 단계(디코딩, 사후 처리, 학습)에서 구조적 오토마타를 탐구하고, AMR에 대한 디코딩도 추가로 평가합니다. 디코딩을 위해, 잔여 길이 제한 내에서 검증된 완성을 유지하면서 모델이 선호하는 토큰 역시 완성이 가능할 때마다 해당 토큰을 따르는 수락 경로 디코딩(APD)을 도입합니다. APD는 토큰을 철회하지 않고도 수락되는 출력을 보장합니다. 엄격한 길이 제한 하에서 APD는 DRS에서 국소 마스킹 대비 Smatch++를 각각 5.1점 및 2.4점 향상시키고, AMR에서는 6.4점 및 3.2점 향상시켜 토큰을 철회하는 동일한 제한 하의 롤백 방식과 비견할 만한 성능을 보입니다. 사후 처리의 경우, 유효하지 않은 출력은 동일한 입력에 집중됩니다. 제한된 학습 데이터 환경에서는 8개의 비제약 샘플링 결과 입력의 10.2%에서 유효한 출력이 전혀 나오지 않은 반면, 전체 데이터 환경에서는 0.4%에 불과했으며, 이 경우 필터링만으로도 APD와 0.2 Smatch++ 포인트 이내의 차이를 보였습니다. 학습의 경우, 오토마톤이 거부한 토큰에 페널티를 부여하면 학습의 10% 시점에서 검증 세트 Smatch++가 3.6~11.0점 향상되지만, 데이터가 제한된 환경에서 학습 초반 6분의 1 이내에 교차 엔트로피만으로도 거부된 토큰에 할당되는 확률이 14%에서 2% 미만으로 감소함에 따라 이러한 이점은 점차 줄어듭니다.
롱 컨텍스트(long-context) 서빙은 하드웨어가 유효 처리량(throughput)에 도달하기 전에 KV 메모리가 먼저 소진된다. HiSparse는 히스토리를 호스트 메모리에 저장하고 선택된 항목만 가져오지만, 모든 로드(load)에서 단일 캐시 예산(budget)을 사용하며 DSpark 및 DFlash2와 같은 드래프트 모델(draft-model) 추측 디코딩(speculative decoding)을 처리하지 못한다. 우리는 계층적 희소(hierarchical sparse) KV 위에서 추측 디코딩을 지원하고 상주(residency)를 부하(load)에 맞춰 적응시키는 Adaptive DHiSparse를 제시한다. 이 방법은 필요한 작업 집합(working set)을 보호하고, GPU 메모리가 여유 있을 때 추가 히스토리를 유지하며, 나머지는 정확한 융합 검색(fused retrieval)을 통해 복구한다. DSpark와 DFlash2는 각자의 제안(proposal) 및 수락(acceptance) 규칙을 그대로 유지한다: 쿼리는 누락된 행(missing-row) 검색을 공유하고, 드래프트 KV는 윈도우 경계(window-bounded) 링을 사용하며, 검증(verification) 스테이징은 층(layer) 그룹 간에 재사용된다. 100K 컨텍스트와 동시성(concurrency) 128에서, DFlash2를 사용한 GLM-5.3은 디코드 GPU당 793 출력 토큰(token)/s를 달성하며, 이는 HiSparse의 1.45배, 추측 디코딩을 사용하지 않는 적응형 서빙의 1.27배이다. HiSparse와 동일한 지연 시간(latency) 한계를 유지할 때, 이 방법은 동시성을 최대 4배까지, 출력을 토큰당 한계에 따라 2.3배에서 3.7배까지 유지한다. 500K 컨텍스트와 동시성 512에서 DeepSeek-V4.1-Flash에 대해, DSpark를 사용한 Adaptive DHiSparse는 DSpark를 사용한 상주(resident) 서빙의 출력보다 1.85배 높은 출력을 달성한다. 로드 단계(load step)에서, 선택적 상주(residency)는 저부하 상황에서 호스트 페치(fetch) 비율을 선택된 행의 14.2%에서 1.4%로 줄이고, 압박(pressure) 상황에서는 메모리를 해제한다.
적은 수의 센서로부터 공간적으로 분해된 플라즈마 동역학을 재구성하는 것은 진단, 축소 모델링(reduced-order modelling) 및 제어에 필수적이지만, 희소한 측정값이 다중 스케일이면서 영역에 의존적인 자유도를 불완전하게 제약하기 때문에 여전히 어려운 과제로 남아 있다. Shallow Recurrent Decoder(SHRED)는 측정 이력(measurement history)을 활용하여 공간적 희소성(sparsity) 문제를 부분적으로 해결하지만, 완전 연결(fully connected) 디코더는 스케일 전반에 걸친 공간 구조를 해석하거나 명시적인 매개변수(parameter) 의존성을 다루는 명시적 메커니즘을 제공하지 않는다.
본 연구에서는 Recurrent Multiscale Affine-modulated Inference Network(ReMAIN)를 제안한다. 이는 SHRED의 순환적(recurrent) 시간 인코딩을 유지하면서도, 디코더를 U-Net으로 대체하여 특징(feature) 계층 구조가 특징별 선형 변조(feature-wise linear modulation)를 통해 순환 상태(recurrent state)에 의해 조건화되도록 한다. 시간적 표현(temporal representation)은 U-Net 전반에 걸쳐 조밀한 사전 정보(dense prior)와 스케일별 변조를 동시에 제공한다. 매개변수적 확장(parametric extension)은 운영 조건과 센서 이력을 함께 임베딩하여, 지배 동역학이 운영 영역(operating regime)에 따라 변화할 때 재구성이 이에 적응할 수 있도록 한다.
ReMAIN은 먼저 다양한 동역학을 대표하는 6개의 1차원 비선형 편미분방정식(PDE)에 대해 SHRED와 벤치마크(benchmark) 비교된다. 모든 벤치마크에서 ReMAIN은 미학습(unseen) 궤적에 대한 재구성 오차를 줄이고, 급격한 전환(sharp transition), 국소 극값(localized extrema), 미세 스케일 변화를 더 충실하게 해석한다. 이후 매개변수 조건화 모델(parameter-conditioned model)은 수직 축방향 전기장과 반경 방향 자기장을 받는 무충돌(collisionless) E × B 플라즈마에 대해 시연되며, 이때 전기장 강도가 운영 매개변수(operating parameter) 역할을 한다. ReMAIN은 고차원의 다중 스케일 플라즈마 상태를 재구성하며, 학습(training)에서 제외된 전기장 강도에서도 영역에 의존적인 시공간적 동역학을 복원한다.
종합적으로, ReMAIN은 희소 센서 기반의 전체 상태 재구성을 개선하며, 매개변수 조건화를 통해 플라즈마 운영 영역 전반에 걸쳐 일반화된다.
단일의 공유된 학습률(learning rate) 하에서 매개변수 효율적 미세조정(PEFT) 레시피들을 비교하는 것은 흔하지만 결함이 있는 방식입니다. 비교 대상군들의 학습 가능한 매개변수 수가 크게 다를 경우, 공유된 학습률은 매개변수가 더 많은 실험군의 성능 평균을 낮추는 동시에 분산을 부풀려, 실제 효과가 아님에도 가장 작은 실험군에 대해 마치 다중 시드에서 유의미한 것처럼 보이는 큰 이점을 인위적으로 만들어낼 수 있습니다. 본 연구는 사후(post-hoc) SVD 기반 KV 캐시(KV-cache) 압축이라는 구체적인 설정에서 이러한 교란 요인을 기록합니다. 이 설정에서는 이미 사전학습된 모델의 키/값(key/value) 가중치를 다운 프로젝션("인코더")과 업 프로젝션("디코더")으로 분해하여 저차원(다중 헤드 잠재 어텐션 스타일) 캐시로 변환한 뒤, 짧은 미세조정("치유", healing)을 통해 절단(truncation)으로 손실된 정확도를 복구합니다. 공유된 학습률 하에서는 디코더를 동결(freezing)하고 인코더만 치유하는 것이 디코더만 치유하거나 두 요소를 모두 치유하는 것보다 확실히 우수한 것처럼 보이지만, 각 실험군에 맞게 튜닝된 자체 학습률을 부여하면 이러한 겉보기 이점은 사라집니다. 대신 인코더 전용 치유는 다른 대안들과 동등한 성능에 도달하면서도, 실제로 측정된 학습 가능 매개변수 수를 3배 줄이고 옵티마이저(optimizer) 상태 메모리를 3배 절감합니다. 우리는 비전-언어 모델(vision-language model; Qwen2.5-VL-3B-Instruct)에서 이 프로토콜이 다루는 단일 압축률에 대해 실험군별 학습률 튜닝과 구성당 3개의 시드를 사용하여 이러한 동등성을 검증했으며, 두 가지 백본에 걸친 텍스트 전용 테스트베드에서도 이를 재현했습니다. 따라서 인코더 전용 치유는 학습 시점에 저차원 KV 캐시 압축을 소급 적용(retrofiting)하기 위한 저메모리 드롭인(drop-in) 레시피이며, 본 논문에서 문서화하고 수정한 공유 학습률의 함정은 학습 가능한 매개변수 수가 서로 다른 미세조정 레시피들을 비교할 때 주의해야 할 경고성 결과를 제공합니다.
현재의 트랜스포머는 위치들 사이에서 풍부한 잠재 잔차 스트림을 버리고, 매 새로운 위치마다 잠재 추론 맥락을 다시 구성하여 잠재적인 추론 역량을 활용하지 못하고 있다. State Stream Transformer(SST) V2는 각 디코더 층에서 FFN 기반의 비선형 순환을 통해 연속 잠재 공간에서의 매개변수 효율적인 추론을 가능하게 하며, 이때 잠재 상태들은 학습된 블렌드를 통해 전체 시퀀스에 걸쳐 수평으로 스트리밍된다. 이와 동일한 메커니즘은 추론 시점에서 위치별 연속적인 잠재 숙고를 지원하여, 토큰을 확정하기 전에 추가적인 FLOPs를 추상적 추론 탐색에 할당할 수 있게 한다. 2단계 병렬 학습 절차는 순차적 순환을 근사하여 공동 학습을 계산적으로 실용적으로 만든다. 은닉 상태 분석에 따르면, 상태 스트림은 연속 잠재 공간에서 명확하고 내용에 의존적인 재구성을 통해 추론을 촉진하며, LM 헤드는 그 결과로 나타나는 잠재 신념 상태를 출력 분포를 통해 드러내고, 상태 스트림은 이를 이후 위치들에 영향을 미치도록 전달한다. 학습된 프로브는 첫 번째로 생성된 토큰에서 이미 잠재 상태가 이후의 모든 위치에서 추가적인 잠재 계산 하에 최종 답이 유지될지 또는 무너질지를 예측함을 보여준다. 단지 소규모의 GSM8K 예제 데이터셋만을 사용하여 기존 27B 백본에 공동 학습시키고, 각 질문을 토큰당 1회에서 4회의 순환적 순전파 깊이로 라우팅하는 오라클을 사용하여 평가한 결과, SST는 분포 외 GPQA-Diamond에서 61.11%의 아키텍처 역량 상한을 달성했으며, 이는 미세조정 매칭 기준선 대비 +15.15포인트의 향상이고, 동일한 기준선에 남아 있는 GSM8K 오류를 46% 줄인다. 이러한 결과들은 비선형 순환을 효율적으로 학습시키는 방법을 제시하며, 상태 스트림이 그 외에는 학습이 매칭된 트랜스포머가 가진 역량을 넘어서는 추가적인 추론 역량을 제공함을 보여준다.