PyTorchKR Papers

최신 논문

총 107편
태그·기간 등 상세 필터

카테고리 탐색

로그인 후 이 태그 구독

arXiv:1912.10150공식 코드 · Papers with Code

Learning Diverse Stochastic Human-Action Generators by Learning Smooth Latent Transitions

Human-motion generation is a long-standing challenging task due to the requirement of accurately modeling complex and diverse dynamic patterns. Most existing methods adopt sequence models such as RNN to directly model transitions in the original action space. Due to high dimensionality and potential noise, such modeling of action transitions is particularly challenging. In this paper, we focus on skeleton-based action generation and propose to model smooth and diverse transitions on a latent space of action sequences with much lower dimensionality. Conditioned on a latent sequence, actions are generated by a frame-wise decoder shared by all latent action-poses. Specifically, an implicit RNN is defined to model smooth latent sequences, whose randomness (diversity) is controlled by noise from the input. Different from standard action-prediction methods, our model can generate action sequences from pure noise without any conditional action poses. Remarkably, it can also generate unseen actions from mixed classes during training. Our model is learned with a bi-directional generative-adversarial-net framework, which not only can generate diverse action sequences of a particular class or mix classes, but also learns to classify action sequences within the same model. Experimental results show the superiority of our method in both diverse action-sequence generation and classification, relative to existing methods.

arXiv:1904.08784

Efficient Motion Planning for Automated Lane Change based on Imitation Learning and Mixed-Integer Optimization

Intelligent motion planning is one of the core components in automated vehicles, which has received extensive interests. Traditional motion planning methods suffer from several drawbacks in terms of optimality, efficiency and generalization capability. Sampling based methods cannot guarantee the optimality of the generated trajectories. Whereas the optimization-based methods are not able to perform motion planning in real-time, and limited by the simplified formalization. In this work, we propose a learning-based approach to handle those shortcomings. Mixed Integer Quadratic Problem based optimization (MIQP) is used to generate the optimal lane-change trajectories which served as the training dataset for learning-based action generation algorithms. A hierarchical supervised learning model is devised to make the fast lane-change decision. Numerous experiments have been conducted to evaluate the optimality, efficiency, and generalization capability of the proposed approach. The experimental results indicate that the proposed model outperforms several commonly used motion planning baselines.

arXiv:1902.11109

Actions Generation from Captions

Sequence transduction models have been widely explored in many natural language processing tasks. However, the target sequence usually consists of discrete tokens which represent word indices in a given vocabulary. We barely see the case where target sequence is composed of continuous vectors, where each vector is an element of a time series taken successively in a temporal domain. In this work, we introduce a new data set, named Action Generation Data Set (AGDS) which is specifically designed to carry out the task of caption-to-action generation. This data set contains caption-action pairs. The caption is comprised of a sequence of words describing the interactive movement between two people, and the action is a captured sequence of poses representing the movement. This data set is introduced to study the ability of generating continuous sequences through sequence transduction models. We also propose a model to innovatively combine Multi-Head Attention (MHA) and Generative Adversarial Network (GAN) together. In our model, we have one generator to generate actions from captions and three discriminators where each of them is designed to carry out a unique functionality: caption-action consistency discriminator, pose discriminator and pose transition discriminator. This novel design allowed us to achieve plausible generation performance which is demonstrated in the experiments.

arXiv:1809.00786공식 코드 · Papers with Code

Mapping Instructions to Actions in 3D Environments with Visual Goal Prediction

We propose to decompose instruction execution to goal prediction and action generation. We design a model that maps raw visual observations to goals using LINGUNET, a language-conditioned image generation network, and then generates the actions required to complete them. Our model is trained from demonstration only without external resources. To evaluate our approach, we introduce two benchmarks for instruction following: LANI, a navigation task; and CHAI, where an agent executes household instructions. Our evaluation demonstrates the advantages of our model decomposition, and illustrates the challenges posed by our new benchmarks.

arXiv:1706.02444

Predictive Coding-based Deep Dynamic Neural Network for Visuomotor Learning

This study presents a dynamic neural network model based on the predictive coding framework for perceiving and predicting the dynamic visuo-proprioceptive patterns. In our previous study [1], we have shown that the deep dynamic neural network model was able to coordinate visual perception and action generation in a seamless manner. In the current study, we extended the previous model under the predictive coding framework to endow the model with a capability of perceiving and predicting dynamic visuo-proprioceptive patterns as well as a capability of inferring intention behind the perceived visuomotor information through minimizing prediction error. A set of synthetic experiments were conducted in which a robot learned to imitate the gestures of another robot in a simulation environment. The experimental results showed that with given intention states, the model was able to mentally simulate the possible incoming dynamic visuo-proprioceptive patterns in a top-down process without the inputs from the external environment. Moreover, the results highlighted the role of minimizing prediction error in inferring underlying intention of the perceived visuo-proprioceptive patterns, supporting the predictive coding account of the mirror neuron systems. The results also revealed that minimizing prediction error in one modality induced the recall of the corresponding representation of another modality acquired during the consolidative learning of raw-level visuo-proprioceptive patterns.

arXiv:1805.10416

Human Action Generation with Generative Adversarial Networks

Inspired by the recent advances in generative models, we introduce a human action generation model in order to generate a consecutive sequence of human motions to formulate novel actions. We propose a framework of an autoencoder and a generative adversarial network (GAN) to produce multiple and consecutive human actions conditioned on the initial state and the given class label. The proposed model is trained in an end-to-end fashion, where the autoencoder is jointly trained with the GAN. The model is trained on the NTU RGB+D dataset and we show that the proposed model can generate different styles of actions. Moreover, the model can successfully generate a sequence of novel actions given different action labels as conditions. The conventional human action prediction and generation models lack those features, which are essential for practical applications.

arXiv:2406.00636v2cs.CV

T2LM: 다중 문장으로부터의 장기 3D 인간 동작 생성

T2LM: Long-Term 3D Human Motion Generation from Multiple Sentences

본 논문에서는 장기 3D 인간 동작 생성이라는 어려운 문제를 다룹니다. 구체적으로, 여러 문장의 스트림(즉, 문단)에서 부드럽게 연결된 동작의 긴 시퀀스를 생성하는 것을 목표합니다. 이전 장기 동작 생성 접근법은 대부분 순환(recurrent) 방법에 기반하여, 이전에 생성한 모션 청크를 다음 단계의 입력으로 사용했습니다. 하지만 이 접근법에는 두 가지 단점이 있습니다: 1) 비용이 많이 드는 순차 데이터셋에 의존한다; 2) 각 단계에서 생성된 모션 사이에 비현실적인 간격이 발생한다. 이러한 문제를 해결하기 위해, 순차 데이터 없이 학습할 수 있는 단순하지만 효과적인 연속적 장기 생성 프레임워크 T2LM을 제안합니다. T2LM은 두 가지 구성 요소로 이루어져 있습니다: 모션을 잠재 벡터 시퀀스로 압축하기 위해 학습된 1D 컨볼루션 VQVAE, 그리고 입력 텍스트에 따라 잠재 시퀀스를 예측하는 트랜스포머 기반 텍스트 인코더. 추론 시, 문장 시퀀스가 연속적인 잠재 벡터 스트림으로 변환됩니다. 이것은 VQVAE 디코더에 의해 모션으로 디코딩되며, 국소 시간 수용 영역을 가진 1D 컨볼루션의 사용은 학습 시퀀스와 생성 시퀀스 사이의 시간적 불일치를 방지합니다. VQ-VAE에 대한 이 단순한 제약은 짧은 시퀀스만으로 학습할 수 있게 하고, 더 부드러운 전환을 만들어냅니다. T2LM은 순차 데이터가 필요한 제약을 극복하는 동시에 이전 장기 생성 모델들을 능가하고, SOTA 단일 동작 생성 모델들과도 경쟁력을 보입니다.