PyTorchKR Papers

최신 논문

총 2,023편
태그·기간 등 상세 필터

카테고리 탐색

로그인 후 이 태그 구독

arXiv:1904.09117공식 코드 · Papers with Code

SelFlow: Self-Supervised Learning of Optical Flow

We present a self-supervised learning approach for optical flow. Our method distills reliable flow estimations from non-occluded pixels, and uses these predictions as ground truth to learn optical flow for hallucinated occlusions. We further design a simple CNN to utilize temporal information from multiple frames for better flow estimation. These two principles lead to an approach that yields the best performance for unsupervised optical flow learning on the challenging benchmarks including MPI Sintel, KITTI 2012 and 2015. More notably, our self-supervised pre-trained model provides an excellent initialization for supervised fine-tuning. Our fine-tuned models achieve state-of-the-art results on all three datasets. At the time of writing, we achieve EPE=4.26 on the Sintel benchmark, outperforming all submitted methods.

arXiv:1904.00230공식 코드 · Papers with Code

MortonNet: Self-Supervised Learning of Local Features in 3D Point Clouds

We present a self-supervised task on point clouds, in order to learn meaningful point-wise features that encode local structure around each point. Our self-supervised network, named MortonNet, operates directly on unstructured/unordered point clouds. Using a multi-layer RNN, MortonNet predicts the next point in a point sequence created by a popular and fast Space Filling Curve, the Morton-order curve. The final RNN state (coined Morton feature) is versatile and can be used in generic 3D tasks on point clouds. In fact, we show how Morton features can be used to significantly improve performance (+3% for 2 popular semantic segmentation algorithms) in the task of semantic segmentation of point clouds on the challenging and large-scale S3DIS dataset. We also show how MortonNet trained on S3DIS transfers well to another large-scale dataset, vKITTI, leading to an improvement over state-of-the-art of 3.8%. Finally, we use Morton features to train a much simpler and more stable model for part segmentation in ShapeNet. Our results show how our self-supervised task results in features that are useful for 3D segmentation tasks, and generalize well to other datasets.

arXiv:1903.11412공식 코드 · Papers with Code

Self-Supervised Learning via Conditional Motion Propagation

Intelligent agent naturally learns from motion. Various self-supervised algorithms have leveraged motion cues to learn effective visual representations. The hurdle here is that motion is both ambiguous and complex, rendering previous works either suffer from degraded learning efficacy, or resort to strong assumptions on object motions. In this work, we design a new learning-from-motion paradigm to bridge these gaps. Instead of explicitly modeling the motion probabilities, we design the pretext task as a conditional motion propagation problem. Given an input image and several sparse flow guidance vectors on it, our framework seeks to recover the full-image motion. Compared to other alternatives, our framework has several appealing properties: (1) Using sparse flow guidance during training resolves the inherent motion ambiguity, and thus easing feature learning. (2) Solving the pretext task of conditional motion propagation encourages the emergence of kinematically-sound representations that poss greater expressive power. Extensive experiments demonstrate that our framework learns structural and coherent features; and achieves state-of-the-art self-supervision performance on several downstream tasks including semantic segmentation, instance segmentation, and human parsing. Furthermore, our framework is successfully extended to several useful applications such as semi-automatic pixel-level annotation. Project page: "http://mmlab.ie.cuhk.edu.hk/projects/CMP/".

arXiv:1903.02330공식 코드 · Papers with Code

Self-Supervised Learning of 3D Human Pose using Multi-view Geometry

Training accurate 3D human pose estimators requires large amount of 3D ground-truth data which is costly to collect. Various weakly or self supervised pose estimation methods have been proposed due to lack of 3D data. Nevertheless, these methods, in addition to 2D ground-truth poses, require either additional supervision in various forms (e.g. unpaired 3D ground truth data, a small subset of labels) or the camera parameters in multiview settings. To address these problems, we present EpipolarPose, a self-supervised learning method for 3D human pose estimation, which does not need any 3D ground-truth data or camera extrinsics. During training, EpipolarPose estimates 2D poses from multi-view images, and then, utilizes epipolar geometry to obtain a 3D pose and camera geometry which are subsequently used to train a 3D pose estimator. We demonstrate the effectiveness of our approach on standard benchmark datasets i.e. Human3.6M and MPI-INF-3DHP where we set the new state-of-the-art among weakly/self-supervised methods. Furthermore, we propose a new performance measure Pose Structure Score (PSS) which is a scale invariant, structure aware measure to evaluate the structural plausibility of a pose with respect to its ground truth. Code and pretrained models are available at https://github.com/mkocabas/EpipolarPose

arXiv:1903.01000공식 코드 · Papers with Code

Self-Supervised Learning of Face Representations for Video Face Clustering

Analyzing the story behind TV series and movies often requires understanding who the characters are and what they are doing. With improving deep face models, this may seem like a solved problem. However, as face detectors get better, clustering/identification needs to be revisited to address increasing diversity in facial appearance. In this paper, we address video face clustering using unsupervised methods. Our emphasis is on distilling the essential information, identity, from the representations obtained using deep pre-trained face networks. We propose a self-supervised Siamese network that can be trained without the need for video/track based supervision, and thus can also be applied to image collections. We evaluate our proposed method on three video face clustering datasets. The experiments show that our methods outperform current state-of-the-art methods on all datasets. Video face clustering is lacking a common benchmark as current works are often evaluated with different metrics and/or different sets of face tracks.

arXiv:1902.09103공식 코드 · Papers with Code

Beyond Photometric Loss for Self-Supervised Ego-Motion Estimation

Accurate relative pose is one of the key components in visual odometry (VO) and simultaneous localization and mapping (SLAM). Recently, the self-supervised learning framework that jointly optimizes the relative pose and target image depth has attracted the attention of the community. Previous works rely on the photometric error generated from depths and poses between adjacent frames, which contains large systematic error under realistic scenes due to reflective surfaces and occlusions. In this paper, we bridge the gap between geometric loss and photometric loss by introducing the matching loss constrained by epipolar geometry in a self-supervised framework. Evaluated on the KITTI dataset, our method outperforms the state-of-the-art unsupervised ego-motion estimation methods by a large margin. The code and data are available at https://github.com/hlzz/DeepMatchVO.

arXiv:1902.07766공식 코드 · Papers with Code

Dense Depth Estimation in Monocular Endoscopy with Self-supervised Learning Methods

We present a self-supervised approach to training convolutional neural networks for dense depth estimation from monocular endoscopy data without a priori modeling of anatomy or shading. Our method only requires monocular endoscopic videos and a multi-view stereo method, e.g., structure from motion, to supervise learning in a sparse manner. Consequently, our method requires neither manual labeling nor patient computed tomography (CT) scan in the training and application phases. In a cross-patient experiment using CT scans as groundtruth, the proposed method achieved submillimeter mean residual error. In a comparison study to recent self-supervised depth estimation methods designed for natural video on in vivo sinus endoscopy data, we demonstrate that the proposed approach outperforms the previous methods by a large margin. The source code for this work is publicly available online at https://github.com/lppllppl920/EndoscopyDepthEstimation-Pytorch.

arXiv:1901.09005공식 코드 · Papers with Code

Revisiting Self-Supervised Visual Representation Learning

Unsupervised visual representation learning remains a largely unsolved problem in computer vision research. Among a big body of recently proposed approaches for unsupervised learning of visual representations, a class of self-supervised techniques achieves superior performance on many challenging benchmarks. A large number of the pretext tasks for self-supervised learning have been studied, but other important aspects, such as the choice of convolutional neural networks (CNN), has not received equal attention. Therefore, we revisit numerous previously proposed self-supervised models, conduct a thorough large scale study and, as a result, uncover multiple crucial insights. We challenge a number of common practices in selfsupervised visual representation learning and observe that standard recipes for CNN design do not always translate to self-supervised representation learning. As part of our study, we drastically boost the performance of previously proposed techniques and outperform previously published state-of-the-art results by a large margin.

arXiv:1901.02004공식 코드 · Papers with Code

Self-Supervised Learning from Web Data for Multimodal Retrieval

Self-Supervised learning from multimodal image and text data allows deep neural networks to learn powerful features with no need of human annotated data. Web and Social Media platforms provide a virtually unlimited amount of this multimodal data. In this work we propose to exploit this free available data to learn a multimodal image and text embedding, aiming to leverage the semantic knowledge learnt in the text domain and transfer it to a visual model for semantic image retrieval. We demonstrate that the proposed pipeline can learn from images with associated textwithout supervision and analyze the semantic structure of the learnt joint image and text embedding space. We perform a thorough analysis and performance comparison of five different state of the art text embeddings in three different benchmarks. We show that the embeddings learnt with Web and Social Media data have competitive performances over supervised methods in the text based image retrieval task, and we clearly outperform state of the art in the MIRFlickr dataset when training in the target data. Further, we demonstrate how semantic multimodal image retrieval can be performed using the learnt embeddings, going beyond classical instance-level retrieval problems. Finally, we present a new dataset, InstaCities1M, composed by Instagram images and their associated texts that can be used for fair comparison of image-text embeddings.

arXiv:1811.03879공식 코드 · Papers with Code

Cross and Learn: Cross-Modal Self-Supervision

In this paper we present a self-supervised method for representation learning utilizing two different modalities. Based on the observation that cross-modal information has a high semantic meaning we propose a method to effectively exploit this signal. For our approach we utilize video data since it is available on a large scale and provides easily accessible modalities given by RGB and optical flow. We demonstrate state-of-the-art performance on highly contested action recognition datasets in the context of self-supervised learning. We show that our feature representation also transfers to other tasks and conduct extensive ablation studies to validate our core contributions. Code and model can be found at https://github.com/nawidsayed/Cross-and-Learn.

arXiv:1810.10191공식 코드 · Papers with Code

Making Sense of Vision and Touch: Self-Supervised Learning of Multimodal Representations for Contact-Rich Tasks

Contact-rich manipulation tasks in unstructured environments often require both haptic and visual feedback. However, it is non-trivial to manually design a robot controller that combines modalities with very different characteristics. While deep reinforcement learning has shown success in learning control policies for high-dimensional inputs, these algorithms are generally intractable to deploy on real robots due to sample complexity. We use self-supervision to learn a compact and multimodal representation of our sensory inputs, which can then be used to improve the sample efficiency of our policy learning. We evaluate our method on a peg insertion task, generalizing over different geometry, configurations, and clearances, while being robust to external perturbations. Results for simulated and real robot experiments are presented.

arXiv:1807.02110공식 코드 · Papers with Code

TextTopicNet - Self-Supervised Learning of Visual Features Through Embedding Images on Semantic Text Spaces

The immense success of deep learning based methods in computer vision heavily relies on large scale training datasets. These richly annotated datasets help the network learn discriminative visual features. Collecting and annotating such datasets requires a tremendous amount of human effort and annotations are limited to popular set of classes. As an alternative, learning visual features by designing auxiliary tasks which make use of freely available self-supervision has become increasingly popular in the computer vision community. In this paper, we put forward an idea to take advantage of multi-modal context to provide self-supervision for the training of computer vision algorithms. We show that adequate visual features can be learned efficiently by training a CNN to predict the semantic textual context in which a particular image is more probable to appear as an illustration. More specifically we use popular text embedding techniques to provide the self-supervision for the training of deep CNN. Our experiments demonstrate state-of-the-art performance in image classification, object detection, and multi-modal retrieval compared to recent self-supervised or naturally-supervised approaches.

arXiv:1806.11191공식 코드 · Papers with Code

CR-GAN: Learning Complete Representations for Multi-view Generation

Generating multi-view images from a single-view input is an essential yet challenging problem. It has broad applications in vision, graphics, and robotics. Our study indicates that the widely-used generative adversarial network (GAN) may learn "incomplete" representations due to the single-pathway framework: an encoder-decoder network followed by a discriminator network. We propose CR-GAN to address this problem. In addition to the single reconstruction path, we introduce a generation sideway to maintain the completeness of the learned embedding space. The two learning pathways collaborate and compete in a parameter-sharing manner, yielding considerably improved generalization ability to "unseen" dataset. More importantly, the two-pathway framework makes it possible to combine both labeled and unlabeled data for self-supervised learning, which further enriches the embedding space for realistic generations. The experimental results prove that CR-GAN significantly outperforms state-of-the-art methods, especially when generating from "unseen" inputs in wild conditions.

arXiv:1703.05446공식 코드 · Papers with Code

Look into Person: Self-supervised Structure-sensitive Learning and A New Benchmark for Human Parsing

Human parsing has recently attracted a lot of research interests due to its huge application potentials. However existing datasets have limited number of images and annotations, and lack the variety of human appearances and the coverage of challenging cases in unconstrained environment. In this paper, we introduce a new benchmark "Look into Person (LIP)" that makes a significant advance in terms of scalability, diversity and difficulty, a contribution that we feel is crucial for future developments in human-centric analysis. This comprehensive dataset contains over 50,000 elaborately annotated images with 19 semantic part labels, which are captured from a wider range of viewpoints, occlusions and background complexity. Given these rich annotations we perform detailed analyses of the leading human parsing approaches, gaining insights into the success and failures of these methods. Furthermore, in contrast to the existing efforts on improving the feature discriminative capability, we solve human parsing by exploring a novel self-supervised structure-sensitive learning approach, which imposes human pose structures into parsing results without resorting to extra supervision (i.e., no need for specifically labeling human joints in model training). Our self-supervised learning framework can be injected into any advanced neural networks to help incorporate rich high-level knowledge regarding human joints from a global perspective and improve the parsing results. Extensive evaluations on our LIP and the public PASCAL-Person-Part dataset demonstrate the superiority of our method.

arXiv:1708.02190공식 코드 · Papers with Code

Intrinsically Motivated Goal Exploration Processes with Automatic Curriculum Learning

Intrinsically motivated spontaneous exploration is a key enabler of autonomous developmental learning in human children. It enables the discovery of skill repertoires through autotelic learning, i.e. the self-generation, self-selection, self-ordering and self-experimentation of learning goals. We present an algorithmic approach called Intrinsically Motivated Goal Exploration Processes (IMGEP) to enable similar properties of autonomous learning in machines. The IMGEP architecture relies on several principles: 1) self-generation of goals, generalized as parameterized fitness functions; 2) selection of goals based on intrinsic rewards; 3) exploration with incremental goal-parameterized policy search and exploitation with a batch learning algorithm; 4) systematic reuse of information acquired when targeting a goal for improving towards other goals. We present a particularly efficient form of IMGEP, called AMB, that uses a population-based policy and an object-centered spatio-temporal modularity. We provide several implementations of this architecture and demonstrate their ability to automatically generate a learning curriculum within several experimental setups. One of these experiments includes a real humanoid robot exploring multiple spaces of goals with several hundred continuous dimensions and with distractors. While no particular target goal is provided to these autotelic agents, this curriculum allows the discovery of diverse skills that act as stepping stones for learning more complex skills, e.g. nested tool use.

arXiv:1806.01260공식 코드 · Papers with Code

Digging Into Self-Supervised Monocular Depth Estimation

Per-pixel ground-truth depth data is challenging to acquire at scale. To overcome this limitation, self-supervised learning has emerged as a promising alternative for training models to perform monocular depth estimation. In this paper, we propose a set of improvements, which together result in both quantitatively and qualitatively improved depth maps compared to competing self-supervised methods. Research on self-supervised monocular training usually explores increasingly complex architectures, loss functions, and image formation models, all of which have recently helped to close the gap with fully-supervised methods. We show that a surprisingly simple model, and associated design choices, lead to superior predictions. In particular, we propose (i) a minimum reprojection loss, designed to robustly handle occlusions, (ii) a full-resolution multi-scale sampling method that reduces visual artifacts, and (iii) an auto-masking loss to ignore training pixels that violate camera motion assumptions. We demonstrate the effectiveness of each component in isolation, and show high quality, state-of-the-art results on the KITTI benchmark.

arXiv:1806.06811공식 코드 · Papers with Code

Temporal coherence-based self-supervised learning for laparoscopic workflow analysis

In order to provide the right type of assistance at the right time, computer-assisted surgery systems need context awareness. To achieve this, methods for surgical workflow analysis are crucial. Currently, convolutional neural networks provide the best performance for video-based workflow analysis tasks. For training such networks, large amounts of annotated data are necessary. However, collecting a sufficient amount of data is often costly, time-consuming, and not always feasible. In this paper, we address this problem by presenting and comparing different approaches for self-supervised pretraining of neural networks on unlabeled laparoscopic videos using temporal coherence. We evaluate our pretrained networks on Cholec80, a publicly available dataset for surgical phase segmentation, on which a maximum F1 score of 84.6 was reached. Furthermore, we were able to achieve an increase of the F1 score of up to 10 points when compared to a non-pretrained neural network.

arXiv:2502.20612v2cs.LG공식 코드 · Papers with Code

자기지도 대조 학습을 위한 전역 거짓 부정의 즉각적인 탐지

Discovering Global False Negatives On the Fly for Self-supervised Contrastive Learning

자기지도 대조 학습(self-supervised contrastive learning)에서 음성 쌍(negative pair)은 일반적으로 앵커(anchor) 이미지와 앵커를 제외한 전체 데이터셋에서 추출된 샘플을 사용해 구성됩니다. 그러나 이러한 방식은 유사한 의미를 지닌 음성 쌍, 즉 "거짓 부정(false negatives)"을 생성할 수 있으며, 이로 인해 이들의 임베딩이 부당하게 멀어지게 됩니다. 이러한 문제를 해결하기 위해, 본 논문에서는 학습(training) 중에 각 앵커 데이터의 거짓 부정을 식별하기 위한 임계값을 즉석에서(on the fly) 자동으로 학습하는 최적화 기반 접근 방식인 GloFND를 소개합니다. 기존의 거짓 부정 탐지 방법들과 달리, 우리의 접근 방식은 미니배치(mini-batch) 내에서 국소적으로 탐지하는 대신 전체 데이터셋에 걸쳐 전역적으로 거짓 부정을 탐지합니다. 또한, 반복(iteration)당 계산 비용은 데이터셋 크기와 무관하게 일정하게 유지됩니다. 이미지 및 이미지-텍스트 데이터에 대한 실험 결과는 제안된 방법의 효과를 입증합니다. 본 연구의 구현 코드는 https://github.com/vibalcam/GloFND 에서 확인할 수 있습니다.

arXiv:2502.02118v2cs.LG공식 코드 · Papers with Code

BRIDLE: 양자화를 활용한 일반화 자기지도 학습

BRIDLE: Generalized Self-supervised Learning with Quantization

자기지도 학습(self-supervised learning)은 다양한 도메인에서 레이블 없는 데이터로부터 의미 있는 표현을 학습하는 강력한 접근법으로, 대규모 레이블 데이터셋에 대한 의존을 줄여 왔다. 자연어 처리(NLP)에서 깊은 양방향 문맥을 포착한 BERT의 성공에 영감을 받아, BEATs와 같은 모델이 벡터 양자화(VQ)를 사용해 오디오 신호에 양방향 학습 패러다임을 확장하는 등 유사한 프레임워크가 오디오 등 다른 모달리티에도 적용되었다. 그러나 이러한 프레임워크는 양자화에 단일 코드book에 의존한다는 점 등의 과제에 직면하며, 이는 신호의 복잡하고 다면적인 성격을 충분히 포착하지 못할 수 있다. 또한 codebook 활용의 비효율은 code vector가 저활용되는 결과를 낳는다. 이러한 한계를 해결하기 위해, 우리는 양방향 학습 과정에 잔차 양자화(RQ)를 통합하고 오디오·이미지·비디오 사전학습(pre-training)에 일반화된 자기지도 인코더 사전학습 프레임워크인 BRIDLE(Bidirectional Residual Quantization Interleaved Discrete Learning Encoder)를 제안한다. 다층 계층적 codebook을 사용하는 RQ는 잠재 공간에서 세밀한 이산화를 가능하게 하여 표현 품질을 향상시킨다. BRIDLE는 인코더와 tokenizer 사이의 교차(interleaved) 학습 절차를 포함한다. 우리는 분류 벤치마크를 사용한 오디오 이해 과제에서 BRIDLE를 평가하여 최신(state-of-the-art) 결과를 달성했으며, 이미지 분류 및 비디오 분류 과제에서도 경쟁력 있는 성능을 보였고, 하류(downstream) 성능에서 기존 VQ 방식 대비 일관된 개선을 확인했다.

arXiv:2502.14424v4stat.ML공식 코드 · Papers with Code

생성 학습을 표현 학습으로 확장하기: 분포 매칭 관점의 자기지도 전이학습

Bringing Generative Learning to Representation Learning: Self-Supervised Transfer Learning as Distribution Matching

대부분의 자기지도학습(self-supervised learning) 목적 함수는 표현 붕괴(collapse)를 방지하지만, 목표 표현의 확률 법칙(representation law)은 명시하지 않은 채 남겨둡니다. 본 논문에서는 표현 학습을 분포 매칭(Distribution Matching, DM)으로 정식화하여, 인코더가 유도하는 분포가 명시적인 기하학적 기준 분포(reference law)와 일치하도록 증강 불변(augmentation-invariant) 인코더를 학습합니다. 기준 분포는 학습된 표현 분포가 어떤 형태를 띠어야 하는지 명시하며, 별도로 선택된 불일치도(discrepancy) 척도는 이 목표로부터의 편차를 어떻게 측정할지 결정합니다. 본 연구에서는 말로우스 거리(Mallows distance)를 사용합니다. DM 프레임워크는 방향의 역관계를 드러냅니다. 즉, 생성 학습이 다루기 쉬운 기준 분포를 데이터로 매핑하는 반면, 표현 학습은 데이터를 설계된 기준 분포로 매핑합니다. 본 연구는 모집단 목적 함수를 클래스 중심 분리 및 분류(classification) 오차와 연결하고, 비점근적 신경 시브(neural-sieve) 보장을 증명합니다. 시뮬레이션 및 이미지 벤치마크 결과는 다양체 정류(manifold rectification), 미세 구조 보존, 그리고 서로 다른 레이블 공간 간의 전이(transfer) 성능을 보여줍니다.