Papers with Code 아카이브

3D Reconstruction

태스크의 하위 분류와 논문·코드·평가·기법·데이터셋을 탐색하세요.

하위 태스크
7
평가표
10
보존된 평가 기록
44

3D Reconstruction is the task of creating a 3D model or representation of an object or scene from 2D images or other data sources. The goal of 3D reconstruction is to create a virtual representation of an object or scene that can be used for a variety of purposes, such as visualization, animation, simulation, and analysis. It can be used in fields such as computer vision, robotics, and virtual reality.

Image: Gwak et al

데이터셋

출처에 같은 태스크 이름으로 연결된 데이터셋입니다. 동명 태스크가 포함될 수 있으므로, 비교할 때는 평가 결과의 조건도 확인하세요.

DeePhy

DeePhy is a novel DeepFake Phylogeny dataset consisting of 5040 DeepFake videos generated using three different generation techniques. It is one of the first datasets which incorporates the concept of Deepfake Phylogeny which refers to the idea of generation of DeepFakes using multiple generation techniques in a sequential manner. The dataset can be used for the tasks of (i) DeepFake Detection ,(ii) Model Attribution of DeepFakes and (iii) Prediction of the sequential order of DeepFake techniques employed to create phylogenetic deepfakes. It will facilitate advancements in real-life scenarios of plagiarism detection, forgery detection, and reverse engineering of deepfakes.

DRACO20K

DRACO20K dataset is used for evaluating object canonicalization on methods that estimate a canonical frame from a monocular input image. Provides: 1. Mixed Reality Multi-view RGB-D images rendered from ShapeNet objects 2. Camera poses 3. NOCS maps 4. Semantic 2D keypoints with visibility 5. Object-centric mask

Drunkard's Dataset

Estimating camera motion in deformable scenes poses a complex and open research challenge. Most existing non-rigid structure from motion techniques assume to observe also static scene parts besides deforming scene parts in order to establish an anchoring reference. However, this assumption does not hold true in certain relevant application cases such as endoscopies. To tackle this issue with a common benchmark, we introduce the Drunkard’s Dataset, a challenging collection of synthetic data targeting visual navigation and reconstruction in deformable environments. This dataset is the first large set of exploratory camera trajectories with ground truth inside 3D scenes where every surface exhibits non-rigid deformations over time. Simulations in realistic 3D buildings lets us obtain a vast amount of data and ground truth labels, including camera poses, RGB images and depth, optical flow and normal maps at high resolution and quality.

ENRICH

A new synthetic, multi-purpose dataset - called ENRICH - for testing photogrammetric and computer vision algorithms. Compared to existing datasets, ENRICH offers higher resolution images also rendered with different lighting conditions, camera orientation, scales, and field of view. Specifically, ENRICH is composed of three sub-datasets: ENRICH-Aerial, ENRICH-Square, and ENRICH-Statue, each exhibiting different characteristics. The proposed dataset is useful for several photogrammetry and computer vision-related tasks, such as the evaluation of hand-crafted and deep learning-based local features, effects of ground control points (GCPs) configuration on the 3D accuracy, and monocular depth estimation. Each zip file in the root is relative to a specific dataset: - ENRICH-Aerial, is an aerial image block of the city of Launceston, Australia. The acquisition is performed by simulating a typical oblique aerial camera with five views (nadir and four oblique views). - ENRICH-Square, is a ground-level dataset of a square captured by four cameras, each one moving on a different path with different focal length, orientation, and lighting conditions. - ENRICH-Statue, is a ground-level dataset portraying a statue (placed in the center of the ENRICH-Square scene), acquired using four cameras moving on different paths with different focal lengths, orientations, and lighting conditions. Be sure to check the README file in the dataset root for information on folder structure and file contents. Please refer to the related paper (https://doi.org/10.1016/j.isprsjprs.2023.03.002) for information about the generation method and the purpose of ENRICH.

GOTCHA

We release the dataset for non-commercial research. Submit requests <a href="https://forms.gle/6WPEGNWbYoEe6bte8" target="_blank">here</a>. The dataset consists of three parts: Original Samples, Deepfakes, and the Fidelity Score Dataset. The dataset is accessible via Box.com, which offers high availability worldwide and allows for access tracking. ### Dataset Statistics: Number of folder (or videos): - Original : 675 (673k frames with 3 frame sizes each) - Fake (DFL) : 26,085 (5.0M frames) - Fake (Adaptive - DFL) : 7,215 (1.3M frames) - Fake (FSGAN) : 25,433 (4.6M frames) - Fake (LIA) : 25,662 (5.0M frames) DFL quantized ready-to-use models : 60 Fidelity Score Dataset: - Fake : 49,603 videos (32 frames each, 1.35M frames total) - Real : 816 videos (32 frames each, 22k frames total) Human evaluation dataset: 200 videos ### Original Samples While the paper focuses on 8 challenges, we collected 13 in total, which we share in full. The data is provided as frames extracted from raw HD videos recorded at 60 fps. We downsample the frames to approximately 10 fps for ease of use without any expected loss in generation fidelity. The images inside each zip file are face-only. We provide three convenient sizes: 224 x 224, 512 x 512, and 1024 x 1024 pixels. Each zip file contains folders numbered according to the following challenges/tasks: 0. The camera moves around for about two minutes, capturing the face from different angles while the participant sits still. 1. The participant rotates their head from side to side, then looks up and down. Each side should be held for about 5 seconds, rotating as comfortably as possible, totaling around 25 seconds. 2. The participant covers their eyes with a hand, followed by covering the left half, the right half, and finally, the lower half of the face. 3. The participant puts on the provided sunglasses and then takes them off. 4. The participant wears the provided clear glasses, ensuring they reflect a lamp light shining. After setting up the reflection, recording starts. Finally, the participant removes the glasses. 5. The participant moves a green cloth in front of their face 6. The participant puts on a face mask and counts from 1 to 10 out loud. Then, they remove the facemask. 7. The participant presses a finger against a cheek. 8. The participant sticks out a (small) portion of their tongue. 9. The participant laughs for 10 seconds, then frowns, as if angry, for another 10 seconds. 10. The participant slowly stands up and then sits back down. 11. The participant counts from 1 to 50 with distinct breaks between each utterance. 12. The room light is dimmed, and a flashing light is kept on the face for 10 seconds. TODO: Verify each challenge index above corresponds to that challenge in the zips. Note: A few challenges are missing from a handful of participants. ### Fakes Using 47 participants, we create deepfakes between all 2,209 (47 x 47) combinations (including self deepfake), using three deepfake generators — DeepFaceLab, FSGANv2, and LIA. 1. DFL (DeepFaceLab): Notorious for generating hyper-realistic deepfakes, this pipeline serves as a baseline for in-the-wild deepfake videos. For our study, we trained individual DFL deepfake generators for each participant using their `no challenge' videos. These videos records the participants in a range of frontal angles while they sit naturally, aiming to mimic the kind of data readily accessible online for non-celebrity individuals. Training continued until convergence for approximately around 300,000 iterations. DFL (Adaptive Adversary): We utilized heavily trained celebrity deepfake generators provided by the community as an adaptive adversary. Such models are trained on upwards of 2M iterations. The DFL models trained by us and the ones provided by the community are available under dfm_models folder, including the hyperparameters in states.dat file. The models can be loaded directly into Deepfacelive to generate deepfakes. 2. FSGAN (Face Swapping Generative Adversarial Network): This corresponds to the second version of FSGAN. Similar to LIA, this model is also target-agnostic, hence we utilized a pre-trained model made available by the authors for our study. Access its release at https://github.com/YuvalNirkin/fsgan 3. LIA (Latent Image Animator): This pipeline is a facial reenactment method outlined in. Given its target-agnostic nature—meaning it does not require specific target data during inference—we employed a pre-trained model for our experiments. Access its release at https://github.com/wyhsirius/LIA As a rule of thumb, An imposter outer face and target is the inner face, in case of faceswaps. While Box offers quick zipping of multiple files, we have placed the files into zips (<10 GB each), for convenient access. Each zip under fakes (except Adaptive Adversary) is organized in path : “target/challenge/imposter/.jpg”. For Adaptive Adversary, the order is opposite, i.e., “imposter/challenge/target/.jpg” ### Fidelity Score Dataset (or a Snapshot of the full dataset) The training subset contains the following subselections: - Original and deepfake videos with four challenges, one from each category and ‘no challenge’. - 32 frames per sample, - 35 out of 47 target identities, and - Deepfake videos created using only DFL. The validation dataset consists of the remaining 12 out of 47 target identities, while test one includes the celebrity deepfakes. The dataset is the smallest snapshot of the full dataset. The code for loading the dataset and training a fidelity score function can be accessed at https://github.com/mittalgovind/gotcha-deepfakes. ### Human evaluation We release the instruments we used for human evaluation at https://app.gorilla.sc/openmaterials/693684 (with active preview).

MBW - Zoo Dataset

Dataset page: https://github.com/mosamdabhi/MBW-Data MBW - Zoo is a challenging dataset consisting image frames of tail-end distribution categories (such as Fish, Colobus Monkeys, Chimpanzees, etc.) with their corresponding 2D, 3D, and Bounding-Box labels generated from minimal human intervention. Some of the prominent use cases of this dataset include not only sparse 2D and 3D landmark prediction. The data was collected by two smartphone cameras without any constraints: meaning no guidance or instructions were given as to how the data should be collected. The intention was to mimic the data captured casually by anyone holding a smartphone grade camera. Due to this reason, the cameras were continuously moving in space changing their extrinsics with respect to each other, capturing an in-the-wild dynamic scene. This dataset could be used to benchmark robust algorithms in various computer vision tasks.

Pano3D

Pano3D is a new benchmark for depth estimation from spherical panoramas. Its goal is to drive progress for this task in a consistent and holistic manner. The Pano3D 360 depth estimation benchmark provides a standard Matterport3D train and test split, as well as a secondary GibsonV2 partioning for testing and training as well. The latter is used for zero-shot cross dataset transfer performance assessment and decomposes it into 3 different splits, each one focusing on a specific generalization axis.

Parcel3D

Synthetic dataset of over 13,000 images of damaged and intact parcels with full 2D and 3D annotations in the COCO format. For details see our paper and for visual samples our project page.

AMT Objects

AMT Objects is a large dataset of object centric videos suitable for training and benchmarking models for generating 3D models of objects from a small number of photos of the objects. The dataset consists of multiple views of a large collection of object instances. The dataset contains 7 object categories from the MS COCO classes: apple, sandwich, orange, donut, banana, carrot and hydrant. For each class, annotators were asked to collect a video by looking ‘around’ a class instance, resulting in a turntable video. The dataset contains 169-457 videos per class. For each class, the videos were randomly split into training and testing videos in an 8:1 ratio.

Danish Airs and Grounds

Danish Airs and Grounds (DAG) is a large collection of street-level and aerial images targeting such cases. Its main challenge lies in the extreme viewing-angle difference between query and reference images with consequent changes in illumination and perspective. The dataset is larger and more diverse than current publicly available data, including more than 50 km of road in urban, suburban and rural areas. All images are associated with accurate 6-DoF metadata that allows the benchmarking of visual localization methods.

DEEP-VOICE: DeepFake Voice Recognition

DEEP-VOICE: Real-time Detection of AI-Generated Speech for DeepFake Voice Conversion This dataset contains examples of real human speech, and DeepFake versions of those speeches by using Retrieval-based Voice Conversion. Can machine learning be used to detect when speech is AI-generated? ## Introduction There are growing implications surrounding generative AI in the speech domain that enable voice cloning and real-time voice conversion from one individual to another. This technology poses a significant ethical threat and could lead to breaches of privacy and misrepresentation, thus there is an urgent need for real-time detection of AI-generated speech for DeepFake Voice Conversion. To address the above emerging issues, we are introducing the DEEP-VOICE dataset. DEEP-VOICE is comprised of real human speech from eight well-known figures and their speech converted to one another using Retrieval-based Voice Conversion. For each speech, the accompaniment ("background noise") was removed before conversion using RVC. The original accompaniment is then added back to the DeepFake speech: Overview of the Retrieval-based Voice Conversion process to generate DeepFake speech with Ryan Gosling's speech converted to Margot Robbie. Conversion is run on the extracted vocals before being layered on the original background ambience. (Above: Overview of the Retrieval-based Voice Conversion process to generate DeepFake speech with Ryan Gosling's speech converted to Margot Robbie. Conversion is run on the extracted vocals before being layered on the original background ambience.) ## Dataset There are two forms to the dataset that are made available. First, the raw audio can be found in the "AUDIO" directory. They are arranged within "REAL" and "FAKE" class directories. The audio filenames note which speakers provided the real speech, and which voices they were converted to. For example "Obama-to-Biden" denotes that Barack Obama's speech has been converted to Joe Biden's voice. Second, the extracted features can be found in the "DATASET-balanced.csv" file. This is the data that was used in the below study. The dataset has each feature extracted from one-second windows of audio and are balanced through random sampling. **Note: ** All experimental data is found within the "KAGGLE" directory. The "DEMONSTRATION" directory is used for playing cropped and compressed demos in notebooks due to Kaggle's limitations on file size. A potential use of a successful system could be used for the following: Usage of the real-time system. The end user is notified when the machine learning model has processed the speech audio (e.g. a phone or conference call) and predicted that audio chunks contain AI-generated speech. (Above: Usage of the real-time system. The end user is notified when the machine learning model has processed the speech audio (e.g. a phone or conference call) and predicted that audio chunks contain AI-generated speech.) ## Kaggle The dataset is available on the Kaggle data science platform. The Kaggle page can be found by clicking here: Dataset on Kaggle ## Attribution This dataset was produced from the study "Real-time Detection of AI-Generated Speech for DeepFake Voice Conversion" The preprint can be found on ArXiv by clicking here: Real-time Detection of AI-Generated Speech for DeepFake Voice Conversion ## License This dataset is provided under the MIT License: Permission is hereby granted, free of charge, to any person obtaining a copy of this software and associated documentation files (the "Software"), to deal in the Software without restriction, including without limitation the rights to use, copy, modify, merge, publish, distribute, sublicense, and/or sell copies of the Software, and to permit persons to whom the Software is furnished to do so, subject to the following conditions: The above copyright notice and this permission notice shall be included in all copies or substantial portions of the Software. THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM, OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE SOFTWARE.

GANGen-Detection

This dataset was created to test whether it's possible to build a general-purpose detector that can tell real images apart from fake ones generated by convolutional neural networks (CNNs), no matter which model or dataset was used to create the fake images. To do this, the authors collected fake images generated by 11 different CNN-based image generation models. These models represent a wide range of current image synthesis techniques and include: ProGAN StyleGAN BigGAN CycleGAN StarGAN GauGAN DeepFakes Cascaded Refinement Networks (CRN) Implicit Maximum Likelihood Estimation (IMLE) Second-order Attention Super-Resolution (SOAT-SR) Seeing-in-the-Dark (SID) The dataset includes fake images from each of these models and a set of real images, allowing for binary classification (real vs. fake). The study found that a standard image classifier (like a convolutional neural network) trained on fake images from just one generator (ProGAN) was able to detect fake images from other, completely different generators with surprising accuracy. This suggests that many CNN-generated images, even from different architectures, share common flaws that can be learned and detected. The dataset is useful for research in detecting synthetic media, improving image forensics, and understanding the weaknesses in current generative models. Code and pre-trained models were made available by the authors (https://github.com/chuangchuangtan/GANGen-Detection).

Individualized Deepfake Detection Dataset

The Deepfake face detection task involves a facial image of unknown authenticity for testing. While most deepfake detection methods take only the image as input, our literature demonstrates that conditioning the deepfake detector on identity—i.e., knowing whose deepfake face the picture might be—can enhance detection performance. Existing deepfake detection datasets, such as FaceForensics++ and DFDC, do not include identity information for authentic and deepfake faces. This dataset contains facial images of 45 specific individuals, divided into train and test sets, including a total of 23k authentic and 22k deepfake images. Having a specific individual's images in both the train and test sets allows us to assess detection performance for that individual. The dataset is curated so that the train and test sets are from two independent sources. The train images are curated from the CelebDFv2 dataset, and the test images are curated from the CACD dataset. Deepfake faces are generated using FaceswapGAN, utilizing a portion of the training images to train the reconstruction model. The test deepfake images are faceswapped with another identity not included in our celebrity list. On the other hand, the training deepfake images are reconstructed images of that person. The deepfake detection method proposed in our paper requires reconstructing both the training and test images. The reconstructed test and train images are also available in this dataset. It is worth mentioning that reconstructing the training deepfake images produces doubly reconstructed images.

Indoor and outdoor DFD dataset

The dfd_indoor dataset contains 110 images for training and 29 images for testing. The dfd_outdoor dataset contains 34 images for tests; no ground truth was given for this dataset, as the depth sensor only works on indoor scenes. Source: DFD dataset

Replay

Replay is a collection of multi-view, multi-modal videos of humans interacting socially. Each scene is filmed in high production quality, from different viewpoints with several static cameras, as well as wearable action cameras, and recorded with a large array of microphones at different positions in the room. The full Replay dataset consists of 68 scenes of social interactions between people, such as playing boarding games, exercising, or unwrapping presents. Each scene is about 5 minutes long and filmed with 12 cameras, static and dynamic. Audio is captured separately by 12 binaural microphones and additional near-range microphones for each actor and for each egocentric video. All sensors are temporally synchronized, undistorted, geometrically calibrated, and color calibrated.

Robust e-NeRF Synthetic Event Dataset

This synthetic event dataset is used in Robust e-NeRF to study the collective effect of camera speed profile, contrast threshold variation and refractory period on the quality of NeRF reconstruction from a moving event camera. It is simulated using an improved version of ESIM with three different camera configurations of increasing difficulty levels (i.e. easy, medium and hard) on seven Realistic Synthetic $360^{\circ}$ scenes (adopted in the synthetic experiments of NeRF), resulting in a total of 21 sequence recordings. Please refer to the Robust e-NeRF paper for more details. The dataset allows for a retrospective comparison between event-based and image-based NeRF reconstruction methods, as the event sequences were simulated under highly similar conditions as the NeRF synthetic dataset. In particular, we adopt the same camera intrinsics and camera distance to the object at the origin. Furthermore, the event camera travels in a hemi-/spherical spiral motion about the object, thereby having a similar camera pose distribution for training. Apart from that, we also use the same test camera poses/views. Nonetheless, this new synthetic event dataset is not only specific to NeRF reconstruction, but also suitable for novel view synthesis, 3D reconstruction, localization and SLAM in general.

ShiftySpeech

ShiftySpeech: A Large-Scale Synthetic Speech Dataset with Distribution Shifts ## 🔥 Key Features - 3000+ hours of synthetic speech - Diverse Distribution Shifts: The dataset spans 7 key distribution shifts, including: - 📖 Reading Style - 🎙️ Podcast - 🎥 YouTube - 🗣️ Languages (Three different languages) - 🌎 Demographics (including variations in age, accent, and gender) - Multiple Speech Generation Systems: Includes data synthesized from various TTS models and vocoders. Dataset can be downloaded from: Hugging Face

Stained mice brain blood vessels. Confocal-LFM

3D confocal stacks with corresponding 2D Light-field microscope images Confocal: -Single volume dimension: 1287x1287x64. -Number of samples: 362 -Voxel size: 0.086x0.086x0.9 um. -Objective: 40x/1.3 Oil. -Stain: tomato lectin (DyLight594 conjugated, DL-1177, Vector Laboratories). LightField: -Image dimensions 1287x1287. -PixelSize: 3.45 um. -Pixels per lenslet: 33x33. -Lenslet Pitch: 112 um. -MLA2Sensor distance: 2500um. -Tube-lens focal length: 165mm. -Objective: 40x/0.9 Air. H5 containing: -volData: confocal volumes 1287x1287x64 voxels. -LFData: LF 4D tensor 33x33x39x39 (Angular coord x, angular coord y, spatial coord x, spatial coord y). -gridCoords: image grid positions.

Synthetic Human Model Dataset

A synthetic dataset for evaluating non-rigid 3D human reconstruction based on conventional RGB-D cameras. The dataset consist of seven motion sequences of a single human model. Source: Synthetic Human Model Dataset for Skeleton Driven Non-rigid Motion Tracking and 3D Reconstruction

ThermoScenes

Dataset of paired thermal and RGB images comprising ten diverse scenes—six indoor and four outdoor scenes— for 3D scene reconstruction and novel view synthesis (e.g. with NeRF).

TwinSynths

The TwinSynths dataset is a novel benchmark designed to overcome common limitations found in earlier synthetic image datasets, such as low image quality, inadequate content preservation, and limited class diversity. TwinSynths generates pairs of images where each synthetic image is visually identical to its real counterpart, ensuring that the essential content remains intact while showcasing the unique architectural features of the generative models used. TwinSynths comprises two subsets: TwinSynths-GAN This subset uses a GAN generator architecture which trained from scratch on individual real images using a mean-squared error loss to ensure pixel-level fidelity. By fixing the latent vector input, the method produces synthetic images that closely mirror the original content. The GAN subset consists of 8,000 generated images spanning 80 classes selected from ImageNet. TwinSynths-DM For the diffusion model-based subset, DDIM inversion is used to maintain the content integrity of the original images. By applying a noise-adding forward process followed by a text-conditioned denoising procedure (using class name prompts), this generates synthetic images that are highly similar to their real counterparts. The same set of ImageNet classes is used as in the GAN subset, allowing for a consistent evaluation across different generative methods.

CVGL

CVGL Camera Calibration Dataset consists of 49 camera configurations with town 1 having 25 configurations while town 2 having 24 configurations. The parameters modified for generating the configurations include fov, x, y, z, pitch, yaw, and roll. Here, fov is the field of view, (x, y, z) is the translation while (pitch, yaw, and roll) is the rotation between the cameras. The total number of image pairs is 79, 320, out of which 18, 083 belong to Town 1 while 61, 237 belong to Town 2, the difference in the number of images is due to the length of the tracks. Source: Multi-task Learning for Camera Calibration Image Source: https://arxiv.org/pdf/2211.12432v1.pdf

DeepSpeak Dataset v1.0

The DeepSpeak dataset contains over 43 hours of real and deepfake footage of people talking and gesturing in front of their webcams. The source data was collected from a diverse set of participants in their natural environments and the deepfakes were generated using state-of-the-art open-source lip-sync and face-swap software.

PINet

We propose a new light field image database called “PINet” inheriting the hierarchical structure from WordNet. It consists of 7549 LIs captured by Lytro Illum, which is much larger than the existing databases. The images are manually annotated to 178 categories according to WordNet, such as cat, camel, bottle, fans, etc. The registered depth maps are also provided. Each image is generated by processing the raw LI from the camera by Light Field Toolbox v0.4 for demosaicing and devignetting. PINet is the largest light field image dataset. More details can be found in https://github.com/VincentChandelier/SADN

출처: Papers with Code · CC BY-SA 4.0. 아카이브 자료와 새로 수집한 논문은 별도로 관리합니다.