Novel View Synthesis

Neural Radiance Fields (NeRF) is a method for synthesizing novel views of complex scenes by optimizing an underlying continuous volumetric scene function using a sparse set of input views. The dataset contains three parts with the first 2 being synthetic renderings of objects called Diffuse Synthetic 360◦ and Realistic Synthetic 360◦ while the third is real images of complex scenes. Diffuse Synthetic 360◦ consists of four Lambertian objects with simple geometry. Each object is rendered at 512x512 pixels from viewpoints sampled on the upper hemisphere. Realistic Synthetic 360◦ consists of eight objects of complicated geometry and realistic non-Lambertian materials. Six of them are rendered from viewpoints sampled on the upper hemisphere and the two left are from viewpoints sampled on a full sphere with all of them at 800x800 pixels. The real images of complex scenes consist of 8 forward-facing scenes captured with a cellphone at a size of 1008x756 pixels.

KITTI (Karlsruhe Institute of Technology and Toyota Technological Institute) is one of the most popular datasets for use in mobile robotics and autonomous driving. It consists of hours of traffic scenarios recorded with a variety of sensor modalities, including high-resolution RGB, grayscale stereo cameras, and a 3D laser scanner. Despite its popularity, the dataset itself does not contain ground truth for semantic segmentation. However, various researchers have manually annotated parts of the dataset to fit their necessities. Álvarez et al. generated ground truth for 323 images from the road detection challenge with three classes: road, vertical, and sky. Zhang et al. annotated 252 (140 for training and 112 for testing) acquisitions – RGB and Velodyne scans – from the tracking challenge for ten object categories: building, sky, road, vegetation, sidewalk, car, pedestrian, cyclist, sign/pole, and fence. Ros et al. labeled 170 training images and 46 testing images (from the visual odometry challenge) with 11 classes: building, tree, sky, car, sign, road, pedestrian, fence, pole, sidewalk, and bicyclist. Source: A Review on Deep Learning Techniques Applied to Semantic Segmentation Image Source: http://www.cvlibs.net/datasets/kitti/eval_object.php?obj_benchmark=3d

ShapeNet is a large scale repository for 3D CAD models developed by researchers from Stanford University, Princeton University and the Toyota Technological Institute at Chicago, USA. The repository contains over 300M models with 220,000 classified into 3,135 classes arranged using WordNet hypernym-hyponym relationships. ShapeNet Parts subset contains 31,693 meshes categorised into 16 common object classes (i.e. table, chair, plane etc.). Each shapes ground truth contains 2-5 parts (with a total of 50 part classes). Source: A review on deep learning techniques for 3D sensed data classification Image Source: ShapeNet: An Information-Rich 3D Model Repository

The SYNTHIA dataset is a synthetic dataset that consists of 9400 multi-viewpoint photo-realistic frames rendered from a virtual city and comes with pixel-level semantic annotations for 13 classes. Each frame has resolution of 1280 × 960. Source: Orientation-aware Semantic Segmentation on Icosahedron Spheres Image Source: https://synthia-dataset.net/

Mip-NeRF 360 is an extension to the Mip-NeRF that uses a non-linear parameterization, online distillation, and a novel distortion-based regularize to overcome the challenge of unbounded scenes. The dataset consists of 9 scenes with 5 outdoors and 4 indoors, each containing a complex central object or area with a detailed background.

Local Light Field Fusion (LLFF) is a practical and robust deep learning solution for capturing and rendering novel views of complex real-world scenes for virtual exploration. The dataset consists of both renderings and real images of natural scenes. The synthetic images are rendered from the SUNCG and UnrealCV where SUNCG contains 45000 simplistic house and room environments with texture-mapped surfaces and low geometric complexity. UnrealCV contains a few large-scale environments modeled and rendered with extreme detail. The real images are 24 scenes captured from a handheld cellphone.

KITTI-360 is a large-scale dataset that contains rich sensory information and full annotations. It is the successor of the popular KITTI dataset, providing more comprehensive semantic/instance labels in 2D and 3D, richer 360 degree sensory information (fisheye images and pushbroom laser scans), very accurate and geo-localized vehicle and camera poses, and a series of new challenging benchmarks.

BlendedMVS is a novel large-scale dataset, to provide sufficient training ground truth for learning-based MVS. The dataset was created by applying a 3D reconstruction pipeline to recover high-quality textured meshes from images of well-selected scenes. Then, these mesh models were rendered to color images and depth maps. Source: BlendedMVS: A Large-scale Dataset for Generalized Multi-view Stereo Networks

OmniObject3D is a large vocabulary 3D object dataset with massive high-quality real-scanned 3D objects. OmniObject3D has several appealing properties: 1) Large Vocabulary: It comprises 6,000 scanned objects in 190 daily categories, sharing common classes with popular 2D datasets (e.g., ImageNet and LVIS), benefiting the pursuit of generalizable 3D representations. 2) Rich Annotations: Each 3D object is captured with both 2D and 3D sensors, providing textured meshes, point clouds, multiview rendered images, and multiple real-captured videos. 3) Realistic Scans: The professional scanners support highquality object scans with precise shapes and realistic appearances.

We present a benchmark for image-based 3D reconstruction. The benchmark sequences were acquired outside the lab, in realistic conditions. Ground-truth data was captured using an industrial laser scanner. The benchmark includes both outdoor scenes and indoor environments. High-resolution video sequences are provided as input, supporting the development of novel pipelines that take advantage of video input to increase reconstruction fidelity. We report the performance of many image-based 3D reconstruction pipelines on the new benchmark. The results point to exciting challenges and opportunities for future work. Paper: Arno Knapitsch, Jaesik Park, Qian-Yi Zhou, and Vladlen Koltun. Tanks and temples: Benchmarking large-scale scene reconstruction. In ACM Transactions on Graphics (TOG), 2017.

ACID consists of thousands of aerial drone videos of different coastline and nature scenes on YouTube. Structure-from-motion is used to get camera poses. Image source: https://arxiv.org/pdf/2012.09855v2.pdf

This dataset focus on two blur types: camera motion blur and defocus blur. For each type of blur we synthesize $5$ scenes using Blender. We manually place multi-view cameras to mimic real data capture. To render images with camera motion blur, we randomly perturb the camera pose, and then linearly interpolate poses between the original and perturbed poses for each view. We render images from interpolated poses and blend them in linear RGB space to generate the final blurry images. For defocus blur, we use the built-in functionality to render depth-of-field images. We fix the aperture and randomly choose a focus plane between the nearest and furthest depth. We also captured $20$ real world scenes with $10$ scenes for each blur type for a qualitative study. The camera used was a Canon EOS RP with manual exposure mode. We captured the camera motion blur images by manually shaking the camera during exposure, while the reference images are taken using a tripod. To capture defocus images, we choose a large aperture. We compute the camera poses of blurry and reference images in the real world scenes using the COLMAP.

We introduce our new dataset, Spaces, to provide a more challenging shared dataset for future view synthesis research. Spaces consists of 100 indoor and outdoor scenes, captured using a 16-camera rig. For each scene, we captured image sets at 5-10 slightly different rig positions (within ∼10cm of each other). This jittering of the rig position provides a flexible dataset for view synthesis, as we can mix views from different rig positions for the same scene during training. We calibrated the intrinsics and the relative pose of the rig cameras using a standard structure from motion approach, using the nominal rig layout as a prior. We corrected exposure differences . For our main experiments we undistort the images and downsample them to a resolution of 800 × 480. We use 90 scenes from the dataset for training and hold out 10 for evaluation.

ScanNet++ is a large scale dataset with 450+ 3D indoor scenes containing sub-millimeter resolution laser scans, registered 33-megapixel DSLR images, and commodity RGB-D streams from iPhone. The 3D reconstructions are annotated with long-tail and label-ambiguous semantics to benchmark semantic understanding methods, while the coupled DSLR and iPhone captures enable benchmarking of novel view synthesis methods in high-quality and commodity settings.

The PhotoShape dataset consists of photorealistic, relightable, 3D shapes produced by the work proposed in the work of Park et al. (2021).

The shiny folder contains 8 scenes with challenging view-dependent effects used in our paper. We also provide additional scenes in the shiny_extended folder. The test images for each scene used in our paper consist of one of every eight images in alphabetical order. Each scene contains the following directory structure: scene/ dense/ cameras.bin images.bin points3D.bin project.ini images/ image_name1.png image_name2.png ... image_nameN.png images_distort/ image_name1.png image_name2.png ... image_nameN.png sparse/ cameras.bin images.bin points3D.bin project.ini database.db hwf_cxcy.npy planes.txt poses_bounds.npy - dense/ folder contains COLMAP's output [1] after the input images are undistorted. - images/ folder contains undistorted images. (We use these images in our experiments.) - images_distort/ folder contains raw images taken from a smartphone. - sparse/ folder contains COLMAP's sparse reconstruction output [1]. Our poses_bounds.npy is similar to the LLFF[2] file format with a slight modification. This file stores a Nx14 numpy array, where N is the number of cameras. Each row in this array is split into two parts of sizes 12 and 2. The first part, when reshaped into 3x4, represents the camera extrinsic (camera-to-world transformation), and the second part with two dimensions stores the distances from that point of view to the first and last planes (near, far). These distances are computed automatically based on the scene’s statistics using LLFF’s code. (For details on how these are computed, see this code) hwf_cxcy.npy stores the camera intrinsic (height, width, focal length, principal point x, principal point y) in a 1x5 numpy array. planes.txt stores information about the MPI planes. The first two numbers are the distances from a reference camera to the first and last planes (near, far). The third number tells whether the planes are placed equidistantly in the depth space (0) or inverse depth space (1). The last number is the padding size in pixels on all four sides of each of the MPI planes. I.e., the total dimension of each plane is (H + 2 * padding, W + 2 * padding). References: - [1]: COLMAP structure from motion (Schönberger and Frahm, 2016). - [2]: Local Light Field Fusion: Practical View Synthesis with Prescriptive Sampling Guidelines (Mildenhall et al., 2019).

We introduce Stanford-ORB, a new real-world 3D Object inverse Rendering Benchmark. Recent advances in inverse rendering have enabled a wide range of real-world applications in 3D content generation, moving rapidly from research and commercial use cases to consumer devices. While the results continue to improve, there is no real-world benchmark that can quantitatively assess and compare the performance of various inverse rendering methods. Existing real-world datasets typically only consist of the shape and multi-view images of objects, which are not sufficient for evaluating the quality of material recovery and object relighting. Methods capable of recovering material and lighting often resort to synthetic data for quantitative evaluation, which on the other hand does not guarantee generalization to complex real-world environments. We introduce a new dataset of real-world objects captured under a variety of natural scenes with ground-truth 3D scans, multi-view images, and environment lighting. Using this dataset, we establish the first comprehensive real-world evaluation benchmark for object inverse rendering tasks from in-the-wild scenes, and compare the performance of various existing methods.

RTMV is a large-scale synthetic dataset for novel view synthesis consisting of ∼300k images rendered from nearly 2000 complex scenes using high-quality ray tracing at high resolution (1600 × 1600 pixels). The dataset is orders of magnitude larger than existing synthetic datasets for novel view synthesis, thus providing a large unified benchmark for both training and evaluation. Using 4 distinct sources of high-quality 3D meshes, the scenes of our dataset exhibit challenging variations in camera views, lighting, shape, materials, and textures. The dataset consists of scenes from four different environments, namely Google Scanned Objects, ABC, Bricks and Amazon Berkeley. Each scene has 150 renders at a 1600 x 1600 resolution. Description adapted from: http://www.cs.umd.edu/~mmeshry/projects/rtmv/

RealEstate10K is a large dataset of camera poses corresponding to 10 million frames derived from about 80,000 video clips, gathered from about 10,000 YouTube videos. For each clip, the poses form a trajectory where each pose specifies the camera position and orientation along the trajectory. These poses are derived by running SLAM and bundle adjustment algorithms on a large set of videos.

This is a gun detection dataset with 51K annotated gun images for gun detection and other 51K cropped gun chip images for gun classification collected from a few different sources.

X3D is a dataset containing 15 scenes and covering 4 applications for X-ray 3D reconstruction. More specifically, the X3D dataset includes the scenes of (1) medicine: jaw, leg, chest, foot, abdomen, aneurism, pelvis, pancreas, head (2) biology: carp, bonsai (3) security: box, backpack (4) industry: engine, teapot Two main tasks can be evaluated, i.e., (1) Novel View Synthesis (2) CT Reconstruction

OMMO is a new benchmark for several outdoor NeRF-based tasks, such as novel view synthesis, surface reconstruction, and multi-modal NeRF. It contains complex objects and scenes with calibrated images, point clouds and prompt annotations. Source: A Large-Scale Outdoor Multi-modal Dataset and Benchmark for Novel View Synthesis and Implicit Scene Reconstruction Image Source: https://arxiv.org/pdf/2301.06782v1.pdf

This is dataset for high dynamic range novel view synthesis. It is collected by HDR-NeRF and recalibrated by HDR-GS for the research of 3DGS-based algorithms. This dataset contains 8 synthetic scenes and 4 real scenes.

We present a large-scale dataset for 3D urban scene understanding. Compared to existing datasets, our dataset consists of 75 outdoor urban scenes with diverse backgrounds, encompassing over 15,000 images. These scenes offer 360◦ hemispherical views, capturing diverse foreground objects illuminated under various lighting conditions. Additionally, our dataset encompasses scenes that are not limited to forward-driving views, addressing the limitations of previous datasets such as limited overlap and coverage between camera views. The closest pre-existing dataset for generalizable evaluation is DTU [2] (80 scenes) which comprises mostly indoor objects and does not provide multiple foreground objects or background scenes. We use the Parallel Domain synthetic data generation to render high-fidelity 360◦ scenes. We select 3 different maps i.e. SF 6thAndMission, SF GrantAndCalifornia and SF VanNessAveAndTurkSt and sample 75 different scenes across all 3 maps as our backgrounds (All 75 scenes across 3 maps are significantly different road scenes from each other, captured at different viewpoints in the city). We select 20 different cars in 50 different textures for training and randomly sample from 1 to 4 cars to render in a scene. We refer to this dataset as NeRDS 360: NeRF for Reconstruction, Decomposition and Scene Synthesis of 360◦ outdoor scenes. In total, we generate 15k renderings by sampling 200 cameras in a hemispherical dome at a fixed distance from the center of cars. We held out 5 scenes with 4 different cars and different backgrounds for testing, comprising 100 cameras distributed uniformly sampled in the upper hemisphere, different from the camera distributions used for training. We use the diverse validation camera distribution to test our approach’s ability to generalize to unseen viewpoints as well as unseen scenes during training. Our dataset and the corresponding task is extremely challenging due to occlusions, diversity of backgrounds, and rendered objects with various lightning and shadows. Our task entails reconstructing 360◦ hemispherical views of complete scenes using a handful of observations i.e. 1 to 5 as shown by red cameras whereas evaluating using all 100 hemispherical views, hence our task requires strong priors for novel view synthesis of outdoor scenes. Tasks our dataset support: - Generaliazable Novel view synthesis (Few shot evaluation) - Novel view synthesis (Overfitting evaluation) - 6D pose estimation - Object editing - Depth estimation - Semantic Segmentation - Instance Segmentation

RefRef is a synthetic dataset and benchmark designed for the task of reconstructing scenes with complex refractive and reflective objects. Our dataset consists of 50 objects categorized based on their geometric and material complexity: single-material convex objects, single-material non-convex objects, and multi-material non-convex objects, where the materials have different colors, opacities, and refractive indices. Each object is placed in three distinct bounded environments and one unbounded environment, resulting in 150 unique scenes with diverse geometries, material properties, and backgrounds. Our dataset provides a controlled setting for evaluating and developing 3D reconstruction and novel view synthesis methods that handle complex optical effects.

The new dataset contains around 1,500 train videos and 290 test videos, with 50 frames per video on average. The dataset was obtained after processing the manually captured video sequences of static real-life urban scenes. The main property of the dataset is the abundance of close objects and, consequently, the larger prevalence of occlusions. According to the introduced heuristic, the mean area of occluded image parts for SWORD is approximately five times larger than for RealEstate10k data (14% vs 3% respectively). This rationalizes the collection and usage of SWORD and explains that SWORD allows training more powerful models despite being of smaller size.

Synthetic (Blender) Dataset for forward facing scenes Toe vaualte NVS quality and camera parameter accuracy.

The UASOL an RGB-D stereo dataset, that contains 160902 frames, filmed at 33 different scenes, each with between 2 k and 10 k frames. The frames show different paths from the perspective of a pedestrian, including sidewalks, trails, roads, etc. The images were extracted from video files with 15 fps at HD2K resolution with a size of 2280 × 1282 pixels. The dataset also provides a GPS geolocalization tag for each second of the sequences and reflects different climatological conditions. It also involved up to 4 different persons filming the dataset at different moments of the day. We propose a train, validation and test split to train the network. Additionally, we introduce a subset of 676 pairs of RGB Stereo images and their respective depth, which we extracted randomly from the entire dataset. This given test set is introduced to make comparability possible between the different methods trained with the dataset.

This dataset presents a vision and perception research dataset collected in Rome, featuring RGB data, 3D point clouds, IMU, and GPS data. We introduce a new benchmark targeting visual odometry and SLAM, to advance the research in autonomous robotics and computer vision. This work complements existing datasets by simultaneously addressing several issues, such as environment diversity, motion patterns, and sensor frequency. It uses up-to-date devices and presents effective procedures to accurately calibrate the intrinsic and extrinsic of the sensors while addressing temporal synchronization. During recording, we cover multi-floor buildings, gardens, urban and highway scenarios. Combining handheld and car-based data collections, our setup can simulate any robot (quadrupeds, quadrotors, autonomous vehicles). The dataset includes an accurate 6-dof ground truth based on a novel methodology that refines the RTK-GPS estimate with LiDAR point clouds through Bundle Adjustment. All sequences divided in training and testing are accessible through our website.

The Deep Blending Dataset comprises 19 diverse scenes, offering comprehensive resources for free-viewpoint image-based rendering (IBR). Each scene includes input images, COLMAP reconstructions (SfM & MVS), global textured meshes from RealityCapture, refined depth maps, per-view meshes, and camera poses. This dataset is ideal for training and evaluating novel view synthesis and blending algorithms in both indoor and outdoor environments.

This is the dataset for the CGF 2021 paper "DONeRF: Towards Real-Time Rendering of Compact Neural Radiance Fields using Depth Oracle Networks". Please note the original creators of the individual 3D scenes themselves (individual license files can be found in the individual .zip archives in the dataset): Bulldozer by "Heinzelnisse" (CC-BY-NC): https://www.blendswap.com/blend/11490 Forest by Robin Tran (CC-BY-SA 3.0): https://cloud.blender.org/p/gallery/5fbd186ec57d586577c57417 Classroom by Christophe Seux (CC-0): https://download.blender.org/demo/test/classroom.zip San Miguel by Guillermo M. Leal Llaguno (CC-BY 3.0): https://casual-effects.com/g3d/data10/index.html# Pavillon by Hamza Cheggour / "eMirage" (CC-BY): https://download.blender.org/demo/test/pabellon_barcelona_v1.scene_.zip Barbershop by Blender Animation Studio (CC-BY): https://svn.blender.org/svnroot/bf-blender/trunk/lib/benchmarks/cycles/barbershop_interior/

IBL-NeRF Dataset. Contains multi-view images with its intrinsic components.

Real-world dataset on forward facing scenes with different camera intrinisc parameters.

A high-quality captured dataset for object relighting. Covering a wide range of geometry and material.

A high-quality synthetic dataset for object relighting. Covering a wide range of geometry and material.

We present SILVR, a dataset of light field images for six-degrees-of-freedom navigation in large fully-immersive volumes. The SILVR dataset is short for "Synthetic Immersive Large-Volume Ray" dataset. ## Properties Our dataset exhibits the following properties: - synthetic: Rendered using Blender 3.0 with Cycles, the images are perfect and do not need any calibration. Camera positions and lens configurations are known exactly and provided in the corresponding JSON files. - large interpolation volume: The camera configurations span a relatively large volume (a couple of meters in diameter). - large field of view: In order to maximize the interpolation volume (a.k.a: the walkable volume of light), the images are rendered using fisheye lenses with a field of view of 180°. - immersive: Thanks to the large field of view and positioning of the viewpoints, every point within the interpolation volume has a full panoramic field of view of light information available. - realism: The selected scenes have reasonable realism. - depth maps: As the images are computer-generated renders, we provide depth maps for every image. - specularities and reflections: The scenes exhibit some specularities or reflections, including mirrors. Reflections and mirrors always have the depth of the surface, and not the apparent depth of the reflections. - volumetrics: Some volumetrics are also present (fire, smoke, fog) in the garden scene. - densly rendered: The camera setup is rather dense (around 10cm spacing between cameras).