Video Prediction

ConvLSTM is a type of recurrent neural network for spatio-temporal prediction that has convolutional structures in both the input-to-state and state-to-state transitions. The ConvLSTM determines the future state of a certain cell in the grid by the inputs and past states of its local neighbors. This can easily be achieved by using a convolution operator in the state-to-state and input-to-state transitions (see Figure). The key equations of ConvLSTM are shown below, where $∗$ denotes the convolution operator and $\odot$ the Hadamard product: $$ i\_{t} = \sigma\left(W\_{xi} ∗ X\_{t} + W\_{hi} ∗ H\_{t−1} + W\_{ci} \odot \mathcal{C}\_{t−1} + b\_{i}\right) $$ $$ f\_{t} = \sigma\left(W\_{xf} ∗ X\_{t} + W\_{hf} ∗ H\_{t−1} + W\_{cf} \odot \mathcal{C}\_{t−1} + b\_{f}\right) $$ $$ \mathcal{C}\_{t} = f\_{t} \odot \mathcal{C}\_{t−1} + i\_{t} \odot \text{tanh}\left(W\_{xc} ∗ X\_{t} + W\_{hc} ∗ \mathcal{H}\_{t−1} + b\_{c}\right) $$ $$ o\_{t} = \sigma\left(W\_{xo} ∗ X\_{t} + W\_{ho} ∗ \mathcal{H}\_{t−1} + W\_{co} \odot \mathcal{C}\_{t} + b\_{o}\right) $$ $$ \mathcal{H}\_{t} = o\_{t} \odot \text{tanh}\left(C\_{t}\right) $$ If we view the states as the hidden representations of moving objects, a ConvLSTM with a larger transitional kernel should be able to capture faster motions while one with a smaller kernel can capture slower motions. To ensure that the states have the same number of rows and same number of columns as the inputs, padding is needed before applying the convolution operation. Here, padding of the hidden states on the boundary points can be viewed as using the state of the outside world for calculation. Usually, before the first input comes, we initialize all the states of the LSTM to zero which corresponds to "total ignorance" of the future.

VisuoSpatial Foresight is a method for robotic fabric manipulation that leverages a combination of RGB and depth information to learn goal conditioned fabric manipulation policies for a variety of long horizon tasks.

DVD-GAN DBlock is a residual block for the discriminator used in the DVD-GAN architecture for video generation. Unlike regular residual blocks, 3D convolutions are employed due to the application to multiple frames in a video.

DVD-GAN is a generative adversarial network for video generation built upon the BigGAN architecture. DVD-GAN uses two discriminators: a Spatial Discriminator $\mathcal{D}\_{S}$ and a Temporal Discriminator $\mathcal{D}\_{T}$. $\mathcal{D}\_{S}$ critiques single frame content and structure by randomly sampling $k$ full-resolution frames and judging them individually. The temporal discriminator $\mathcal{D}\_{T}$ must provide $G$ with the learning signal to generate movement (not evaluated by $\mathcal{D}\_{S}$). The input to $G$ consists of a Gaussian latent noise $z \sim N\left(0, I\right)$ and a learned linear embedding $e\left(y\right)$ of the desired class $y$. Both inputs are 120-dimensional vectors. $G$ starts by computing an affine transformation of $\left[z; e\left(y\right)\right]$ to a $\left[4, 4, ch\_{0}\right]$-shaped tensor. $\left[z; e\left(y\right)\right]$ is used as the input to all class-conditional Batch Normalization layers throughout $G$. This is then treated as the input (at each frame we would like to generate) to a Convolutional GRU. This RNN is unrolled once per frame. The output of this RNN is processed by two residual blocks. The time dimension is combined with the batch dimension here, so each frame proceeds through the blocks independently. The output of these blocks has width and height dimensions which are doubled (we skip upsampling in the first block). This is repeated a number of times, with the output of one RNN + residual group fed as the input to the next group, until the output tensors have the desired spatial dimensions. The spatial discriminator $\mathcal{D}\_{S}$ functions almost identically to BigGAN’s discriminator. A score is calculated for each of the uniformly sampled $k$ frames (default $k = 8$) and the $\mathcal{D}\_{S}$ output is the sum over per-frame scores. The temporal discriminator $\mathcal{D}\_{T}$ has a similar architecture, but pre-processes the real or generated video with a $2 \times 2$ average-pooling downsampling function $\phi$. Furthermore, the first two residual blocks of $\mathcal{D}\_{T}$ are 3-D, where every convolution is replaced with a 3-D convolution with a kernel size of $3 \times 3 \times 3$. The rest of the architecture follows BigGAN.

TSRUc, or Transformation-based Spatial Recurrent Unit c, is a modification of a ConvGRU used in the TriVD-GAN architecture for video generation. Instead of computing the reset gate $r$ and resetting $h\_{t−1}$, the TSRUc computes the parameters of a transformation $\theta$, which we use to warp $h\_{t−1}$. The rest of our model is unchanged (with $\hat{h}\_{t-1}$ playing the role of $h'\_{t}$ in $c$’s update equation from ConvGRU. The TSRUc module is described by the following equations: $$ \theta\_{h,x} = f\left(h\_{t−1}, x\_{t}\right) $$ $$ \hat{h}\_{t-1} = w\left(h\_{t-1}; \theta\_{h, x}\right) $$ $$ c = \rho\left(W\_{c} \star\_{n}\left[\hat{h}\_{t-1};x\_{t}\right] + b\_{c} \right) $$ $$ u = \sigma\left(W\_{u} \star\_{n}\left[h\_{t-1};x\_{t}\right] + b\_{u} \right) $$ $$ h\_{t} = u \odot h\_{t-1} + \left(1-u\right) \odot c $$ In these equations $\sigma$ and $\rho$ are the elementwise sigmoid and ReLU functions respectively and the $\star\_{n}$ represents a convolution with a kernel of size $n \times n$. Brackets are used to represent a feature concatenation.

TSRUp, or Transformation-based Spatial Recurrent Unit p, is a modification of a ConvGRU used in the TriVD-GAN architecture for video generation. It largely follows TSRUc, but computes $\theta$, $u$ and $c$ in parallel given $x\_{t}$ and $h\_{t−1}$, yielding the following replacement for the $c$ update equation: $$ c = \rho\left(W\_{c} \star\_{n}\left[h\_{t-1}; x\_{t}\right] + b\_{c} \right) $$ In these equations $\sigma$ and $\rho$ are the elementwise sigmoid and ReLU functions respectively and the $\star\_{n}$ represents a convolution with a kernel of size $n \times n$. Brackets are used to represent a feature concatenation.

TSRUs, or Transformation-based Spatial Recurrent Unit p, is a modification of a ConvGRU used in the TriVD-GAN architecture for video generation. It largely follows TSRUc, but computes each intermediate output in a fully sequential manner: like in TSRUc, $c$ is given access to $\hat{h}\_{t-1}$, but additionally, $u$ is given access to both outputs $\hat{h}\_{t-1}$ and $c$, so as to make an informed decision prior to mixing. This yields the following replacement for $u$: $$ u = \sigma\left(W\_{u} \star\_{n}\left[\hat{h}\_{t-1};c\right] + b\_{u} \right) $$ In these equations $\sigma$ and $\rho$ are the elementwise sigmoid and ReLU functions respectively and the $\star\_{n}$ represents a convolution with a kernel of size $n \times n$. Brackets are used to represent a feature concatenation.