Multi-Person Pose Estimation

Region of Interest Align, or RoIAlign, is an operation for extracting a small feature map from each RoI in detection and segmentation based tasks. It removes the harsh quantization of RoI Pool, properly aligning the extracted features with the input. To avoid any quantization of the RoI boundaries or bins (using $x/16$ instead of $[x/16]$), RoIAlign uses bilinear interpolation to compute the exact values of the input features at four regularly sampled locations in each RoI bin, and the result is then aggregated (using max or average).

Mask R-CNN extends Faster R-CNN to solve instance segmentation tasks. It achieves this by adding a branch for predicting an object mask in parallel with the existing branch for bounding box recognition. In principle, Mask R-CNN is an intuitive extension of Faster R-CNN, but constructing the mask branch properly is critical for good results. Most importantly, Faster R-CNN was not designed for pixel-to-pixel alignment between network inputs and outputs. This is evident in how RoIPool, the de facto core operation for attending to instances, performs coarse spatial quantization for feature extraction. To fix the misalignment, Mask R-CNN utilises a simple, quantization-free layer, called RoIAlign, that faithfully preserves exact spatial locations. Secondly, Mask R-CNN decouples mask and class prediction: it predicts a binary mask for each class independently, without competition among classes, and relies on the network's RoI classification branch to predict the category. In contrast, an FCN usually perform per-pixel multi-class categorization, which couples segmentation and classification.

Represent and associate with a composite of primitive fields.

Dynamic Keypoint Head is an output head for pose estimation that are conditioned on each instance (person), and can encode the instance concept in the dynamically-generated weights of their filters. They are used in the FCPose architecture. The Figure shows the core idea. $F$ denotes a level of feature maps. "Rel. Coord." means the relative coordinates, denoting the relative offsets from the locations of $F$ to the location where the filters are generated. Refer to the text for details. $f\_{\theta\_{i}}$ is the dynamically-generated keypoint head for the $i$-th person instance. Note that each person instance has its own keypoint head.

FCPose is a fully convolutional multi-person pose estimation framework using dynamic instance-aware convolutions. Different from existing methods, which often require ROI (Region of Interest) operations and/or grouping post-processing, FCPose eliminates the ROIs and grouping pre-processing with dynamic instance aware keypoint estimation heads. The dynamic keypoint heads are conditioned on each instance (person), and can encode the instance concept in the dynamically-generated weights of their filters. Overall, FCPose is built upon the one-stage object detector FCOS. The controller that generates the weights of the keypoint heads is attached to the FCOS heads. The weights $\theta\_{i}$ generated by the controller is used to fulfill the keypoint head $f$ for the instance $i$. Moreover, a keypoint refinement module is introduced to predict the offsets from each location of the heatmaps to the ground-truth keypoints. Finally, the coordinates derived from the predicted heatmaps are refined by the offsets predicted by the keypoint refinement module, resulting in the final keypoint results. "Rel. coord." is a map of the relative coordinates from all the locations of the feature maps $F$ to the location where the weights are generated. The relative coordinate map is concatenated to $F$ as the input to the keypoint head.