Hierarchical-Split Block is a representational block for multi-scale feature representations. It contains many hierarchical split and concatenate connections within one single residual block. Specifically, ordinary feature maps in deep neural networks are split into $s$ groups, each with $w$ channels. As shown in the Figure, only the first group of filters can be straightly connected to next layer. The second group of feature maps are sent to a convolution of $3 \times 3$ filters to extract features firstly, then the output feature maps are split into two sub-groups in the channel dimension. One sub-group of feature maps straightly connected to next layer, while the other sub-group is concatenated with the next group of input feature maps in the channel dimension. The concatenated feature maps are operated by a set of $3 \times 3$ convolutional filters. This process repeats several times until the rest of input feature maps are processed. Finally, features maps from all input groups are concatenated and sent to another layer of $1 \times 1$ filters to rebuild the features.
LiteSeg is a lightweight architecture for semantic segmentation that uses a deeper version of Atrous Spatial Pyramid Pooling module (ASPP) and applies short and long residual connections, and depthwise separable convolution, resulting in a faster, more efficient model.
MiVOS is a video object segmentation model which decouples interaction-to-mask and mask propagation. By decoupling interaction from propagation, MiVOS is versatile and not limited by the type of interactions. It uses three modules: Interaction-to-Mask, Propagation and Difference-Aware Fusion. Trained separately, the interaction module converts user interactions to an object mask, which is then temporally propagated by our propagation module using a novel top-filtering strategy in reading the space-time memory. To effectively take the user's intent into account, a novel difference-aware module is proposed to learn how to properly fuse the masks before and after each interaction, which are aligned with the target frames by employing the space-time memory.
PQ-Transformer, or PointQuad-Transformer, is a Transformer-based architecture that predicts 3D objects and layouts simultaneously, using point cloud inputs. Unlike existing methods that either estimate layout keypoints or edges, room layouts are directly parameterized as a set of quads. Along with the quad representation, a physical constraint loss function is used that discourages object-layout interference. Given an input 3D point cloud of $N$ points, the point cloud feature learning backbone extracts $M$ context-aware point features of $\left(3+C\right)$ dimensions, through sampling and grouping. A voting module and a farthest point sampling (FPS) module are used to generate $K\_{1}$ object proposals and $K\_{2}$ quad proposals respectively. Then the proposals are processed by a transformer decoder to further refine proposal features. Through several feedforward layers and non-maximum suppression (NMS), the proposals become the final object bounding boxes and layout quads.
PSFR-GAN is a semantic-aware style transformation framework for face restoration. Given a pair of LQ face image and its corresponding parsing map, we first generate a multi-scale pyramid of the inputs, and then progressively modulate different scale features from coarse-to-fine in a semantic-aware style transfer way. Compared with previous networks, the proposed PSFR-GAN makes full use of the semantic (parsing maps) and pixel (LQ images) space information from different scales of inputs.
\begin{equation} QA\left( x \right) = \sigma\left( f\left( x \right)^{1x1} \right) + x \end{equation} Quick Attention takes in the feature map as an input WxHxC (Width x Height x Channels) and creates two instances of the input feature map then it performs the 1x1xC convolution on the first instance and calculates the sigmoid activations after that it is added with the second instance to generate the final attention map as output which is of same dimensions as of input.
Model to adapt: We use Ultra Fast Structure-aware Deep Lane Detection (UFLD) as baseline and strictly adopt its training scheme and hyperparameters. UFLD treats lane detection as a row-based classification problem and utilizes the row anchors defined by TuSimple. Unsupervised Domain Adaptation with SGPCS: SGPCS builds upon PCS and performs in-domain contrastive learning and crossdomain self-supervised learning via cluster prototypes. We reformulate the pseudo label selection mechanism from SGADA. For each lane, we select the highest confidence value from the griding cells of each row anchor. Based on their griding cell position, the confidence values are divided into two cases: absent lane points and present lane points. Thereby, the last griding cell represents absent lane points as in. For each case, we calculate the mean confidence over the corresponding lanes. We then use the thresholds defined by SGADA to decide whether the prediction is treated as a pseudo label. Our overall objective function comprises the in-domain and cross-domain loss from PCS, the losses defined by UFLD, and our adopted pseudo loss from SGADA. We adjust the momentum for memory bank feature updates to 0.5 and use spherical K-means with K = 2,500 to cluster them into prototypes.
State-Aware Tracker is a pipeline for semi-supervised video object segmentation. It takes each target object as a tracklet, which not only makes the pipeline more efficient but also filters distractors to facilitate target modeling. For more stable and robust performance over video sequences, SAT gets awareness for each state and makes self-adaptation via two feedback loops. One loop assists SAT in generating more stable tracklets. The other loop helps to construct a more robust and holistic target representation.
Video Panoptic Segmentation Network, or VPSNet, is a model for video panoptic segmentation. On top of UPSNet, which is a method for image panoptic segmentation, VPSNet is designed to take an additional frame as the reference to correlate time information at two levels: pixel-level fusion and object-level tracking. To pick up the complementary feature points in the reference frame, a flow-based feature map alignment module is introduced along with an asymmetric attention block that computes similarities between the target and reference features to fuse them into one-frame shape. Additionally, to associate object instances across time, an object track head is added which learns the correspondence between the instances in the target and reference frames based on their RoI feature similarity.