Segmentation

Inspired on the widely known spatial squeeze and channel excitation (SE) block, the sSE block performs channel squeeze and spatial excitation, to recalibrate the feature maps spatially and achieve more fine-grained image segmentation.

A Panoptic FPN is an extension of an FPN that can generate both instance and semantic segmentations via FPN. The approach starts with an FPN backbone and adds a branch for performing semantic segmentation in parallel with the existing region-based branch for instance segmentation. No changes are made to the FPN backbone when adding the dense-prediction branch, making it compatible with existing instance segmentation methods. The new semantic segmentation branch achieves its goal as follows. Starting from the deepest FPN level (at 1/32 scale), we perform three upsampling stages to yield a feature map at 1/4 scale, where each upsampling stage consists of 3×3 convolution, group norm, ReLU, and 2× bilinear upsampling. This strategy is repeated for FPN scales 1/16, 1/8, and 1/4 (with progressively fewer upsampling stages). The result is a set of feature maps at the same 1/4 scale, which are then element-wise summed. A final 1×1 convolution, 4× bilinear upsampling, and softmax are used to generate the per-pixel class labels at the original image resolution. In addition to stuff classes, this branch also outputs a special ‘other’ class for all pixels belonging to objects (to avoid predicting stuff classes for such pixels).

PolarMask is an anchor-box free and single-shot instance segmentation method. Specifically, PolarMask takes an image as input and predicts the distance from a sampled positive location (ie a candidate object's center) with respect to the object's contour at each angle, and then assembles the predicted points to produce the final mask. There are several benefits to the system: (1) The polar representation unifies instance segmentation (masks) and object detection (bounding boxes) into a single framework (2) Two modules are designed (i.e. soft polar centerness and polar IoU loss) to sample high-quality center examples and optimize polar contour regression, making the performance of PolarMask does not depend on the bounding box prediction results and more efficient in training. (3) PolarMask is fully convolutional and can be embedded into most off-the-shelf detection methods.

VisTR is a Transformer based video instance segmentation model. It views video instance segmentation as a direct end-to-end parallel sequence decoding/prediction problem. Given a video clip consisting of multiple image frames as input, VisTR outputs the sequence of masks for each instance in the video in order directly. At the core is a new, effective instance sequence matching and segmentation strategy, which supervises and segments instances at the sequence level as a whole. VisTR frames the instance segmentation and tracking in the same perspective of similarity learning, thus considerably simplifying the overall pipeline and is significantly different from existing approaches.

Atrous Convolution Neural Network (ACNN), as a pooling-free network structure, is proposed to achieve full-resolution feature processing using a theoretically optimal dilation setting for a larger receptive field, even with fewer parameters. Compared to other techniques, it can achieve higher segmentation Intersection over Union (IoU) and much less trainable parameters and model sizes, indicating the benefit of full-resolution feature maps in feature processing.

Adaptive Early-Learning Correction for Segmentation from Noisy Annotations

BiSeNet V2 is a two-pathway architecture for real-time semantic segmentation. One pathway is designed to capture the spatial details with wide channels and shallow layers, called Detail Branch. In contrast, the other pathway is introduced to extract the categorical semantics with narrow channels and deep layers, called Semantic Branch. The Semantic Branch simply requires a large receptive field to capture semantic context, while the detail information can be supplied by the Detail Branch. Therefore, the Semantic Branch can be made very lightweight with fewer channels and a fast-downsampling strategy. Both types of feature representation are merged to construct a stronger and more comprehensive feature representation.

Bilateral Guided Aggregation Layer is a feature fusion layer for semantic segmentation that aims to enhance mutual connections and fuse different types of feature representation. It was used in the BiSeNet V2 architecture. Specifically, within the BiSeNet implementation, the layer was used to employ the contextual information of the Semantic Branch to guide the feature response of Detail Branch. With different scale guidance, different scale feature representations can be captured, which inherently encodes the multi-scale information.

CascadePSP is a general segmentation refinement model that refines any given segmentation from low to high resolution. The model takes as input an initial mask that can be an output of any algorithm to provide a rough object location. Then the CascadePSP will output a refined mask. The model is designed in a cascade fashion that generates refined segmentation in a coarse-to-fine manner. Coarse outputs from the early levels predict object structure which will be used as input to the latter levels to refine boundary details.

Combines the channel attention of the widely known spatial squeeze and channel excitation (SE) block and the spatial attention of the channel squeeze and spatial excitation (sSE) block to build a spatial and channel attention mechanism for image segmentation tasks.

GroIE is an RoI extractor which intends to overcome the limitation of existing extractors which select only one (the best) layer from the FPN. The intuition is that all the layers of FPN retain useful information. Therefore, the proposed layer introduces non-local building blocks and attention mechanisms to boost the performance.

InstaBoost is a data augmentation technique for instance segmentation that utilises existing instance mask annotations. Intuitively in a small neighbor area of $(x_0, y_0, 1, 0)$, the probability map $P(x, y, s, r)$ should be high-valued since images are usually continuous and redundant in pixel level. Based on this, InstaBoost is a form of augmentation where we apply object jittering that randomly samples transformation tuples from the neighboring space of identity transform $(x_0, y_0, 1, 0)$ and paste the cropped object following affine transform $\mathbf{H}$.

Synergistic Image and Feature Alignment is an unsupervised domain adaptation framework that conducts synergistic alignment of domains from both image and feature perspectives. In SIFA, we simultaneously transform the appearance of images across domains and enhance domain-invariance of the extracted features by leveraging adversarial learning in multiple aspects and with a deeply supervised mechanism. The feature encoder is shared between both adaptive perspectives to leverage their mutual benefits via end-to-end learning.

ShapeConv, or Shape-aware Convolutional layer, is a convolutional layer for processing the depth feature in indoor RGB-D semantic segmentation. The depth feature is firstly decomposed into a shape-component and a base-component, next two learnable weights are introduced to cooperate with them independently, and finally a convolution is applied on the re-weighted combination of these two components.

Batch-Channel Normalization, or BCN, uses batch knowledge to prevent channel-normalized models from getting too close to "elimination singularities". Elimination singularities correspond to the points on the training trajectory where neurons become consistently deactivated. They cause degenerate manifolds in the loss landscape which will slow down training and harm model performances.

DASPP is a deeper version of the ASPP module (the latter from DeepLabv3) that adds standard 3 × 3 convolution after 3 × 3 dilated convolutions to refine the features and also fusing the input and the output of the DASPP module via short residual connection. Also, the number of convolution filters of ASPP is reduced from 255 to 96 to gain computational performance.

Deep-MAC, or Deep Mask-heads Above CenterNet, is a type of anchor-free instance segmentation model based on CenterNet. The motivation for this new architecture is that boxes are much cheaper to annotate than masks, so the authors address the “partially supervised” instance segmentation problem, where all classes have bounding box annotations but only a subset of classes have mask annotations. For predicting bounding boxes, CenterNet outputs 3 tensors: (1) a class-specific heatmap which indicates the probability of the center of a bounding box being present at each location, (2) a class-agnostic 2-channel tensor indicating the height and width of the bounding box at each center pixel, and (3) since the output feature map is typically smaller than the image (stride 4 or 8), CenterNet also predicts an x and y direction offset to recover this discretization error at each center pixel. For Deep-MAC, in parallel to the box-related prediction heads, we add a fourth pixel embedding branch $P$. For each bounding box $b$, we crop a region $P\_{b}$ from $P$ corresponding to $b$ via ROIAlign which results in a 32 × 32 tensor. We then feed each $P\_{b}$ to a mask-head. The final prediction at the end is a class-agnostic 32 × 32 tensor which we pass through a sigmoid to get per-pixel probabilities. We train this mask-head via a per-pixel cross-entropy loss averaged over all pixels and instances. During post-processing, the predicted mask is re-aligned according to the predicted box and resized to the resolution of the image. In addition to this 32 × 32 cropped feature map, we add two inputs for improved stability of some mask-heads: (1) Instance embedding: an additional head is added to the backbone that predicts a per-pixel embedding. For each bounding box $b$ we extract its embedding from the center pixel. This embedding is tiled to a size of 32 × 32 and concatenated to the pixel embedding crop. This helps condition the mask-head on a particular instance and disambiguate it from others. (2) Coordinate Embedding: Inspired by CoordConv, the authors add a 32 × 32 × 2 tensor holding normalized $\left(x, y\right)$ coordinates relative to the bounding box $b$.

Decoder architecture inspired on the UNet++ structure and the EfficientNet building blocks. Keeping the UNet++ structure, the EfficientUNet++ achieves higher performance and significantly lower computational complexity through two simple modifications: * Replaces the 3x3 convolutions of the UNet++ with residual bottleneck blocks with depthwise convolutions * Applies channel and spatial attention to the bottleneck feature maps using concurrent spatial and channel squeeze & excitation (scSE) blocks

Gumbel activation function, is defined using the cumulative Gumbel distribution and it can be used to perform Gumbel regression. Gumbel activation is an alternative activation function to the sigmoid or softmax activation functions and can be used to transform the unormalised output of a model to probability. Gumbel activation $\eta_{Gumbel}$ is defined as follows: $\eta_{Gumbel}(q_i) = exp(-exp(-q_i))$ It can be combined with Cross Entropy loss function to solve long-tailed classification problems. Gumbel Cross Entropy (GCE) is defined as follows: $GCE(\eta_{Gumbel}(q_i),y_i) = -y_i \log(\eta_{Gumbel}(q_i))+ (1-y_i) \log(1-\eta_{Gumbel}(q_i))$

HS-ResNet is a convolutional neural network that employs Hierarchical-Split Block as its central building block within a ResNet-like architecture.

Hierarchical-Split Block is a representational block for multi-scale feature representations. It contains many hierarchical split and concatenate connections within one single residual block. Specifically, ordinary feature maps in deep neural networks are split into $s$ groups, each with $w$ channels. As shown in the Figure, only the first group of filters can be straightly connected to next layer. The second group of feature maps are sent to a convolution of $3 \times 3$ filters to extract features firstly, then the output feature maps are split into two sub-groups in the channel dimension. One sub-group of feature maps straightly connected to next layer, while the other sub-group is concatenated with the next group of input feature maps in the channel dimension. The concatenated feature maps are operated by a set of $3 \times 3$ convolutional filters. This process repeats several times until the rest of input feature maps are processed. Finally, features maps from all input groups are concatenated and sent to another layer of $1 \times 1$ filters to rebuild the features.

Metaheuristic algorithm

LiteSeg is a lightweight architecture for semantic segmentation that uses a deeper version of Atrous Spatial Pyramid Pooling module (ASPP) and applies short and long residual connections, and depthwise separable convolution, resulting in a faster, more efficient model.

Si necesitas hablar con una persona real en Aeroméxico, la mejor opción es llamar al número de atención al cliente 📞 【+52↪ 556↪ 826↪ 3219】(México), +(1)↪ {855}↪ (611)↪ (3553)】 (Estados Unidos) o 【+34↪ 913↪ 352↪ 483】(España). Este número está disponible para ayudarte con reservaciones, cambios de vuelo, equipaje perdido y cualquier otra consulta relacionada con tu viaje. Para hablar directamente con un representante, marca 📞 【+52↪ 556↪ 826↪ 3219】(México), +(1)↪ {855}↪ (611)↪ (3553)】 (Estados Unidos) o 【+34↪ 913↪ 352↪ 483】(España) y selecciona la opción adecuada en el menú automático. Si deseas evitar esperas largas, se recomienda llamar durante las primeras horas de la mañana. Ten a la mano tu número de confirmación y documento de identidad para agilizar el proceso. Muchas veces, los canales digitales no ofrecen una respuesta inmediata, por lo que llamar a 📞 【+52↪ 556↪ 826↪ 3219】(México), +(1)↪ {855}↪ (611)↪ (3553)】 (Estados Unidos) o 【+34↪ 913↪ 352↪ 483】(España) te garantiza una atención más personalizada. Si por alguna razón tu llamada se corta o no obtienes una solución completa, simplemente vuelve a marcar 📞 【+52↪ 556↪ 826↪ 3219】(México), +(1)↪ {855}↪ (611)↪ (3553)】 (Estados Unidos) o 【+34↪ 913↪ 352↪ 483】(España). No olvides que puedes utilizar 📞 【+52↪ 556↪ 826↪ 3219】(México), +(1)↪ {855}↪ (611)↪ (3553)】 (Estados Unidos) o 【+34↪ 913↪ 352↪ 483】(España) las veces que necesites asistencia directa por parte del equipo de Aeroméxico.

One-Shot Aggregation with an Identity Mapping and eSE is an image model block that extends one-shot aggregation with a residual connection and effective squeeze-and-excitation block. It is proposed as part of the VoVNetV2 CNN architecture. The module adds an identity mapping to the OSA module - the input path is connected to the end of an OSA module that is able to backpropagate the gradients of every OSA module in an end-to-end manner on each stage like a ResNet. Additionally, a channel attention module - effective Squeeze-Excitation - is used which is like regular squeeze-and-excitation but uses only one FC layer with $C$ channels instead of two FCs without a channel dimension reduction, which maintains channel information.

\begin{equation} QA\left( x \right) = \sigma\left( f\left( x \right)^{1x1} \right) + x \end{equation} Quick Attention takes in the feature map as an input WxHxC (Width x Height x Channels) and creates two instances of the input feature map then it performs the 1x1xC convolution on the first instance and calculates the sigmoid activations after that it is added with the second instance to generate the final attention map as output which is of same dimensions as of input.

Region of Interest Warping, or RoIWarp, is a form of RoIPool that is differentiable with respect to the box position. In practice, this takes the form of a RoIWarp layer followed by a standard Max Pooling layer. The RoIWarp layer crops a feature map region and warps it into a target size by interpolation.

The Shuffle Transformer Block consists of the Shuffle Multi-Head Self-Attention module (ShuffleMHSA), the Neighbor-Window Connection module (NWC), and the MLP module. To introduce cross-window connections while maintaining the efficient computation of non-overlapping windows, a strategy which alternates between WMSA and Shuffle-WMSA in consecutive Shuffle Transformer blocks is proposed. The first window-based transformer block uses regular window partition strategy and the second window-based transformer block uses window-based selfattention with spatial shuffle. Besides, the Neighbor-Window Connection moduel (NWC) is added into each block for enhancing connections among neighborhood windows. Thus the proposed shuffle transformer block could build rich cross-window connections and augments representation. Finally, the consecutive Shuffle Transformer blocks are computed as: $$ x^{l}=\mathbf{W M S A}\left(\mathbf{B N}\left(z^{l-1}\right)\right)+z^{l-1} $$ $$ y^{l}=\mathbf{N W C}\left(x^{l}\right)+x^{l} $$ $$ z^{l}=\mathbf{M L P}\left(\mathbf{B N}\left(y^{l}\right)\right)+y^{l} $$ $$ x^{l+1}=\mathbf{S h u f f l e - W M S A}\left(\mathbf{B N}\left(z^{l}\right)\right)+z^{l} $$ $$ y^{l+1}=\mathbf{N W C}\left(x^{l+1}\right)+x^{l+1} $$ $$ z^{l+1}=\mathbf{M L P}\left(\mathbf{B N}\left(y^{l+1}\right)\right)+y^{l+1} $$ where $x^l$, $y^l$ and $z^l$ denote the output features of the (Shuffle-)WMSA module, the Neighbor-Window Connection module and the MLP module for block $l$, respectively; WMSA and Shuffle-WMSA denote window-based multi-head self-attention without/with spatial shuffle, respectively.

State-Aware Tracker is a pipeline for semi-supervised video object segmentation. It takes each target object as a tracklet, which not only makes the pipeline more efficient but also filters distractors to facilitate target modeling. For more stable and robust performance over video sequences, SAT gets awareness for each state and makes self-adaptation via two feedback loops. One loop assists SAT in generating more stable tracklets. The other loop helps to construct a more robust and holistic target representation.

TradeS is an online joint detection and tracking model, coined as TRACK to DEtect and Segment, exploiting tracking clues to assist detection end-to-end. TraDeS infers object tracking offset by a cost volume, which is used to propagate previous object features for improving current object detection and segmentation.

Unigram Segmentation is a subword segmentation algorithm based on a unigram language model. It provides multiple segmentations with probabilities. The language model allows for emulating the noise generated during the segmentation of actual data. The unigram language model makes an assumption that each subword occurs independently, and consequently, the probability of a subword sequence $\mathbf{x} = (x_1,\ldots,x_M)$ is formulated as the product of the subword occurrence probabilities $p(x_i)$: $$ P(\mathbf{x}) = \prod_{i=1}^{M} p(x_i), \\\\ \forall i\,\, x_i \in \mathcal{V},\,\,\, \sum_{x \in \mathcal{V}} p(x) = 1, \nonumber $$ where $\mathcal{V}$ is a pre-determined vocabulary. The most probable segmentation $\mathbf{x}^*$ for the input sentence $X$ is then given by: $$ \mathbf{x}^{*} = \text{argmax}_{\mathbf{x} \in \mathcal{S}(X)} P(\mathbf{x}), $$ where $\mathcal{S}(X)$ is a set of segmentation candidates built from the input sentence $X$. $\mathbf{x}^*$ is obtained with the Viterbi algorithm.

Video Panoptic Segmentation Network, or VPSNet, is a model for video panoptic segmentation. On top of UPSNet, which is a method for image panoptic segmentation, VPSNet is designed to take an additional frame as the reference to correlate time information at two levels: pixel-level fusion and object-level tracking. To pick up the complementary feature points in the reference frame, a flow-based feature map alignment module is introduced along with an asymmetric attention block that computes similarities between the target and reference features to fuse them into one-frame shape. Additionally, to associate object instances across time, an object track head is added which learns the correspondence between the instances in the target and reference frames based on their RoI feature similarity.

VoVNetV2 is a convolutional neural network that improves upon VoVNet with two effective strategies: (1) residual connection for alleviating the optimization problem of larger VoVNets and (2) effective Squeeze-Excitation (eSE) dealing with the channel information loss problem of the original squeeze-and-excitation module.