Dimensionality Reduction

Feature selection, also known as variable selection, attribute selection or variable subset selection, is the process of selecting a subset of relevant features (variables, predictors) for use in model construction.

An ECA block has similar formulation to an SE block including a squeeze module for aggregating global spatial information and an efficient excitation module for modeling cross-channel interaction. Instead of indirect correspondence, an ECA block only considers direct interaction between each channel and its k-nearest neighbors to control model complexity. Overall, the formulation of an ECA block is: \begin{align} s = F_\text{eca}(X, \theta) & = \sigma (\text{Conv1D}(\text{GAP}(X))) \end{align} \begin{align} Y & = s X \end{align} where $\text{Conv1D}(\cdot)$ denotes 1D convolution with a kernel of shape $k$ across the channel domain, to model local cross-channel interaction. The parameter $k$ decides the coverage of interaction, and in ECA the kernel size $k$ is adaptively determined from the channel dimensionality $C$ instead of by manual tuning, using cross-validation: \begin{equation} k = \psi(C) = \left | \frac{\log_2(C)}{\gamma}+\frac{b}{\gamma}\right |\text{odd} \end{equation} where $\gamma$ and $b$ are hyperparameters. $|x|\text{odd}$ indicates the nearest odd function of $x$. Compared to SENet, ECANet has an improved excitation module, and provides an efficient and effective block which can readily be incorporated into various CNNs.

Parametric UMAP is a non-parametric graph-based dimensionality reduction algorithm that extends the second step of UMAP to a parametric optimization over neural network weights, learning a parametric relationship between data and embedding.

DELG is a convolutional neural network for image retrieval that combines generalized mean pooling for global features and attentive selection for local features. The entire network can be learned end-to-end by carefully balancing the gradient flow between two heads – requiring only image-level labels. This allows for efficient inference by extracting an image’s global feature, detected keypoints and local descriptors within a single model. The model is enabled by leveraging hierarchical image representations that arise in CNNs, which are coupled to generalized mean pooling and attentive local feature detection. Secondly, a convolutional autoencoder module is adopted that can successfully learn low-dimensional local descriptors. This can be readily integrated into the unified model, and avoids the need of post-processing learning steps, such as PCA, that are commonly used. Finally, a procedure is used that enables end-to-end training of the proposed model using only image-level supervision. This requires carefully controlling the gradient flow between the global and local network heads during backpropagation, to avoid disrupting the desired representations.