기법

Two-Way Dense Layer is an image model block used in the PeleeNet architectures. Motivated by GoogLeNet, the 2-way dense layer is used to get different scales of receptive fields. One way of the layer uses a 3x3 kernel size. The other way of the layer uses two stacked 3x3 convolution to learn visual patterns for large objects.

U2-Net is a two-level nested U-structure architecture that is designed for salient object detection (SOD). The architecture allows the network to go deeper, attain high resolution, without significantly increasing the memory and computation cost. This is achieved by a nested U-structure: on the bottom level, with a novel ReSidual U-block (RSU) module, which is able to extract intra-stage multi-scale features without degrading the feature map resolution; on the top level, there is a U-Net like structure, in which each stage is filled by a RSU block.

A Unitary RNN is a recurrent neural network architecture that uses a unitary hidden to hidden matrix. Specifically they concern dynamics of the form: $$ h\_{t} = f\left(Wh\_{t−1} + Vx\_{t}\right) $$ where $W$ is a unitary matrix $\left(W^{†}W = I\right)$. The product of unitary matrices is a unitary matrix, so $W$ can be parameterised as a product of simpler unitary matrices: $$ h\_{t} = f\left(D\_{3}R\_{2}F^{−1}D\_{2}PR\_{1}FD\_{1}h\_{t−1} + Vxt\right) $$ where $D\_{3}$, $D\_{2}$, $D\_{1}$ are learned diagonal complex matrices, and $R\_{2}$, $R\_{1}$ are learned reflection matrices. Matrices $F$ and $F^{−1}$ are the discrete Fourier transformation and its inverse. P is any constant random permutation. The activation function $f\left(h\right)$ applies a rectified linear unit with a learned bias to the modulus of each complex number. Only the diagonal and reflection matrices, $D$ and $R$, are learned, so Unitary RNNs have fewer parameters than LSTMs with comparable numbers of hidden units. Source: Associative LSTMs

Video-Audio-Text Transformer, or VATT, is a framework for learning multimodal representations from unlabeled data using convolution-free Transformer architectures. Specifically, it takes raw signals as inputs and extracts multidimensional representations that are rich enough to benefit a variety of downstream tasks. VATT borrows the exact architecture from BERT and ViT except the layer of tokenization and linear projection reserved for each modality separately. The design follows the same spirit as ViT that makes the minimal changes to the architecture so that the learned model can transfer its weights to various frameworks and tasks. VATT linearly projects each modality into a feature vector and feeds it into a Transformer encoder. A semantically hierarchical common space is defined to account for the granularity of different modalities and noise contrastive estimation is employed to train the model.

VisTR is a Transformer based video instance segmentation model. It views video instance segmentation as a direct end-to-end parallel sequence decoding/prediction problem. Given a video clip consisting of multiple image frames as input, VisTR outputs the sequence of masks for each instance in the video in order directly. At the core is a new, effective instance sequence matching and segmentation strategy, which supervises and segments instances at the sequence level as a whole. VisTR frames the instance segmentation and tracking in the same perspective of similarity learning, thus considerably simplifying the overall pipeline and is significantly different from existing approaches.

VoiceFilter-Lite is a single-channel source separation model that runs on the device to preserve only the speech signals from a target user, as part of a streaming speech recognition system. In this architecture, the voice filtering model operates as a frame-by-frame frontend signal processor to enhance the features consumed by the speech recognizer, without reconstructing audio signals from the features. The key contributions are (1) A system to perform speech separation directly on ASR input features; (2) An asymmetric loss function to penalize oversuppression during training, to make the model harmless under various acoustic environments, (3) An adaptive suppression strength mechanism to adapt to different noise conditions.

Vokenization is an approach for extrapolating multimodal alignments to language-only data by contextually mapping language tokens to their related images ("vokens") by retrieval. Instead of directly supervising the language model with visually grounded language datasets (e.g., MS COCO) these relative small datasets are used to train the vokenization processor (i.e. the vokenizer). Vokens are generated for large language corpora (e.g., English Wikipedia), and the visually-supervised language model takes the input supervision from these large datasets, thus bridging the gap between different data sources.

XGrad-CAM, or Axiom-based Grad-CAM, is a class-discriminative visualization method and able to highlight the regions belonging to the objects of interest. Two axiomatic properties are introduced in the derivation of XGrad-CAM: Sensitivity and Conservation. In particular, the proposed XGrad-CAM is still a linear combination of feature maps, but able to meet the constraints of those two axioms.

ZoomNet is a 2D human whole-body pose estimation technique. It aims to localize dense landmarks on the entire human body including face, hands, body, and feet. ZoomNet follows the top-down paradigm. Given a human bounding box of each person, ZoomNet first localizes the easy-to-detect body keypoints and estimates the rough position of hands and face. Then it zooms in to focus on the hand/face areas and predicts keypoints using features with higher resolution for accurate localization. Unlike previous approaches which usually assemble multiple networks, ZoomNet has a single network that is end-to-end trainable. It unifies five network heads including the human body pose estimator, hand and face detectors, and hand and face pose estimators into a single network with shared low-level features.

k-Sparse Autoencoders are autoencoders with linear activation function, where in hidden layers only the $k$ highest activities are kept. This achieves exact sparsity in the hidden representation. Backpropagation only goes through the the top $k$ activated units. This can be achieved with a ReLU layer with an adjustable threshold.

reSGLD proposes to simulate a high-temperature particle for exploration and a low-temperature particle for exploitation and allows them to swap simultaneously. Moreover, a correction term is included to avoid biases.

The time-causal limit kernel is a temporal smoothing kernel that is (i) time-causal, (ii) time-recursive and (iii) obeys temporal scale covariance. This kernel constitutes the limit case of coupling an infinite number of truncated exponential kernels in cascade, with specifically chosen time constants to obtain temporal scale covariance. For practical purposes, the infinite convolution operation can often be well approximated by a moderate number (4-8) truncated exponential kernels coupled in cascade. The discrete implementation can, in turn, be performed by a set of first-order recursive filters coupled in cascade.

Atrous Convolution Neural Network (ACNN), as a pooling-free network structure, is proposed to achieve full-resolution feature processing using a theoretically optimal dilation setting for a larger receptive field, even with fewer parameters. Compared to other techniques, it can achieve higher segmentation Intersection over Union (IoU) and much less trainable parameters and model sizes, indicating the benefit of full-resolution feature maps in feature processing.

ACKTR, or Actor Critic with Kronecker-factored Trust Region, is an actor-critic method for reinforcement learning that applies trust region optimization using a recently proposed Kronecker-factored approximation to the curvature. The method extends the framework of natural policy gradient and optimizes both the actor and the critic using Kronecker-factored approximate curvature (K-FAC) with trust region.

Adaptive Early-Learning Correction for Segmentation from Noisy Annotations

Trainable activation function as a sigmoid-based generalization of ReLU, Swish and SiLU.

ALAE, or Adversarial Latent Autoencoder, is a type of autoencoder that attempts to overcome some of the limitations of generative adversarial networks. The architecture allows the latent distribution to be learned from data to address entanglement (A). The output data distribution is learned with an adversarial strategy (B). Thus, we retain the generative properties of GANs, as well as the ability to build on the recent advances in this area. For instance, we can include independent sources of stochasticity, which have proven essential for generating image details, or can leverage recent improvements on GAN loss functions, regularization, and hyperparameters tuning. Finally, to implement (A) and (B), AE reciprocity is imposed in the latent space (C). Therefore, we can avoid using reconstruction losses based on simple $\mathcal{l}\){2}$ norm that operates in data space, where they are often suboptimal, like for the image space. Since it works on the latent space, rather than autoencoding the data space, the approach is named Adversarial Latent Autoencoder (ALAE).

The Adaptive Locally Connected Neuron (ALCN) is a topology aware, and locally adaptive -focusing neuron: $$a = f\:\Bigg( \sum_{i=1}^{m} w_{i}\phi\left( \tau\left(i\right),\Theta\right) x_{i} + b \Bigg) %f\:\Bigg(\mathbf{X(W \circ \Phi) + b} \Bigg) $$

Adversarial-Learned Loss for Domain Adaptation is a method for domain adaptation that combines adversarial learning with self-training. Specifically, the domain discriminator has to produce different corrected labels for different domains, while the feature generator aims to confuse the domain discriminator. The adversarial process finally leads to a proper confusion matrix on the target domain. In this way, ALDA takes the strengths of domain-adversarial learning and self-training based methods.

ARM-Net is an adaptive relation modeling network tailored for structured data, and a lightweight framework ARMOR based on ARM-Net for relational data analytics. The key idea is to model feature interactions with cross features selectively and dynamically, by first transforming the input features into exponential space, and then determining the interaction order and interaction weights adaptively for each cross feature. The authors propose a novel sparse attention mechanism to dynamically generate the interaction weights given the input tuple, so that we can explicitly model cross features of arbitrary orders with noisy features filtered selectively. Then during model inference, ARM-Net can specify the cross features being used for each prediction for higher accuracy and better interpretability.

Automatic Structured Variational Inference (ASVI) is a fully automated method for constructing structured variational families, inspired by the closed-form update in conjugate Bayesian models. These convex-update families incorporate the forward pass of the input probabilistic program and can therefore capture complex statistical dependencies. Convex-update families have the same space and time complexity as the input probabilistic program and are therefore tractable for a very large family of models including both continuous and discrete variables.

The area under the receiver operating characteristics (ROC) Curve, referred to as AUC, is a well-known performance measure in the supervised learning domain. Due to its compelling features, it has been employed in a number of studies to evaluate and compare the performance of different classifiers. In this work, we explore AUC as a performance measure in the unsupervised learning domain, more specifically, in the context of cluster analysis. In particular, we elaborate on the use of AUC as an internal/relative measure of clustering quality, which we refer to as Area Under the Curve for Clustering (AUCC). We show that the AUCC of a given candidate clustering solution has an expected value under a null model of random clustering solutions, regardless of the size of the dataset and, more importantly, regardless of the number or the (im)balance of clusters under evaluation. In addition, we elaborate on the fact that, in the context of internal/relative clustering validation as we consider, AUCC is actually a linear transformation of the Gamma criterion from Baker and Hubert (1975), for which we also formally derive a theoretical expected value for chance clusterings. We also discuss the computational complexity of these criteria and show that, while an ordinary implementation of Gamma can be computationally prohibitive and impractical for most real applications of cluster analysis, its equivalence with AUCC actually unveils a much more efficient algorithmic procedure. Our theoretical findings are supported by experimental results. These results show that, in addition to an effective and robust quantitative evaluation provided by AUCC, visual inspection of the ROC curves themselves can be useful to further assess a candidate clustering solution from a broader, qualitative perspective as well.

AccoMontage is a model for accompaniment arrangement, a type of music generation task involving intertwined constraints of melody, harmony, texture, and music structure. AccoMontage generates piano accompaniments for folk/pop songs based on a lead sheet (i.e. a melody with chord progression). It first retrieves phrase montages from a database while recombining them structurally using dynamic programming. Second, chords of the retrieved phrases are manipulated to match the lead sheet via style transfer. Lastly, the system offers controls over the generation process. In contrast to pure deep learning approaches, AccoMontage uses a hybrid pathway, in which rule-based optimization and deep learning are both leveraged.

Please enter a description about the method here

AdaFisher – an adaptive second-order optimizer that leverages a block-diagonal approximation to the Fisher information matrix for adaptive gradient preconditioning.

AdaSmooth is a stochastic optimization technique that allows for per-dimension learning rate method for SGD. It is an extension of Adagrad and AdaDelta that seek to reduce its aggressive, monotonically decreasing learning rate. Instead of accumulating all past squared gradients, Adadelta restricts the window of accumulated past gradients to a fixed size $w$ while AdaSmooth adaptively selects the size of the window. Given the window size $M$, the effective ratio is calculated by $$e_t = \frac{s_t}{n_t}= \frac{| x_t - x_{t-M}|}{\sum_{i=0}^{M-1} | x_{t-i} - x_{t-1-i}|}\\ = \frac{| \sum_{i=0}^{M-1} \Delta x_{t-1-i}|}{\sum_{i=0}^{M-1} | \Delta x_{t-1-i}|}.$$ Given the effective ratio, the scaled smoothing constant is obtained by: $$c_t = ( \rho_2- \rho_1) \times e_t + (1-\rho_2),$$ The running average $E\left[g^{2}\right]\_{t}$ at time step $t$ then depends only on the previous average and current gradient: $$ E\left[g^{2}\right]\_{t} = c_t^2 \odot g_{t}^2 + \left(1-c_t^2 \right)\odot E[g^2]_{t-1} $$ Usually $\rho_1$ is set to around $0.5$ and $\rho_2$ is set to around 0.99. The update step the follows: $$ \Delta x_t = -\frac{\eta}{\sqrt{E\left[g^{2}\right]\_{t} + \epsilon}} \odot g_{t}, $$ which is incorporated into the final update: $$x_{t+1} = x_{t} + \Delta x_t.$$ The main advantage of AdaSmooth is its faster convergence rate and insensitivity to hyperparameters.

Adam-mini is a memory-efficient Adam variant that achieves on-par or better performance than AdamW with 45% to 50% less memory footprint. Adam-mini reduces the memory footprint by cutting down the learning rate resources in Adam (i.e., $ {1}/{\sqrt{v}} $). The authors find that ≥ 90% of these learning rates in $ v $ could be harmlessly removed if they (1) carefully partition the parameters into blocks following their proposed principle on Hessian structure; (2) assign a single but good learning rate to each parameter block. They further find that, for each of these parameter blocks, there exists a single high-quality learning rate that can outperform Adam, provided that sufficient resources are available to search it out.

AlignPS, or Feature-Aligned Person Search Network, is an anchor-free framework for efficient person search. The model employs the typical architecture of an anchor-free detection model (i.e., FCOS). An aligned feature aggregation (AFA) module is designed to make the model focus more on the re-id subtask. Specifically, AFA reshapes some building blocks of FPN to overcome the issues of region and scale misalignment in re-id feature learning. A deformable convolution is exploited to make the re-id embeddings adaptively aligned with the foreground regions. A feature fusion scheme is designed to better aggregate features from different FPN levels, which makes the re-id features more robust to scale variations. The training procedures of re-id and detection are also optimized to place more emphasis on generating robust re-id embeddings.

An All-Attention Layer is an attention module and layer for transformers that merges the self-attention and feedforward sublayers into a single unified attention layer. As opposed to the two-step mechanism of the Transformer layer, it directly builds its representation from the context and a persistent memory block without going through a feedforward transformation. The additional persistent memory block stores, in the form of key-value vectors, information that does not depend on the context. In terms of parameters, these persistent key-value vectors replace the feedforward sublayer.

In this work, we present a conceptually simple and effective method to train a strong bilingual multimodal representation model. Starting from the pretrained multimodal representation model CLIP released by OpenAI, we switched its text encoder with a pretrained multilingual text encoder XLM-R, and aligned both languages and image representations by a two-stage training schema consisting of teacher learning and contrastive learning. We validate our method through evaluations of a wide range of tasks. We set new state-of-the-art performances on a bunch of tasks including ImageNet-CN, Flicker30k- CN, and COCO-CN. Further, we obtain very close performances with CLIP on almost all tasks, suggesting that one can simply alter the text encoder in CLIP for extended capabilities such as multilingual understanding. Our models and code are available at https://github.com/FlagAI-Open/FlagAI.

Anycost GAN is a type of generative adversarial network for image synthesis and editing. Given an input image, we project it into the latent space with encoder $E$ and backward optimization. We can modify the latent code with user input to edit the image. During editing, a sub-generator of small cost is used for fast and interactive preview; during idle time, the full cost generator renders the final, high-quality output. The outputs from the full and sub-generators are visually consistent during projection and editing.