기법

SRU, or Simple Recurrent Unit, is a recurrent neural unit with a light form of recurrence. SRU exhibits the same level of parallelism as convolution and feed-forward nets. This is achieved by balancing sequential dependence and independence: while the state computation of SRU is time-dependent, each state dimension is independent. This simplification enables CUDA-level optimizations that parallelize the computation across hidden dimensions and time steps, effectively using the full capacity of modern GPUs. SRU also replaces the use of convolutions (i.e., ngram filters), as in QRNN and KNN, with more recurrent connections. This retains modeling capacity, while using less computation (and hyper-parameters). Additionally, SRU improves the training of deep recurrent models by employing highway connections and a parameter initialization scheme tailored for gradient propagation in deep architectures. A single layer of SRU involves the following computation: $$ \mathbf{f}\_{t} =\sigma\left(\mathbf{W}\_{f} \mathbf{x}\_{t}+\mathbf{v}\_{f} \odot \mathbf{c}\_{t-1}+\mathbf{b}\_{f}\right) $$ $$ \mathbf{c}\_{t} =\mathbf{f}\_{t} \odot \mathbf{c}\_{t-1}+\left(1-\mathbf{f}\_{t}\right) \odot\left(\mathbf{W} \mathbf{x}\_{t}\right) \\ $$ $$ \mathbf{r}\_{t} =\sigma\left(\mathbf{W}\_{r} \mathbf{x}\_{t}+\mathbf{v}\_{r} \odot \mathbf{c}\_{t-1}+\mathbf{b}\_{r}\right) \\ $$ $$ \mathbf{h}\_{t} =\mathbf{r}\_{t} \odot \mathbf{c}\_{t}+\left(1-\mathbf{r}\_{t}\right) \odot \mathbf{x}\_{t} $$ where $\mathbf{W}, \mathbf{W}\_{f}$ and $\mathbf{W}\_{r}$ are parameter matrices and $\mathbf{v}\_{f}, \mathbf{v}\_{r}, \mathbf{b}\_{f}$ and $\mathbf{b}_{v}$ are parameter vectors to be learnt during training. The complete architecture decomposes to two sub-components: a light recurrence and a highway network, The light recurrence component successively reads the input vectors $\mathbf{x}\_{t}$ and computes the sequence of states $\mathbf{c}\_{t}$ capturing sequential information. The computation resembles other recurrent networks such as LSTM, GRU and RAN. Specifically, a forget gate $\mathbf{f}\_{t}$ controls the information flow and the state vector $\mathbf{c}\_{t}$ is determined by adaptively averaging the previous state $\mathbf{c}\_{t-1}$ and the current observation $\mathbf{W} \mathbf{x}_{+}$according to $\mathbf{f}\_{t}$.

Inspired by the spatial propagation mechanism utilized in the depth completion task \cite{NLSPN}, we introduce a normal incorporated non-local disparity propagation module in which we hub NDP to generate non-local affinities and offsets for spatial propagation at the disparity level. The motivation lies that the sampled pixels for edges and occluded regions are supposed to be selected. The propagation process aggregates disparities via plane affinity relations, which alleviates the phenomenon of disparity blurring at object edges due to frontal parallel windows. And the disparities in occluded areas are also optimized at the same time by being propagated from non-occluded areas where the predicted disparities are with high confidence.

Temporal Graph Network, or TGN, is a framework for deep learning on dynamic graphs represented as sequences of timed events. The memory (state) of the model at time $t$ consists of a vector $\mathbf{s}_i(t)$ for each node $i$ the model has seen so far. The memory of a node is updated after an event (e.g. interaction with another node or node-wise change), and its purpose is to represent the node's history in a compressed format. Thanks to this specific module, TGNs have the capability to memorize long term dependencies for each node in the graph. When a new node is encountered, its memory is initialized as the zero vector, and it is then updated for each event involving the node, even after the model has finished training.

An Accumulating Eligibility Trace is a type of eligibility trace where the trace increments in an accumulative way. For the memory vector $\textbf{e}\_{t} \in \mathbb{R}^{b} \geq \textbf{0}$: $$\mathbf{e\_{0}} = \textbf{0}$$ $$\textbf{e}\_{t} = \nabla{\hat{v}}\left(S\_{t}, \mathbf{\theta}\_{t}\right) + \gamma\lambda\textbf{e}\_{t}$$

Adaptive Dropout is a regularization technique that extends dropout by allowing the dropout probability to be different for different units. The intuition is that there may be hidden units that can individually make confident predictions for the presence or absence of an important feature or combination of features. Dropout will ignore this confidence and drop the unit out 50% of the time. Denote the activity of unit $j$ in a deep neural network by $a\_{j}$ and assume that its inputs are {$a\_{i}: i < j$}. In dropout, $a\_{j}$ is randomly set to zero with probability 0.5. Let $m\_{j}$ be a binary variable that is used to mask, the activity $a\_{j}$, so that its value is: $$ a\_{j} = m\_{j}g \left( \sum\_{i: i<j}w\_{j, i}a\_{i} \right)$$ where $w\_{j,i}$ is the weight from unit $i$ to unit $j$ and $g\left(·\right)$ is the activation function and $a\_{0} = 1$ accounts for biases. Whereas in standard dropout, $m\_{j}$ is Bernoulli with probability $0.5$, adaptive dropout uses adaptive dropout probabilities that depends on input activities: $$ P\left(m\_{j} = 1\mid{\{a\_{i}: i < j\}}\right) = f \left( \sum\_{i: i<j}\pi{\_{j, i}a\_{i}} \right) $$ where $\pi\_{j, i}$ is the weight from unit $i$ to unit $j$ in the standout network or the adaptive dropout network; $f(·)$ is a sigmoidal function. Here 'standout' refers to a binary belief network is that is overlaid on a neural network as part of the overall regularization technique.

A BiGAN, or Bidirectional GAN, is a type of generative adversarial network where the generator not only maps latent samples to generated data, but also has an inverse mapping from data to the latent representation. The motivation is to make a type of GAN that can learn rich representations for us in applications like unsupervised learning. In addition to the generator $G$ from the standard GAN framework, BiGAN includes an encoder $E$ which maps data $\mathbf{x}$ to latent representations $\mathbf{z}$. The BiGAN discriminator $D$ discriminates not only in data space ($\mathbf{x}$ versus $G\left(\mathbf{z}\right)$), but jointly in data and latent space (tuples $\left(\mathbf{x}, E\left(\mathbf{x}\right)\right)$ versus $\left(G\left(z\right), z\right)$), where the latent component is either an encoder output $E\left(\mathbf{x}\right)$ or a generator input $\mathbf{z}$.

Enhanced Sequential Inference Model or ESIM is a sequential NLI model proposed in Enhanced LSTM for Natural Language Inference paper.

Inception v2 is the second generation of Inception convolutional neural network architectures which notably uses batch normalization. Other changes include dropping dropout and removing local response normalization, due to the benefits of batch normalization.

BCI MI signal Classification Framework using Fuzzy integrals. Paper: Ko, L. W., Lu, Y. C., Bustince, H., Chang, Y. C., Chang, Y., Ferandez, J., ... & Lin, C. T. (2019). Multimodal fuzzy fusion for enhancing the motor-imagery-based brain computer interface. IEEE Computational Intelligence Magazine, 14(1), 96-106.

Squared ReLU is an activation function used in the Primer architecture in the feedforward block of the Transformer layer. It is simply squared ReLU activations. The effectiveness of higher order polynomials can also be observed in other effective Transformer nonlinearities, such as GLU variants like ReGLU and point-wise activations like approximate GELU. However, squared ReLU has drastically different asymptotics as $x \rightarrow \inf$ compared to the most commonly used activation functions: ReLU, GELU and Swish. Squared ReLU does have significant overlap with ReGLU and in fact is equivalent when ReGLU’s $U$ and $V$ weight matrices are the same and squared ReLU is immediately preceded by a linear transformation with weight matrix $U$. This leads the authors to believe that squared ReLUs capture the benefits of these GLU variants, while being simpler, without additional parameters, and delivering better quality.

The Contour Proposal Network (CPN) detects possibly overlapping objects in an image while simultaneously fitting pixel-precise closed object contours. The CPN can incorporate state of the art object detection architectures as backbone networks into a fast single-stage instance segmentation model that can be trained end-to-end.

Please enter a description about the method here

Gradient Checkpointing is a method used for reducing the memory footprint when training deep neural networks, at the cost of having a small increase in computation time.

MATE is a Transformer architecture designed to model the structure of web tables. It uses sparse attention in a way that allows heads to efficiently attend to either rows or columns in a table. Each attention head reorders the tokens by either column or row index and then applies a windowed attention mechanism. Unlike traditional self-attention, Mate scales linearly in the sequence length.

MEI introduces the multi-partition embedding interaction technique with block term tensor format to systematically address the efficiency--expressiveness trade-off in knowledge graph embedding. It divides the embedding vector into multiple partitions and learns the local interaction patterns from data instead of using fixed special patterns as in ComplEx or SimplE models. This enables MEI to achieve optimal efficiency--expressiveness trade-off, not just being fully expressive. Previous methods such as TuckER, RESCAL, DistMult, ComplEx, and SimplE are suboptimal restricted special cases of MEI.

MelGAN is a non-autoregressive feed-forward convolutional architecture to perform audio waveform generation in a GAN setup. The architecture is a fully convolutional feed-forward network with mel-spectrogram $s$ as input and raw waveform $x$ as output. Since the mel-spectrogram is at a 256× lower temporal resolution, the authors use a stack of transposed convolutional layers to upsample the input sequence. Each transposed convolutional layer is followed by a stack of residual blocks with dilated convolutions. Unlike traditional GANs, the MelGAN generator does not use a global noise vector as input. To deal with 'checkerboard artifacts' in audio, instead of using PhaseShuffle, MelGAN uses kernel-size as a multiple of stride. Weight normalization is used for normalization. A window-based discriminator, similar to a PatchGAN is used for the discriminator.

Primer is a Transformer-based architecture that improves upon the Transformer architecture with two improvements found through neural architecture search: squared RELU activations in the feedforward block, and depthwise convolutions added to the attention multi-head projections: resulting in a new module called Multi-DConv-Head-Attention.

A R(2+1)D convolutional neural network is a network for action recognition that employs R(2+1)D convolutions in a ResNet inspired architecture. The use of these convolutions over regular 3D Convolutions reduces computational complexity, prevents overfitting, and introduces more non-linearities that allow for a better functional relationship to be modeled.

Self-supervised Equivariant Attention Mechanism, or SEAM, is an attention mechanism for weakly supervised semantic segmentation. The SEAM applies consistency regularization on CAMs from various transformed images to provide self-supervision for network learning. To further improve the network prediction consistency, SEAM introduces the pixel correlation module (PCM), which captures context appearance information for each pixel and revises original CAMs by learned affinity attention maps. The SEAM is implemented by a siamese network with equivariant cross regularization (ECR) loss, which regularizes the original CAMs and the revised CAMs on different branches.

ShuffleNet V2 Downsampling Block is a block for spatial downsampling used in the ShuffleNet V2 architecture. Unlike the regular ShuffleNet V2 block, the channel split operator is removed so the number of output channels is doubled.

Skip-gram Word2Vec is an architecture for computing word embeddings. Instead of using surrounding words to predict the center word, as with CBow Word2Vec, Skip-gram Word2Vec uses the central word to predict the surrounding words. The skip-gram objective function sums the log probabilities of the surrounding $n$ words to the left and right of the target word $w\_{t}$ to produce the following objective: $$J\_\theta = \frac{1}{T}\sum^{T}\_{t=1}\sum\_{-n\leq{j}\leq{n}, \neq{0}}\log{p}\left(w\_{j+1}\mid{w\_{t}}\right)$$

Spatial Broadcast Decoder is an architecture that aims to improve disentangling, reconstruction accuracy, and generalization to held-out regions in data space. It provides a particularly dramatic benefit when applied to datasets with small objects. Source: Watters et al. Image source: Watters et al.

TD_INLINE_MATH_1 is a generalisation of TD_INLINE_MATH_2 reinforcement learning algorithms, but it employs an eligibility trace $\lambda$ and $\lambda$-weighted returns. The eligibility trace vector is initialized to zero at the beginning of the episode, and it is incremented on each time step by the value gradient, and then fades away by $\gamma\lambda$: $$ \textbf{z}\_{-1} = \mathbf{0} $$ $$ \textbf{z}\_{t} = \gamma\lambda\textbf{z}\_{t-1} + \nabla\hat{v}\left(S\_{t}, \mathbf{w}\_{t}\right), 0 \leq t \leq T$$ The eligibility trace keeps track of which components of the weight vector contribute to recent state valuations. Here $\nabla\hat{v}\left(S\_{t}, \mathbf{w}\_{t}\right)$ is the feature vector. The TD error for state-value prediction is: $$ \delta\_{t} = R\_{t+1} + \gamma\hat{v}\left\(S\_{t+1}, \mathbf{w}\_{t}\right) - \hat{v}\left(S\_{t}, \mathbf{w}\_{t}\right) $$ In TD_INLINE_MATH_1, the weight vector is updated on each step proportional to the scalar TD error and the vector eligibility trace: $$ \mathbf{w}\_{t+1} = \mathbf{w}\_{t} + \alpha\delta\mathbf{z}\_{t} $$ Source: Sutton and Barto, Reinforcement Learning, 2nd Edition

Please enter a description about the method here

The principle of parsimony, also known as Occam's razor, elucidates the preference for the simplest explanation that provides optimal results when faced with multiple options. Thus, we can assert that the principle of parsimony is justified by "the assumption that is both the simplest and contains all the necessary information required to comprehend the experiment at hand." This principle finds application in various scenarios or events in our daily lives, including predictions in Data Science models. It is widely recognized that a less complex model will produce more stable predictions, exhibit greater resilience to noise and disturbances, and be more manageable for maintenance and analysis. Additionally, reducing the number of features can lead to further cost savings by diminishing the use of sensors, lowering energy consumption, minimizing information acquisition costs, reducing maintenance requirements, and mitigating the necessity to retrain models due to feature fluctuations caused by noise, outliers, data drift, etc. The concurrent optimization of hyperparameters (HO) and feature selection (FS) for achieving Parsimonious Model Selection (PMS) is an ongoing area of active research. Nonetheless, the effective selection of appropriate hyperparameters and feature subsets presents a challenging combinatorial problem, frequently requiring the application of efficient heuristic methods.

CoVe, or Contextualized Word Vectors, uses a deep LSTM encoder from an attentional sequence-to-sequence model trained for machine translation to contextualize word vectors. $\text{CoVe}$ word embeddings are therefore a function of the entire input sequence. These word embeddings can then be used in downstream tasks by concatenating them with $\text{GloVe}$ embeddings: $$ v = \left[\text{GloVe}\left(x\right), \text{CoVe}\left(x\right)\right]$$ and then feeding these in as features for the task-specific models.

E-BRANCHFORMER: BRANCHFORMER WITH ENHANCED MERGING FOR SPEECH RECOGNITION

Inception-A is an image model block used in the Inception-v4 architecture.

Inception-B is an image model block used in the Inception-v4 architecture.

Inception-C is an image model block used in the Inception-v4 architecture.