기법

The Invertible 1x1 Convolution is a type of convolution used in flow-based generative models that reverses the ordering of channels. The weight matrix is initialized as a random rotation matrix. The log-determinant of an invertible 1 × 1 convolution of a $h \times w \times c$ tensor $h$ with $c \times c$ weight matrix $\mathbf{W}$ is straightforward to compute: $$ \log | \text{det}\left(\frac{d\text{conv2D}\left(\mathbf{h};\mathbf{W}\right)}{d\mathbf{h}}\right) | = h \cdot w \cdot \log | \text{det}\left(\mathbf{W}\right) | $$

Linear classifier using random convolutional kernels applied to time series.

Chain-of-thought prompts contain a series of intermediate reasoning steps, and they are shown to significantly improve the ability of large language models to perform certain tasks that involve complex reasoning (e.g., arithmetic, commonsense reasoning, symbolic reasoning, etc.)

How Do I Get a Human at Expedia? If you’re having trouble with an Expedia booking—such as a flight change, refund delay, or technical issue—the fastest way to resolve it is to speak directly with a human agent by calling +1-805>330>4056.. While Expedia offers automated menus and digital support options, reaching a live person ensures your concern is handled promptly and effectively. When you dial +1-805>330>4056., follow these tips to get past the automated system: ● Press “0” repeatedly or say “Agent” or “Representative” ● Avoid long voice prompts; repeat “Talk to someone” ● Call during off-peak hours (early morning or late evening) These steps can help you bypass long hold times and connect faster with a real Expedia support agent at +1-805>330>4056.. When to Speak to a Human at Expedia Contacting a live agent at +1-805>330>4056. is best for the following situations: ● Refunds not received ● Cancellations or itinerary issues ● Name or flight changes ● Missing reservations or booking errors ● Unexpected charges or billing problems While Expedia’s Help Center and app offer helpful tools, calling +1-805>330>4056. provides more personalized and immediate assistance. What You Should Have Ready Before Calling +1-805>330>4056. To make your call more efficient, gather the following: ● Your booking confirmation number ● The email address used to make your Expedia reservation ● A clear description of the issue ● Any supporting documents (e.g., screenshots or receipts) This information will help the representative at +1-805>330>4056. resolve your issue more quickly. Other Ways to Reach a Human at Expedia If you’re unable to connect via phone, here are some alternatives—though phone remains the most direct method via +1-805>330>4056.: ● Live Chat: Available in the Help Center (expedia.com/help) ● Social Media: Contact Expedia through Twitter (@Expedia) or Facebook ● Callback Requests: Use the Help Center to schedule a callback Even when using online channels, it’s often recommended to follow up with a phone call to +1-805>330>4056. for thorough resolution. Final Thoughts To get a human at Expedia, call +1-805>330>4056. and use the voice or keypad shortcuts to reach a live agent. Whether you're dealing with a refund delay, flight issue, or app problem, speaking to someone directly at +1-805>330>4056. ensures your case is handled with urgency and clarity.

RandomHorizontalFlip is a type of image data augmentation which horizontally flips a given image with a given probability. Image Credit: Apache MXNet

The Huber loss function describes the penalty incurred by an estimation procedure f. Huber (1964) defines the loss function piecewise by[1] L δ ( a ) = { 1 2 a 2 for | a | ≤ δ , δ ⋅ ( | a | − 1 2 δ ) , otherwise. {\displaystyle L_{\delta }(a)={\begin{cases}{\frac {1}{2}}{a^{2}}&{\text{for }}|a|\leq \delta ,\\delta \cdot \left(|a|-{\frac {1}{2}}\delta \right),&{\text{otherwise.}}\end{cases}}} This function is quadratic for small values of a, and linear for large values, with equal values and slopes of the different sections at the two points where | a | = δ |a|=\delta . The variable a often refers to the residuals, that is to the difference between the observed and predicted values a = y − f ( x ) a=y-f(x), so the former can be expanded to[2] L δ ( y , f ( x ) ) = { 1 2 ( y − f ( x ) ) 2 for | y − f ( x ) | ≤ δ , δ ⋅ ( | y − f ( x ) | − 1 2 δ ) , otherwise. {\displaystyle L_{\delta }(y,f(x))={\begin{cases}{\frac {1}{2}}(y-f(x))^{2}&{\text{for }}|y-f(x)|\leq \delta ,\\delta \ \cdot \left(|y-f(x)|-{\frac {1}{2}}\delta \right),&{\text{otherwise.}}\end{cases}}} The Huber loss is the convolution of the absolute value function with the rectangular function, scaled and translated. Thus it "smoothens out" the former's corner at the origin. .. math:: \ell(x, y) = L = {l_1, ..., l_N}^T with .. math:: l_n = \begin{cases} 0.5 (x_n - y_n)^2, & \text{if } |x_n - y_n| < delta \ delta * (|x_n - y_n| - 0.5 * delta), & \text{otherwise } \end{cases}

Layer-wise Adaptive Rate Scaling, or LARS, is a large batch optimization technique. There are two notable differences between LARS and other adaptive algorithms such as Adam or RMSProp: first, LARS uses a separate learning rate for each layer and not for each weight. And second, the magnitude of the update is controlled with respect to the weight norm for better control of training speed. $$m\_{t} = \beta\_{1}m\_{t-1} + \left(1-\beta\_{1}\right)\left(g\_{t} + \lambda{x\_{t}}\right)$$ $$x\_{t+1}^{\left(i\right)} = x\_{t}^{\left(i\right)} - \eta\_{t}\frac{\phi\left(|| x\_{t}^{\left(i\right)} ||\right)}{|| m\_{t}^{\left(i\right)} || }m\_{t}^{\left(i\right)} $$

The Griffin-Lim Algorithm (GLA) is a phase reconstruction method based on the redundancy of the short-time Fourier transform. It promotes the consistency of a spectrogram by iterating two projections, where a spectrogram is said to be consistent when its inter-bin dependency owing to the redundancy of STFT is retained. GLA is based only on the consistency and does not take any prior knowledge about the target signal into account. This algorithm expects to recover a complex-valued spectrogram, which is consistent and maintains the given amplitude $\mathbf{A}$, by the following alternative projection procedure: $$ \mathbf{X}^{[m+1]} = P\_{\mathcal{C}}\left(P\_{\mathcal{A}}\left(\mathbf{X}^{[m]}\right)\right) $$ where $\mathbf{X}$ is a complex-valued spectrogram updated through the iteration, $P\_{\mathcal{S}}$ is the metric projection onto a set $\mathcal{S}$, and $m$ is the iteration index. Here, $\mathcal{C}$ is the set of consistent spectrograms, and $\mathcal{A}$ is the set of spectrograms whose amplitude is the same as the given one. The metric projections onto these sets $\mathcal{C}$ and $\mathcal{A}$ are given by: $$ P\_{\mathcal{C}}(\mathbf{X}) = \mathcal{GG}^{†}\mathbf{X} $$ $$ P\_{\mathcal{A}}(\mathbf{X}) = \mathbf{A} \odot \mathbf{X} \oslash |\mathbf{X}| $$ where $\mathcal{G}$ represents STFT, $\mathcal{G}^{†}$ is the pseudo inverse of STFT (iSTFT), $\odot$ and $\oslash$ are element-wise multiplication and division, respectively, and division by zero is replaced by zero. GLA is obtained as an algorithm for the following optimization problem: $$ \min\_{\mathbf{X}} || \mathbf{X} - P\_{\mathcal{C}}\left(\mathbf{X}\right) ||^{2}\_{\text{Fro}} \text{ s.t. } \mathbf{X} \in \mathcal{A} $$ where $ || · ||\_{\text{Fro}}$ is the Frobenius norm. This equation minimizes the energy of the inconsistent components under the constraint on amplitude which must be equal to the given one. Although GLA has been widely utilized because of its simplicity, GLA often involves many iterations until it converges to a certain spectrogram and results in low reconstruction quality. This is because the cost function only requires the consistency, and the characteristics of the target signal are not taken into account.

HRNet, or High-Resolution Net, is a general purpose convolutional neural network for tasks like semantic segmentation, object detection and image classification. It is able to maintain high resolution representations through the whole process. We start from a high-resolution convolution stream, gradually add high-to-low resolution convolution streams one by one, and connect the multi-resolution streams in parallel. The resulting network consists of several ($4$ in the paper) stages and the $n$th stage contains $n$ streams corresponding to $n$ resolutions. The authors conduct repeated multi-resolution fusions by exchanging the information across the parallel streams over and over.

A Pyramid Pooling Module is a module for semantic segmentation which acts as an effective global contextual prior. The motivation is that the problem of using a convolutional network like a ResNet is that, while the receptive field is already larger than the input image, the empirical receptive field is much smaller than the theoretical one especially on high-level layers. This makes many networks not sufficiently incorporate the momentous global scenery prior. The PPM is an effective global prior representation that addresses this problem. It contains information with different scales and varying among different sub-regions. Using our 4-level pyramid, the pooling kernels cover the whole, half of, and small portions of the image. They are fused as the global prior. Then we concatenate the prior with the original feature map in the final part.

The methon to overcome catastrophic forgetting in neural network while continual learning

There are at least eight notable examples of models from the literature that can be described using the Message Passing Neural Networks (MPNN) framework. For simplicity we describe MPNNs which operate on undirected graphs $G$ with node features $x_{v}$ and edge features $e_{vw}$. It is trivial to extend the formalism to directed multigraphs. The forward pass has two phases, a message passing phase and a readout phase. The message passing phase runs for $T$ time steps and is defined in terms of message functions $M_{t}$ and vertex update functions $U_{t}$. During the message passing phase, hidden states $h_{v}^{t}$ at each node in the graph are updated based on messages $m_{v}^{t+1}$ according to $$ m_{v}^{t+1} = \sum_{w \in N(v)} M_{t}(h_{v}^{t}, h_{w}^{t}, e_{vw}) $$ $$ h_{v}^{t+1} = U_{t}(h_{v}^{t}, m_{v}^{t+1}) $$ where in the sum, $N(v)$ denotes the neighbors of $v$ in graph $G$. The readout phase computes a feature vector for the whole graph using some readout function $R$ according to $$ \hat{y} = R(\\{ h_{v}^{T} | v \in G \\}) $$ The message functions $M_{t}$, vertex update functions $U_{t}$, and readout function $R$ are all learned differentiable functions. $R$ operates on the set of node states and must be invariant to permutations of the node states in order for the MPNN to be invariant to graph isomorphism.

MobileNet is a type of convolutional neural network designed for mobile and embedded vision applications. They are based on a streamlined architecture that uses depthwise separable convolutions to build lightweight deep neural networks that can have low latency for mobile and embedded devices.

TransE is an energy-based model that produces knowledge base embeddings. It models relationships by interpreting them as translations operating on the low-dimensional embeddings of the entities. Relationships are represented as translations in the embedding space: if $\left(h, \mathcal{l}, t\right)$ holds, the embedding of the tail entity $t$ should be close to the embedding of the head entity $h$ plus some vector that depends on the relationship $\mathcal{l}$.

Adaptive Softmax is a speedup technique for the computation of probability distributions over words. The adaptive softmax is inspired by the class-based hierarchical softmax, where the word classes are built to minimize the computation time. Adaptive softmax achieves efficiency by explicitly taking into account the computation time of matrix-multiplication on parallel systems and combining it with a few important observations, namely keeping a shortlist of frequent words in the root node and reducing the capacity of rare words.

Barlow Twins is a self-supervised learning method that applies redundancy-reduction — a principle first proposed in neuroscience — to self supervised learning. The objective function measures the cross-correlation matrix between the embeddings of two identical networks fed with distorted versions of a batch of samples, and tries to make this matrix close to the identity. This causes the embedding vectors of distorted version of a sample to be similar, while minimizing the redundancy between the components of these vectors. Barlow Twins does not require large batches nor asymmetry between the network twins such as a predictor network, gradient stopping, or a moving average on the weight updates. Intriguingly it benefits from very high-dimensional output vectors.

A Deep Belief Network (DBN) is a multi-layer generative graphical model. DBNs have bi-directional connections (RBM-type connections) on the top layer while the bottom layers only have top-down connections. They are trained using layerwise pre-training. Pre-training occurs by training the network component by component bottom up: treating the first two layers as an RBM and training, then treating the second layer and third layer as another RBM and training for those parameters. Source: Origins of Deep Learning Image Source: Wikipedia

Local SGD is a distributed training technique that runs SGD independently in parallel on different workers and averages the sequences only once in a while.

DeepWalk learns embeddings (social representations) of a graph's vertices, by modeling a stream of short random walks. Social representations are latent features of the vertices that capture neighborhood similarity and community membership. These latent representations encode social relations in a continuous vector space with a relatively small number of dimensions. It generalizes neural language models to process a special language composed of a set of randomly-generated walks. The goal is to learn a latent representation, not only a probability distribution of node co-occurrences, and so as to introduce a mapping function $\Phi \colon v \in V \mapsto \mathbb{R}^{|V|\times d}$. This mapping $\Phi$ represents the latent social representation associated with each vertex $v$ in the graph. In practice, $\Phi$ is represented by a $|V| \times d$ matrix of free parameters.

This study proposes an efficient metaheuristic algorithm called the Artemisinin Optimization (AO) algorithm. This algorithm draws inspiration from the process of artemisinin medicine therapy for malaria, which involves the comprehensive eradication of malarial parasites within the human body. AO comprises three optimization stages: a comprehensive eliminations phase simulating global exploration, a local clearance phase for local exploitation, and a post-consolidation phase to enhance the algorithm's ability to escape local optima. In the experimental, this paper conducts a qualitative analysis experiment on the AO, explaining its characteristics in searching for the optimal solution. Subsequently, AO is then tested on the classical IEEE CEC 2014, and the latest IEEE CEC 2022 benchmark function sets to assess its adaptability. Comparative analyses are conducted against eight well-established algorithms and eight high-performance improved algorithms. Statistical analyses of convergence curves and qualitative metrics revealed AO's robust competitiveness. Lastly, the AO is incorporated into breast cancer pathology image segmentation applications. Using 15 authentic medical images at six threshold levels, AO's segmentation performance is compared against eight distinguished algorithms. Experimental results demonstrated AO's superiority in terms of image segmentation accuracy, Feature Similarity Index (FSIM), Peak Signal-to-Noise Ratio (PSNR), and Structural Similarity Index (SSIM) over the contrast algorithms. These results emphasize AO's efficiency and its potential in real-world optimization applications. The source codes

Adaptive Input Embeddings extend the adaptive softmax to input word representations. The factorization assigns more capacity to frequent words and reduces the capacity for less frequent words with the benefit of reducing overfitting to rare words.

Exit whenever the model is confident enough allowing early exiting from hidden layers

A Residual GRU is a gated recurrent unit (GRU) that incorporates the idea of residual connections from ResNets.

CBHG is a building block used in the Tacotron text-to-speech model. It consists of a bank of 1-D convolutional filters, followed by highway networks and a bidirectional gated recurrent unit (BiGRU). The module is used to extract representations from sequences. The input sequence is first convolved with $K$ sets of 1-D convolutional filters, where the $k$-th set contains $C\_{k}$ filters of width $k$ (i.e. $k = 1, 2, \dots , K$). These filters explicitly model local and contextual information (akin to modeling unigrams, bigrams, up to K-grams). The convolution outputs are stacked together and further max pooled along time to increase local invariances. A stride of 1 is used to preserve the original time resolution. The processed sequence is further passed to a few fixed-width 1-D convolutions, whose outputs are added with the original input sequence via residual connections. Batch normalization is used for all convolutional layers. The convolution outputs are fed into a multi-layer highway network to extract high-level features. Finally, a bidirectional GRU RNN is stacked on top to extract sequential features from both forward and backward context.

Tacotron is an end-to-end generative text-to-speech model that takes a character sequence as input and outputs the corresponding spectrogram. The backbone of Tacotron is a seq2seq model with attention. The Figure depicts the model, which includes an encoder, an attention-based decoder, and a post-processing net. At a high-level, the model takes characters as input and produces spectrogram frames, which are then converted to waveforms.

Wasserstein Gradient Penalty Loss, or WGAN-GP Loss, is a loss used for generative adversarial networks that augments the Wasserstein loss with a gradient norm penalty for random samples $\mathbf{\hat{x}} \sim \mathbb{P}\_{\hat{\mathbf{x}}}$ to achieve Lipschitz continuity: $$ L = \mathbb{E}\_{\mathbf{\hat{x}} \sim \mathbb{P}\_{g}}\left[D\left(\tilde{\mathbf{x}}\right)\right] - \mathbb{E}\_{\mathbf{x} \sim \mathbb{P}\_{r}}\left[D\left(\mathbf{x}\right)\right] + \lambda\mathbb{E}\_{\mathbf{\hat{x}} \sim \mathbb{P}\_{\hat{\mathbf{x}}}}\left[\left(||\nabla\_{\tilde{\mathbf{x}}}D\left(\mathbf{\tilde{x}}\right)||\_{2}-1\right)^{2}\right]$$ It was introduced as part of the WGAN-GP overall model.

Cutout is an image augmentation and regularization technique that randomly masks out square regions of input during training. and can be used to improve the robustness and overall performance of convolutional neural networks. The main motivation for cutout comes from the problem of object occlusion, which is commonly encountered in many computer vision tasks, such as object recognition, tracking, or human pose estimation. By generating new images which simulate occluded examples, we not only better prepare the model for encounters with occlusions in the real world, but the model also learns to take more of the image context into consideration when making decisions

Embedding Dropout is equivalent to performing dropout on the embedding matrix at a word level, where the dropout is broadcast across all the word vector’s embedding. The remaining non-dropped-out word embeddings are scaled by $\frac{1}{1-p\_{e}}$ where $p\_{e}$ is the probability of embedding dropout. As the dropout occurs on the embedding matrix that is used for a full forward and backward pass, this means that all occurrences of a specific word will disappear within that pass, equivalent to performing variational dropout on the connection between the one-hot embedding and the embedding lookup. Source: Merity et al, Regularizing and Optimizing LSTM Language Models

Hierarchical Feature Fusion (HFF) is a feature fusion method employed in ESP and EESP image model blocks for degridding. In the ESP module, concatenating the outputs of dilated convolutions gives the ESP module a large effective receptive field, but it introduces unwanted checkerboard or gridding artifacts. To address the gridding artifact in ESP, the feature maps obtained using kernels of different dilation rates are hierarchically added before concatenating them (HFF). This solution is simple and effective and does not increase the complexity of the ESP module.

Transformer-XL (meaning extra long) is a Transformer architecture that introduces the notion of recurrence to the deep self-attention network. Instead of computing the hidden states from scratch for each new segment, Transformer-XL reuses the hidden states obtained in previous segments. The reused hidden states serve as memory for the current segment, which builds up a recurrent connection between the segments. As a result, modeling very long-term dependency becomes possible because information can be propagated through the recurrent connections. As an additional contribution, the Transformer-XL uses a new relative positional encoding formulation that generalizes to attention lengths longer than the one observed during training.

Computer vision is an interesting tool for animal behavior monitoring, mainly because it limits animal handling and it can be used to record various traits using only one sensor. From previous studies, this technic has shown to be suitable for various species and behavior. However it remains challenging to collect individual information, i.e. not only to detect animals and behavior on the video frames, but also to identify them. Animal identification is a prerequisite to gather individual information in order to characterize individuals and compare them. A common solution to this problem, known as multiple objects tracking, consists in detecting the animals on each video frame, and then associate detections to a unique animal ID. Association of detections between two consecutive frames are generally made to maintain coherence of the detection locations and appearances. To extract appearance information, a common solution is to use a convolutional neural network (CNN), trained on a large dataset before running the tracking algorithm. For farmed animals, designing such network is challenging as far as large training dataset are still lacking. In this article, we proposed an innovative solution, where the CNN used to extract appearance information is parameterized using offline unsupervised training. The algorithm, named Wizard, was evaluated for the purpose of goats monitoring in outdoor conditions. 17 annotated videos were used, for a total of 4H30, with various number of animals on the video (from 3 to 8) and different level of color differences between animals. First, the ability of the algorithm to track the detected animals was evaluated. When animals were detected, the algorithm found the correct animal ID in 94.82% of the frames. When tracking and detection were evaluated together, we found that Wizard found the correct animal ID in 86.18% of the video length. In situations where the animal detection rate could be high, Wizard seems to be a suitable solution for individual behavior analysis experiments based on computer vision.

A regularization criterion that, differently from dropout and its variants, is deterministic rather than random. It grounds on the empirical evidence that feature descriptors with larger L2-norm and highly-active nodes are strongly correlated to confident class predictions. Thus, the criterion guides towards dropping a percentage of the most active nodes of the descriptors, proportionally to the estimated class probability

Weight Tying improves the performance of language models by tying (sharing) the weights of the embedding and softmax layers. This method also massively reduces the total number of parameters in the language models that it is applied to. Language models are typically comprised of an embedding layer, followed by a number of Transformer or LSTM layers, which are finally followed by a softmax layer. Embedding layers learn word representations, such that similar words (in meaning) are represented by vectors that are near each other (in cosine distance). [Press & Wolf, 2016] showed that the softmax matrix, in which every word also has a vector representation, also exhibits this property. This leads them to propose to share the softmax and embedding matrices, which is done today in nearly all language models. This method was independently introduced by Press & Wolf, 2016 and Inan et al, 2016. Additionally, the Press & Wolf paper proposes Three-way Weight Tying, a method for NMT models in which the embedding matrix for the source language, the embedding matrix for the target language, and the softmax matrix for the target language are all tied. That method has been adopted by the Attention Is All You Need model and many other neural machine translation models.