기법

SC-GPT is a multi-layer Transformer neural language model, trained in three steps: (i) Pre-trained on plain text, similar to GPT-2; (ii) Continuously pretrained on large amounts of dialog-act labeled utterances corpora to acquire the ability of controllable generation; (iii) Fine-tuned for a target domain using very limited amounts of domain labels. Unlike GPT-2, SC-GPT generates semantically controlled responses that are conditioned on the given semantic form, similar to SC-LSTM but requiring much less domain labels to generalize to new domains. It is pre-trained on a large set of annotated NLG corpus to acquire the controllable generation ability, and fine-tuned with only a few domain-specific labels to adapt to new domains.

SCARLET-NAS is a type of neural architecture search that utilises a learnable stabilizer to calibrate feature deviation, named the Equivariant Learnable Stabilizer (ELS). Previous one-shot approaches can be limited by fixed-depth search spaces. With SCARLET-NAS, we use the equivariant learnable stabilizer on each skip connection. This can lead to improved convergence, more reliable evaluation, and retained equivalence. The third benefit is deemed most important by the authors for scalability.

Spatial CNN with UNet based Encoder-decoder and ConvLSTM

SERLU, or Scaled Exponentially-Regularized Linear Unit, is a type of activation function. The new function introduces a bump-shaped function in the region of negative input. The bump-shaped function has approximately zero response to large negative input while being able to push the output of SERLU towards zero mean statistically. $$ \text{SERLU}\left(x\right)) = \lambda\_{serlu}x \text{ if } x \geq 0 $$ $$ \text{SERLU}\left(x\right)) = \lambda\_{serlu}\alpha\_{serlu}xe^{x} \text{ if } x < 0 $$ where the two parameters $\lambda\_{serlu} > 0$ and $\alpha\_{serlu} > 0$ remain to be specified.

SETSe is a deterministic physics based graph embedding algorithm. It embeds weighted feature rich networks. It treats each edge as a spring and each node as a bead whose movement is constrained by the graph adjacency matrix so that the nodes move in parallel planes enforcing a minimum distance between neighboring nodes. The node features act as forces moving the nodes up and down. The network converges to the embedded state when the force produced by each node is equal and opposite to the sum of the forces exerted by its edges, creating a net force of 0. SETSe has no conventional loss function and does not attempt to place similar nodes close to each other.

Model to adapt: We use Ultra Fast Structure-aware Deep Lane Detection (UFLD) as baseline and strictly adopt its training scheme and hyperparameters. UFLD treats lane detection as a row-based classification problem and utilizes the row anchors defined by TuSimple. Unsupervised Domain Adaptation with SGPCS: SGPCS builds upon PCS and performs in-domain contrastive learning and crossdomain self-supervised learning via cluster prototypes. We reformulate the pseudo label selection mechanism from SGADA. For each lane, we select the highest confidence value from the griding cells of each row anchor. Based on their griding cell position, the confidence values are divided into two cases: absent lane points and present lane points. Thereby, the last griding cell represents absent lane points as in. For each case, we calculate the mean confidence over the corresponding lanes. We then use the thresholds defined by SGADA to decide whether the prediction is treated as a pseudo label. Our overall objective function comprises the in-domain and cross-domain loss from PCS, the losses defined by UFLD, and our adopted pseudo loss from SGADA. We adjust the momentum for memory bank feature updates to 0.5 and use spherical K-means with K = 2,500 to cluster them into prototypes.

SKEP is a self-supervised pre-training method for sentiment analysis. With the help of automatically-mined knowledge, SKEP conducts sentiment masking and constructs three sentiment knowledge prediction objectives, so as to embed sentiment information at the word, polarity and aspect level into pre-trained sentiment representation. In particular, the prediction of aspect-sentiment pairs is converted into multi-label classification, aiming to capture the dependency between words in a pair. SKEP contains two parts: (1) Sentiment masking recognizes the sentiment information of an input sequence based on automatically-mined sentiment knowledge, and produces a corrupted version by removing these informations. (2) Sentiment pre-training objectives require the transformer to recover the removed information from the corrupted version. The three prediction objectives on top are jointly optimized: Sentiment Word (SW) prediction (on $\left.\mathrm{x}\_{9}\right)$, Word Polarity (SP) prediction (on $\mathrm{x}\_{6}$ and $\mathbf{x}\_{9}$ ), Aspect-Sentiment pairs (AP) prediction (on $\mathbf{x}\_{1}$ ). Here, the smiley denotes positive polarity. Notably, on $\mathrm{x}\_{6}$, only SP is calculated without SW, as its original word has been predicted in the pair prediction on $\mathbf{x}\_{1}$.

Please enter a description about the method here

SMITH, or Siamese Multi-depth Transformer-based Hierarchical Encoder, is a Transformer-based model for document representation learning and matching. It contains several design choices to adapt self-attention models for long text inputs. For the model pre-training, a masked sentence block language modeling task is used in addition to the original masked word language model task used in BERT, to capture sentence block relations within a document. Given a sequence of sentence block representation, the document level Transformers learn the contextual representation for each sentence block and the final document representation.

SPP-Net is a convolutional neural architecture that employs spatial pyramid pooling to remove the fixed-size constraint of the network. Specifically, we add an SPP layer on top of the last convolutional layer. The SPP layer pools the features and generates fixed-length outputs, which are then fed into the fully-connected layers (or other classifiers). In other words, we perform some information aggregation at a deeper stage of the network hierarchy (between convolutional layers and fully-connected layers) to avoid the need for cropping or warping at the beginning.

In our proposed model, SPPF-AMP, besides passing the output of the convolutional layer from a sequence of max pooling layers with kernel sizes of 5, it is also passed through two average pooling layers with kernel sizes of 9 and 13. Then the output of each max pooling layer, each average pooling layer, and the convolutional layer are concatenated and passed to the next convolutional layer.

Structurally Regularized Deep Clustering, or SRDC, is a deep network based discriminative clustering method for domain adaptation that minimizes the KL divergence between predictive label distribution of the network and an introduced auxiliary one. Replacing the auxiliary distribution with that formed by ground-truth labels of source data implements the structural source regularization via a simple strategy of joint network training.

Please enter a description about the method here

Semi-Supervised Knowledge Distillation is a type of knowledge distillation for person re-identification that exploits weakly annotated data by assigning soft pseudo labels to YouTube-Human to improve models' generalization ability. SSKD first trains a student model (e.g. ResNet-50) and a teacher model (e.g. ResNet-101) using labeled data from multi-source domain datasets. Then, SSKD develops an auxiliary classifier to imitate the soft predictions of unlabeled data generated by the teacher model. Meanwhile, the student model is also supervised by hard labels and predicted soft labels by the teacher model for labeled data.

STraTA, or Self-Training with Task Augmentation, is a self-training approach that builds on two key ideas for effective leverage of unlabeled data. First, STraTA uses task augmentation, a technique that synthesizes a large amount of data for auxiliary-task fine-tuning from target-task unlabeling texts. Second, STRATA performs self-training by further fine-tuning the strong base model created by task augmentation on a broad distribution of pseudo-labeled data. In task augmentation, we train an NLI data generation model and use it to synthesize a large amount of in-domain NLI training data for each given target task, which is then used for auxiliary (intermediate) fine-tuning. The self-training algorithm iteratively learns a better model using a concatenation of labeled and pseudo-labeled examples. At each iteration, we always start with the auxiliary-task model produced by task augmentation and train on a broad distribution of pseudo-labeled data.

The ScaledSoftSign is a modification of SoftSign activation function that has trainable parameters. $$ScaledSoftSign(x) = \frac{\alpha x}{\beta + |x|}$$

Self-Adjusting Smooth L1 Loss is a loss function used in object detection that was introduced with RetinaMask. This is an improved version of Smooth L1. For Smooth L1 loss we have: $$ f(x) = 0.5 \frac{x^{2}}{\beta} \text{ if } |x| < \beta $$ $$ f(x) = |x| -0.5\beta \text{ otherwise } $$ Here a point $\beta$ splits the positive axis range into two parts: $L2$ loss is used for targets in range $[0, \beta]$, and $L1$ loss is used beyond $\beta$ to avoid over-penalizing utliers. The overall function is smooth (continuous, together with its derivative). However, the choice of control point ($\beta$) is heuristic and is usually done by hyper parameter search. Instead, with self-adjusting smooth L1 loss, inside the loss function the running mean and variance of the absolute loss are recorded. We use the running minibatch mean and variance with a momentum of $0.9$ to update these two parameters.

Seq2Edits is an open-vocabulary approach to sequence editing for natural language processing (NLP) tasks with a high degree of overlap between input and output texts. In this approach, each sequence-to-sequence transduction is represented as a sequence of edit operations, where each operation either replaces an entire source span with target tokens or keeps it unchanged. For text normalization, sentence fusion, sentence splitting & rephrasing, text simplification, and grammatical error correction, the approach improves explainability by associating each edit operation with a human-readable tag. Rather than generating the target sentence as a series of tokens, the model predicts a sequence of edit operations that, when applied to the source sentence, yields the target sentence. Each edit operates on a span in the source sentence and either copies, deletes, or replaces it with one or more target tokens. Edits are generated auto-regressively from left to right using a modified Transformer architecture to facilitate learning of long-range dependencies.

Per contattare Delta Air Lines dall’Italia, chiama il numero dedicato (+3902).305.66.320, disponibile per ricevere assistenza clienti in italiano. Questo numero ti permette di parlare direttamente con un operatore Delta per ottenere supporto su prenotazioni, check-in, modifiche di volo, richieste di rimborso o informazioni sui bagagli. Il servizio clienti è attivo nei giorni feriali, con orari che possono variare. Componendo il (+3902).305.66.320, accedi al sistema vocale interattivo e, seguendo le istruzioni, potrai rapidamente parlare con un rappresentante. Ti consigliamo di avere il numero di prenotazione a portata di mano per agevolare la procedura e ricevere un’assistenza più efficiente.

The Shifted Rectified Linear Unit, or ShiLU, is a modification of ReLU activation function that has trainable parameters. $$ShiLU(x) = \alpha ReLU(x) + \beta$$

The Shuffle Transformer Block consists of the Shuffle Multi-Head Self-Attention module (ShuffleMHSA), the Neighbor-Window Connection module (NWC), and the MLP module. To introduce cross-window connections while maintaining the efficient computation of non-overlapping windows, a strategy which alternates between WMSA and Shuffle-WMSA in consecutive Shuffle Transformer blocks is proposed. The first window-based transformer block uses regular window partition strategy and the second window-based transformer block uses window-based selfattention with spatial shuffle. Besides, the Neighbor-Window Connection moduel (NWC) is added into each block for enhancing connections among neighborhood windows. Thus the proposed shuffle transformer block could build rich cross-window connections and augments representation. Finally, the consecutive Shuffle Transformer blocks are computed as: $$ x^{l}=\mathbf{W M S A}\left(\mathbf{B N}\left(z^{l-1}\right)\right)+z^{l-1} $$ $$ y^{l}=\mathbf{N W C}\left(x^{l}\right)+x^{l} $$ $$ z^{l}=\mathbf{M L P}\left(\mathbf{B N}\left(y^{l}\right)\right)+y^{l} $$ $$ x^{l+1}=\mathbf{S h u f f l e - W M S A}\left(\mathbf{B N}\left(z^{l}\right)\right)+z^{l} $$ $$ y^{l+1}=\mathbf{N W C}\left(x^{l+1}\right)+x^{l+1} $$ $$ z^{l+1}=\mathbf{M L P}\left(\mathbf{B N}\left(y^{l+1}\right)\right)+y^{l+1} $$ where $x^l$, $y^l$ and $z^l$ denote the output features of the (Shuffle-)WMSA module, the Neighbor-Window Connection module and the MLP module for block $l$, respectively; WMSA and Shuffle-WMSA denote window-based multi-head self-attention without/with spatial shuffle, respectively.

SimAdapter is a module for explicitly learning knowledge from adapters. SimAdapter aims to learn the similarities between the source and target languages during fine-tuning using the adapters, and the similarity is based on an attention mechanism. The detailed composition of the SimAdapter is shown in the Figure. By taking the language-agnostic representations from the backbone model as the query, and the language-specific outputs from multiple adapter as the keys and values, the final output for SimAdapter over attention are computed as (For notation simplicity, we omit the layer index $l$ below): $$ \operatorname{SimAdapter}\left(\mathbf{z}, \mathbf{a}\_{\left\(S\_{1}, S\_{2}, \ldots, S\_{N}\right\)}\right)=\sum_{i=1}^{N} \operatorname{Attn}\left(\mathbf{z}, \mathbf{a}\_{S\_{i}}\right) \cdot\left(\mathbf{a}\_{S\_{i}} \mathbf{W}\_{V}\right) $$ where SimAdapter $(\cdot)$ and $\operatorname{Attn}(\cdot)$ denotes the SimAdapter and attention operations, respectively. Specifically, the attention operation is computed as: $$ \operatorname{Attn}(\mathbf{z}, \mathbf{a})=\operatorname{Softmax}\left(\frac{\left(\mathbf{z} \mathbf{W}\_{Q}\right)\left(\mathbf{a} \mathbf{W}\_{K}\right)^{\top}}{\tau}\right) $$ where $\tau$ is the temperature coefficient, $\mathbf{W}\_{Q}, \mathbf{W}\_{K}, \mathbf{W}\_{V}$ are attention matrices. Note that while $\mathbf{W}\_{Q}, \mathbf{W}\_{K}$ are initialized randomly, $\mathbf{W}\_{V}$ is initialized with a diagonal of ones and the rest of the matrix with small weights $(1 e-6)$ to retain the adapter representations. Furthermore, a regularization term is introduced to avoid drastic feature changes: $$ \mathcal{L}\_{\mathrm{reg}}=\sum\_{i, j}\left(\left(\mathbf{I}\_{V}\right)\_{i, j}-\left(\mathbf{W}\_{V}\right)_{i, j}\right)^{2} $$ where $\mathbf{I}\_{V}$ is the identity matrix with the same size as $\mathbf{W}\_{V}$

The Sinkhorn Transformer is a type of transformer that uses Sparse Sinkhorn Attention as a building block. This component is a plug-in replacement for dense fully-connected attention (as well as local attention, and sparse attention alternatives), and allows for reduced memory complexity as well as sparse attention.

Please enter a description about the method here

Soft Split and Soft Composition are video frame based operations used in the FuseFormer architecture, specifically the FuseFormer blocks. We softly split each frame into overlapped patches and then softly composite them back, by using an unfold and fold operator with patch size $k$ being greater than patch stride $s$. When compositing patches back to its original spatial shape, we add up feature values at each overlapping spatial location of neighboring patches.

SortCut Sinkhorn Attention is a variant of Sparse Sinkhorn Attention where a post-sorting truncation of the input sequence is performed, essentially performing a hard top-k operation on the input sequence blocks within the computational graph. While most attention models mainly re-weight or assign near-zero weights during training, this allows for explicitly and dynamically truncate the input sequence. Specifically: $$ Y = \text{Softmax}\left(Q{\psi\_{S}}\left(K\right)^{T}\_{\left[:n\right]}\right)\psi\_{S}\left(V\right)\_{\left[:n\right]} $$ where $n$ is the Sortfut budget hyperparameter.

Sparse Switchable Normalization (SSN) is a variant on Switchable Normalization where the importance ratios are constrained to be sparse. Unlike $\ell_1$ and $\ell_0$ constraints that impose difficulties in optimization, the constrained optimization problem is turned into feed-forward computation through SparseMax, which is a sparse version of softmax.

SpecGAN is a generative adversarial network method for spectrogram-based, frequency-domain audio generation. The problem is suited for GANs designed for image generation. The model can be approximately inverted. To process audio into suitable spectrograms, the authors perform the short-time Fourier transform with 16 ms windows and 8ms stride, resulting in 128 frequency bins, linearly spaced from 0 to 8 kHz. They take the magnitude of the resultant spectra and scale amplitude values logarithmically to better-align with human perception. They then normalize each frequency bin to have zero mean and unit variance. They clip the spectra to $3$ standard deviations and rescale to $\left[−1, 1\right]$. They then use the DCGAN approach on the result spectra.