기법

A method that can recover the weights of the pre-fine-tuning model using a few low-rank (LoRA) fine-tuned models. In contrast to previous attacks that attempt to recover pre-fine-tuning capabilities, Spectral DeTuning aims to recover the exact pre-fine-tuning weights. Spectral DeTuning can exploit this vulnerability against large-scale models such as a personalized Stable Diffusion and an aligned Mistral.

Please enter a description about the method here

SpreadsheetCoder is a neural network architecture for spreadsheet formula prediction. It is a BERT-based model architecture to represent the tabular context in both row-based and column-based formats. A BERT encoder computes an embedding vector for each input token, incorporating the contextual information from nearby rows and columns. The BERT encoder is initialized from the weights pre-trained on English text corpora, which is beneficial for encoding table headers. To handle cell references, a two-stage decoding process is used inspired by sketch learning for program synthesis. The decoder first generates a formula sketch, which does not include concrete cell references, and then predicts the corresponding cell ranges to generate the complete formula

SqueezeBERT is an efficient architectural variant of BERT for natural language processing that uses grouped convolutions. It is much like BERT-base, but with positional feedforward connection layers implemented as convolutions, and grouped convolution for many of the layers.

Sscs, or Support-set Based Cross-Supervision, is a module for video grounding which consists of two main components: a discriminative contrastive objective and a generative caption objective. The contrastive objective aims to learn effective representations by contrastive learning, while the caption objective can train a powerful video encoder supervised by texts. Due to the co-existence of some visual entities in both ground-truth and background intervals, i.e., mutual exclusion, naively contrastive learning is unsuitable to video grounding. This problem is addressed by boosting the cross-supervision with the support-set concept, which collects visual information from the whole video and eliminates the mutual exclusion of entities. Specifically, in the Figure to the right, two video-text pairs { $V\_{i}, L\_{i}$}, {$V\_{j} , L\_{j}$ } in the batch are presented for clarity. After feeding them into a video and text encoder, the clip-level and sentence-level embedding ( {$X\_{i}, Y\_{i}$} and {$X\_{j} , Y\_{j}$} ) in a shared space are acquired. Base on the support-set module, the weighted average of $X\_{i}$ and $X\_{j}$ is computed to obtain $\bar{X}\_{i}$, $\bar{X}\_{j}$ respectively. Finally, the contrastive and caption objectives are combined to pull close the representations of the clips and text from the same samples and push away those from other pairs

$s \cdot (\mathrm{ReLU}(x))^2 + b$ where $s \in \mathbb{R}$ and $b \in \mathbb{R}$ are shared for all channels and can be set as constants (s=0.8944, b=-0.4472) or learnable parameters.

State-Aware Tracker is a pipeline for semi-supervised video object segmentation. It takes each target object as a tracklet, which not only makes the pipeline more efficient but also filters distractors to facilitate target modeling. For more stable and robust performance over video sequences, SAT gets awareness for each state and makes self-adaptation via two feedback loops. One loop assists SAT in generating more stable tracklets. The other loop helps to construct a more robust and holistic target representation.

StoGCN is a control variate based algorithm which allow sampling an arbitrarily small neighbor size. Presents new theoretical guarantee for the algorithms to converge to a local optimum of GCN.

A Strided EESP unit is based on the EESP Unit but is modified to learn representations more efficiently at multiple scales. Depth-wise dilated convolutions are given strides, an average pooling operation is added instead of an identity connection, and the element-wise addition operation is replaced with a concatenation operation, which helps in expanding the dimensions of feature maps efficiently.

A large amount of information is stored in data tables. Users can search for data tables using a keyword-based query. A table is composed primarily of data values that are organized in rows and columns providing implicit structural information. A table is usually accompanied by secondary information such as the caption, page title, etc., that form the textual information. Understanding the connection between the textual and structural information is an important yet neglected aspect in table retrieval as previous methods treat each source of information independently. In addition, users can search for data tables that are similar to an existing table, and this setting can be seen as a content-based table retrieval. In this paper, we propose StruBERT, a structure-aware BERT model that fuses the textual and structural information of a data table to produce context-aware representations for both textual and tabular content of a data table. StruBERT features are integrated in a new end-to-end neural ranking model to solve three table-related downstream tasks: keyword- and content-based table retrieval, and table similarity. We evaluate our approach using three datasets, and we demonstrate substantial improvements in terms of retrieval and classification metrics over state-of-the-art methods.

StyleMapGAN is a generative adversarial network for real-time image editing. The intermediate latent space has spatial dimensions, and a spatially variant modulation replaces AdaIN. It aims to make the embedding through an encoder more accurate than existing optimization-based methods while maintaining the properties of GANs.

Despite the tantalizing success in a broad of vision tasks, transformers have not yet demonstrated on-par ability as ConvNets in high-resolution image generative modeling. In this paper, we seek to explore using pure transformers to build a generative adversarial network for high-resolution image synthesis. To this end, we believe that local attention is crucial to strike the balance between computational efficiency and modeling capacity. Hence, the proposed generator adopts Swin transformer in a style-based architecture. To achieve a larger receptive field, we propose double attention which simultaneously leverages the context of the local and the shifted windows, leading to improved generation quality. Moreover, we show that offering the knowledge of the absolute position that has been lost in window-based transformers greatly benefits the generation quality. The proposed StyleSwin is scalable to high resolutions, with both the coarse geometry and fine structures benefit from the strong expressivity of transformers. However, blocking artifacts occur during high-resolution synthesis because performing the local attention in a block-wise manner may break the spatial coherency. To solve this, we empirically investigate various solutions, among which we find that employing a wavelet discriminator to examine the spectral discrepancy effectively suppresses the artifacts. Extensive experiments show the superiority over prior transformer-based GANs, especially on high resolutions, e.g., 1024x1024. The StyleSwin, without complex training strategies, excels over StyleGAN on CelebA-HQ 1024x1024, and achieves on-par performance on FFHQ 1024x1024, proving the promise of using transformers for high-resolution image generation.

This is a general approach to convert a neural network into an analytic equation. The technique works as follows: 1. Encourage sparse latent representations 2. Apply symbolic regression to approximate the transformations between in/latent/out layers 3. Compose the symbolic expressions. In the paper, we show that we find the correct known equations, including force laws and Hamiltonians, can be extracted from the neural network. We then apply our method to a non-trivial cosmology example-a detailed dark matter simulation-and discover a new analytic formula which can predict the concentration of dark matter from the mass distribution of nearby cosmic structures. The symbolic expressions extracted from the GNN using our technique also generalized to out-of-distribution data better than the GNN itself. Our approach offers alternative directions for interpreting neural networks and discovering novel physical principles from the representations they learn.

A Synaptic Neural Network (SynaNN) consists of synapses and neurons. Inspired by the synapse research of neuroscience, we built a synapse model with a nonlinear and log-concave synapse function of excitatory and inhibitory probabilities of channels.

Syntax Heat Parse Tree are heatmaps over parse trees, similar to "heat trees" in biology.

It is often desirable to distill the capabilities of large language models (LLMs) into smaller student models due to compute and memory constraints. One way to do this for classification tasks is via dataset synthesis. Prior approaches to synthesis use few-shot prompting, which relies on the LLM's parametric knowledge to generate usable examples. However, this leads to issues of repetition, bias towards popular entities, and stylistic differences from human text. We propose Synthesize by Retrieval and Refinement (SynthesizRR), which uses retrieval augmentation to introduce variety into the dataset synthesis process: as retrieved passages vary, the LLM is seeded with different content to generate its examples. We find that SynthesizRR greatly improves lexical and semantic diversity, similarity to human-written text, and distillation performance,

The Analog Activation Function, or TAAF, introduces a novel approach to non-linear transformations within neural networks by employing a carefully constructed ratio of exponential functions. At its core, TAAF leverages the decaying nature of the exponential function in two distinct ways to shape its activation profile. Imagine a mathematical dance between two exponential curves, one that decays linearly with increasing input, and another that decays quadratically, much faster as the input moves away from zero. TAAF cleverly combines these behaviours in a ratio, where the numerator reflects a standard exponential decay, and the denominator sums this with the rapidly decaying quadratic exponential. This interplay creates a unique, asymmetric S-shaped curve for the activation function. For very negative inputs, the function approaches a constant value, as one exponential term dominates the behaviour. Around zero, the function exhibits a non-linear transition, carefully sculpted by the interaction of both exponential components. As the input becomes increasingly positive, the function smoothly approaches another constant value, completing its characteristic S-shape, but with a distinct asymmetry not found in typical sigmoid or tanh functions. This mathematical design aims to provide a smooth, bounded, and subtly asymmetric non-linearity, potentially leading to more nuanced and stable training dynamics within deep learning models compared to simpler activation functions.

TABBIE is a pretraining objective (corrupt cell detection) that learns exclusively from tabular data. Unlike other approaches, TABBIE provides embeddings of all table substructures (cells, rows, and columns). TABBIE can be seen as a table embedding model trained to detect corrupted cells, inspired by the ELECTRA objective function.

A method to convert a Tree Ensemble model into a Rule list. This makes the AI model more transparent.

A TResNet is a variant on a ResNet that aim to boost accuracy while maintaining GPU training and inference efficiency. They contain several design tricks including a SpaceToDepth stem, Anti-Alias downsampling, In-Place Activated BatchNorm, Blocks selection and squeeze-and-excitation layers.

TSRUc, or Transformation-based Spatial Recurrent Unit c, is a modification of a ConvGRU used in the TriVD-GAN architecture for video generation. Instead of computing the reset gate $r$ and resetting $h\_{t−1}$, the TSRUc computes the parameters of a transformation $\theta$, which we use to warp $h\_{t−1}$. The rest of our model is unchanged (with $\hat{h}\_{t-1}$ playing the role of $h'\_{t}$ in $c$’s update equation from ConvGRU. The TSRUc module is described by the following equations: $$ \theta\_{h,x} = f\left(h\_{t−1}, x\_{t}\right) $$ $$ \hat{h}\_{t-1} = w\left(h\_{t-1}; \theta\_{h, x}\right) $$ $$ c = \rho\left(W\_{c} \star\_{n}\left[\hat{h}\_{t-1};x\_{t}\right] + b\_{c} \right) $$ $$ u = \sigma\left(W\_{u} \star\_{n}\left[h\_{t-1};x\_{t}\right] + b\_{u} \right) $$ $$ h\_{t} = u \odot h\_{t-1} + \left(1-u\right) \odot c $$ In these equations $\sigma$ and $\rho$ are the elementwise sigmoid and ReLU functions respectively and the $\star\_{n}$ represents a convolution with a kernel of size $n \times n$. Brackets are used to represent a feature concatenation.

TSRUp, or Transformation-based Spatial Recurrent Unit p, is a modification of a ConvGRU used in the TriVD-GAN architecture for video generation. It largely follows TSRUc, but computes $\theta$, $u$ and $c$ in parallel given $x\_{t}$ and $h\_{t−1}$, yielding the following replacement for the $c$ update equation: $$ c = \rho\left(W\_{c} \star\_{n}\left[h\_{t-1}; x\_{t}\right] + b\_{c} \right) $$ In these equations $\sigma$ and $\rho$ are the elementwise sigmoid and ReLU functions respectively and the $\star\_{n}$ represents a convolution with a kernel of size $n \times n$. Brackets are used to represent a feature concatenation.

TSRUs, or Transformation-based Spatial Recurrent Unit p, is a modification of a ConvGRU used in the TriVD-GAN architecture for video generation. It largely follows TSRUc, but computes each intermediate output in a fully sequential manner: like in TSRUc, $c$ is given access to $\hat{h}\_{t-1}$, but additionally, $u$ is given access to both outputs $\hat{h}\_{t-1}$ and $c$, so as to make an informed decision prior to mixing. This yields the following replacement for $u$: $$ u = \sigma\left(W\_{u} \star\_{n}\left[\hat{h}\_{t-1};c\right] + b\_{u} \right) $$ In these equations $\sigma$ and $\rho$ are the elementwise sigmoid and ReLU functions respectively and the $\star\_{n}$ represents a convolution with a kernel of size $n \times n$. Brackets are used to represent a feature concatenation.

A Time-aware Large Kernel (TaLK) convolution is a type of temporal convolution that learns the kernel size of a summation kernel for each time-step instead of learning the kernel weights as in a typical convolution operation. For each time-step, a function is responsible for predicting the appropriate size of neighbor representations to use in the form of left and right offsets relative to the time-step.

Unlike synthetic data generators that rely on increasingly complex and resource-heavy architectures, TabularARGN adopts a more focused and efficient model design. These design choices result in: * High Fidelity: TabularARGN achieves synthetic data quality on par with state-of-the-art (SOTA) models * Privacy by Design: TabularARGN only considers privacy-preserving value ranges for sampling, and has built-in privacy protection features. Plus can be trained via DP-SGD for obtaining differential privacy guarantees. * Simplicity: TabularARGN leverages existing building blocks, and thus can be easily implemented within standard deep learning frameworks. * Compute Efficiency: With training speeds up to 100x faster, TabularARGN scales effectively, even for large and complex datasets. * Sampling Flexibility: TabularARGN supports advanced sampling capabilities, including: * Conditional generation to create targeted datasets. * Missing value imputation to handle incomplete data seamlessly. * Fairness adjustments to align with ethical data synthesis goals. * Controlling sampling probabilities via temperature adjustments to balance rule-adherence with data diversity. * Data Versatility: TabularARGN accommodates the heterogeneity of real-world tabular datasets, including: * Multi-variate, mixed-type data (categorical, numerical, date-time, geo-spatial). * Multi-sequence datasets with varying sequence lengths and varying time intervals. * Missing values. * Robustness in Training: TabularARGN delivers high-quality synthetic data with default settings and remains consistent across several training runs.

Talking-Heads Attention is a variation on multi-head attention which includes linear projections across the attention-heads dimension, immediately before and after the softmax operation. In multi-head attention, the different attention heads perform separate computations, which are then summed at the end. Talking-Heads Attention breaks that separation. Two additional learned linear projections are inserted, $P\_{l}$ and $P\_{w}$, which transform the attention-logits and the attention weights respectively, moving information across attention heads. Instead of one "heads" dimension $h$ across the whole computation, we now have three separate heads dimensions: $h\_{k}$, $h$, and $h\_{v}$, which can optionally differ in size (number of "heads"). $h\_{k}$ refers to the number of attention heads for the keys and the queries. $h$ refers to the number of attention heads for the logits and the weights, and $h\_{v}$ refers to the number of attention heads for the values.

TaxoExpan is a self-supervised taxonomy expansion framework. It automatically generates a set of <query concept, anchor concept> pairs from the existing taxonomy as training data. Using such self-supervision data, TaxoExpan learns a model to predict whether a query concept is the direct hyponym of an anchor concept. TaxoExpan features: (1) a position-enhanced graph neural network that encodes the local structure of an anchor concept in the existing taxonomy, and (2) a noise-robust training objective that enables the learned model to be insensitive to the label noise in the self-supervision data.

TayPO, or Taylor Expansion Policy Optimization, refers to a set of algorithms that apply the $k$-th order Taylor expansions for policy optimization. This generalizes prior work, including TRPO as a special case. It can be thought of unifying ideas from trust-region policy optimization and off-policy corrections. Taylor expansions share high-level similarities with both trust region policy search and off-policy corrections. To get high-level intuitions of such similarities, consider a simple 1D example of Taylor expansions. Given a sufficiently smooth real-valued function on the real line $f : \mathbb{R} \rightarrow \mathbb{R}$, the $k$-th order Taylor expansion of $f\left(x\right)$ at $x\_{0}$ is $$f\_{k}\left(x\right) = f\left(x\_{0}\right)+\sum^{k}\_{i=1}\left[f^{(i)}\left(x\_{0}\right)/i!\right]\left(x−x\_{0}\right)^{i}$$ where $f^{(i)}\left(x\_{0}\right)$ are the $i$-th order derivatives at $x\_{0}$. First, a common feature shared by Taylor expansions and trust-region policy search is the inherent notion of a trust region constraint. Indeed, in order for convergence to take place, a trust-region constraint is required $|x − x\_{0}| < R\left(f, x\_{0}\right)^{1}$. Second, when using the truncation as an approximation to the original function $f\_{K}\left(x\right) \approx f\left(x\right)$, Taylor expansions satisfy the requirement of off-policy evaluations: evaluate target policy with behavior data. Indeed, to evaluate the truncation $f\_{K}\left(x\right)$ at any $x$ (target policy), we only require the behavior policy "data" at $x\_{0}$ (i.e., derivatives $f^{(i)}\left(x\_{0}\right)$).

The table uniformity approach is proposed to solve the problem of dialect determination. The method is based on consistency measurement over a table Γδ , which has been returned by parsing a CSV file with a dialect ρδ , and the dispersion of records along with the inference of raw data types from fields.

Temporal Distribution Characterization, or TDC, is a module used in the AdaRNN architecture to characterize the distributional information in a time series. Based on the principle of maximum entropy, maximizing the utilization of shared knowledge underlying a times series under temporal covariate shift can be done by finding periods which are most dissimilar to each other, which is also considered as the worst case of temporal covariate shift since the cross-period distributions are the most diverse. TDC achieves this goal for splitting the time-series by solving an optimization problem whose objective can be formulated as: $$ \max \_{0<K \leq K\_{0}} \max \_{n\_{1}, \cdots, n\_{K}} \frac{1}{K} \sum_{1 \leq i \neq j \leq K} d\left(\mathcal{D}\_{i}, \mathcal{D}\_{j}\right) $$ $$ \text { s.t. } \forall i, \Delta_{1}<\left|\mathcal{D}\_{i}\right|<\Delta_{2} ; \sum_{i}\left|\mathcal{D}\_{i}\right|=n $$ where $d$ is a distance metric, $\Delta\_{1}$ and $\Delta\_{2}$ are predefined parameters to avoid trivial solutions (e.g., very small values or very large values may fail to capture the distribution information), and $K\_{0}$ is the hyperparameter to avoid over-splitting. The metric $d(\cdot, \cdot)$ above can be any distance function, e.g., Euclidean or Editing distance, or some distribution-based distance / divergence, like MMD [14] and KL-divergence. The learning goal of the optimization problem (1) is to maximize the averaged period-wise distribution distances by searching $K$ and the corresponding periods so that the distributions of each period are as diverse as possible and the learned prediction model has better a more generalization ability.

Temporal Distribution Matching, or TDM, is a module used in the AdaRNN architecture to match the distributions of the discovered periods to build a time series prediction model $\mathcal{M}$ Given the learned time periods, the TDM module is designed to learn the common knowledge shared by different periods via matching their distributions. Thus, the learned model $\mathcal{M}$ is expected to generalize well on unseen test data compared with the methods which only rely on local or statistical information. Within the context of AdaRNN, Temporal Distribution Matching aims to adaptively match the distributions between the RNN cells of two periods while capturing the temporal dependencies. TDM introduces the importance vector $\mathbf{\alpha} \in \mathbb{R}^{\hat{V}}$ to learn the relative importance of $V$ hidden states inside the RNN, where all the hidden states are weighted with a normalized $\alpha$. Note that for each pair of periods, there is an $\mathbf{\alpha}$, and we omit the subscript if there is no confusion. In this way, we can dynamically reduce the distribution divergence of cross-periods. Given a period-pair $\left(\mathcal{D}\_{i}, \mathcal{D}\_{j}\right)$, the loss of temporal distribution matching is formulated as: $$ \mathcal{L}\_{t d m}\left(\mathcal{D}\_{i}, \mathcal{D}\_{j} ; \theta\right)=\sum_{t=1}^{V} \alpha\_{i, j}^{t} d\left(\mathbf{h}\_{i}^{t}, \mathbf{h}\_{j}^{t} ; \theta\right) $$ where $\alpha\_{i, j}^{t}$ denotes the distribution importance between the periods $\mathcal{D}\_{i}$ and $\mathcal{D}\_{j}$ at state $t$. All the hidden states of the RNN can be easily computed by following the standard RNN computation. Denote by $\delta(\cdot)$ the computation of a next hidden state based on a previous state. The state computation can be formulated as $$ \mathbf{h}\_{i}^{t}=\delta\left(\mathbf{x}\_{i}^{t}, \mathbf{h}\_{i}^{t-1}\right) $$ The final objective of temporal distribution matching (one RNN layer) is: $$ \mathcal{L}(\theta, \mathbf{\alpha})=\mathcal{L}\_{\text {pred }}(\theta)+\lambda \frac{2}{K(K-1)} \sum\_{i, j}^{i \neq j} \mathcal{L}\_{t d m}\left(\mathcal{D}\_{i}, \mathcal{D}\_{j} ; \theta, \mathbf{\alpha}\right) $$ where $\lambda$ is a trade-off hyper-parameter. Note that in the second term, we compute the average of the distribution distances of all pairwise periods. For computation, we take a mini-batch of $\mathcal{D}_{i}$ and $\mathcal{D}\_{j}$ to perform forward operation in RNN layers and concatenate all hidden features. Then, we can perform TDM using the above equation.

TopK Copy is a cross-attention guided copy mechanism for entity extraction where only the Top-$k$ important attention heads are used for computing copy distributions. The motivation is that that attention heads may not equally important, and that some heads can be pruned out with a marginal decrease in overall performance. Attention probabilities produced by insignificant attention heads may be noisy. Thus, computing copy distributions without these heads could improve the model’s ability to infer the importance of each token in the input document.

Topographic VAE is a method for efficiently training deep generative models with topographically organized latent variables. The model learns sets of approximately equivariant features (i.e. "capsules") directly from sequences and achieves higher likelihood on correspondingly transforming test sequences. The combined color/rotation transformation in input space $\tau\_{g}$ becomes encoded as a $\mathrm{Roll}$ within the capsule dimension. The model is thus able decode unseen sequence elements by encoding a partial sequence and Rolling activations within the capsules. This resembles a commutative diagram.

TradeS is an online joint detection and tracking model, coined as TRACK to DEtect and Segment, exploiting tracking clues to assist detection end-to-end. TraDeS infers object tracking offset by a cost volume, which is used to propagate previous object features for improving current object detection and segmentation.

Unsupervised knowledge distillation from a pretrained language model to itself, by alternating between its bi- and cross-encoder forms.