Synthetic Data Generation

Wasserstein GAN, or WGAN, is a type of generative adversarial network that minimizes an approximation of the Earth-Mover's distance (EM) rather than the Jensen-Shannon divergence as in the original GAN formulation. It leads to more stable training than original GANs with less evidence of mode collapse, as well as meaningful curves that can be used for debugging and searching hyperparameters.

Wasserstein Gradient Penalty Loss, or WGAN-GP Loss, is a loss used for generative adversarial networks that augments the Wasserstein loss with a gradient norm penalty for random samples $\mathbf{\hat{x}} \sim \mathbb{P}\_{\hat{\mathbf{x}}}$ to achieve Lipschitz continuity: $$ L = \mathbb{E}\_{\mathbf{\hat{x}} \sim \mathbb{P}\_{g}}\left[D\left(\tilde{\mathbf{x}}\right)\right] - \mathbb{E}\_{\mathbf{x} \sim \mathbb{P}\_{r}}\left[D\left(\mathbf{x}\right)\right] + \lambda\mathbb{E}\_{\mathbf{\hat{x}} \sim \mathbb{P}\_{\hat{\mathbf{x}}}}\left[\left(||\nabla\_{\tilde{\mathbf{x}}}D\left(\mathbf{\tilde{x}}\right)||\_{2}-1\right)^{2}\right]$$ It was introduced as part of the WGAN-GP overall model.

Wasserstein GAN + Gradient Penalty, or WGAN-GP, is a generative adversarial network that uses the Wasserstein loss formulation plus a gradient norm penalty to achieve Lipschitz continuity. The original WGAN uses weight clipping to achieve 1-Lipschitz functions, but this can lead to undesirable behaviour by creating pathological value surfaces and capacity underuse, as well as gradient explosion/vanishing without careful tuning of the weight clipping parameter $c$. A Gradient Penalty is a soft version of the Lipschitz constraint, which follows from the fact that functions are 1-Lipschitz iff the gradients are of norm at most 1 everywhere. The squared difference from norm 1 is used as the gradient penalty.

A group of methods designed to generate outliers in tabular data, emphasizing the importance of such techniques as supported by extreme value theory. These methods are crucial for modeling, analyzing, and understanding anomalous or rare events within structured datasets.

It is often desirable to distill the capabilities of large language models (LLMs) into smaller student models due to compute and memory constraints. One way to do this for classification tasks is via dataset synthesis. Prior approaches to synthesis use few-shot prompting, which relies on the LLM's parametric knowledge to generate usable examples. However, this leads to issues of repetition, bias towards popular entities, and stylistic differences from human text. We propose Synthesize by Retrieval and Refinement (SynthesizRR), which uses retrieval augmentation to introduce variety into the dataset synthesis process: as retrieved passages vary, the LLM is seeded with different content to generate its examples. We find that SynthesizRR greatly improves lexical and semantic diversity, similarity to human-written text, and distillation performance,