Chatbot

Meena is a multi-turn open-domain chatbot trained end-to-end on data mined and filtered from public domain social media conversations. This 2.6B parameter neural network is simply trained to minimize perplexity of the next token. A seq2seq model is used with the Evolved Transformer as the main architecture. The model is trained on multi-turn conversations where the input sequence is all turns of the context and the output sequence is the response.

ERNIE-GEN is a multi-flow sequence to sequence pre-training and fine-tuning framework which bridges the discrepancy between training and inference with an infilling generation mechanism and a noise-aware generation method. To make generation closer to human writing patterns, this framework introduces a span-by-span generation flow that trains the model to predict semantically-complete spans consecutively rather than predicting word by word. Unlike existing pre-training methods, ERNIE-GEN incorporates multi-granularity target sampling to construct pre-training data, which enhances the correlation between encoder and decoder.

Factorized Dense Synthesized Attention is a synthesized attention mechanism, similar to dense synthesized attention, but we factorize the outputs to reduce parameters and prevent overfitting. It was proposed as part of the Synthesizer architecture. The factorized variant of the dense synthesizer can be expressed as follows: $$A, B = F\_{A}\left(X\_{i}\right), F\_{B}\left(X\_{i}\right)$$ where $F\_{A}\left(.\right)$ projects input $X\_{i}$ into $a$ dimensions, $F\_B\left(.\right)$ projects $X\_{i}$ to $b$ dimensions, and $a \text{ x } b = l$. The output of the factorized module is now written as: $$ Y = \text{Softmax}\left(C\right)G\left(X\right) $$ where $C = H\_{A}\left(A\right) * H\_{B}\left(B\right)$, where $H\_{A}$, $H\_{B}$ are tiling functions and $C \in \mathbb{R}^{l \text{ x } l}$. The tiling function simply duplicates the vector $k$ times, i.e., $\mathbb{R}^{l} \rightarrow \mathbb{R}^{lk}$. In this case, $H\_{A}\left(\right)$ is a projection of $\mathbb{R}^{a} \rightarrow \mathbb{R}^{ab}$ and $H\_{B}\left(\right)$ is a projection of $\mathbb{R}^{b} \rightarrow \mathbb{R}^{ba}$. To avoid having similar values within the same block, we compose the outputs of $H\_{A}$ and $H\_{B}$.

Factorized Random Synthesized Attention, introduced with the Synthesizer architecture, is similar to factorized dense synthesized attention but for random synthesizers. Letting $R$ being a randomly initialized matrix, we factorize $R$ into low rank matrices $R\_{1}, R\_{2} \in \mathbb{R}^{l\text{ x}k}$ in the attention function: $$ Y = \text{Softmax}\left(R\_{1}R\_{2}^{T}\right)G\left(X\right) . $$ Here $G\left(.\right)$ is a parameterized function that is equivalent to $V$ in Scaled Dot-Product Attention. For each head, the factorization reduces the parameter costs from $l^{2}$ to $2\left(lk\right)$ where $k << l$ and hence helps prevent overfitting. In practice, we use a small value of $k = 8$. The basic idea of a Random Synthesizer is to not rely on pairwise token interactions or any information from individual token but rather to learn a task-specific alignment that works well globally across many samples.

PanGu-$α$ is an autoregressive language model (ALM) with up to 200 billion parameters pretrained on a large corpus of text, mostly in Chinese language. The architecture of PanGu-$α$ is based on Transformer, which has been extensively used as the backbone of a variety of pretrained language models such as BERT and GPT. Different from them, there's an additional query layer developed on top of Transformer layers which aims to explicitly induce the expected output.