Optical Character Recognition (OCR)

TrOCR is an end-to-end Transformer-based OCR model for text recognition with pre-trained CV and NLP models. It leverages the Transformer architecture for both image understanding and wordpiece-level text generation. It first resizes the input text image into $384 × 384$ and then the image is split into a sequence of 16 patches which are used as the input to image Transformers. Standard Transformer architecture with the self-attention mechanism is leveraged on both encoder and decoder parts, where wordpiece units are generated as the recognized text from the input image.

PP-OCR is an OCR system that consists of three parts, text detection, detected boxes rectification and text recognition. The purpose of text detection is to locate the text area in the image. In PP-OCR, Differentiable Binarization (DB) is used as text detector which is based on a simple segmentation network. It integrates feature extraction and sequence modeling. It adopts the Connectionist Temporal Classification (CTC) loss to avoid the inconsistency between prediction and label.

Siamese U-Net model with a pre-trained ResNet34 architecture as an encoder for data efficient Change Detection

Fast-OCR is a new lightweight detection network that incorporates features from existing models focused on the speed/accuracy trade-off, such as YOLOv2, CR-NET, and Fast-YOLOv4.

The Fast-YOLOv4-SmallObj model is a modified version of Fast-YOLOv4 to improve the detection of small objects. Seven layers were added so that it predicts bounding boxes at 3 different scales instead of 2.

PGNet is a point-gathering network for reading arbitrarily-shaped text in real-time. It is a single-shot text spotter, where the pixel-level character classification map is learned with proposed PG-CTC loss avoiding the usage of character-level annotations. With a PG-CTC decoder, we gather high-level character classification vectors from two-dimensional space and decode them into text symbols without NMS and RoI operations involved, which guarantees high efficiency. Additionally, reasoning the relations between each character and its neighbors, a graph refinement module (GRM) is proposed to optimize the coarse recognition and improve the end-to-end performance.

CDCC-NET is a multi-task network that analyzes the detected counter region and predicts 9 outputs: eight float numbers referring to the corner positions (x0/w, y0/h, ... , x3/w, y3/h) and an array containing two float numbers regarding the probability of the counter being legible/operational or illegible/faulty.

** LayoutReader** is a sequence-to-sequence model for reading order detection that uses both textual and layout information, where the layout-aware language model LayoutLM is leveraged as an encoder. The generation step in the encoder-decoder structure tis modified to generate the reading order sequence. In the encoding stage, LayoutReader packs the pair of source and target segments into a contiguous input sequence of LayoutLM and carefully designs the self-attention mask to control the visibility between tokens. As shown in the Figure, LayoutReader allows the tokens in the source segment to attend to each other while preventing the tokens in the target segment from attending to the rightward context. If 1 means allowing and 0 means preventing, the detail of the mask $M$ is as follows: $$ M\_{i, j}= \begin{cases}1, & \text { if } i<j \text { or } i, j \in \operatorname{src} \\ 0, & \text { otherwise }\end{cases} $$ where $i, j$ are the indices in the packed input sequence, so they may be from source or target segments; $i, j \in$ src means both tokens are from source segment. In the decoding stage, since the source and target are reordered sequences, the prediction candidates can be constrained to the source segment. Therefore, we ask the model to predict the indices in the source sequence. The probability is calculated as follows: $$ \mathcal{P}\left(x_{k}=i \mid x_{<k}\right)=\frac{\exp \left(e_{i}^{T} h\_{k}+b\_{k}\right)}{\sum_{j} \exp \left(e\_{j}^{T} h_{k}+b\_{k}\right)} $$ where $i$ is an index in the source segment; $e\_{i}$ and $e\_{j}$ are the $\mathrm{i}$-th and $\mathrm{j}$-th input embeddings of the source segment; $h\_{k}$ is the hidden states at the $\mathrm{k}$-th time step; $b\_{k}$ is the bias at the $\mathrm{k}$-th time step.