Handwritten Text Recognition (HTR) is the task of automatically identifying and transcribing handwritten text from images or scanned documents into machine-readable text. The goal is to develop a system capable of accurately interpreting diverse handwriting styles, accounting for variations in alignment, stroke, spacing, and noise. This task involves detecting handwritten regions within an image, extracting the text content, and converting it into a structured digital format, enabling further search, indexing, or data analysis.
TrOCR is an end-to-end Transformer-based OCR model for text recognition with pre-trained CV and NLP models. It leverages the Transformer architecture for both image understanding and wordpiece-level text generation. It first resizes the input text image into $384 × 384$ and then the image is split into a sequence of 16 patches which are used as the input to image Transformers. Standard Transformer architecture with the self-attention mechanism is leveraged on both encoder and decoder parts, where wordpiece units are generated as the recognized text from the input image.