Most existing word segmentation methods output one single segmentation solution. This paper provides an analysis of word segmentation performance when more than one solutions are taken into account. Towards this investigation, a deep neural network with multiple thresholds is applied to generate multiple candidates for segmentation. As a test-bed, the well-known bidirectional long short-term memory (BiLSTM) units are used with eleven contexts in a deep neural network. As performance indices, three measures; recall, precision and f-measure, are plotted with respect to various thresholds for both boundary level and word level evaluation. By a number of experiments, the result shows that the multi-candidate word segmentation can help us increase the recalls while maintaining the precisions.
Word segmentation is a fundamental pre-processing step for Thai Natural Language Processing. The current off-the-shelf solutions are not benchmarked consistently, so it is difficult to compare their trade-offs. We conducted a speed and accuracy comparison of the popular systems on three different domains and found that the state-of-the-art deep learning system is slow and moreover does not use sub-word structures to guide the model. Here, we propose a fast and accurate neural Thai Word Segmenter that uses dilated CNN filters to capture the environment of each character and uses syllable embeddings as features. Our system runs at least 5.6x faster and outperforms the previous state-of-the-art system on some domains. In addition, we develop the first ML-based Thai orthographical syllable segmenter, which yields syllable embeddings to be used as features by the word segmenter.