TurkicNLP: An NLP Toolkit for Turkic Languages

Turkic languages are spoken by about 200 million people and are written in Latin, Cyrillic, and Perso-Arabic scripts, yet the NLP tools that exist for them are scattered across languages, scripts, and incompatible software. We present TurkicNLP, an open-source Python library that offers a single pipeline for 24 Turkic languages, covering tokenization, morphological analysis, POS tagging, dependency parsing, named entity recognition, language identification, sentence embeddings, and machine translation. The library is designed for cross-lingual use: it detects and converts between scripts, maps 23 languages into the shared Common Turkic Alphabet, and includes multilingual models that share parameters across the family and transfer to languages without annotated data. Our evaluation shows the benefits of this design. The multilingual parser outperforms per-language parsers, the multilingual morphological analyzer complements finite-state analyzers where their lexicons are small, and both produce usable analyses for languages unseen in training. Transliteration is almost fully reversible, and converting text to the Common Turkic Alphabet removes most of the additional tokenization cost that large language models incur on Cyrillic- and Arabic-script Turkic languages. Code, models, and documentation are available at: https://github.com/turkic-nlp/turkicnlp.