BM-TOK
Balancing multilingual token vocabularies
How to build the vocabulary of a multilingual language model so that less-resourced languages and non-Latin scripts are represented fairly.
Tanja works on the structure of text data and multilingual text processing. Her research is driven by the idea that establishing scientific facts about information density in text is a way towards language models that are fairer across languages, more robust and more sustainable.
She holds a PhD in computational linguistics from the University of Geneva. Before joining IDSIA in 2025, she led the Text Group and co-directed the Language and Space Lab at the University of Zurich (2013–2024), and was a visiting scholar at the University of Cambridge and the IT University of Copenhagen.
Projects
ELDI
Dialect identification in short texts
Developed methods to recognise the dialect of short texts such as social media posts, focusing on varieties of Arabic.
INCdid
Dialect identification in noisy text
Developed methods to identify dialects in short, noisy texts, especially between closely related language varieties.