Language detection
Identifying which language a piece of text is written in.
- Status
- ML experiment
- Year
- 2024
- Role
- Solo
- Area
- Machine learning
In plain English
Before a computer can translate or search text, it often needs to know what language the text is in. This project compared a simple, classical approach with a large modern model on a dataset covering 28 languages.
How it works
- Clean the text by lowercasing it and removing symbols.
- Break the text into character n-grams (short runs of letters, like "th" or "ing").
- Score them with TF-IDF (how distinctive a pattern is) and train a random forest (many decision trees voting).
- Tune the settings with grid search and compare the results with a fine-tuned XLM-RoBERTa model.
What I did
I built both approaches solo, using a public Kaggle dataset and pretrained models from Hugging Face.
What I learned
The simple character-pattern model outperformed the large transformer in my notebooks. A strong simple baseline gives an honest bar for anything fancier.