LanguageDetection.Ai 1.0.4

Language Detection

Detect the language of a text using naive a Bayesian filter with generated language profiles from Wikipedia abstract xml, 99% over precision for 51 languages. Original author: Nakatani Shuyo.

.NET Port of Language Detection Library for Java by @shuyo Forked from TechnikEmpire/language-detection

This package has been updated to C# 11 and .NET 7 and all external dependencies has been removed. The execution has also been optimized a bit to use threads and other improvements. The algorithm is now detecting languages with all 51 languages added in 0.5 ms down from 1.12 ms. Feel free to send pull requests to this repo

The Naive Bayesian filter

The Naive Bayesian filter, which is a classification method based on Bayes' Theorem, works on the principle of considering each feature to be independent of one another. In the context of language detection, these "features" could be the frequency of certain words, characters, or n-grams (sequences of n characters) in a text.

Here's a high-level overview of how it works:

Training Phase: During training, the filter uses a set of labeled training data (in this case, the Wikipedia abstract XML data for each language) to calculate the prior probability of each class (language) and the conditional probability of each feature (word, character, or n-gram) given each class. The prior probability of a class is the overall likelihood of that class in the training set, while the conditional probability of a feature given a class is the likelihood of that feature occurring in instances of that class.

Prediction Phase: To predict the class of a new, unlabeled instance (in this case, a piece of text whose language we want to detect), the filter first transforms the instance into a feature vector. It then applies Bayes' Theorem to calculate the posterior probability of each class given this feature vector. The class with the highest posterior probability is chosen as the prediction.

In the context of this Language Detector, the "naive" assumption of the Naive Bayesian filter—that every feature is independent of every other feature—is not strictly true, as words and characters in a language are often dependent on each other. However, despite this assumption, the Naive Bayesian filter often performs well in practice and is particularly effective for language detection due to its ability to handle many features and its robustness to irrelevant features.

It's important to note that the accuracy of the Naive Bayesian filter heavily depends on the quality and representativeness of the training data. The language profiles generated from Wikipedia abstract XML data provide a broad and diverse sample of language use, contributing to the high precision of this Language Detector.

Supported Languages

This library provides language detection support for the following languages:

These language codes follow the ISO 639-2/T standard. Make sure to use the correct language code when invoking the language detection methods.

Install

dotnet add package LanguageDetection.Ai

Use

using LanguageDetection;

Load all supported languages

LanguageDetector detector = new LanguageDetector();
detector.AddAllLanguages();
Assert.Equal("dan", detector.Detect("Denne tekst er skrevet i dansk"));

or a small subset

LanguageDetector detector = new LanguageDetector();
detector.AddLanguages("dan", "eng", "swe");
Assert.Equal("dan", detector.Detect("Denne tekst er skrevet i dansk"));

You can also change parameters

LanguageDetector detector = new LanguageDetector();
detector.RandomSeed = 1;
detector.ConvergenceThreshold = 0.9;
detector.MaxIterations = 50;

License

Apache 2.0