Classifying Bolivia’s Indigenous languages with machine learning

La Paz. 2021

We use Machine Learning techniques for language classification.

Bolivia is working to preserve its Indigenous languages, but learning resources remain difficult to find. We therefore proposed a machine-learning tool for language classification, a foundation for tasks such as translation, sentiment analysis and speech-to-text or text-to-speech conversion. The model uses natural language processing (NLP) and achieved over 99% accuracy in the project’s evaluation.

Since 2006, Bolivia has been a leader in the defense and vindication of indigenous peoples and cultures in its territory and in the world.

"Today, the 36 native languages in Bolivia are official languages. In Bolivia, an indigenous language must initially be spoken and taught" [1].

Does this work translate into better opportunities to learn and use Bolivia’s Indigenous languages?

Problem description

Although Bolivia is working to preserve its Indigenous languages, learning resources remain hard to access. Technology can help: identifying a text’s language is a first step towards translation tools and systems that convert between text and speech.
However, technology can currently be used as an ally to solve this problem and language detection is an initial and essential point to create tools for automatic text translation, text-to-speech conversion, voice-to-text, voice-to-speech, among many other applications.

Objective

Create a tool with Machine Learning capable of identifying and classifying languages originating from Bolivia, to speed up related tasks such as translation, information retrieval, etc.

Limits and scope

LIMITS: Due to the difficulty of obtaining a sufficiently large data set of the most spoken languages in Bolivia (Quechua, Guaraní, Aymara), we only focus on the Quechua language.

SCOPES: The identification tool, in a first stage, will be able to classify the language of phrases as either Quechua or Spanish.

Methodology

For language classification using Machine Learning, an incremental iterative methodology was used, which involves the following phases:

Figure 1. Methodology (Dataiku)

Data capture

After creating the dataset, we realized that it was unbalanced because the number of phrases in Quechua doubled that of Spanish, for that reason we decided to balance the data by adding Spanish phrases obtained from a Kaggle dataset.

Figure 2. Data capture (Image extracted from a website)

Pre-processing

At this stage, various forms of pre-processing were carried out, from feature engineering to vectorization. Which are described below.

Cleaning irrelevant characters

The Spanish phrases from Kaggle had characters from other languages and irrelevant for the classification, which is why before joining with the dataset that was available, all those characters from the Kaggle phrases were cleaned. Once the dataset was joined, there were still characters that did not provide information such as: digits, punctuation marks, etc. and therefore a cleaning of these characters was carried out.

Techniques implemented

While examining the dataset, we found an incorrectly tabulated entry and marked it as null. The initial dataset contained only each sentence’s text and its label, so we did not apply dimensionality reduction at that stage.

Analysis of the features

Each language has its own grammatical rules and the Quechua language is no stranger to that, therefore the rules of this language were investigated and certain interesting characteristics were noticed that differentiate it from other languages, to mention a few:

The Quechua alphabet has 28 consonants (some consonants are different from Spanish, such as: ch', chh, qh, p') and 3 vowels (a, i, u)
Quechua consonants are classified according to the mode of articulation, some of these are:

Plosives (p, t, k, q)
Aspirated (ph, th, chh, kh, qh)
Glottalized (p', t', ch', k', q')
Semiconsonants (w, y)

3. To differentiate the gender of a person, the words are used: warmi and qhari

4. The interrogation in Quechua is made by adding the suffix -chu to the word.

These and many more characteristics of Quechua grammar, as well as Spanish grammar, were taken into account to create the graphs, which allow us to corroborate these differences between the grammatical rules in the dataset.

One of the graphs we made was the frequency of vowels by language in the dataset (Figure 3). The frequencies of the vowels were calculated according to the number of characters in each sentence.

Figure 3. Vowel frequency (Own elaboration)

Other graphs we made were the frequency of consonants such as: K, H, M, R (Figure 4) and special characters such as: á, é, í, ó, ú, ä, ü, ' (Figure 4), depending on the language.

Figure 4. Character histogram (Own creation)

These graphs allowed us to clarify some doubts about the grammatical differences between the Spanish and Quechua languages, and represented a key point to carry out the pre-processing of the data.

Due to the differences in some letters and characters used for each language, in addition to certain suffixes or prefixes, numbering and other characteristics, we decided to vectorize the phrases according to the n-gram character model. To capture important characteristics in both languages, we delimit the n-gram model from 1 to 5, this for performance reasons and also because we consider that this number allows us to abstract those grammatical characteristics that we mentioned above.

Figure 5: Pre-processing and vectorization of the features (Own creation)

In addition to vectorizing the phrases according to the frequency of unique characters that each phrase has, the TF-IDF frequency was applied, a statistical measure that evaluates how relevant a term is for a document in a collection. In this case each term is represented by each character and the document is represented by the language phrase in the dataset (collection of phrases).

Model selection and evaluation

With the data ready, the prediction models were built. Classification models were used because we have a binary classification problem, there are only two possible labels, “Quechua” and “Spanish”.

Therefore, the following supervised learning algorithms were used:

Naïve Bayes
Support vector Machine
Logistic regression

To find the best combination of hyperparameters, GridSearch from the Sklearn library was used.

The confusion matrices for the 3 models are:

Figure 6. Confusion matrices, balanced accuracy and execution time of the 3 models (Own elaboration)

Classification of new data

Once we have selected our best model, it is necessary to test it with some phrases never seen before, so we test phrases that only contain words from one language, phrases that contain words from both languages that is common in the colloquial speech of Quechua speakers and finally a meaningless phrase. Our selected model behaves well with the first two types of phrases, however, when entering meaningless phrases that do not belong to either Spanish or Quechua, these phrases are directly classified as Quechua, this is because as we handle a binary model, the entered text must be tagged with one label or another.

Figure 7. Model results (own elaboration)

Conclusion

In the project’s evaluation, the support vector machine (SVM) model achieved the best predictive result, with balanced accuracy above 99%.

Specifically, the model works well for classifying Spanish and Quechua given the high precision, and f1 scores for these two languages.

Although SVM achieved the highest accuracy, logistic regression offered a good balance between accuracy and execution time: it also exceeded 99% and ran faster, which matters for real-time applications.

It was possible to abstract some characteristics of the Quechua language, making it possible to carry out the same analysis with other languages originating from Bolivia.

Future work

Although this problem appears to be simple, it is a necessary step to: