HaPyness: Analysis of sentiments in the population of Aragon
The Aragón Feliz team explores sentiment analysis to study how the emotional well-being of the population of Aragón is expressed.
A learning project shared by the community. The article documents its team, methods and results; it does not establish that the prototype is ready for real-world use.
This archive includes automatic translations. Audio, video and text inside images remain in their original language.
Zaragoza. First Edition. 2022
¡How our lives have changed in recent years!
A global pandemic, the attack on the US Capitol, Storm Filomena, a volcanic eruption and the outbreak of war in Europe… and, amid it all, an AI course in Zaragoza. That was the backdrop to this project in 2022.
One windy Friday afternoon, three intrepid Zaragoza residents began our journey in this area.
We are the team Happy Aragon, we come from different worlds, but we share a common interest in Artificial Intelligence, and we seek to expand and apply the knowledge acquired during the program. Saturdays AI. The team is made up of:
Fernando Jiménez
Héctor Soria
Virginia Navarro
And thanks to the help of our mentors of the program Saturdays AI, who have given us the basis, we have been able to undertake this project.
When Saturdays AI-Zaragoza asked us to develop a project based on AI, we asked ourselves several questions.
And the Covid-19 health contingency has caused exceptional circumstances with a high impact on society, massive confinement, loneliness, uncertainty, economic, health and social crisis, which has generated different types of reactions in the population, increasing the present pathologies, both physical and psychological, economic problems, among others.
Some data
According to the Confederation of Mental Health Spain in the #saludmentalvscovid19 study (https://consaludmental.org/centro-documentacion/salud-mental-covid19/)
Source Connsaludmental.org
The data is clear, this whole situation has already had a high impact on society and will probably manifest itself even more over the coming years.
It concerns us personally, but… is it a widespread concern?
Different sources show that yes, it is an issue that is of concern in all age ranges, let's see for example this survey carried out on a younger profile.
Source statista
But what could we do? How could we reverse this effect? How could we tip the balance towards the other opposite pole: Happiness?
To begin with, what is Happiness and how do we achieve that Happiness? What makes each of us happy?
According to this survey we can see what specific aspects are valued when thinking about Happiness.
Source statista
And the following questions What arose for us then were:
How were we before?
How were we during the highest peaks of the pandemic?
How did that state modulate as we navigated the waves and the different curves?
How are we really now?
Ultimately…
How do we become aware of the emotional journey we have suffered?
How have we been specifying it in language?
But above all...
How do we translate it into something measurable that we can analyze?
Are we able to anticipate and, through words, detect states and launch possible alerts that, with the proper tools, facilitate change?
What is the problem to be solved?
Our first approaches
With all the previous questions we were aware of the magnitude of the problem we were considering.
To begin with, how were we going to be able to measure all this? Is it possible to generate some indicator and timely information about the emotional impact on people and how they, consciously or unconsciously, reflect it through their language?
Is it possible through sentiment analysis algorithms and topics, detect concerns related to the health contingency, as well as other general situations that may affect each individual?
One of the main handicaps we face is the correct interpretation of language, for which there are numerous techniques and resources within NLP (Natural Language Processing). In our case, we focus on Spanish, a language in which, although widely used, truly efficient algorithms and AI models that give accurate results are still being developed and improved, unlike English.
Well, after all this reflection, this was then going to be our initial challenge.
Problem
Given the situation of pandemic lived throughout the last year we want to analyze how this situation has been able affect perception general to psychological level of the citizens of Aragon, and how has it been able to influence this state with respect to different themes (inferred or prefixed) in your comments in different media, social networks, press, etc.
Maybe too ambitious?…
Maybe so, so we tried to limit our scope and from there evaluate if it was viable to do it, so we focused on the following problem.
Bounded problem
There is a high complexity when making a sentiment analysis (need for manually labeled datasets by experts, time to perform this manual annotation, the ambiguity of language and subjectivity of emotions, impact of the language with which we are going to work and high risk of errors). Being aware of this complexity, we developed our main objectives.
What is our objective?
Realistic goal
Provide society with a tool that gives a classification or automatic labeling of Happiness and overcomes the complexity of a manual evaluation, facilitate this work and reduce ambiguity and error.
Demonstrate that the tool is viable and develop all the methodology necessary to achieve this automatic labeling.
Objetivo WOW
Develop a model that analyzes the evolution of citizen sentiment in Aragon regarding different themes (extracted from the context itself or predefined based on a standard categorization), taking into account different sources such as social networks, press, etc. And based on the analysis carried out, which make recommendations and provide tools in order to mitigate the negative impact on your emotional state.
Available to the general public, public organizations and the media.
Objective with Social Impact
Our guiding axis is included within the Sustainable Development Goals, specifically in 3. Health and well-being, although being so broad it has an impact on many more objectives.
Let's get to work. Project Development
For the execution of the project we have followed the following iterative methodology based on the CRISP-DM standard (Cross Industry Standard Process for Data Mining).
Phase I. UNDERSTANDING THE PROBLEM
At this point in the project, we work on the identification of resources, restrictions, problems and processes involved.
As we have been commenting, without a doubt the pandemic period has been a stage of high emotional impact, but in this very global concept, do we all understand these emotions in the same way?
We start with emotions
Source Universe of emotions
Language problems
We will work on written texts, so we began to consider what possible problems we could encounter when managing and analyzing this type of information.
Furthermore, our model, to the extent possible, should recognize and consider different relevant linguistic phenomena to autonomously interpret users' opinions. Thus, in practice, you should take into account and be able to recognize irony, denial and emotions, among other issues.
As a final reflection, based on all of the above, we saw clearly the difficulty in extracting emotions from texts, which led us to consider building our own dictionary and, as a first approach, considering a semi-automatic classification algorithm.
Phase II. UNDERSTANDING THE DATA
And we continue with data, the workhorse of all projects.
A. Data sources and problems in obtaining them
We did not have a labeled dataset specific to Aragon and the pandemic and we considered how to solve the problem without using texts from other areas and periods. But, even so, the available ones have been used to make a qualitative comparison of our solution.
We found different data sources that could be useful to us and we were browsing them to see which could be most valuable to us, since we needed a previously categorized dataset.
Labeled data
Initially we use the corpus of TASS, so we go to the Spanish Society for Natural Language Processing (SEPLN), through which the TASS, which is a Sentiment Analysis Workshop in Spanish organized every year http://tass.sepln.org/
The folder with the data was requested from the organization and we were granted permission to use it.
Reviewing the data we find different archives of tweets, grouped by countries and years. Not all have the same categorization, different files have different schemes depending on which edition of the TASS Some of them are labeled with their feeling (polarity — positive/negative/neutral, or with emotions like happy/sad).
The corpus files are in XML format, which we have processed and adapted so that our model understands it, transforming the data into a .csv format, “Comma Separated Values”, to be able to proceed with the next phase.
Unlabeled data
Regarding the data to evaluate the model, we followed two paths:
1) On the one hand, we used the test data from the Corpus TASS, whose file structure is similar to this:
2) and on the other hand we collect data directly from Twitter thanks to the tool Tweepy which allowed us to search for user publications on this social network through its API, during the pandemic period. The selection of publications was carried out with the following criteria: Aragon, covid, date, geolocation,…
To be able to do this process, we create a Twitter/developer account to start downloading the tweets.
We use this article as a reference:
It took a while to give us access, but we finally got it and were able to continue advancing in this part.
The file structure is similar to this:
B. Características What we would be interested in considering
And we begin to consider what variables could be relevant in our study to help us achieve our goal (although we probably do not have all of them).
Phase III. DATA PREPARATION
- Exploratory Data Analysis (EDA)
For our analysis, the initial dataset has the following description:
As we have mentioned, the corpus contains multiple levels of polarity. However, there are differences between different files, for example some files only have the levels Positive, Negative and Neutral, while others include an additional level called None.Therefore, if we wanted to work with all the files together, we would have to convert the polarity into a dichotomous (binary) variable with the values (Positive=1, Negative=0).
In our case, we started from those that started from positive-negative and then moved on to Happy-Sad, which was more suited to the categorization that we could get from our dictionary that is associated with the terms Happiness/Sadness.
- Data pretreatment and cleaning
Prior to analysis and modeling we need to do a noise elimination and normalization process.
We create the function limpia_tweet() that will eliminate stopwords and some peculiar characters from Twitter, later we adjusted this cleaning due to the noise of the data and the problems that the code was presenting to us.
1. Stopwords such as: de, con, por, …among others were eliminated.
2. Any type of mention, RTs, #, links, URLs was eliminated.
3. Any type of punctuation was removed.
4. Abbreviations and texts with cyber jargons were searched such as: “u” for “you”, “ur” for “your”, “k” for “that”, among others.
5. Emojis were eliminated, and any type of alphanumeric value, numerical characters, linked to currencies, percentages, etc...
6. The data was explored, analyzing the most common words in the Tweets.
Once the process has been done, the clean text is stored in a new 'tweets_transform' column:
- Text processing
In order to be able to mathematically treat preprocessed tweets, we have to extract and structure the information contained in the text. To do this, we will use the class sklearn.feature_extraction.CountVectorizer.
CountVectorizer converts the text column into a matrix in which each word is a column whose value is the number of times that word appears in each tweet.
This way we can work with these vectors instead of plain text.
For each tweet we will carry out the following process:
1. Tokenization: This step converts a text string into a list of words (tokens)
2. Conversion to lowercase
3. Delete stopwords. Stopwords are words that are very frequent but do not provide great syntactic value. Examples of stopwords in Spanish would be: de, por, con…
4. Semantic filters (Stemming/lemmatization) is the process by which we transform each word into its root. For example, the words marvelous, marvel or marvel share the same root and are considered the same word after stemming.
5. Creation of a bag of words.
with WordCloud We visualize the most used words in the dataset:
- Data quality verification
We check whether this data is sufficient or whether it is necessary to process some more data (data, geolocation, etc.) or construct new data based on existing information (assessment).
- Construction of a data structure for subsequent training and evaluation of the model
A. Geolocation data
B. Data Date. Transformation to adapt them to the format necessary for the evolution graph.
Phase IV. MODELING
How can we address it?
Variant 1: Solve the problem with classification techniques, specifically “sentiment analysis”, which has sentiment classes or labels: “positive”, “negative” or “neutral”. To do this, it will be necessary to obtain a data set with said tags that are related to Aragon (for example, Twitter data sets) or search for textual information (for example, regional newspapers, etc.) and previously classify them using sentiment tags (through a manual process, or automated using dictionaries).
Variant 2: Solve the problem by applying grouping techniques. In this case, it would be necessary to collect textual information (e.g. regional newspapers, Twitter, etc.) and group the information, and analyze these groups.
All the code and data used, in case you want to consult it in more detail, they are available in our Github repository: https://github.com/FJDevelop/HaPyness-Saturdays-AI
A. Dictionary-based valuation model (semi-supervised classification)
Creating a dictionary
In order to be able to do this semi-automatic classification process, we needed to have a dictionary with which to compare and classify. Given the difficulty of finding one especially valid in Spanish, we decided to create our own dictionary, and these were the steps we followed:
- Dictionary Creation initial that allows us to detect the values Happy (+1) and Sad (-1), and thereby make an automatic assessment. Source: http://www.ideasafines.com.ar/buscador-ideas-relacionadas.php
- We perform a bias analysis words (30% Positive 70% Negative).
- We carry out an assessment of the word volume 3000 (since it could influence us later on the processing and execution of the code)
- Search for alternatives. Assessment of the possibility of using a reduced dictionary.
- Decision on the final dictionary.
The dictionary is initially based on 3000 words related to 'happiness' and 'sadness'.
After intensive post-processing and formatting, the vocabulary has been adjusted to 2600 words with as little ambiguity as possible.
The 3000 words have been reviewed manually to eliminate ambiguities and errors in assessment.
In generating this dictionary, following this reference document, which explains how to extract relevant features from a textual corpus:
https://www.geeksforgeeks.org/using-countvectorizer-to-extracting-features-from-text/
We use CountVectorizer: converts texts (= processed tweets) into rows, converts them to lowercase (but preserves the accents), and in each column the words used appear (their frequency in the sentence), in alphabetical order. The example shown here is very simple.
From the resulting object we can extract:
– a dictionary with the words found (obtained directly using ‘vectorizer.vocabulary’)
– the matrix to analyze, with the frequencies (in a few simple steps)
Problem:
There were thousands of words and it took a lot of time, so, from the initial result, we decided to limit it to the most useful tabs (they are the yellow ones), enough for the “Happy/Sad” rating, since working with more values (Very happy, happy, sad, very sad) would involve much more development time and we wanted to advance in the rest of the methodology and evaluation of our solution. The input data to this process is shown below, where together with the manually prepared dictionary, the labeled corpus is analyzed and mapped to the Happiness/Sadness labels to complete the assessment vocabulary.
We create a model which executes the following algorithm:
a. Vocabulary matters
Prepare Spanish stopwords from the vocabulary
Apply steemer functions
b. Prepare the stemmed vocabulary
c. Read the tweets
d. Apply cleaning filters
e. Rate feelings about texts with the steemed vocabulary, creating a new calculated rating field.
f. Save the evaluated tweets in different files according to the comparative result against the tagged texts (Ok, errors, suspicious)
Example of comparison between semi-automatic and manual valuation
Regarding the file with the suspicious words, we performed an analysis to check if our process was working correctly.
Suspicious words
We realized that the ones that appeared made some sense, if they were removed it seemed that it could get worse, so we would need to have another list of "interesting words" to look at before removing them, which represented a considerable additional effort, so we decided to address it later if we had time.
As a result, we considered that without context the words are often ambiguous, so since with what we had obtained up to that moment, we obtained 60% correctness, and it does not seem easy to increase quantitatively, we decided to continue with the OUT_es file with the two columns (Corpus_Value and Calculated_Value) with which we could feed the AI part.
B. Traditional Machine Learning models (supervised classification)
The objective of this Machine Learning problem is to predict the sentiment of the tweets included in our file using the TASS corpus as training data (data to train the predictive model).
It should be noted that the dataset has been divided into TRAIN and EVAL/DEV
The 15 most used words before stemming
Numerical Representation
After stemming, we create a bag of words (Bag of Words, BoW), which is a representation of each document based on the unique words that appear among all available documents, from the union of all the phrases together with the number of times each word appears.
With the idea of compensating for those with a higher frequency, we apply TF-IDF (TermFrequency–Inverse DocumentFrequency), which tells us the relevance of a term in the document, taking into account the area in which it is usually used (corpus).
ML Algorithms Selection
Below we list the set of machine learning algorithms that we have considered in this phase (the original manual labeling as true classes has been used):
Random Forest Classifier
SVM
XGBoost
Application taking into account the Rating module
Random Forest
SVM
XGBoost
Since RF performs best with human assessment, in terms of classification, we wanted to improve the RF model for algorithmic assessment. To do this, we use a GridSearchCV in both types of evaluation, human and algorithmic. In the case of human assessment it seems that the result is not improved, it seems that we are at the maximum achievable value although it is also likely that it is a local maximum and not a global one.
On the other hand, in the algorithmic assessment we managed to improve the result of the model, using the same quality parameters: Accuracy, Recall, Precision and F1.
☝ hey! that we improved significantly! Let's save the model, just in case?
C. Grouping…
We applied the Kmeans model, but the truth is that it was not very useful to us, we did not get very clear conclusions nor did we obtain very clear sets of terms, perhaps we could achieve better results by carrying out additional data cleaning work or perhaps have a greater volume of tweets to be able to draw conclusions.
TSNEt-DistributedStochasticNeighborEmbedding
In order to visualize words in 2D we modify different parameters of the word2vec function. The ones that have the most influence on our system are the maximum number of words considered and the number of times a word has to be repeated in the text to be considered representative. As we can see from tsne's representation, it is difficult to draw conclusions. The text should be better cleaned to avoid dependency on accents or finish eliminating characters that only add noise such as 'ji' or 'san'.
D. Application on TEST data (unlabeled)
We follow the same steps on the obtained and not previously tagged tweets to evaluate the power of the models against previously unseen texts and from other domains. Below we see the results.
We start from an exploratory analysis of the data and their pretreatment.
We see which are the most common words
We apply the valuation module to obtain the calculated valuation:
We now apply Machine Learning models and analyze results (taking the calculated valuation as a reference):
Random Forest Classifier
SVM
XGBoost
The RF model, again, is the most promising, so we again use GridSearchCV to see if we can improve the classification.
? There is not much improvement so far, it seems that it is not very good at generalizing...
Phase V. EVALUATION OF RESULTS
All the code and data used, in case you want to consult it in more detail, are available at https://github.com/FJDevelop/HaPyness-Saturdays-AI
A. Comparison of ML models
Among the different ML models using manually labeled classes directly, the SVM is the one that yields slightly better results.
B. Comparison of ML Models vs Valuation
It can be seen that the valuation model has a performance close to traditional automatic ML models.
C. Evaluation on directly obtained tweets
Evaluation of the results of the models in relation to the evaluation of previously unlabeled tweets
Using the best pre-trained and saved models (model.pkl), some differences are observed:
Therefore: comparing the models in different scenarios
As you can see in the image, the automatic titration on a sample de novo has a lower classification capacity than in the original sample. Sample heterogeneity and less efficient cleaning leads to lower power.
D. Analysis of the variation of the perception of Happiness/Sadness
We transform the date field, and classify those tweets that belong to a pre-covid phase or that correspond to the covid stage
A greater imbalance is observed in the covid stage, with comments closer to sadness than in the precovid period, where despite sadness predominating, the results were much closer
Visualization of evolution
We see that using Machine Learning the result predicts more negative tweets, in general.
Phase VI. Deployment
To put the final models and the results obtained into production, we use Streamlit which is an open source Python library that makes it easy to create and share custom web applications for machine learning and data science.
Our application in streamlit is a Happiness evaluator which allows you to carry out the analysis of the text you enter, resulting in its evaluation, as well as the words that have determined that evaluation.
On the other hand, it allows the visualization of the evolution of feelings regarding Happiness/Sadness in the selected sample.
Conclusions
The conclusions obtained after the development and resolution of the general objective are:
With respect to the linked to our objectives:
We have managed to develop a tool of automatic labeling for Happiness and sadness with a satisfactory percentage of success (63% with respect to the corpus and 53.2% with respect to the Machine Learning model with respect to automatic labeling), which can facilitate the work and avoid the complexity of manual labeling.
Additionally, we have developed a model that analyzes the evolution of feeling in this case of citizens in Aragon regarding the emotions happiness and sadness inferred from their comments on social media, which allows us to clearly see the impact that the pandemic or other significant events have had on their perception and well-being.
At a general level, the main conclusions that we have obtained are:
– The concept of Happiness and Sadness it is not common to all people both in the expression and in the evaluation, so this can affect the evaluation due to the difficulty of standardizing the expression of a feeling.
– The complexity of interpreting tweets is so great that it can have a great impact on both the subjectivity of the person and that of the moment of the evaluation. To solve this, a vote could be carried out among several people at the vocabulary level and at the tweet level.
– Risk of confusion between positivity-negativity and happiness-sadness when evaluating words.
– Source bias chosen Twitter (social networks more biased than others, 280-character message can limit the expression of emotion, accessible to all types of the population?).
– Bias in language, which is transferred to our dictionary. Weighting of happy and sad words (30 -70).
– Risk of loss of information in pretreatment: In the process of pre-processing the tweets in order to facilitate the process, we have eliminated elements that could expand the analysis, for example, emoticons or hahtags (#), which may have distorted the final assessment (evolution phase 20.0).
– Importance of context in analysis and assessment.
– The results of the three models that we have used have been very similar, although the modelo Random Forest It seems that it is the one that best adapts to the classification and domain under study, according to the results, it may be due to the volume of the sample, although it could evolve by searching for better parameters in SVM and XGboost or testing neural networks.
– There are numerous techniques and resources within NLP (Natural Language Processing) in English, but they are not as developed in spanish, which made application difficult in our case.
Improvements and Evolution
As possible proposals for the evolution of the tool
ML Analysis Features:
- Explore and further analyze Kmeans results
- Explore and analyze the options provided by Word Embeddings
- Improve results of models with hyper parameters
- Using NLP techniques, perform semantic searches (cluster grouping, topics, profile similarity, trend prediction)
- Explore neural networks
Sources:
- Extend the analysis to a higher volume of tweets
- Apply our model to other types of sources (social networks, press,…)
- Analysis of the accounts that generate the tweets
Emotion analysis:
- Expand the dictionary to more categories of emotions
- Incorporate emoticon analysis, hashtag in the evaluation
- Incorporate analysis of sarcasm and denial in the assessment
- Incorporate context analysis (of the tweet itself and the context of the tweet itself)
- Incorporate emotional analysis
- Personalized analysis of tweets linked to a person to assess their emotional state
Display:
- Incorporate geolocation maps of tweets for analysis of ratings by area.
- Customized emotion analysis radial chart
Alerts and recommendations:
- Alert system based on emotional state.
- Specific thematic recommendations adapted to this personalized analysis.
- Generic recommendations accessible to all profiles
- Identification of critical profiles and associated alerts (suicide prevention, depression rates...)
Acknowledgments
To the entire Saturdays AI team, and above all to the team of Saturdays AI Zaragoza, especially to Rosa Montañés, Mª del Carmen Rodríguez, Rocío Aznar and Rafael del Hoyo since without them it would not have been possible.
The experience has been very pleasant and recommended, with a very effective learning methodology basada en la práctica, ¡Thank you for all your support, patience and dedication!
and to you, for having come this far in reading the article.
References:
Github
All the code and data used are available in https://github.com/FJDevelop/HaPyness-Saturdays-AI/tree/main
Vocabulary related to happiness and sadness http://www.ideasafines.com.ar/buscador-ideas-relacionadas.php
Datasets
Tass Corpus:
Personalized access to the TASS Corpus
http://tass.sepln.org/tass_data/download.php
https://gplsi.dlsi.ua.es/sepln15/es/taller-de-analisis-de-sentimientos-en-la-sepln-tass
Analysis of Feelings
NRC Word-Emotion Association Lexicon (aka EmoLex)
https://saifmohammad.com/WebPages/NRC-Emotion-Lexicon.htm
Sentiment Analysis in Python
https://neptune.ai/blog/sentiment-analysis-python-textblob-vs-vader-vs-flair
Developments
Sentimental example Analysis in Spanish
https://pybonacci.org/2015/11/24/como-hacer-analisis-de-sentimiento-en-espanol-2/
Análsis de tweets
https://www.cienciadedatos.net/documentos/py25-text-mining-python.html
Information of interest
Tool to carry out the Demo.
Streamlit: https://docs.streamlit.io/library/get-started/create-an-app
Covid information
Mental health data on coronavirus impact
https://consaludmental.org/sala-prensa/manifiesto-salud-mental-covid-19/
Survey statistical data
Members
- Fernando Jiménez
- Héctor Soria
- Virginia Navarro
Project presentation: Demoday
Repository
The code for this project can be found at GitHub
More artificial intelligence!
The mission of Saturdays.ai is to make artificial intelligence more accessible (#ai4all) through intensive courses and programs where projects for good are carried out (#ai4good).
Explore our AI master’s programme at https://saturdays.ai/master-ia-online/
If you want to learn more about artificial intelligence join our community at community.saturdays.ai or visit us on our website www.saturdays.ai Join us!
A history of the Saturdays.AI community. Originally published on Apr 7, 2022.
← More community stories
