MOODY.AI

Editorial note: this project classifies facial expressions using its dataset’s labels. That task does not, by itself, establish a person’s emotions, attention or wellbeing. The results described should be read as those of an educational prototype.

Evidence on facial expressions and emotions

Guadalajara. Third Edition. 2021

Our experience at Saturdays.AI

Our team is made up of 5 people from various professional areas and levels of experience, as well as different places of origin, which further enriched our project due to the different points of view. Our project was named Moody.AI

team logo

Members:

   Raúl Sandoval

   Karla Ramírez

   Aldo Álvarez

   Israel Castillo

   Jaime Chacoff

When we formed the team, each member proposed a project. We shared an interest in image processing, so we chose to explore facial-expression classification. Our own experience of online classes initially drew us towards education and the difficulties children might face.

Initially, the project established creating a tool that would allow detecting, at an early stage, children's negative emotions during virtual classes since it would be of great help to school authorities and the children's parents.

However, reviewing the topic in more depth we came across certain barriers, such as the fact that we did not have a dataset specialized in children, which would require explicit consent from parents to be able to capture images of their faces, among other things.

By continuing to investigate the effects of the pandemic and virtual communications, we discovered that the impact is also significant on adults working remotely (home office).

Considering that we could also have a significant social impact by focusing the project on adults, it was decided to change the focus of the project and orient it to the detection of emotions in adults who are experiencing the modality of remote work (home office).

The fact that the datasets available on the Internet are mostly from adults also had an influence. Likewise, an important aspect was the possibility of testing the faces of the team members (all of legal age) and whose participation was voluntary.

In conclusion, we worked on a model for the detection of emotions through facial expression, for adults working remotely (home office). There was a focus on negative emotions, so that this tool helps the human resources areas of companies to detect the state of mind in the work environment and with this information they can take the necessary actions.

Introduction

NOM-035-STPS-2018: has the objective of establishing the elements to identify, analyze and prevent psychosocial risk factors, as well as to promote a favorable organizational environment in the workplace.

Today, what emotional health needs is more attention, visibility and less stigma. To this end, the use of Artificial Intelligence in projects with positive social impact was proposed, considering living in a time of pandemic as a factor.

Initially, the project established having a tool that would allow detecting, at an early stage, children's negative emotions during virtual classes since it would be of great help to school authorities and the children's parents.

Considering that we can have a broader social impact by including adults in this project, and taking into account that to use the faces of minors it is necessary to have consent from parents or guardians, it was decided to change the focus of the project and direct it to the detection of emotions in adults who are experiencing the modality of remote work (home office).

The problem

Due to the global COVID-19 pandemic, the population has had to remain in quarantine for more than a year. A significant percentage of companies have implemented remote work so as not to slow down operations during the pandemic. Now work is carried out in a “home office” mode, which has generated problems that have affected remote working collaborators. Among these problems, the lack of coexistence with work colleagues, the lack of distance between personal life and work life, and anxiety due to uncertainty and confinement stand out. All of these problems end up reducing the emotional stability of an individual, which has great repercussions on their performance and general well-being.

There are psychological and social studies that highlight the enormous negative impact that the pandemic can have on an individual's mental stability. In this way, it is considered to be of great help for companies to be alerted to visualize in real time the negative emotions detected in their collaborators, which can be: sadness, anger, fear or disgust. With this, a preliminary detection of negative emotional behaviors would be made, which will serve as a first guide to follow up the case with a professional consultation.

Dataset

An existing, freely available online dataset was used, which was created by Pierre-Luc Carrier and Aaron Courville as part of a research project [3]. Facial expressions from 35,685 images are classified in this file. Each image is 48x48 grayscale pixels in size and is cropped so that only a single person's face is included. Each face in the data set is classified based on facial expressions. The classes into which expressions are divided are: happiness, neutral, sadness, anger, surprise, disgust and fear.

Model

All layers have been organized sequentially using TensorFlow 2.5 and Keras. The network resembles a VGG-16 type convolutional network with several CONV convolutional layers, maintaining the padding and always with a 3×3 kernel except in the first 2 convolutional layers that have a 5×5 kernel.

MaxPooling layers between the convolutional layers reduce the spatial dimensions while retaining relevant information. We also added BatchNorm and Dropout layers to stabilise training and reduce overfitting.

moodyNet: VGG-16 type convolutional network

Each convolutional layer has a ReLU (Rectified Linear Unit) activation so that negative values are not transferred to the next layer.

After the convolutional layers, a flatten operation converts the resulting matrix into a vector. It then passes through four dense layers with ReLU activation and an output layer with one unit per class and a softmax activation. The output values lie between 0 and 1 and sum to 1, representing the estimated probability of each class.

The optimizer that has been used is ADAM.

Results

Loss and accuracy across training epochs

Conclusions

The prototype classifies facial expressions in the visual material it receives and lets us explore the frequency of different labels, including those associated with negative emotions. We intended to investigate ways to support workplace wellbeing, but these classifications do not provide a reliable emotional profile or identify mental health problems. Any application in that setting would require independent validation, voluntary participation and privacy safeguards.

We wanted to put people’s wellbeing at the centre, beyond workplace performance. That goal requires listening to employees, respecting their autonomy and ensuring that a prediction based on their face never replaces their own account or a professional assessment.

Future projection

We are sure that this first version of Moody.ai can be perfected and expandable to more sophisticated versions with different approaches. Among the ideas that have emerged in the team for subsequent versions of Moody.ai are the following applications:

• Automate emotion detection and trigger events based on these detections. For example, identifying, in real time, faces of tired transport drivers, blinking more frequently than normal, yawning, etc.
• Detection of employees' emotions on specific topics discussed in remote work environments and for companies to identify them early.
• Identify, in real time, early people with negative emotions carrying out potentially risky activities: machinery operators, production line personnel, construction workers, etc.

Further development of Moody.ai could help us learn more about computer vision and its limitations. The proposed applications are research directions, not demonstrated capabilities. Adapting the model to another context would require new data, bias assessment and targeted testing; making it larger would not guarantee better results.

Project presentation: DemoDay

Repository

In the following repository you will find the code used to develop this application: https://github.com/SaturdaysAI/Projects/tree/master/Guadalajara/March2021/EmotionsDetector-main

More artificial intelligence!

Saturdays.AI makes AI more accessible through shared learning and projects with a social purpose. Our motto is AI4ALL IS AI4GOOD.

Explore the available courses

Meet the Saturdays.AI community