Detecting emotions through images with Artificial Intelligence

This archive includes automatic translations. Audio, video and text inside images remain in their original language.

Editorial note: this project classifies facial expressions using its dataset’s labels. That task does not, by itself, establish a person’s emotions, attention or wellbeing. The results described should be read as those of an educational prototype.

Evidence on facial expressions and emotions

team logo

Donostia. First edition. 2020

Introduction

In everyday life, how many times does it happen to us that we ask a person how they are, and the answer is positive while their face indicates the opposite? How many times have you gone to the hairdresser and pretended to leave happy when you really didn't like the final result? Have you ever wanted to receive feedback from thousands of people at or during a conference?

Carl Rogers quote

If we start to reflect on the aforementioned issues, we will probably realize that we often lie when dealing with emotions, but does our face also lie?

Problem definition

Many times more importance is given to what is said with the voice than to what is said with the facial expression, making it easier to lie or hide reality with the former. With this project we wanted, in addition to deploying a real artificial intelligence project, to do something that could be useful, that could detect people's emotions based on their face, through an image or by detecting the face with a webcam. Emotions arise when something relevant happens. They appear quickly, automatically, and change our focus of attention.

Emotional intelligence is something that has become very relevant in recent years, the importance of perceiving, using, understanding and managing emotions, both those corresponding to oneself and those of others. To do this, it is clear that we need real emotions, so we wanted to facilitate the way in which emotions can be perceived. Will a machine be able to detect and classify emotions better than humans?

Facial Expression Recogniser will be an application in charge of detecting emotions in real time. In this first version, images will be used and, then, its function will be to classify emotions as soon as the camera can detect faces.

Dataset

The dataset used for the development of this project, which was obtained from Kaggle, consisted of a series of images divided into folders based on facial expression. The folder labels were divided according to the following classification:

0 — Angry

1 — Disgust

2 — Fear

3 — Happy emotions

4 — Sad

5 — Surprise

6 — Neutral

The main objective of the project was to detect and classify emotions according to these labels. For this prediction, images obtained through the webcam would be used.

data train

The approach

As mentioned previously, when describing the dataset used, it was seen that it had images and the corresponding emotion labels. This has made the EDA process less important in this project.

However, some analysis and transformation of the data has been necessary. To begin with, dataframes had to be created from the dataset. To do this, the photos we had were converted to pixels, and two columns were created in said table, one related to emotion and the other to pixels.

Dataset format

Furthermore, it should be noted that from the beginning there was a classification of the dataset between train and test, so the transformation of images to pixels was done twice, thus ending with two dataframes: train_data and test_data

Table size

Building the model

The project carried out is based on Deep Learning, which is why the use of neural networks has been necessary. In our case, convolutional neural networks have been used, which are mainly used for artificial vision tasks, as they are very effective in image classification and segmentation, among other applications.

For this purpose, a sequential model has been presented, allowing sequential layers to be stacked in order from input to output.

The layers added to the model have been:

– Conv2D

– Batch Normalization

– MaxPooling2D

–Flatten

– Dense

– Activation

– Dropout

To create the aforementioned model, Tensorflow and Keras have been used. The latter is a Neural Networks library written in Python. It is capable of running on top of TensorFlow. The latter meets the needs of systems capable of building and training neural networks to detect and decipher patterns and correlations.

After creating the model, training of the model began. At first, the model was trained with a single epoch, which in addition to taking a long time, only obtained an accuracy of 0.29. This is why the training of the model had to be modified, increasing the epochs, changing the steps to take in each epoch, etc.

Furthermore, due to a saving problem, a callback had to be created to the ModelCheckpoint, so that it would store a checkpoint every time an epoch ended, thus, a final model with more epochs could be obtained.

In the end, the model obtained has achieved a final accuracy of 0.9602. This indicates the accuracy of what was trained. However, if we analyze the val_accuracy, which refers to how well your model works in general for cases outside the training set, the value obtained was 0.6035.

However, if we calculate the precision of the model with the dataset used for testing, we will see that the accuracy is quite low, 0.1733, which implies there is a lot of room for improvement in this model.

Prediction

Once we had the model ready, we had to predict and test it. To do this, it was coded in a way that indicated those emotions that could be considered in the facial expression of the entered image, and according to the percentage, concluded with the most significant feeling. For example:

Image to predict

We introduce this first image, where it is evident that the boy is showing some anger. In this way, our model has classified it as follows:

Emphasizing that anger is the feeling that predominates in the image. If we use our model, in order to detect some other emotion, we will see that it also works.

Predicted image

As mentioned in the conclusions, the intention was to incorporate face detection through webcams and thus be able to detect emotions in a more real way.

Conclusions

Once the project was finished, in a group reflection, we commented on how much we had learned in its development, in addition to realizing how far technology can go, and to be more precise, artificial intelligence.

We have seen that great advances are being made in this area of facial recognition, there are models that recognize faces even when wearing a mask, and the applications of this technology are only limited by our imagination. From a marketing point of view, collecting feedback from users and customers is a very important process, but obtaining this information is usually costly; almost no one stops to fill out a form to say what our experience has been like unless it was negative.

That is why if we are able to detect red spots in the user experience without requiring them to make any additional effort, the service could be improved, and thanks to this technology this is possible.

Next steps

The next steps that will be taken with this project are directly related to the problems that have been encountered in the completion of the project. The first difficulty suffered by the team was the one corresponding to the deployment in Amazon Web Services, which was supposed to facilitate training, causing the project to come to a standstill given the inexperience of the team members with said tool. This has meant that training could not be done on Amazon's servers, which has taken a long time and made any modification and execution of the model difficult. This is why, soon, an attempt will be made to carry out this deployment in order to work in a more efficient and effective way.

This problem caused the deviation suffered over time to be very high, which made it difficult to complete all the functionality that we anticipated in the first instance. Furthermore, this was also directly related to the small margin we had left to train the model, which implies that the efficiency and precision of the model is not optimal, and there is still a fairly large margin for improvement. This is why, through more training, a higher quality model could be obtained.

We did not consider having so many problems when we defined the project we wanted to carry out, due to the inexperience we had in this area. One of the functionalities that we proposed at the beginning was the incorporation of a Web Cam that would allow us to take photos instantly and be able to classify the emotions in said image, in order to make it more real. However, due to lack of time, it is an aspect that has not been able to be developed but would be the first thing we would do in the future.

It would be very useful to integrate our model with webcams at the entrances/exits of all those places that want to assess customer satisfaction or experience in said place. For example, restaurants, shops, conferences, etc.

Furthermore, in the future it would be great to program the application to be able to recognize an image taken at the exit of a place with that image taken at the entrance of the same person. This would make it possible to efficiently compare the results of all users, and it would be a huge step forward since we would provide the company/institution that uses it with business intelligence. If we added to that a series of graphs that display the results in a kind of dashboard, it could help managers make various decisions based on customer satisfaction.

Members

Project presentation: DemoDay

Open video or resource

Repository

In the following repository you will find the code used to develop this application: https://github.com/SaturdaysAI/Projects/tree/master/Donostia/Donostia2020/Facial_Expression_Saturdays

More artificial intelligence!

The mission of Saturdays.ai is to make artificial intelligence more accessible (#ai4all) through intensive courses and programs where projects for good are carried out (#ai4good).

Explore our AI master’s programme at https://saturdays.ai/master-ia-online/

If you want to learn more about artificial intelligence join our community at community.saturdays.ai or visit us on our website www.saturdays.ai Join us!