Real-time emotion recognition

Editorial note: this project classifies facial expressions using its dataset’s labels. That task does not, by itself, establish a person’s emotions, attention or wellbeing. The results described should be read as those of an educational prototype.

Evidence on facial expressions and emotions

Quito. 2021

Eng. Benjamin Villegas Mendez, Eng. Mauricio Oroza Herbas

INTRODUCTION

Faced with the covid 19 pandemic, all institutions that are dedicated to learning were affected since teaching classes would not be the same for a long time. Thus, the migration to virtual classes is now a fact. This migration brings with it many advantages and disadvantages. This project focuses on the great problem of loss of attention and interest on the part of students, resulting in only 40% of students actually maintaining attention and being able to take advantage of the class taught, whether at the college, undergraduate and even postgraduate level. This project suggests the solution of providing feedback to the teacher through emotional recognition of students' faces in real time. Having this type of information at hand, the teacher can modify the content of his class or use techniques that can regain the attention of the majority of his students to make his class effective. The proposed solution consists of a recurrent convolutional neural network with ResNet (residual network) that, through the webcam, recognizes what emotion the student is having. An integration of 3 different parts of datasets was made for training and techniques were used to improve the performance of the system. The result points to a usable facial emotion recognition system that could later be implemented in virtual class and meeting software.

PROBLEM DESCRIPTION

The problem lies in the difficulty on the part of students to maintain attention in virtual classes, resulting in 6 out of 10 students losing attention in the class taught. Unlike face-to-face classes, teachers have little or no feedback about the attention state of their students, making it difficult for the teacher to take actions to maintain their students' attention.

OBJECTIVES

The general objective is the creation of a deep learning model that can predict in real time the emotions of a person in front of their webcam, within a programming environment.

The particular objectives include selecting a dataset that may be suitable for the intended purpose, implementing a facial recognition system only, and selecting an appropriate classification model to create the general emotion recognition model.

DATASET SELECTION

A search was made for datasets related to emotions and a list of the most used datasets was found as shown in the figure below:

The main criterion for selection was to search for datasets that are freely available that are within the list presented above, which are FER2013 and CK+. For this project, a mixture of both datasets was used, including an own dataset created. The first 2 datasets are currently widely used for facial recognition and have a significant amount of data relevant to the ongoing project. Due to cost issues, fragments of these datasets that are available for free were searched and integrated into a single one. The datasets were reviewed manually to eliminate possible images that do not correspond to the emotion, data was added and data augmentation was used.

MODEL SELECTION

Since this is an image classification problem with 7 possible outputs (corresponding to 6 basic detectable facial emotions and one neutral), several possible computer vision and image classification architectures were studied, among which we can mention: AlexNet, VGG, inception and ResNet. The latter being chosen for the proposed model given its versatility in training deep networks without presenting the problem of gradient vanishing.

In the following table you can see its error percentage compared to the image classification dataset from the 2015 ImageNet competition.

The principle of operation of residual networks (ResNet) can be understood by analyzing what its authors call convolutional residual block.

If we assume that the output of the block F(x) has the same dimension as its input x, this block is known as the residual identity block, which can be summarized by the following formula:

x = x + F(x)

Where for simplicity the output of the block will also be called x.

The resulting data from the residual Identity block are not modified, this is because the tensor that results from the convolutional stages F(x) is of the same dimensions as the original data, so it is not necessary to adjust the dimensions to add them.

In the event that the output dimensions and the input tensor are different because the number of channels is increased or the dimensions of the tensor are modified, it will be necessary to apply operations to x to make the dimensions congruent.

Therefore, in order to be able to add the residual link with the processed data, a convolutional layer (without activation) is added to the bridge link with a step of two and with the required number of channels.

X = [Vx + b] + F(x)

This block is known as a convolutional residual block.

In the case of the model implemented for emotion recognition, a 9-block ResNet network was implemented so that the training is not too extensive or consumes too many GPU and memory resources allocated in the designed notebook.

TECHNIQUES IMPLEMENTED

Given the nature of the composite dataset used, whose images contain variation in sizes, color and grayscale images, and for some of the detected emotions there are fewer images for the training stage than others, the following data augmentation techniques were used:

• Random horizontal flip: Used to randomly mirror images horizontally.
• Color Jitter: To randomly adjust the brightness and contrast of images.
• Resize: To parameterize the size of the images before entering the training
• Random rotation: To randomly rotate images by 20 degrees from their original position.
• Grayscale: to make all input images grayscale.
• To Tensor: To convert images into tensors prior to training

Some techniques were also used to make the training more optimal:

• Learning Rate Scheduling: To have a variable learning rate as the training progresses in each epoch, so that around 30 percent of the epochs have the maximum value of the learning rate.
• Weight Decay: Prevents weights from being too heavy.
• Gradient Clipping: Limits gradient values to a small range to avoid unwanted changes in parameters due to large gradient values.

It is worth mentioning that in the construction of the ResNet model, well-known optimization techniques were also used, such as: batch normalization, maxpool and dropout.

In the case of face detection for subsequent classification, the pre-trained Open CV model “Haar cascade frontal face” was used, whose call to said model is made within a JavaScript script for Webcam management within Colab in real time.

MODEL EVALUATION

Within the function developed in JavaScript for Webcam access in real time, there are both the pre-trained face detection model and the trained model for the classification of emotions, both work in real time, thus managing to detect emotions in a matter of seconds while saving the predictions obtained to make graphs at the end of the execution of the function as in the following graph

The following image shows a screenshot of the model in full operation detecting the emotion of anger.

ANALYSIS OF THE RESULTS

40 epochs were used for training the designed model, achieving an accuracy of 70% in a training time of approximately 40 minutes. It is worth mentioning that the accuracy throughout the epoch does not fluctuate too much, this after having modified the dataset manually by eliminating images considered noise.

It is possible to see the change of learning rate during training in the next image.

The model's predictions are generally accurate considering that Google Colab has limited GPU and memory resources allocated to each notebook.

CONCLUSIONS

-A good dataset is essential for good detection, when obtaining fragments of datasets for free, it was evident that they had erroneous images that did not correspond to the emotion to be detected, that is why each image of the dataset had to be analyzed, cleaned and added data to be able to have an accurate result. It is recommended to access datasets prepared by universities or institutions dedicated to research.

-A pre-trained model was used for face recognition, and subsequently our own model was created capable of detecting emotions in real time with an accuracy of 70%.

-The use of residual neural networks that benefited from a shorter training time with acceptable results.

-The real-time emotion recognition system was implemented within the colab environment with javaScript functions, otherwise it would not be possible to manage a webCam within colab.

-The data is saved and different graphs are presented at the end of the detection so that the teacher can view the data obtained throughout his class.

• Further applications of this model can be used in emotion detection in marketing, content creation, and product development.

The complete code of both the notebook used for training and the notebook used for the execution of the model itself can be found at:

https://github.com/benjorocker94/Face-emotion-recognition-with-cola

BIBLIOGRAPHY

• Mellouck & Handouzi. (2020). Facial emotion recognition using deep learning: review and insights. Procedia Computer Science. https://www.sciencedirect.com/science/article/pii/S1877050920318019
• Minae et al. (2019). Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network
• Rivera. (2019). The Residual Network (Residual Network, ResNet).http://personal.cimat.mx:8181/~mrivera/cursos/aprendizaje\_profundo/resnet/resnet.html#par%C3%A1metros-del-modelo-vs-la-exactitud-accuracy
• Bok. (2020). Getting Started with Albumentation: Winning Deep Learning Image Augmentation Technique in PyTorch example. https://towardsdatascience.com/getting-started-with-albumentation-winning-deep-learning-image-augmentation-technique-in-pytorch-47aaba0ee3f8
• Rohan Singh. (2020). Emotion Detection Using Pytorch. https://medium.com/swlh/emotion-detection-using-pytorch-4f6fbfd14b2e

Repository

In the following repository you will find the code used to develop that application: https://github.com/SaturdaysAI/Projects/tree/master/Quito/2021/Face-emotion-recognition-with-colab-main

More artificial intelligence!

The mission of Saturdays.ai is to make artificial intelligence more accessible (#ai4all) while carrying out projects for good (#ai4good). The workshops we carry out are part of the program AI 4 Schools so that anyone “learn by doing” AI regardless of their background or starting point.

Explore our AI master’s programme at https://saturdays.ai/master-ia-online/

If you want to learn more about artificial intelligence join our community through this link or visit us on our website www.saturdays.ai Join us!