Real-time facial expression classification
A deep learning and computer vision prototype for classifying facial expressions, with its results and limitations.
A learning project shared by the community. The article documents its team, methods and results; it does not establish that the prototype is ready for real-world use.
Editorial note: this project classifies facial expressions using its dataset’s labels. That task does not, by itself, establish a person’s emotions, attention or wellbeing. The results described should be read as those of an educational prototype.
Evidence on facial expressions and emotions
Quito. 2021
Eng. Benjamin Villegas Mendez, Eng. Mauricio Oroza Herbas
INTRODUCTION
The COVID-19 pandemic accelerated the move to online classes and created challenges for student engagement. This project explored facial expression classification as a possible source of information for teachers. The prototype used a residual convolutional network, ResNet, images from three sources and data augmentation. Its outputs are labels for facial expressions captured by a webcam: they do not, by themselves, measure a student’s attention, interest or emotions. Any potential educational use would require further evaluation and attention to privacy and consent.
PROBLEM DESCRIPTION
Online classes can make it difficult to sustain engagement and understand how students are following a session. The team wanted to explore this problem, although the prototype cannot quantify how many students are paying attention.
OBJECTIVES
The main objective was to build a deep learning model that classified facial expressions in real time using a webcam.
The particular objectives include selecting a dataset that may be suitable for the intended purpose, implementing a facial recognition system only, and selecting an appropriate classification model to create the general emotion recognition model.
DATASET SELECTION
A search was made for datasets related to emotions and a list of the most used datasets was found as shown in the figure below:
The main criterion for selection was to search for datasets that are freely available that are within the list presented above, which are FER2013 and CK+. For this project, a mixture of both datasets was used, including an own dataset created. The first 2 datasets are currently widely used for facial recognition and have a significant amount of data relevant to the ongoing project. Due to cost issues, fragments of these datasets that are available for free were searched and integrated into a single one. The datasets were reviewed manually to eliminate possible images that do not correspond to the emotion, data was added and data augmentation was used.
MODEL SELECTION
Since this is an image classification problem with 7 possible outputs (corresponding to 6 basic detectable facial emotions and one neutral), several possible computer vision and image classification architectures were studied, among which we can mention: AlexNet, VGG, inception and ResNet. The latter being chosen for the proposed model given its versatility in training deep networks without presenting the problem of gradient vanishing.
In the following table you can see its error percentage compared to the image classification dataset from the 2015 ImageNet competition.
The principle of operation of residual networks (ResNet) can be understood by analyzing what its authors call convolutional residual block.
If we assume that the output of the block F(x) has the same dimension as its input x, this block is known as the residual identity block, which can be summarized by the following formula:
x = x + F(x)
Where for simplicity the output of the block will also be called x.
In an identity block, the skip connection needs no transformation: F(x) and the input x have the same dimensions and can be added directly. This does not mean the output is identical to the input.
In the event that the output dimensions and the input tensor are different because the number of channels is increased or the dimensions of the tensor are modified, it will be necessary to apply operations to x to make the dimensions congruent.
Therefore, in order to be able to add the residual link with the processed data, a convolutional layer (without activation) is added to the bridge link with a step of two and with the required number of channels.
X = [Vx + b] + F(x)
This block is known as a convolutional residual block.
In the case of the model implemented for emotion recognition, a 9-block ResNet network was implemented so that the training is not too extensive or consumes too many GPU and memory resources allocated in the designed notebook.
TECHNIQUES IMPLEMENTED
Given the nature of the composite dataset used, whose images contain variation in sizes, color and grayscale images, and for some of the detected emotions there are fewer images for the training stage than others, the following data augmentation techniques were used:
- Random horizontal flip: Used to randomly mirror images horizontally.
- Color Jitter: To randomly adjust the brightness and contrast of images.
- Resize: To parameterize the size of the images before entering the training
- Random rotation: To randomly rotate images by 20 degrees from their original position.
- Grayscale: to make all input images grayscale.
- To Tensor: To convert images into tensors prior to training
Some techniques were also used to make the training more optimal:
- Learning Rate Scheduling: To have a variable learning rate as the training progresses in each epoch, so that around 30 percent of the epochs have the maximum value of the learning rate.
- Weight decay: discourages excessively large parameter values.
- Gradient Clipping: Limits gradient values to a small range to avoid unwanted changes in parameters due to large gradient values.
It is worth mentioning that in the construction of the ResNet model, well-known optimization techniques were also used, such as: batch normalization, maxpool and dropout.
In the case of face detection for subsequent classification, the pre-trained Open CV model “Haar cascade frontal face” was used, whose call to said model is made within a JavaScript script for Webcam management within Colab in real time.
MODEL EVALUATION
The webcam function combines face detection with the facial expression classifier. It saves predictions during execution so they can later be displayed in charts.
The following screenshot shows a prediction labelled “angry”. This is a classifier output, not confirmation of the person’s emotional state.
ANALYSIS OF THE RESULTS
40 epochs were used for training the designed model, achieving an accuracy of 70% in a training time of approximately 40 minutes. It is worth mentioning that the accuracy throughout the epoch does not fluctuate too much, this after having modified the dataset manually by eliminating images considered noise.
It is possible to see the change of learning rate during training in the next image.
The team obtained 70% accuracy in its evaluation using the GPU and memory resources available in Google Colab. That result concerns the dataset’s labels, not the measurement of real emotions.
CONCLUSIONS
-A good dataset is essential for good detection, when obtaining fragments of datasets for free, it was evident that they had erroneous images that did not correspond to the emotion to be detected, that is why each image of the dataset had to be analyzed, cleaned and added data to be able to have an accurate result. It is recommended to access datasets prepared by universities or institutions dedicated to research.
-We used a pretrained face detector and our own facial expression classifier, which achieved 70% accuracy in the evaluation described.
-The use of residual neural networks that benefited from a shorter training time with acceptable results.
-The real-time emotion recognition system was implemented within the colab environment with JavaScript functions, otherwise it would not be possible to manage a webcam within colab.
-The data is saved and different graphs are presented at the end of the detection so that the teacher can view the data obtained throughout his class.
- The team proposed possible research directions in marketing, content creation and product development. Their feasibility and appropriateness require independent evaluation; facial expressions cannot simply be assumed to reveal a person’s emotions.
The complete code of both the notebook used for training and the notebook used for the execution of the model itself can be found at:
https://github.com/benjorocker94/Face-emotion-recognition-with-cola
BIBLIOGRAPHY
- Mellouck & Handouzi. (2020). Facial emotion recognition using deep learning: review and insights. Procedia Computer Science. https://www.sciencedirect.com/science/article/pii/S1877050920318019
- Minae et al. (2019). Deep-Emotion: Facial Expression Recognition Using Attentional Convolutional Network
- Rivera. (2019). The Residual Network (Residual Network, ResNet).http://personal.cimat.mx:8181/~mrivera/cursos/aprendizaje\_profundo/resnet/resnet.html#par%C3%A1metros-del-modelo-vs-la-exactitud-accuracy
- Bok. (2020). Getting Started with Albumentation: Winning Deep Learning Image Augmentation Technique in PyTorch example. https://towardsdatascience.com/getting-started-with-albumentation-winning-deep-learning-image-augmentation-technique-in-pytorch-47aaba0ee3f8
- Rohan Singh. (2020). Emotion Detection Using Pytorch. https://medium.com/swlh/emotion-detection-using-pytorch-4f6fbfd14b2e
Repository
In the following repository you will find the code used to develop that application: https://github.com/SaturdaysAI/Projects/tree/master/Quito/2021/Face-emotion-recognition-with-colab-main
More artificial intelligence!
Saturdays.AI makes AI more accessible through shared learning and projects with a social purpose. Our motto is AI4ALL IS AI4GOOD.
A story from the Saturdays.AI community. Originally published on Nov 22, 2021.
← More community stories
