Deep learning to detect driver fatigue and inattention

Latam online. Second Edition. 2021

In Ecuador, cell phone use while driving a vehicle registered an increase in the first nine months of 2021; Between January and September 2021, 15,393 traffic accidents were reported. Of them, 3,989 are related to distractions, for example cell phone use, which represents 26% of the total mentioned. If compared to the same period in 2020, an increase is observed: 240 more accidents caused by this type of distractions. (ANT, 2021)

As can be seen in Figure 1, only in Guayaquil, one of the most important cities in the country, a total of 1,450 traffic accidents are recorded due to driving inattentive to traffic conditions (cell phone, video screens, food, makeup or any other distracting element) labeled as C14; These types of distractors left 1,256 deaths on the road", being one of the first causes of this. Driving while drowsy or in poor physical condition (C03) only records a total of 9 accidents and 11 deaths in 2021.

Classification of states and distractors

There are several types of distractions when driving and this is determined by the National Center for Statistics and Analysis (NCSA), which are classified as follows: (NHTSA, 2021)

Driver status:

• Attentive
• Distraído
• He looked but did not see
• Are you sleepy or have fallen asleep?

Driver-related distractions:

• Eat or drink
• External person, object or event
• Setting radio, cassette or CD
• Other vehicle occupants
• Moving object in the vehicle

Problem description

According to the statistical data obtained, a significant number of people die as a result of traffic accidents. Of all of them, many are caused by what is known as inattention, the main contributing factors of which are both distraction and drowsiness. Therefore, drivers with fatigue can benefit from a system that alerts them when they lose attention.

Based on what was mentioned in previous lines, the following problem arises:

How to prevent a vehicle driver from becoming distracted or drowsy?

General objective

The objective was to design a tool for vehicular environments, which, through Deep Learning techniques, detects both distraction and drowsiness in drivers and can send an audible alert directly to the driver, so that they immediately return their attention to the road and thus prevent a traffic accident from occurring, in addition to being able to monitor the alerts through an administration console, in order to control the personnel who are maneuvering the vehicle.

Tool structure

Figure 2 shows the structure of the tool, its operation will be as follows:

1. Webcam with speaker implemented in the driver's cabin directly frontally.
2. Web server where the algorithm will be executed, which will process the images and make the detections according to the training, whether due to distractions or drowsiness. After processing and detection, it will send a direct alert to the driver, the type of this alert will be audible, with the aim of returning attention to the road.
3. Administration or control panel, after processing and executing the algorithm, a visual or textual alert will also be sent to the administrator of the tool, so that internal control of the acts committed by the driver is kept.

Figure 3 and 4 show the alert that will be displayed in the administrative panel of the tool under design, sending the image of the driver with the distracting object or with the driver's status.

Techniques implemented

A. Inattention detection

To detect driver distractions we need to train an object detection model. Object detection is a computer vision technology that locates and identifies objects in an image. Due to its versatility, it has become the most used computer vision technology in recent years.

The object detection task locates objects in an image and labels them as belonging to a target class, as shown in the image below.

Object detection models achieve this goal by predicting the coordinates X1, X2, Y1, Y2, and object class labels.

For our project we used YOLO V5, YOLO was initially introduced as the first object detection model that combined bounding box prediction and object classification into a single end-to-end differentiable network. It was written and is maintained on a framework called Darknet. YOLO V5 is the first of the YOLO models written in the PyTorch framework and is much lighter and easier to use. That being said, YOLO V5 did not make any major changes to the network architecture in YOLO V4 and does not outperform YOLO V4 on a common benchmark, the dataset used was from COCO dataset.

Figure 6 shows us the architecture used for Yolo V5, which is composed as follows:

Backbone:

Model Backbone acts as a feature extractor from an input image. Feature extractor is nothing but the use of convolutional layers like kernel, stride, batch normalization is applied to the input images to extract important features like edges, shapes, etc. CSP (Cross Stage Partial Networks) are used as backbone in YOLO V5 to extract useful features from an input image.

Neck:

At the neck layer, the network was designed to perform multi-scale predictions on top of the feature pyramid network. Multiscale prediction helps detect objects of different sizes by outputting images to three different grid values. So small grid images detect large objects and large grid images detect small objects.

Head:

The Head model is primarily responsible for the final detection step. It uses anchor boxes to construct final output vectors with class probabilities, objectivity scores, and bounding boxes.

B. Sleep Detection and Vision Distraction

For eye detection, the Dlib library was used in conjunction with the face reference model: shape_predictor_68_face_landmarks.dat, which uses the point detection mask (Figure 7) to determine key locations that make up a face.

Using these tools, it was possible to identify the location of the eyes taking as a reference from point 36 to 41 to outline the left eye and from 42 to 47 the right eye.

With this information it was possible to determine if a human face was in front of the camera, and if so, the Euclidean distance was used (to determine the length of a straight line between two points) and thus establish whether the eyes are open or closed.

Subsequently, if the eyes remain closed for more than two seconds, an alarm is activated whose purpose is to wake up the driver (Figure 3). On the other hand, if the driver is looking in any direction other than the front, after four seconds the alarm is activated to alert about the distraction (Figure 3).

Model evaluation

We use YOLO V5, post-training, evaluate training losses and performance metrics in Weights & Biases.

The mAP at 0.5 and the mAP at 0.5:0.95 are plotted for 200 epochs. mAP@[0.5:0.95] means average mAP over different IoU thresholds, from 0.5 to 0.95, step 0.05 (0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95).

As for the precision and recall for 200 epochs they are given below. The precision was good during the later epochs, but the recall values ​​were lower during the later epochs.

Loss of training and validation:

In object detection, there were a total of 3 losses to calculate

• Box loss: which is a squared loss because detecting the box coordinates is a simple regression task
• Class loss, which is a log loss because it predicts the probability that an object belongs to a class.
• Loss of objects

Generation and preparation of the data set

To train our object detection model we need to monitor its learning with bounding box annotations. We draw a box around each distractor that the detector recognizes: cell phone, food, cigarette and bottles to identify when the driver is distracted or carrying out some activity unrelated to driving. There are many labeling tools (CVAT, LabelImg, VoTT, Roboflow) and large-scale solutions (Scale, AWS Ground Truth), in our case we use Roboflow.

The images to build our dataset were downloaded from Kaggle and COCO Dataset:

• Dataset containing smoking and not-smoking images (smoker vs non-smoker)
• Mobile Images Dataset
• COCO Dataset

Advantages of using guardian

• Innovative, designed with Artificial Intelligence, Deep Learning algorithm
• Accurate,algorithm trained with 92% accuracy of distractors and eye blinks.
• Accessible, the tool can be implemented anywhere in the world.
• Adaptable, the algorithm can be retrained as many times as necessary.
• Lightweight and easy to handle, it does not require installation and its administration is very easy.
• Reliable, tool designed for the cloud, capable of staying active 24/7

Conclusion

Artificial intelligence has increased its possibilities, especially since the emergence of technologies such as neural networks and machine learning.

By using Deep Learning to identify patterns in photos or videos and detect distractors or the driver's state of drowsiness, it is possible to send an audible alert that would return immediate attention and probably prevent a traffic accident, thus reducing the mortality rate from these causes.

The project was designed as a functional system for vehicular environments, capable of detecting fatigue and distracting objects while driving a vehicle, in addition to issuing both types of alerts (audio and textual). The model manages to detect distractors correctly by 92%.

The algorithm will be able to continue training and expanding the distracting objects for detection.

Future plans

This project intends to take the following companies:

• Mass transportation companies (Land terminal, bus station)
• Executive transportation companies (Uber, Cabify, Taxi Friend)

Members

• Ing. Carlos Watson (cwatsonm@gmail.com)
• Eng. Josué Huamán (esai.huaman@gmail.com)
• Ing. Yolanda C. Sarmiento (yolitasarmiento@hotmail.com)
• Karyme Rodriguez (karyme.rodriguez@cetys.edu.mx)
• Joanna Araiza (joanna.araiza7829@alumnos.udg.mx)

Project presentation: DemoDay

Open video or resource

Our repository on Github

https://github.com/SaturdaysAI/Projects/tree/master/LATAM_remote/GuardIAn

More artificial intelligence!

The mission of Saturdays.ai is to make artificial intelligence more accessible (#ai4all) through intensive courses and programs where projects for good are carried out (#ai4good).

If you want to learn more about artificial intelligence join our community at community.saturdays.ai or visit us on our website www.saturdays.ai Join us!