Deep learning to detect driver fatigue and inattention

Latam online. Second Edition. 2021

In Ecuador, cell phone use while driving a vehicle registered an increase in the first nine months of 2021; Between January and September 2021, 15,393 traffic accidents were reported. Of them, 3,989 are related to distractions, for example cell phone use, which represents 26% of the total mentioned. If compared to the same period in 2020, an increase is observed: 240 more accidents caused by this type of distractions. (ANT, 2021)

Figure 1 includes crash categories related to distraction (C14) and drowsiness or poor physical condition (C03). The original text did not clearly distinguish the figures and their geographical scope, so we do not present them as reliable fatality estimates for Guayaquil.

Classification of states and distractors

There are several types of distractions when driving and this is determined by the National Center for Statistics and Analysis (NCSA), which are classified as follows: (NHTSA, 2021)

Driver status:

• Attentive
• Distracted
• Looked but did not see
• Drowsy or asleep

Driver-related distractions:

• Eating or drinking
• External person, object or event
• Adjusting the radio, cassette or CD player
• Other vehicle occupants
• Moving object in the vehicle

Problem description

According to the statistical data obtained, a significant number of people die as a result of traffic accidents. Of all of them, many are caused by what is known as inattention, the main contributing factors of which are both distraction and drowsiness. Therefore, drivers with fatigue can benefit from a system that alerts them when they lose attention.

Based on what was mentioned in previous lines, the following problem arises:

How to prevent a vehicle driver from becoming distracted or drowsy?

General objective

Design a deep learning prototype that estimates signs of distraction or drowsiness and generates an audible alert. An alert-review dashboard was also proposed. The project explores this functionality but does not establish that alerts prevent crashes or replace driver rest.

Tool structure

Figure 2 shows the structure of the tool, its operation will be as follows:

1. A forward-facing webcam and speaker installed in the driver’s cabin.
2. Web server where the algorithm will be executed, which will process the images and make the detections according to the training, whether due to distractions or drowsiness. After processing and detection, it will send a direct alert to the driver, the type of this alert will be audible, with the aim of returning attention to the road.
3. An administration panel for reviewing visual or text alerts. Real-world use would require decisions about access to images, retention periods and how potential errors are reviewed.

Figure 3 and 4 show the alert that will be displayed in the administrative panel of the tool under design, sending the image of the driver with the distracting object or with the driver's status.

Techniques implemented

A. Inattention detection

To detect driver distractions we need to train an object detection model. Object detection is a computer vision technology that locates and identifies objects in an image. Due to its versatility, it has become the most used computer vision technology in recent years.

The object detection task locates objects in an image and labels them as belonging to a target class, as shown in the image below.

Object detection models achieve this goal by predicting the coordinates X1, X2, Y1, Y2, and object class labels.

We used YOLOv5, a PyTorch object-detection implementation. The YOLO family combines localisation and classification in a network that processes an image in one pass. This project used COCO resources and a custom dataset of distracting objects.

Figure 6 shows us the architecture used for Yolo V5, which is composed as follows:

Backbone:

The backbone extracts image features through convolutional blocks. In the version used, CSP blocks help represent patterns such as edges and shapes.

Neck:

The neck combines features at different scales, allowing the detector to handle large and small objects using feature maps at different resolutions.

Head:

The detection head uses anchor boxes to produce bounding boxes, objectness scores and class probabilities.

B. Detecting drowsiness and gaze diversion

For eye detection, the Dlib library was used in conjunction with the face reference model: shape_predictor_68_face_landmarks.dat, which uses the point detection mask (Figure 7) to determine key locations that make up a face.

Using these tools, it was possible to identify the location of the eyes taking as a reference from point 36 to 41 to outline the left eye and from 42 to 47 the right eye.

With this information it was possible to determine if a human face was in front of the camera, and if so, the Euclidean distance was used (to determine the length of a straight line between two points) and thus establish whether the eyes are open or closed.

Subsequently, if the eyes remain closed for more than two seconds, an alarm is activated whose purpose is to wake up the driver (Figure 3). On the other hand, if the driver is looking in any direction other than the front, after four seconds the alarm is activated to alert about the distraction (Figure 3).

Model evaluation

After training YOLOv5, we examined losses and performance metrics in Weights & Biases.

The mAP at 0.5 and the mAP at 0.5:0.95 are plotted for 200 epochs. mAP@[0.5:0.95] means average mAP over different IoU thresholds, from 0.5 to 0.95, step 0.05 (0.5, 0.55, 0.6, 0.65, 0.7, 0.75, 0.8, 0.85, 0.9, 0.95).

As for the precision and recall for 200 epochs they are given below. The precision was good during the later epochs, but the recall values ​​were lower during the later epochs.

Loss of training and validation:

In object detection, there were a total of 3 losses to calculate

• Box loss: assesses how well predicted bounding boxes match annotations.
• Classification loss: assesses each object’s predicted category.
• Objectness loss: assesses predicted object presence.

Generation and preparation of the data set

To train our object detection model we need to monitor its learning with bounding box annotations. We draw a box around each distractor that the detector recognizes: cell phone, food, cigarette and bottles to identify when the driver is distracted or carrying out some activity unrelated to driving. There are many labeling tools (CVAT, LabelImg, VoTT, Roboflow) and large-scale solutions (Scale, AWS Ground Truth), in our case we use Roboflow.

The images to build our dataset were downloaded from Kaggle and COCO Dataset:

• Dataset containing smoking and not-smoking images (smoker vs non-smoker)
• Mobile Images Dataset
• COCO Dataset

Features of the GuardIAn prototype

• Innovative, designed with Artificial Intelligence, Deep Learning algorithm
• Team-reported result: 92% in the distractor-detection evaluation. This is not a safety guarantee and does not establish the same performance for drowsiness detection.
• Accessible, the tool can be implemented anywhere in the world.
• Adaptable, the algorithm can be retrained as many times as necessary.
• Lightweight and easy to handle, it does not require installation and its administration is very easy.
• Designed for cloud deployment; continuous availability would require suitable infrastructure and operational support.

Conclusion

Artificial intelligence has increased its possibilities, especially since the emergence of technologies such as neural networks and machine learning.

The prototype identifies visual patterns and generates alerts. Its potential to reduce crashes would need study under real conditions, including false alarms, missed detections and driver behaviour.

The project was designed as a functional system for vehicular environments, capable of detecting fatigue and distracting objects while driving a vehicle, in addition to issuing both types of alerts (audio and textual). The model manages to detect distractors correctly by 92%.

The algorithm will be able to continue training and expanding the distracting objects for detection.

Future plans

We would like to explore potential applications with the following types of company:

• Mass transportation companies (Land terminal, bus station)
• Private-hire transport companies (Uber, Cabify, Taxi Amigo)

Members

• Ing. Carlos Watson (cwatsonm@gmail.com)
• Eng. Josué Huamán (esai.huaman@gmail.com)
• Ing. Yolanda C. Sarmiento (yolitasarmiento@hotmail.com)
• Karyme Rodriguez (karyme.rodriguez@cetys.edu.mx)
• Joanna Araiza (joanna.araiza7829@alumnos.udg.mx)

Project presentation: DemoDay

Open video or resource

Our repository on Github

https://github.com/SaturdaysAI/Projects/tree/master/LATAM_remote/GuardIAn

More artificial intelligence!

Saturdays.AI makes AI more accessible through shared learning and projects with a social purpose. Our motto is AI4ALL IS AI4GOOD.

Meet the Saturdays.AI community