Focus On Driving: Neural Networks applied to Road Safety

Donostia. Second Edition. 2021

Introduction

According to the General Directorate of Traffic (DGT), the distracted driving It is one of the main causes of traffic accidents, being the cause of 1 in 3 fatal accidents (33%), ahead of speed (29%) and alcohol (26%).

After a minute and a half of talking on the cell phone (even hands-free), the driver does not perceive 40% of the signals, his average speed drops by 12%, his heart rate accelerates sharply during the call and it takes longer to react.

The danger caused by improper use of a mobile phone can be comparable to driving with excess alcohol.

More than 1 in 3 Spaniards admit to having spoken on the phone, read or written messages while driving in recent years.

A driver who talks while driving:

• Loses the ability to maintain a constant speed.

• It does not maintain sufficient safety distance from the vehicle in front.

• The reaction time increases considerably: between half and two seconds, depending on the reflexes of each driver.

Figure 1. Images of common distractions while driving.

This makes the detection of these distractions essential to improve the road safety. If they could be “detected” in order to warn the driver that something is wrong, the number of accidents could be radically reduced.

In this project we will be focused on detecting some of the most common distractions, such as using a cell phone, looking in the mirror or drinking something while driving, as well as identifying them.

Objective

The objective of this project is to adjust a model of Machine Learning capable of Identify and classify different distractions to which we are exposed whenever we drive.

To do this, we will work with a technique of Deep Learning known as “Convolutional Neural Networks” (CNN or “Convolutional Neural Network”).

We will test some neural network architectures, all of them enhanced with pre-trained networks for image classification (transfer learning).

Image Set

The set of images that we have used in the framework of this project comes from a contest launched on the platform kaggle about 5 years ago (press here to see it).

All images have been obtained through a camera installed inside the car, always with a driver inside and in various everyday situations.

The photos have been taken in a controlled environment: a truck dragged the car through the streets, so that the drivers did not have to really drive.

To train the models, we have had a total of 22,424 images, classified into 10 categories:

Figure 2. Description of the 10 categories of the experiment.

Exploratory Data Analysis

As part of the stage of “Exploratory Data Analysis”, we have verified that all the images had the same dimension: (480, 640, 3).

After random sampling, we could affirm that the images were well classified throughout the 10 categories of the experiment. This is surely due to the fact that it is a set of images from a controlled experiment that was entered into a contest on the platform. kaggle.

Furthermore, the images were distributed in a reasonable way. balanced between the 10 categories, so we have practically been able to “jump” to the “EDA” stage to go directly to the training phase of the models. cnn.

Figure 3. Distribution of the number of images among the 10 categories of the experiment.

They were also available on the platform kaggle 79,725 images for testing, without any type of classification or label.

It is important to mention that the training and testing images were divided into kaggle in such a way that The same driver could not appear in both groups. (training and testing).

In general, the 10 categories could be considered “exclusive", except for the categories "c0” (driving safely) and “c9” (talking to passenger), which are more similar to each other and could be considered “not exclusive”, since, in some situations, they have in common the 2 hands placed on the steering wheel.

Figure 4. Images of 2 drivers, the first from the “c0” category, and the second, from the “c9” category.

Implementation

In general terms, we have followed the following steps to adjust the Convolutional Neural Networks presented in this project:

1. Download and pre-process driver images.
2. Build and train a model to classify driver images.
3. Check the fit of the model and improve it using different techniques.

Transfer Learning

Normally, there is no need to train a Convolutional Neural Network from scratch.

Modern Convolutional Networks, trained with huge data sets like ImageNet, it can take weeks on multiple GPUs.

On the other hand, an increasingly common practice is the use of pre-trained networks as a fixed feature extractor, or as an initial network for finer adjustments.

The pre-trained network VGG It is interesting for the objective of this project because, apart from its simplicity, it generates excellent results. The idea is to maintain all the layers of the VGG network, connecting its final layer to our own classifier.

In this way, we could use the VGG network as a fixed feature extractor for our images, leaving the training of a simpler classifier pending at the end of everything (top model).

In this project, we will use the pre-trained network “VGG16", as a starting point. For the most fine tuning, we will use 3 final coats, proposed in this link (“Hands-on Transfer Learning with Keras and the VGG16 Model”).

From the result of this first adjustment, we will build new models, with more layers. We will also use the resource of Data Augmentation, and we will even dare to change the “VGG16” for the “VGG19", more complex pre-trained neural network, since it has 3 more layers than the "VGG16”.

Amazon Web Services

The adjustments of this first model and all the others presented in this article have been made in SageMaker of Amazon Web Services. For this, we have had the credit of 1.000$ that amazon granted us for being part of the family Saturdays.AI.

We have chosen the computing center located in 'US East (Ohio)us-east-2', since this center had GPU machines. The machine chosen to open the instance of our notebook in SageMaker has been: 'ml.p3.2xlarge’.

Parameters and Evaluation of Models

A summary of the parameters of the fitted models will be presented below in table mode.

In any case, it is worth mentioning that, in all models, we have set batch = 32 and Epochs = 25, so that there was a basis of comparability between the model fits.

To evaluate the fitted models, in addition to the graphs of accuracy and of loss generated by the algorithm, we have classified 1,002 images, which from now on we will call 'Test Data’.

These images have been chosen randomly from the 79,725 images not classified for testing, with the aim of having a measure of accuracy on a set of images totally unrelated to the data used for training ('Test Data’ Accuracy).

We have also prepared a code in Google Colab (“model_test.ipynb”), in order to visualize the result of the model in each image, as you can see in the GIF below.

Figure 5. Output of “model_test.ipynb”.

Model 1: VGG16 + 3 Layers

As mentioned above, for the first model (“base”) we have used the pre-trained network “VGG16”, and for the finest adjustment, we have used 3 final coats.

The result of the adjustment of this model could be considered satisfactory, taking into account that this has been the first attempt at modeling the project:

• ‘Test Data’ Accuracy: 80,44%

Figure 6. A_ccuracy_ graphs and loss of Model 1.

Model 2: VGG16 + 3 Layers + Data Augmentation

To improve the previous model, we have considered using the resource of “Data Augmentation”.

At first, we did not know what some of the various parameters available to make the Data Augmentation, nor what their ranks meant.

To solve this, we did a series of tests, to be able to decide which parameters of Data Augmentation we wanted to use in our set of images and their respective ranges.

Being a controlled experiment, the set of images used required a Data Augmentation controlled and discreet, taking into account that exaggerated changes applied to the original images could be counterproductive.

For example, we have decided not to do “rotations”, since we ran the risk of generating “unnatural” images.

The parameters of Data Augmentation that we have used have been: “zoom”, “width”, “height” and “brightness” Below are some examples.

Figure 7. Zoom_range = [0.95, 1.05]

Figure 8. width_shift_range = 0.1

Note: The images have that tone due to RGB preprocessing.

The result of the adjustment of this model has improved considerably compared to the previous one (Model 1), so we have decided to continue using Data Augmentation in the following models adjusted within the framework of this project.

• ‘Test Data’ Accuracy: 83,73%

Figure 9. A_ccuracy_ graphs and loss of Model 2.

Model 3: VGG16 + 5 Layers + Data Augmentation

In order to continue improving the previous adjustment, we have increased 2 layers to the final neural network (top model), maintaining the same parameters of Data Augmentation of the previous model.

A significant improvement is observed in the accuracy of the model in 'Test Data’:

• ‘Test Data’ Accuracy: 88,12%

Figure 10. A_ccuracy_ graphs and loss of Model 3.

Model 4: VGG19 + 5 Layers + Data Augmentation

At this point, since the results had improved considerably, we decided to do something different, such as a changing the pre-trained network, to continue exploring new paths towards improvement.

We have used the network “VGG19", which contains 3 more layers that the networkVGG16”.

In this case, the results of accuracy They have not improved compared to Model 3:

• ‘Test Data’ Accuracy: 84,53%

Figure 11. A_ccuracy_ graphs and loss of Model 4.

So we have decided to use the network again “VGG16” for the adjustment of the following model.

Model 5: VGG16 + 9 Layers + Data Augmentation

To continue exploring other possible improvements regarding the accuracy of the model, we have decided considerably increase the number of layers, from 5 to 9, maintaining the Data Augmentation previous.

In this case, we have not improved the accuracy regarding the Model 3, although the result has been quite acceptable:

• ‘Test Data’ Accuracy: 86,63%

Figure 12. A_ccuracy_ graphs and loss of Model 5.

Summary Table

The table below details the 5 adjusted models and the line of reasoning followed to adjust them, step by step.

In blue, the changes implemented from the first model (“base”), and the best results of accuracy (> 85%).

Figure 13. Parameters and results of the 5 adjusted models.

Confusion Matrix: Model 3

The percentages of correctness by category from the adjustment of the Model 3 They are generally quite satisfactory.

This is reflected in the Confusion Matrix of the model.

Figure 14. Model 3 Confusion Matrix.

The following categories stand out for their high success rate (>95%): “c3” (typing mobile phone - left hand), “c5” (manipulating the radio) and “c7” (looking in the back).

Figure 15. Correct percentages by category.

The decrease in the percentage of correct answers in the category “c9” (talking to passenger). This happens because the categories “c0” and “c9” could be considered the “less exclusive”, as mentioned previously.

If we take into account “c9” and “c0” both as correct classifications for “c9”, this percentage of success for “c9” increases by 30,2% (from 54.5% to 85.7%), approaching the correct percentages of the other categories.

We have carried out this calculation to try to quantify the impact on accuracy for the fact of having worked with a couple of categories “somewhat less exclusive”.

If the categories “c0” and “c9"would have been defined in a way"more exclusive”, surely we would have achieved some adjustments with better accuracy from the beginning.

On the other hand, we understand that the study has to reflect everyday situations, since they will be the reality to be faced when deploying any model in real life.

Conclusions

The great computational capacity The fact that this type of technique requires has meant that we had to dedicate an important part of the project resource to solve execution problems of the codes used.

We have had to learn to use the services of Amazon Web Services to make it viable, and the models of this project have been able to be trained thanks to the GPUs available from this platform.

Below are some conclusions and lessons learned after developing this project:

1. Not always an increase in layers to the Neural Network is going to improve the accuracy model fit. This fact has been seen in the adjustments of Models 3 and 5.
2. The resource of Data Augmentation must be used with judgment, after an analysis of the initial set of images, since its indiscriminate use can be counterproductive.
3. The teamwork has been fundamental to the success of this project. The different profiles between the members of the project team and the project coordination team Saturdays.AI Donostia has been key to adding a broad and enriching vision.

The road has not been easy and has required many hours of training. But believe us:it has been worth it!

Next Steps

As next steps, we consider the following:

• “Adjust a second, more specific model that learns to detect differences between categories.”c0” and “c9”.

• Retrain the Model 4 (VGG19 + 5 layers + Data Augmentation), but now with more “epochs". The idea is to observe if with more training the network "VGG19"converges to the same results observed with the network"VGG16”.

• Run the Model 3 in one Jetson Nano Nvidia, and through a camera installed in the car, check the effectiveness of the model in detecting inappropriate behavior.

Work Team

And to say goodbye, we would like to introduce ourselves.

Hello. my name is Imanol Areizaga and I am an Industrial Engineer. Since my first work experiences I have been working on digitalization processes in the industrial field. On the way to Data Science you discover new horizons step by step and in this step that I am taking together with Saturdays.AI I am interested in how to apply Deep Learning in the Industrial field. A very demanding area, in which the efficiency of the models has to be very close to 100%.

Hello everyone. my name is Henry Corazza Sef and I'm a statistician. Although I have chosen to work on the implementation of the 6 Sigma Methodology in the industrial area, I am still very interested in Data Science in general. After this course, I have discovered another passion: the enigmatic universe of Neural Networks! ?

We thank the mentors and organizers of Saturdays.AI Donostia, especially to Beñat Galdós, Joxemari Gallastegi and Imanol Echeverria, for his professionalism, dedication and enthusiasm.

Repository

In the following repository you will find the code used to develop this application: https://github.com/SaturdaysAI/Projects/tree/master/Donostia/Donostia2021/focusondriving-main

More artificial intelligence!

The mission of Saturdays.ai is to make artificial intelligence more accessible (#ai4all) through intensive courses and programs where projects for good are carried out (#ai4good).

Find out about our artificial intelligence master's degree in https://saturdays.ai/master-ia-online/

If you want to learn more about artificial intelligence join our community at community.saturdays.ai or visit us on our website www.saturdays.ai Join us!