GANs: Artificial Intelligence to color videos
La Paz. Deep Learning. 2021 Over the last decade, the explosion in the amount of data available, the optimization of algorithms and the constant evolution of computing power have allowed intelligence…
A learning project shared by the community. The article documents its team, methods and results; it does not establish that the prototype is ready for real-world use.
This archive includes automatic translations. Audio, video and text inside images remain in their original language.
La Paz. Deep Learning. 2021
Over the past decade, the explosion in the amount of data available, the optimization of algorithms, and the constant evolution of computing power have enabled artificial intelligence (AI) to perform more and more human tasks [1]. And for some time now, artificial intelligence has become a powerful tool to take us back to the past. This is demonstrated by the increasing number of images and videos restored through different artificial intelligence techniques, such as the ability to colorize black and white videos.
In terms of image and video colorization, Artificial Intelligence has evolved and significant improvement has been observed in recent years. Since manual coloring requires a large amount of time and is an expensive process, the application of artificial intelligence techniques is a great option for this type of task because it is capable of working on its own and coloring automatically in a short period of time [2].
Problematic
There are few film records that capture the daily life of our country in the past, which is why we wanted to have another perspective on these videos by adding color to the few we found.
Objective
Extract images from old videos to give them color using a GAN network, to later generate a color video.
Dataset
The dataset could be obtained from videos found on YouTube, documentary type and recordings of the different landscapes of Bolivia, 10 different videos were used since there are old color videos, which have a more opaque color, other videos of the landscapes with a more defined color but with a quality no higher than 480p, and another with a higher quality. Not all of these frames were taken since it was decided to use the significant frames and thus not have several frames from the same scene.
Model selection and techniques used
The architecture used for coloring using Artificial Intelligence was Unet-GAN due to its great capacity to synthesize the characteristics of the images, which greatly improves the final results.
A generative adversarial network (GAN) has two networks. One is the discriminator that will try to discriminate between real and fake images. And the second network of the GAN is the generator that will be responsible for generating false images but very close to the original images. Rather, we can say that the false images will be in the same distribution as the original images.
UNET is basically used in image segmentation problems where the input and output dimensions are the same. GAN tries to generate new images from random noise. So, the GAN generator has very low dimensions at the input, but the output is high dimensional. You can use UNET as a generator in the GAN (pix2pix uses this).
In this case, it will generate fake images not from random noise but from another reference image. It has other options like deconvolution network, autoencoder, etc.
At the beginning, videos were obtained, the videos were segmented into frames that were used as ground truths, the images were standardized to a size and the RGB frames were converted to grayscale to use them as inputs. At the end of the colorization process, the resulting frames were joined to convert them back into video.
Analysis of results
In graph 1 we can observe the loss of the discriminator and the generator and it can be seen that there is an exponential trend and that the loss is stationary at the value 0.4 for the discriminator and 0 for the generator.
Graph 1. Loss of the discriminator and the generator
It can be seen in the training images with 600 epochs that the color is correct, but not as intense as in the real RGB images. It can also be seen that there are errors when there is a brighter section in the image, such as sun rays or white sections.
With the test images you can see that the images are colored but there is still a problem with the brightness sections and that there is a greater predisposition to the range of reds and blues.
In the images of the black and white video chosen to test our model, there is a good coloration and since it is a recording of the city of La Paz-Bolivia, there are more images of buildings, it is not necessary to have a great variety of colors, the most intense color that is seen is in the sky.
Conclusion and recommendations
It was possible to implement a model with Unet-GAN architecture capable of coloring black and white images and with the set of colored images a color video was obtained. This whole process allowed us to become familiar with Artificial Intelligence models of the Deep Unsupervised Learning type, preparation of datasets, Python language, libraries specialized in Deep learning and Colab as a tool for the implementation of our model. The results are acceptable, however, there are some shortcomings that need to be corrected, such as certain bright images that are part of our training dataset that present errors when reproduced by the GAN, and some colorizations are not complete, this is because the dataset did not have many color variations, therefore it is advisable to expand the color palette with which the training is carried out.project_color.mp4Edit descriptiondrive.google.com
Bibliography
[1] A basic introduction to GANs (Generative Adversarial Networks). (n.d.). Retrieved December 4, 2021, from https://ichi.pro/es/una-introduccion-basica-a-las-gan-generative-adversarial-networks-217887110266867
[2] film colorization, colorize video, colorize black and white videos, colorize video software, film colorization software, colorization of film. (n.d.). Retrieved December 4, 2021, from https://pixbim.com/film-colorization
Codes
- Code for collecting the dataset from videos downloaded from YouTube
- Code for training the model
- Code for the implementation of previously saved model weights
Project presentation: DemoDay
Repository
In the following repository you will find the code used to develop that application: https://github.com/SaturdaysAI/Projects/tree/master/Lapaz/2021.DL/Coloracion-de-videos-main
More artificial intelligence!
The mission of Saturdays.ai is to make artificial intelligence more accessible (#ai4all) through intensive courses and programs where projects for good are carried out (#ai4good).
Explore our AI master’s programme at https://saturdays.ai/master-ia-online/
If you want to learn more about artificial intelligence join our community at community.saturdays.ai or visit us on our website www.saturdays.ai Join us! Saturdays.AI
WRITTEN BY
Daniela Centellas Yucra
Saturdays.AI
Saturdays.AI is an impact-focused organization on a mission to empower diverse individuals to learn Artificial Intelligence in a collaborative and project-based way, beyond the conventional path of traditional education.
A history of the Saturdays.AI community. Originally published on Jan 2, 2022.
← More community stories
