GANs: Artificial Intelligence to color videos

La Paz. Deep Learning. 2021

Over the past decade, the explosion in the amount of data available, the optimization of algorithms, and the constant evolution of computing power have enabled artificial intelligence (AI) to perform more and more human tasks [1]. And for some time now, artificial intelligence has become a powerful tool to take us back to the past. This is demonstrated by the increasing number of images and videos restored through different artificial intelligence techniques, such as the ability to colorize black and white videos.

In terms of image and video colorization, Artificial Intelligence has evolved and significant improvement has been observed in recent years. Since manual coloring requires a large amount of time and is an expensive process, the application of artificial intelligence techniques is a great option for this type of task because it is capable of working on its own and coloring automatically in a short period of time [2].

The problem

There are few film records that capture the daily life of our country in the past, which is why we wanted to have another perspective on these videos by adding color to the few we found.

Objective

Extract images from old videos to give them color using a GAN network, to later generate a color video.

Dataset

We built the dataset from ten YouTube videos: documentaries and recordings of Bolivian landscapes. They included older colour footage with muted tones, videos up to 480p and higher-quality material. We selected representative frames to avoid including many near-duplicates from the same scene.

Model selection and techniques used

The architecture used for coloring using Artificial Intelligence was Unet-GAN due to its great capacity to synthesize the characteristics of the images, which greatly improves the final results.

A generative adversarial network (GAN) combines a generator and a discriminator. The generator creates images, while the discriminator tries to distinguish them from real ones. Training aims to bring the generated images closer to the distribution of the original data.

U-Net is commonly used for segmentation, with similarly sized inputs and outputs. It can also serve as the generator in a conditional GAN, as in pix2pix, where the output depends on an input image rather than only random noise.

In this case, it will generate fake images not from random noise but from another reference image. It has other options like deconvolution network, autoencoder, etc.

We extracted video frames, standardised their dimensions and kept the RGB images as reference targets. We converted them to greyscale for use as inputs. After colourising the frames, we combined them to reconstruct the video.

Analysis of results

In graph 1 we can observe the loss of the discriminator and the generator and it can be seen that there is an exponential trend and that the loss is stationary at the value 0.4 for the discriminator and 0 for the generator.

Graph 1. Loss of the discriminator and the generator

After 600 epochs, the training examples showed plausible colours, although less intense than the originals. Errors also appeared in bright regions such as sunlight or white surfaces.

Test images still showed errors in bright regions and a tendency towards red and blue tones.

The black-and-white test video showed buildings in La Paz. We found the colourisation visually plausible, with particularly vivid sky colours. These are model estimates, not a verified recovery of the original historical colours.

Conclusion and recommendations

We implemented a U-Net-GAN model to colourise black-and-white frames and reconstruct a video. The process gave us practice with data preparation, Python, deep learning and Google Colab. Bright areas and incomplete colourisation remain problems. Expanding the dataset’s range of colours and scenes is a potential improvement to evaluate.Watch the colourised video

Bibliography

[1] A basic introduction to GANs (Generative Adversarial Networks). (n.d.). Accessed 4 December 2021 at https://ichi.pro/es/una-introduccion-basica-a-las-gan-generative-adversarial-networks-217887110266867

[2] film colorization, colorize video, colorize black and white videos, colorize video software, film colorization software, colorization of film. (n.d.). Accessed 4 December 2021 at https://pixbim.com/film-colorization

Codes