Digitization of photographs and Restoration of photographs with Artificial Intelligence

La Paz. Deep Learning. 2021

La Paz 2021. Artificial Intelligence techniques have many current applications in the field of photographs. One of them has to do with image manipulation, a field in which our project falls.

In particular, image restoration is the process of recovering an image from a degraded version. Image restoration is a case study that is normally treated with image processing. It is based on the idea that the image has gone through a degradation function and noise has been added; The restoration will then consist of reversing the process and recovering the original image.

The purpose of this process is to “compensate” or “undo” those defects that generated the degradation of the image, obtaining an estimate of the original image.

Degradation can come from different sources, such as motion blur, noise, camera blur, or a combination of all of these. We will also talk about degradation when referring to those printed photographs or negatives that suffered any type of wear due to the aging of the paper, dust, stains, etc.

Traditionally, once the type of degradation has been identified, it is possible to process the image with an image processing treatment. There are different studies and specializations regarding these processes, especially to reverse the noise added to the degradation function. Each case is unique, so processing time can be very long. At the same time, we can find photography studios that offer restoration services using digital editing techniques, which is expensive.

Image restoration example

On the other hand, we have the photographic negative films produced by analog cameras that are preserved by various types of users, from historians and librarians to those who simply want to preserve family memories. These films must normally go through a chemical process (developed) to obtain the physical photograph that describes the scene that was captured with the analog camera.

Digital negative development

There are techniques for manipulating the digital image of a negative to obtain the image that would correspond to the one developed in the traditional process. These techniques follow a sequence of steps that consist of adjusting certain image parameters that go beyond color inversion, something that must also be done image by image.

DESCRIPTION OF THE PROBLEM and OBJECTIVE

The selection of the image processing technique that should be applied to the degraded image comes from a visual appreciation and is subjective. Many times we must apply several techniques in cascade to obtain an optimal result. This process is time-consuming, is specific to each case that arises, and in most cases cannot be parallelized.

What if we could use artificial intelligence to create a single model that is capable of restoring the degraded image without the need to categorize the type of degradation, how to also develop photographs from film negatives?

This is precisely our objective. We will use Artificial Intelligence techniques to build a photo restoration model that will be able to recover the original image from the degradations described and/or from photographic negatives. To make the challenge more interesting we will also include images in black and white format that must be colored; and also images with suppressed regions, that is, we will cover regions of the scene simulating stains or scratches that could have completely eliminated these regions. Although there are models that perform certain functions that we describe, one of the difficulties will be to combine all these restorations (colorization, development, restoration and even creative generation) in a single model.

MODEL SELECTION

The project presented is based on several Fast.ai libraries, which provide photo manipulation tools using artificial intelligence, datasets for training and even pre-trained models that are a good starting point. The general architecture chosen is to work with a GAN (Generative Adversarial Network), which is appropriate to solve our problem.

The general structure used consists of an image generator, which tries to create images with the best possible quality, whether in color, resolution, texture... On the other hand, a discriminator is trained that must distinguish between the real images given as input and the images generated by the generator. Finally, a third training is done where the two previous models are combined, which creates the GAN: in this process both models compete and force each other to improve.

Our generator model is already pre-trained and consists of two parts. The first are the convolutions that take an input image to increasingly smaller dimensions, and the other part is when the reverse path is done to recover a new image of the same dimensions as the one that is used as input. The total architecture is UNET, which has the peculiarity that certain information from the image reduction process is sent directly to its counterpart (when the dimensions are equivalent). This process can be seen in the following image.

Join

Likewise, the different operations and convolutions used are based on a Resnet34 CNN (Convolutional Neural Network) architecture (34 layers deep), which uses residual or identity blocks during convolution operations. When we load this model, only the right part (approx. 20 million parameters) can be trained using the freeze() option, or the entire model (approx. 40 million parameters) can be trained with the unfreeze() option. At this point in the process, and seeing that a Mean Squared Error loss function (MSELossFlat()) is used, the generator only tries to get as close as possible to the pixel values ​​of the original image. However, this is not enough to capture, for example, the pixel environment, specifically certain textures that are very important to obtain a correct image with good resolution. That is why the generator is used along with the discriminator in a GAN structure.

The discriminator (also called critic) uses a cross-entropy loss function with logits allowing a binary classification (BCEWithLogitsLoss()), which is well suited to its objective. Before training the GAN, the discriminator can be trained between images created by the generator and the real inputs. However, what we are most interested in is training the GAN.

In this stage, the two generator and discriminator models are trained together. The idea is to use a switcher that will decide if it is time to train the generator to improve the images created and confuse the discriminator, or on the contrary train the discriminator when a certain threshold of images are being classified as real when in reality they are created by the generator.

This process can be carried out iteratively by changing the size of the images with which the model is working, going from smaller dimensions to larger ones. In this way, the model is progressively improving. This technique is used, for example, to improve the resolution of the images given to the Artificial Intelligence model and thus improve the photographs.

PREPARATION AND CONSTRUCTION OF THE DATA SET

It is well known that one of the most important aspects in building any intelligent model is data. Thus, we placed emphasis on obtaining, preparing and constructing a dataset that could provide us with the entire spectrum of degradations that we require our model to be able to reconstruct and that we have the necessary quantity and variety.

We work with the VOC2012 dataset and ColorizationDataSet as initial data (varied color images without defects). We decided to use 9895 images in total. An artificial negativization process was applied to one half thanks to image preprocessing and for the other half, black and white images were taken into account. A second image processing was applied to all these images where some type of degradation was applied to 65% such as jpeg compression, salt and pepper noise, blur, among others; and at 25% a more important degradation was applied, such as holes or spots in various sectors of the images. Below we can see several examples of this treatment.

Artificially generated degraded images

To train the model, the dataset was divided into a train set of 8906 images and a test set of 989 images.

MODEL EVALUATION

To train our model, we use the GPU provided by Google Colab. Before training the GAN, the generator and the discriminator can be trained with their respective loss functions detailed in point 4.

For the generator, we see that there is a certain improvement as the number of epochs increases, both for the train set and the test set. Keep in mind that this is only part of the training of the generated, which will actually be done during the training of the GAN.

Loss function curve in one of the Generator trainings

For the discriminator, it can also be trained alone, and at this stage we can reach an accuracy of up to 95%, which shows that before training the GAN, it is very easy for it to distinguish real images from those produced by the generator.

During the training of the GAN, other somewhat modified loss functions are used for both the generator and discriminator. However, to evaluate the final model, a visual verification is needed since the loss functions of both do not provide us with relevant information because both functions will improve and worsen depending on which model is being trained.

ANALYSIS OF RESULTS

To train the Artificial Intelligence model specialized in photographs, a data set of 9895 images of photographic negatives (artificial) and 9895 color images was prepared, each of the negatives has a related color image.

The data set was divided into 2 groups, one for training and the other for validation, with the following detail:

– Train (8906 images)

– Valid (989 images)

The following groups of images were considered for training:

a) Black and white images.

b) Images without degradation.

c) Images with Gaussian noise.

d) Images with degradation and suppression of regions

Train Results:

– Black and white images

Image: image2669.jpg

From left to right: Original image, black and white gradient image, prediction in training

– Negative images without degradation

Image: 2007_000039.jpg

From left to right: Original Image, Negative Image, prediction in training

We can see that the model does a good job of coloring certain negative images, finding the correct color in most objects in the image.

– Negative images with Gaussian noise.

Image: 2007_003118.jpg

From left to right: Original image, negative image with noise, prediction in training

Noise restoration towards better resolution is not fully realized, even for the train set. We can conclude that more training time is needed to continue with the process.

– Negative images with excised regions

Image: 2007_000033.jpg

From left to right: Original Image, Negative image with region suppression, prediction in training

Image: 2007_000027.jpg

From left to right: Original Image, Negative image with region suppression, prediction in training

For degraded images with entire parts missing, the model recognizes what color to give to the dark area. At the moment, the resolution is poor, but with more training, this may improve.

Test Set Results:

– Black and white images.

Image: image0476.jpg

From left to right: Original image, black and white gradient image, prediction in testing

– Negative images without degradation.

Image: image1027.jpg

From left to right: Original Image, Negative Image, prediction in testing

– Images with Gaussian noise.

Image: image0850.jpg

From left to right: Original Image, Image with noise, prediction in testing

– Images with Degradation

Image: image4988.jpg

From left to right: Original Image, Negative image with region suppression, prediction in training

Results of images that are not part of the dataset:

Below we present model prediction results on completely new images that were not used during training. We can observe that depending on the image in black and white, the colorization is done in an acceptable way in some but almost not at all in others. Likewise, in the second image it is observed that the result image has improved the image quality in terms of the noise that the original presents. As for negative images, the model does a good job of detecting the objects in the image and coloring them accordingly.

CONCLUSION

This Deep Learning project allowed us to become familiar with certain image manipulation techniques, Python code, libraries specialized in Deep Learning as well as model training techniques in data science, using Artificial Intelligence applying it to photographs. The results are acceptable considering the limitations in training time as well as in GPU that were had. We saw that the model can work with both photographic negatives and old black and white photographs that could present slight or strong degradation. However, there is still a lot of room for improvement, as for example we saw that some restorations of negatives have a blue tint in the background, probably due to the fact that only artificially created and not “real” negatives have been used. These negatives may have different chemical qualities, materials... that may vary by model or brand. Likewise, using real gradient images is likely to increase the power of the model.

BIBLIOGRAPHY

• VOC2012 DataSet — http://host.robots.ox.ac.uk/pascal/VOC/voc2012/
• Image Colorization DataSet— https://www.kaggle.com/aayush9753/image-colorization-dataset
• https://www.fast.ai/2019/05/03/decrappify/
• Ziyu Wan, Bo Zhang, Dongdong Chen, Pan Zhang, Dong Chen, Jing Liao, and Fang Wen. Old photo restoration via deep latent space translation. arXiv preprint arXiv:2009.07047, 2020.

Saturdays.AI

• Gans
• Deep Learning
• Image Restoration
• computer vision
• Ai Saturdays

Silvana Dávila

Project presentation: DemoDay

Open video or resource

More Artificial Intelligence!

The mission of Saturdays.ai is to make artificial intelligence more accessible (#ai4all) through intensive courses and programs where projects for good are carried out (#ai4good).

Explore our AI master’s programme at https://saturdays.ai/master-ia-online/

If you want to learn more about artificial intelligence join our community at community.saturdays.ai or visit us on our website www.saturdays.ai Join us!