Manual invoice data detection

La Paz. Deep Learning. 2021

Introduction

The declaration of invoices is one of the duties that a percentage of the Bolivian population has, whether they are dependent or independent taxpayers. Although it is possible to import data from an electronic invoice by scanning a QR code. Performing said transcription of a manual invoice becomes more complicated, slow and time-consuming when transcribing the NIT number, the authorization number and the invoice number of each one. Especially for people who are not very familiar with computer systems, this being an important part of the adult population in Bolivia.

Figure 1. Images of a Manual Invoice

Figure 2. Images of a Manual Invoice and a Digital Invoice

For this reason, this project is a prototype of an optical character recognition system that identifies the previously mentioned values. And therefore allows the user to export said information in a CSV or XLSX file, for easy import into the SIAT invoice declaration platform (Integrated Tax Administration System).

Figure 3. Main screen of the SIAT platform, for importing billing data.

Figure 4. Import button for xls or xlsx files.

Problem description

The problem identified is the amount of time invested when transcribing the information from each of the manual invoices; both for people with limited time due to multiple personal or professional activities, and for people with little computational skill.

Objective

The project seeks to facilitate the detection of the longest data from manual invoices, these being the company's NIT number, the authorization number, and the invoice number. For subsequent export in CSV or XLSX formats for subsequent import on the My Invoices page of the SIAT (Integrated Tax Administration System).

Model selection

CRAFT

CRAFT is used (Character-Region Awareness For Text detection) [1], this because it allows us to locate individual character regions and link the detected characters to a text instance. You could have used Tesseract which is an ORC module but it fails on texts with curves and irregular shapes in certain types of fonts.

Figure 5. Schematic illustration of the CRAFT Model Architecture

The architecture is based on the CNN convolutional neural network, VGG-16 [2], it is essentially for feature extraction which is used to encode the input of the network into a certain feature representation and the decoding segment of the CRAFT network.

Figure 6. VGG-16 CNN model.

Techniques implemented

The techniques implemented by the CRAFT model are the basic ones for character detection, such as: cuts, rotations and/or color variations.

Figure 7. Character splitting procedure to achieve character-level annotations from word-level annotations: 1) The image is cropped at the word level; 2) The region score is predicted; 3) The algorithm is applied watershed; 4) The character bounding boxes are obtained; 5) Character bounding boxes are unpacked.

Model evaluation

Being a pre-trained KERAS-OCR model, with the CRAFT model [3]. We can see its effectiveness with respect to other similar models for character detection:

.

Figure 8. Comparison of different models, where the ICDAR and MSRA-TD500 datasets were used. Where: P is the Precision, R is Recall, H is the H-mean and FPS is the frames analyzed per second [4].

Analysis of results

Once the invoice image is imported, all the values that are detected in it are obtained. So we cut the image in two, to only have the right part; where the data we are trying to collect is located. Since, as seen in Figure 9, the amount of data is excessive.

Figure 9. Character detection of a manual invoice.

Once we have all the data detected and converted into STRING type, in a dataframe created with Pandas. We compare each of the columns, looking for the desired text. For this reason, we compare the number of characters in the supplier's NIT (8 to 10 characters), the Authorization Number (14 to 16 characters) and the Invoice Number (4 to 6 characters). Therefore, we find the desired value after searching for columns.

Figure 10. Detection of desired values, transformation to a dataframe with Pandas and conversion of the data to STRING for later comparison.

Finally, we save these values in a dataframe created, with the same distribution that the system of my SIAT platform invoices requests. And we export in CSV, or XLSX. And as seen in figure 11.

Figure 11. Export of generated data to CSV and XLSX files.

Conclusion

Through this prototype, you can identify the most relevant and complicated values to transcribe for the majority of people who file their invoice declaration. In addition, importing them from a photo or scanned image drastically reduces data transcription time. Which would be very helpful for people not very familiar with computers, such as a good percentage of older adults in the country.

Although, many data are identified. The biggest problem lies in the quality of the image taken, since if it does not have good sharpness or size you may have problems with the identification of some characters. And that for future work we could try to solve it with more personalized training and not based on a pre-trained one.

In turn, creating a GUI (Graphic User Interface) would help a lot in making this prototype more user-friendly. And therefore, it facilitates the import of images or scanned invoices for character recognition and final export.

Code

https://github.com/albmarale/SaturdaysAIDeepLearning

Bibliography

[1] “PyTorch: Scene Text Detection and Recognition by CRAFT and a Four-Stage Network | by Nikita Saxena | Towards Data Science.” https://towardsdatascience.com/pytorch-scene-text-detection-and-recognition-by-craft-and-a-four-stage-network-ec814d39db05 (accessed Jul. 13, 2021).

[2] M. ul Hassan, “VGG16-Convolutional Network for Classification and Detection,” Available online: https://neurohive.io/en/popular-networks/vgg16/ [accessed 10 April 2019]., 2018.

[3] F. Morales, “keras-ocr — keras_ocr documentation,” 2021, Accessed: 13-Jul-2021. [Online]. Available: https://keras-ocr.readthedocs.io/en/latest/.

[4] Y. Baek, B. Lee, D. Han, S. Yun, and H. Lee, “Character region awareness for text detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 9365–9374.

This project was prepared by:

• Albert Martínez Alegría
• Alvaro Alanoca Huaycho
• Belinda Alcón Sullcani
• Rodrigo Aliaga

for the program Saturdays.AI La Paz.

Project presentation: DemoDay

Repository

In the following repository you will find the code used to develop that application: https://github.com/SaturdaysAI/Projects/tree/master/Lapaz/2021.DL/Deteccion-facturas-manuales-main

More artificial intelligence!

The mission of Saturdays.ai is to make artificial intelligence more accessible (#ai4all) while carrying out social impact projects (#ai4good).

Explore our AI master’s programme at https://saturdays.ai/master-ia-online/

If you want to learn more about this project (and others) join our community in or learn how to create yours in our AI Saturdays program.

WRITTEN BY

Albert Mario Martínez Alegría