Manual invoice data detection
A La Paz prototype that extracts invoice identifiers using optical character recognition and exports them to simplify data entry.
A learning project shared by the community. The article documents its team, methods and results; it does not establish that the prototype is ready for real-world use.
La Paz. Deep Learning. 2021
Introduction
Invoice reporting is an obligation for certain taxpayers in Bolivia, including employees and self-employed workers. Data from some electronic invoices can be imported by scanning a QR code, but manual invoices require users to type the tax identification number (NIT), authorisation number and invoice number. This takes time and can be especially difficult for people with limited experience of digital tools.
Figure 1. Images of a Manual Invoice
Figure 2. Images of a Manual Invoice and a Digital Invoice
We therefore developed an optical character recognition prototype that identifies these values and exports them to a CSV or XLSX file for import into SIAT, Bolivia’s Integrated Tax Administration System.
Figure 3. Main screen of the SIAT platform, for importing billing data.
Figure 4. Import button for xls or xlsx files.
Problem description
The problem identified is the amount of time invested when transcribing the information from each of the manual invoices; both for people with limited time due to multiple personal or professional activities, and for people with little computational skill.
Objective
The project aims to simplify extraction of the longest identifiers on manual invoices: the company’s NIT, the authorisation number and the invoice number. It then exports them as CSV or XLSX files for import into SIAT’s Mis Facturas section.
Model selection
CRAFT
CRAFT is used (Character-Region Awareness For Text detection) [1], because it locates individual character regions and groups them into text. Tesseract was another OCR option, although curved text and irregular letter shapes posed difficulties for our use case.
Figure 5. Schematic illustration of the CRAFT Model Architecture
The architecture uses VGG-16 [2], a convolutional neural network, to extract features from the input image. These representations feed into CRAFT’s decoding stage.
Figure 6. VGG-16 CNN model.
Techniques implemented
The techniques implemented by the CRAFT model are the basic ones for character detection, such as: cuts, rotations and/or color variations.
Figure 7. Character splitting procedure to achieve character-level annotations from word-level annotations: 1) The image is cropped at the word level; 2) The region score is predicted; 3) The algorithm is applied watershed; 4) The character bounding boxes are obtained; 5) Character bounding boxes are unpacked.
Model evaluation
We used the pretrained CRAFT model included in KERAS-OCR [3]. The following comparison shows its results against other text-detection models:
.
Figure 8. Comparison of different models, where the ICDAR and MSRA-TD500 datasets were used. Where: P is the Precision, R is Recall, H is the H-mean and FPS is the frames analyzed per second [4].
Analysis of results
The system initially detects text across the entire invoice. In our examples, the identifiers we need are in the right-hand half, so we crop the image to reduce the amount of information to process. Figure 9 shows the text detected without that crop.
Figure 9. Character detection of a manual invoice.
We convert the detected text into strings and organise it in a Pandas dataframe. We then look for candidates by length: 8–10 characters for the supplier’s NIT, 14–16 for the authorisation number and 4–6 for the invoice number. These rules help us locate the values in the examples we analysed.
Figure 10. Detection of desired values, transformation to a dataframe with Pandas and conversion of the data to STRING for later comparison.
Finally, we save these values in a dataframe with the structure required by SIAT’s Mis Facturas section and export them as CSV or XLSX, as shown in Figure 11.
Figure 11. Export of generated data to CSV and XLSX files.
Conclusion
Through this prototype, you can identify the most relevant and complicated values to transcribe for the majority of people who file their invoice declaration. In addition, importing them from a photo or scanned image drastically reduces data transcription time. Which would be very helpful for people not very familiar with computers, such as a good percentage of older adults in the country.
Although many values are identified, image quality remains the main limitation. Insufficient resolution or sharpness makes some characters difficult to recognise. Future work could evaluate training specifically for these invoices alongside the pretrained model.
A graphical user interface (GUI) would also make the prototype easier to use, from uploading photographs or scanned invoices to reviewing and exporting the recognised data.
Code
https://github.com/albmarale/SaturdaysAIDeepLearning
Bibliography
[1] “PyTorch: Scene Text Detection and Recognition by CRAFT and a Four-Stage Network | by Nikita Saxena | Towards Data Science.” https://towardsdatascience.com/pytorch-scene-text-detection-and-recognition-by-craft-and-a-four-stage-network-ec814d39db05 (accessed Jul. 13, 2021).
[2] M. ul Hassan, “VGG16-Convolutional Network for Classification and Detection,” Available online: https://neurohive.io/en/popular-networks/vgg16/ [accessed 10 April 2019]., 2018.
[3] F. Morales, “keras-ocr — keras_ocr documentation,” 2021, Accessed: 13-Jul-2021. [Online]. Available: https://keras-ocr.readthedocs.io/en/latest/.
[4] Y. Baek, B. Lee, D. Han, S. Yun, and H. Lee, “Character region awareness for text detection,” in Proceedings of the IEEE/CVF Conference on Computer Vision and Pattern Recognition, 2019, pp. 9365–9374.
This project was prepared by:
- Albert Martínez Alegría
- Alvaro Alanoca Huaycho
- Belinda Alcón Sullcani
- Rodrigo Aliaga
for the program Saturdays.AI La Paz.
Project presentation: DemoDay
Repository
In the following repository you will find the code used to develop that application: https://github.com/SaturdaysAI/Projects/tree/master/Lapaz/2021.DL/Deteccion-facturas-manuales-main
More artificial intelligence!
Saturdays.AI makes AI more accessible through shared learning and projects with a social purpose. Our motto is AI4ALL IS AI4GOOD.
If you want to learn more about this project (and others) join our community or learn to build your own through our AI Saturdays programme.
Written by
A story from the Saturdays.AI community. Originally published on Nov 22, 2021.
← More community stories
