AImagining Tales

AImagining Tales is a project that was born in the fifth edition of the Saturdays AI Madrid, as a proposal for the challenge of creating a social impact project using Artificial Intelligence.

The purpose of this project is to be able to illustrate stories.

Image Generation by AImagining Tales

Who are we?

The members of the team belong to the Deep Learning itinerary. From the beginning we have been very interested in image and text processing, so why not combine both?

• David Cayuelas García
• Marta Diez Diez
• Alejandra De Francisco Rus
• Claudia Lucio Sarsa
• Pablo Rodríguez Lapetra

What is our motivation?

Through this initiative we are committed to the use of Artificial Intelligence to help people (particularly children) with learning problems, providing them with a very attractive and visual tool for representing texts.

Additionally, we seek to promote children's writing, so that the child can imagine and write a story and then see it captured in self-generated images.

In some way it reminds us of the colorful stained glass windows of churches. Its original purpose was to represent biblical passages to transmit to the people, most of whom were illiterate, its teachings, which they were not otherwise able to access.

AImagining Tales was born with the intention of eliminating the main barriers to access to reading and writing.

What is our solution?

We want to build a tool capable of illustrating stories through Artificial Intelligence.

In its basic conception, our tool would process the text of a story as input to generate an image representative of its content. The original texts, accompanied by their corresponding images, would be compiled in a PDF document.

To generate the images we are going to use one of the most innovative existing solutions (models) for image generation: VQGAN+CLIP. However, these solutions present restrictions regarding the size of the text they receive as input, so treatment will be necessary to reduce the number of words.

It has been decided to develop a Natural Language Processing (NLP) model to summarize the original texts. This model will be based on existing summary models, retrained for our specific task: the interpretation and summary of stories.

Finally, text and images will be compiled into a document in PDF format.

With all this, the flow would be the following:

The story would go through an NLP summary model, which would output the summarized text. This result would be the input of the VQGAN+CLIP network, from which its representation would be obtained. Once the stages were completed, the story text and its images would be combined into a PDF document for your use (each step is illustrated below).

1. The dataset: Shmoop Corpus

When training a tool like the summarizer, a very important challenge arises: you need stories or stories summarized by people and, furthermore, that include both descriptions and dialogues. This task is costly and could pose a crucial limitation to the viability of the project.

Fortunately, we found that a group of researchers at the University of Toronto had already faced this problem. In order to work on summary models, they propose their own data set: Shmoop Corpus.

Shmoop Corpus consists of 231 stories with 7,234 chapters in total, with their corresponding summaries. The stories, as indicated in the original paper, have been extracted from Project Gutenberg and the summaries from the Shmoop website, which gives its name to the corpus. The true differentiating element in this dataset is that, in addition, they provide the keys to relate the text-summary pair.

The Shmoop Corpus team provides the set of files that contain both the complete stories organized in folders (one for each chapter) and the files in which the text-summary alignment/keys are indicated. To obtain the summaries from the Shmoop website we have resorted to web scraping practices (web scraping).

Excerpt of an example of a text-abstract pair from the original Shmoop Corpus paper.

To group these data and finally obtain a dataset in the appropriate format to train the model, an aggregation task is carried out on the disaggregated information at different file levels (texts, summaries, text-summary keys).

The result of this processing, a dataframe with a column of paragraphs and another of summaries, is the training input of the summarizer.

2. Word processing: Summarizer

After collecting and processing the data, we are ready to train our summarizing model, hereinafter the summarizer.

A summary model is an NLP model capable of processing text to return a reduced version of it, so that the most important information is retained. The summary obtained can be an extract from the original text, a text generated by the model itself based on its interpretation or a combination of both, as is the case of our summarizer.

Summary of a text after passing it through the Summarizer.

Currently, there are many pre-trained models for this purpose, most of them based on transformers. These are collected on the platform Hugging Face, a project specialized in NLP tasks and that seeks to democratize access to AI by providing tools, datasets, models, etc. open and for the community.

In particular, in this project we have taken as a basis both the tokenizer like him summerizer of the model t5 from Google and we have retrained it with our own data set so that it is able to interpret and summarize not only narrations but also dialogues. This retraining procedure to adapt it to a new task is what is known as transfer learning.

Note to lost surfers: Tokenization is the step prior to training the model on NLP tasks. Typical neural networks are trained by performing a bunch of tensor operations on data in numerical format. When we work with text it is necessary to convert it to numerical format so that the network is able to interpret it. A tokenizer is the component in charge of carrying out said conversion, generally based on tokenization mechanisms. one hot encoding or embeddings.

Once trained with our data, the new model should be able to summarize literary content better than the original. This output summary of summarizer It will be consumed by the image generating model.

3. Image generation: VQGAN + CLIP

After applying the summarizer to our texts, it is time to use these summaries to generate the images that will illustrate our stories. To do this, we have focused on image generation models from text.

After doing the corresponding investigation of the current state of the art (beyond DALL-E and DALL-E 2), we found a very interesting implementation that unites VQGAN and CLIP (Vector Quantized Generative Adversarial Network and Contrastive Language–Image Pre-training).

GAN networks (Generative Adversial Networks) are systems composed of two neural networks, which basically “face off” against each other. A generator that synthesizes images or data, and a discriminator, which scores the performance of the previous one. The process is recursive, so both networks improve their results after the corresponding training.

Our model is based on this concept. VQGAN It is the generative neural network, trained to generate images similar to others. While CLIP is the neural network capable of determining “how well” the input text matches the generated image.

Additionally, in this process you can add a multitude of parameters: reference images, styles, more than a single text, the size of the generated image, etc.

Representation of “Fantasy city on a sunny day” with different styles.

Of course, the challenge has been to understand this already trained model, to know which parameters best fit our project, as well as the input texts that give the best results (not too short, not too long, differences between a single meaningful text or key concepts...).

4. Story generation: PDF generator

Finally, it is time to join the result of all the steps in our illustrated story.

To do this, we have developed a series of functions that create the story in PDF format. Here we join the initial texts, along with the images generated from the summaries.

After considering different Python library options that offer this functionality (ReportLab, PyPDF2), we decided on PyFPDF for the simplest handling.

Obviously, the first page will consist of a cover with a title and an image generated from it, while for the rest of the pages we can find different formats, which differ in the location of the image.

Additionally, we have decided to give our documents a storybook style, with a border and details in the corners.

“Beauty and the Beast” illustrated by AImagining Tales.

Next steps and reflections

This proposal is a first iteration, a proof of concept, the result of the collaborative work of our team over six weeks.

Through our project we want to highlight the immense potential that Artificial Intelligence offers to provide solutions to real current problems, solutions that are within everyone's reach.

There is still more to be done to make AImagining Tales a reality.

And that is how our story ends…

If you have been left wanting more and want to review the code in greater depth, we invite you to visit our Github repository.

More artificial intelligence!

The mission of Saturdays.ai is to make artificial intelligence more accessible (#ai4all) through intensive courses and programs where projects for good are carried out (#ai4good).

Explore our AI master’s programme at https://saturdays.ai/master-ia-online/

If you want to learn more about artificial intelligence join our community at community.saturdays.ai or visit us on our website www.saturdays.ai Join us!