AImagining Tales

AImagining Tales began during the fifth edition of the course run by Saturdays AI Madrid, as a proposal for the challenge of creating a social impact project using Artificial Intelligence.

The purpose of this project is to be able to illustrate stories.

Image Generation by AImagining Tales

Who are we?

Our team followed the Deep Learning track. We were interested in both image and text processing from the start, so we decided to combine them.

• David Cayuelas García
• Marta Díez Díez
• Alejandra De Francisco Rus
• Claudia Lucio Sarsa
• Pablo Rodríguez Lapetra

What is our motivation?

We want to explore whether visual representations of texts can help people with learning difficulties, particularly children. That is the motivation behind this prototype; we have not yet evaluated its educational effectiveness.

Additionally, we seek to promote children's writing, so that the child can imagine and write a story and then see it captured in self-generated images.

In some way it reminds us of the colorful stained glass windows of churches. Its original purpose was to represent biblical passages to transmit to the people, most of whom were illiterate, its teachings, which they were not otherwise able to access.

AImagining Tales aims to make reading and writing more approachable.

What is our solution?

We want to build a tool capable of illustrating stories through Artificial Intelligence.

In its basic conception, our tool would process the text of a story as input to generate an image representative of its content. The original texts, accompanied by their corresponding images, would be compiled in a PDF document.

For image generation, we chose VQGAN+CLIP, a combination of models that interested us at the time of the project. Because input length is limited, we need to summarise passages before generating illustrations.

It has been decided to develop a Natural Language Processing (NLP) model to summarize the original texts. This model will be based on existing summary models, retrained for our specific task: the interpretation and summary of stories.

Finally, text and images will be compiled into a document in PDF format.

With all this, the flow would be the following:

The story would go through an NLP summary model, which would output the summarized text. This result would be the input of the VQGAN+CLIP network, from which its representation would be obtained. Once the stages were completed, the story text and its images would be combined into a PDF document for your use (each step is illustrated below).

1. The dataset: Shmoop Corpus

Training a summariser presents an important challenge: we need narratives containing descriptions and dialogue, paired with human-written summaries. Preparing that material is costly and can limit a project’s feasibility.

Fortunately, we found that a group of researchers at the University of Toronto had already faced this problem. In order to work on summary models, they propose their own data set: Shmoop Corpus.

Shmoop Corpus consists of 231 stories with 7,234 chapters in total, with their corresponding summaries. The stories, as indicated in the original paper, have been extracted from Project Gutenberg and the summaries from the Shmoop website, which gives its name to the corpus. The true differentiating element in this dataset is that, in addition, they provide the keys to relate the text-summary pair.

The Shmoop Corpus team provides the set of files that contain both the complete stories organized in folders (one for each chapter) and the files in which the text-summary alignment/keys are indicated. To obtain the summaries from the Shmoop website we have resorted to web scraping practices (web scraping).

Example of a text-summary pair from the original Shmoop Corpus paper.

To group these data and finally obtain a dataset in the appropriate format to train the model, an aggregation task is carried out on the disaggregated information at different file levels (texts, summaries, text-summary keys).

The result of this processing, a dataframe with a column of paragraphs and another of summaries, is the training input of the summarizer.

2. Word processing: Summarizer

After collecting and processing the data, we are ready to train our summarizing model, hereinafter the summarizer.

A summarisation model aims to produce a shorter version that preserves the main information. Extractive methods select original passages; abstractive methods generate new text. T5 is a generative model that may reuse parts of the original, but its summaries can also omit details or introduce errors. summarizer.

Summary of a text after passing it through the Summarizer.

Currently, there are many pre-trained models for this purpose, most of them based on transformers. These are collected on the platform Hugging Face, a project specialized in NLP tasks and that seeks to democratize access to AI by providing tools, datasets, models, etc. open and for the community.

In particular, in this project we have taken as a basis both the tokenizer like him summariser of the model T5 from Google and we have retrained it with our own data set so that it is able to interpret and summarize not only narrations but also dialogues. This retraining procedure to adapt it to a new task is what is known as transfer learning.

A quick guide: a tokenizer splits text into units called tokens and assigns them numerical IDs. The model transforms those IDs into vector representations, or embeddings. Tokenisation and embedding are related but distinct steps.

By fine-tuning on our data, we aim to improve literary summaries. That improvement needs to be evaluated; retraining does not guarantee it. The text produced by the summarizer will serve as input to the image generator.

3. Image generation: VQGAN + CLIP

After applying the summarizer to our texts, it is time to use these summaries to generate the images that will illustrate our stories. To do this, we have focused on image generation models from text.

After doing the corresponding investigation of the current state of the art (beyond DALL-E and DALL-E 2), we found a very interesting implementation that unites VQGAN and CLIP (Vector Quantized Generative Adversarial Network and Contrastive Language–Image Pre-training).

GAN networks (Generative Adversarial Networks) are systems composed of two neural networks, which basically “face off” against each other. A generator that synthesizes images or data, and a discriminator, which learns to distinguish generated data from real data. Training alternates between the two components, but does not guarantee continuous improvement.

Our model is based on this concept. VQGAN provides the generative image representation, while CLIP scores the correspondence between text and image. That score guides generation towards the input text.

Additionally, in this process you can add a multitude of parameters: reference images, styles, more than a single text, the size of the generated image, etc.

Representation of “Fantasy city on a sunny day” with different styles.

Of course, the challenge has been to understand this already trained model, to know which parameters best fit our project, as well as the input texts that give the best results (not too short, not too long, differences between a single meaningful text or key concepts...).

4. Story generation: PDF generator

Finally, it is time to join the result of all the steps in our illustrated story.

To do this, we have developed a series of functions that create the story in PDF format. Here we join the initial texts, along with the images generated from the summaries.

After considering different Python library options that offer this functionality (ReportLab, PyPDF2), we decided on PyFPDF for the simplest handling.

The first page is a cover with the title and an image generated from it. For subsequent pages, we offer layouts that vary the position of the illustration.

Additionally, we have decided to give our documents a storybook style, with a border and details in the corners.

“Beauty and the Beast” illustrated by AImagining Tales.

Next steps and reflections

This proposal is a first iteration, a proof of concept, the result of the collaborative work of our team over six weeks.

Through our project we want to highlight the immense potential that Artificial Intelligence offers to provide solutions to real current problems, solutions that are within everyone's reach.

There is still more to be done to make AImagining Tales a reality.

And that is how our story ends…

If you have been left wanting more and want to review the code in greater depth, we invite you to visit our Github repository.

More artificial intelligence!

Saturdays.AI makes AI more accessible through shared learning and projects with a social purpose. Our motto is AI4ALL IS AI4GOOD.

Explore the available courses

Meet the Saturdays.AI community