res.txt: Text summary and translation with Machine Learning

La Paz. 2021

Introduction

Text summarization is the process of extracting the most important information from a source text to produce an abbreviated version for a user. We humans perform the summarization task as we have the ability to understand the meaning of a document and extract salient features to summarize documents using our own words. However, automatic methods for text summarization are crucial in today's world, where there is a lack of workers and time to interpret data.

Figure 1:  Amount of data generated annually

Problem

Figure 2: Predominant languages on the internet

Nowadays, with the constant growth of information, being updated on any topic is a necessity. But the sources of information are usually extensive and have a large amount of text, but the most important thing is that the information is in English.

Objective

Develop an application that can help the understanding of a text document in English and shorten the understanding time of a long text.

Proposed solution

Make the application based on DeepLearning methods focused on attention mechanism. These techniques mimic cognitive attention. This causes enhancement in the most important parts of the input data and fades out the rest.

Dataset

CNN Daily Mail is a dataset of more than 300 thousand articles in English written by journalists at CNN and Daily Mail. The size of the dataset is 1.27 GiB in the current version, it supports extractive and abstract summarization, although the original version was created for automatic reading and understanding and answering abstract questions. The data can be used to train a model for abstract and extractive summarization

DL model

Figure 3: BART scheme

BART is an architecture that is based on both BERT and GPT, BART uses the standard seq2seq transformer architecture but, after GPT, we modified the activation functions from ReLU to GeLU. The architecture is closely related to BERT, with the following differences:

1. Each layer of the decoder additionally performs cross-attention on the final hidden layer of the encoder (as in the sequence-to-sequence model of the transformer).
2. BERT uses a feed-forward network before word prediction, which BART does not do.

So in a nutshell it uses text understanding or input from BERT and GPT for output or word generation.

To work with this model, it was done using the base model that HuggingFace provides us.

HuggingFace is an Open Source platform that focuses on NLP, helping us build, train and deploy state-of-the-art models. In this case it helped us build the model and with the dataset.

Model training

The project was carried out based on the models provided by the Hugging Face page, which has a large collection of both models and datasets for NLP (Natural Language Processing) tasks. Of all the options, the BART abstractive summary model was selected.

The code makes use of the HuggingFace “datasets” and “transformers” libraries, as well as nltk to perform the evaluation.

# Imported libraries
from datasets import load_dataset, load_metric
from transformers import AutoTokenizer
from transformers import AutoModelForSeq2SeqLM,
DataCollatorForSeq2Seq, Seq2SeqTrainingArguments, Seq2SeqTrainer
import nltk
import numpy as np
from transformers import pipeline

To perform the fine tuning, the “cnn_dailymail” dataset was used, for the evaluation the “rouge” metric was used, and the base model used was “facebook/bart-base”.

raw_datasets = load_dataset(“cnn_dailymail”, “3.0.0”)
metric = load_metric(“rouge”)
model_checkpoint = “facebook/bart-base”

A tokenizer and a model were loaded referencing the selected base model.

tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint)

From the “cnn_dailymail” dataset, the elements found under the “article” and “highlights” labels were extracted as input and output data respectively.

max_input_length = 512
max_target_length = 128def preprocess_function(examples):
inputs = [doc for doc in examples[“article”]] model_inputs = tokenizer(inputs, max_length=max_input_length, truncation=True)
# Setup the tokenizer for targets
with tokenizer.as_target_tokenizer():
labels = tokenizer(examples[“highlights”], max_length=max_target_length,
truncation=True)model_inputs[“labels”] = labels[“input_ids”] return model_inputstokenized_datasets = raw_datasets.map(preprocess_function, batched=True)

Before training the model, hyperparameter configuration is performed.

batch_size = 4
args = Seq2SeqTrainingArguments(
“BART_Finetuned_CNN_dailymail”,
evaluation_strategy = “epoch”,
learning_rate=2e-5,
per_device_train_batch_size=batch_size,
per_device_eval_batch_size=batch_size,
gradient_accumulation_steps=2,
weight_decay=0.01,
save_total_limit=2,
num_train_epochs=1,
predict_with_generate=True,
fp16=True,
)

A function is created to compute metrics using rouge score.

def compute_metrics(eval_pred):
predictions, labels = eval_pred
decoded_preds = tokenizer.batch_decode(predictions, skip_special_tokens=True)

Replace -100 in the labels as we can’t decode them.

labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)

# Rouge expects a newline after each sentence
decoded_preds = [“\n”.join(nltk.sent_tokenize(pred.strip())) for pred in decoded_preds]decoded_labels = [“\n”.join(nltk.sent_tokenize(label.strip())) for label in decoded_labels]

result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)
# Extract a few results
result = {key: value.mid.fmeasure * 100 for key, value in result.items()}

# Add mean generated length
prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in predictions]result[“gen_len”] = np.mean(prediction_lens)

return {k: round(v, 4) for k, v in result.items()}

Once the model has been initialized, the hyperparameters, datasets, tokenizer and metrics have been defined, we create a Trainer.

trainer = Seq2SeqTrainer(
model,
args,
train_dataset=tokenized_datasets[“train”],
eval_dataset=tokenized_datasets[“validation”],
data_collator=data_collator,
tokenizer=tokenizer,
compute_metrics=compute_metrics
)

We download punkt tokenizer that allows us to transform a text into sentences.

nltk.download('punkt')

Finally we carry out the training and evaluation

trainer.train()trainer.evaluate()

Results

Figure 4: Loss during training

Once the training phase is finished, we can observe the loss throughout the training carried out in the “train” dataset.

Although good results are shown, they can be improved by doing better tuning and training times with the model.

The results obtained when evaluating the model trained with the rouge score metric were the following:

eval_loss: 1.5510817766189575

eval_rouge1: 46.8771

eval_rouge2:23.5647

eval_rougeL: 39.6525

eval_rougeLsum: 43.3625

eval_gen_len: 17.8154

Conclusions

It was possible to create an application that summarizes and translates the given text; it was seen that the results obtained are optimal for the task.

Two models were used to make the application, the first model is the one that was trained, in summary, this model is already in huggingface , for the translation part an already trained huggingface model was used. (https://huggingface.co/Helsinki-NLP/opus-mt-en-es).

Future Job

Improve the inference times that models have when using cloud services.

Expand the types of entries you can have, since now it only supports pure text. (txt, docx)

Members

Sergio Flores. — www.linkedin.com/in/sergio-flores-ll

Anna Montevilla. — www.linkedin.com/in/anna-i-montevilla

Koichi Sato. — www.linkedin.com/in/koichi-sato-316902182

Josmar Suarez. — www.linkedin.com/in/josmar-suarez-l

References

https://arxiv.org/abs/1910.13461v1

https://huggingface.co/datasets/cnn_dailymail

https://huggingface.co/transformers/model_doc/bart.html

https://arxiv.org/abs/1810.04805

https://medium.com/walmartglobaltech/the-journey-of-open-ai-gpt-models-32d95b7b7fb2

More artificial intelligence!

The mission of Saturdays.ai is to make artificial intelligence more accessible (#ai4all) while carrying out projects for good (#ai4good). The workshops we carry out are part of the program AI 4 Schools so that anyone “learn by doing” AI regardless of their background or starting point.

Explore our AI master’s programme at https://saturdays.ai/master-ia-online/

If you want to learn more about artificial intelligence join our community through this link or visit us on our website www.saturdays.ai Join us!

WRITTEN BY

Annita Isabel