res.txt: Text summary and translation with Machine Learning

La Paz. 2021

Introduction

Summarising means selecting a document’s essential information and expressing it more briefly. People do this by understanding the text and reformulating its ideas. Automated methods can help when there is more information than time to read it, although their output still needs review.

Figure 1: Amount of data generated annually

Problem

internet languages

Figure 2: Predominant languages on the internet

Keeping up to date often means reading lengthy sources that, in many fields, are written in English. That can create barriers of both time and language.

Objective

Develop an application that can help the understanding of a text document in English and shorten the understanding time of a long text.

Proposed solution

Build an application using deep learning models with attention mechanisms. These calculate relationships between parts of the input and help the model weight relevant information; they do not literally reproduce human attention.

Dataset

CNN/Daily Mail contains over 300,000 English-language news articles. The version used here occupied about 1.27 GiB and provides articles with reference summaries. It is used for reading comprehension and both extractive and abstractive summarisation.

DL model

Figure 3: BART scheme

BART uses a transformer encoder–decoder architecture. It combines a bidirectional encoder, as in BERT, with an autoregressive decoder, as in GPT, and uses GeLU activations. Its features include:

Each layer of the decoder additionally performs cross-attention on the final hidden layer of the encoder (as in the sequence-to-sequence model of the transformer).
In the architecture described by its authors, BART omits the additional feed-forward layer that BERT uses before word prediction.

In short, the encoder represents the input text and the decoder generates the output text.

We started with the base model available on Hugging Face.

Hugging Face provides open-source tools and a catalogue of models and datasets. We used its libraries and resources to prepare the data and fine-tune the model.

Model training

The following snippets retain the APIs used in this 2021 project. We have restored the code formatting; running it with current versions would require updating the dependencies and their interfaces.

The project was carried out based on the models provided by the Hugging Face page, which has a large collection of both models and datasets for NLP (Natural Language Processing) tasks. Of all the options, the BART abstractive summary model was selected.

The code makes use of the HuggingFace “datasets” and “transformers” libraries, as well as nltk to perform the evaluation.

# Imported libraries
from datasets import load_dataset, load_metric
from transformers import (
    AutoTokenizer, AutoModelForSeq2SeqLM, DataCollatorForSeq2Seq,
    Seq2SeqTrainingArguments, Seq2SeqTrainer, pipeline,
)
import nltk
import numpy as np

To perform the fine tuning, the “cnn_dailymail” dataset was used, for the evaluation the “rouge” metric was used, and the base model used was “facebook/bart-base”.

raw_datasets = load_dataset("cnn_dailymail", "3.0.0")
metric = load_metric("rouge")
model_checkpoint = "facebook/bart-base"

A tokenizer and a model were loaded referencing the selected base model.

tokenizer = AutoTokenizer.from_pretrained(model_checkpoint)
model = AutoModelForSeq2SeqLM.from_pretrained(model_checkpoint)

From the “cnn_dailymail” dataset, the elements found under the “article” and “highlights” labels were extracted as input and output data respectively.

max_input_length = 512
max_target_length = 128

def preprocess_function(examples):
    inputs = [doc for doc in examples["article"]]
    model_inputs = tokenizer(inputs, max_length=max_input_length, truncation=True)
    with tokenizer.as_target_tokenizer():
        labels = tokenizer(examples["highlights"], max_length=max_target_length, truncation=True)
    model_inputs["labels"] = labels["input_ids"]
    return model_inputs

tokenized_datasets = raw_datasets.map(preprocess_function, batched=True)

Before training the model, hyperparameter configuration is performed.

batch_size = 4
args = Seq2SeqTrainingArguments(
 "BART_Finetuned_CNN_dailymail",
 evaluation_strategy = "epoch",
 learning_rate=2e-5,
 per_device_train_batch_size=batch_size,
 per_device_eval_batch_size=batch_size,
 gradient_accumulation_steps=2,
 weight_decay=0.01,
 save_total_limit=2,
 num_train_epochs=1,
 predict_with_generate=True,
 fp16=True,
)

A function is created to compute metrics using rouge score.

def compute_metrics(eval_pred):
    predictions, labels = eval_pred
    decoded_preds = tokenizer.batch_decode(predictions, skip_special_tokens=True)
    # Replace -100 in the labels before decoding.
    labels = np.where(labels != -100, labels, tokenizer.pad_token_id)
    decoded_labels = tokenizer.batch_decode(labels, skip_special_tokens=True)
    # ROUGE expects a newline after each sentence.
    decoded_preds = ["\n".join(nltk.sent_tokenize(pred.strip())) for pred in decoded_preds]
    decoded_labels = ["\n".join(nltk.sent_tokenize(label.strip())) for label in decoded_labels]
    result = metric.compute(predictions=decoded_preds, references=decoded_labels, use_stemmer=True)
    result = {key: value.mid.fmeasure * 100 for key, value in result.items()}
    prediction_lens = [np.count_nonzero(pred != tokenizer.pad_token_id) for pred in predictions]
    result["gen_len"] = np.mean(prediction_lens)
    return {k: round(v, 4) for k, v in result.items()}

Once the model has been initialized, the hyperparameters, datasets, tokenizer and metrics have been defined, we create a Trainer.

data_collator = DataCollatorForSeq2Seq(tokenizer, model=model)
trainer = Seq2SeqTrainer(
 model,
 args,
 train_dataset=tokenized_datasets["train"],
 eval_dataset=tokenized_datasets["validation"],
 data_collator=data_collator,
 tokenizer=tokenizer,
 compute_metrics=compute_metrics
)

We download punkt tokenizer that allows us to transform a text into sentences.

nltk.download('punkt')

Finally we carry out the training and evaluation

trainer.train()
trainer.evaluate()

Results

Figure 4: Loss during training

Once the training phase is finished, we can observe the loss throughout the training carried out in the “train” dataset.

Although good results are shown, they can be improved by doing better tuning and training times with the model.

The results obtained when evaluating the model trained with the rouge score metric were the following:

eval_loss: 1.5510817766189575

eval_rouge1: 46.8771

eval_rouge2:23.5647

eval_rougeL: 39.6525

eval_rougeLsum: 43.3625

eval_gen_len: 17.8154

Conclusions

We built an application that summarises and translates text. Results from this experiment were promising, although ROUGE scores do not guarantee that every summary is accurate or preserves every nuance.

We used two models: a summarisation model fine-tuned by the team and published on huggingface and a pretrained English-to-Spanish translation model:https://huggingface.co/Helsinki-NLP/opus-mt-en-es).