Can AI boost tradition?
MiliaGAN explores generating melodies for Basque improvised verse with generative adversarial networks: tradition, experimentation and plenty of learning.
A learning project shared by the community. The article documents its team, methods and results; it does not establish that the prototype is ready for real-world use.
Tradition, from Latin tradere, meaning “to transmit”, is the passing of practices or beliefs from one generation to the next. This heritage becomes part of who we are.
Innovation introduces new ideas and approaches that change how things are done. It is often seen as a threat to tradition precisely because it changes established practices.
But must it always be that way? Could innovation help sustain and share tradition?
Donostia. 2021
Bertso, doinu and neurri: a glimpse of Basque tradition
Among the various treasures Basque culture has managed to keep alive are bertsoak, improvised verses with rhyme that are sung following a doinu, a melody, that has to match with certain neurri, or verse length.
These verses are usually sung before an audience. The bertsolaris, who improvise verses in Basque, sing about topics set by the moderator, the gaijartzaile, and interact with other bertsolaris, often with sharp wit, in pursuit of the audience’s applause.
What does a modern bertso session look like? The following video gives a taste.
Did you notice that both bertsolaris use the same melody? This is the doinu. These melodies are passed from generation to generation, and knowing them well matters: performers may need to use one suggested by the gaijartzaile (moderator) or other bertsolaris when singing their verses.
Can we improvise on improvisation?
Like rap battles, bertsos unfold on the spot without a script. Improvisation is central to the bertsolari’s craft. Yet the melody, the doinu, is familiar and fixed.
Which is a pity, isn’t it? If improvisation is part of the great art of the bertsolari… Why not provide an improvised doinu so that the bertso experience becomes even more challenging and unique?
With these ideas and our newly acquired deep learning skills, @jperezvisaires and @klopetx had the same thought: perhaps a generative adversarial network could create new doinus.
Could bertsolarism be revolutionized with the use of AI?
Well… maybe not that much, but it was worth a try.
GANs: where technology and tradition could meet
At Saturdays.AI, we explored convolutional and recurrent networks, autoencoders, reinforcement learning and generative adversarial networks. Could a GAN create new doinus, and what would we need to try?
Data! Of course.
Gathering the data
Fortunately, Bertsozale Elkartea maintained a website containing around 3,000 doinus and their metadata. The site is in Basque, if you would like to explore it.
And you know the saying:
It’s easier to ask for forgiveness than to get permission…
So we scraped the website. Thank you, Bertsozale, for your work, and apologies for overloading your servers and retrieving the data without formally asking permission.
We first downloaded the metadata and selected melodies by type and syllable count. We chose zortziko txiki melodies, with seven syllables in the first line and six in the next, reducing the list to around 200.
Midi format
“Wait a moment: download what, exactly?”
Fortunately for us, we had the chance to download the doinus in either mp3 or midi formats.
“MIDI? I know MP3, but midi reminds me of the French word for midday…”
MIDI (Musical Instrument Digital Interface) is a standard for exchanging musical messages. A MIDI 1.0 stream supports 16 channels. Messages describe events such as a note starting or stopping, its pitch and its velocity. Essentially, the file specifies which notes to play, when and for how long; it does not contain a sound recording.
Example of a midi.
Feeding our little generative monster
Once the data was ready, we just needed to feed the GAN.
And our experience of using midi directly for the GAN is perfectly summarized by the following poem:
We used MIDI as input;
well, at least we tried.
We ran into problems
and tried another route.
You know, everyone uses Deep Learning with images, why should we do otherwise?
Instead of feeding MIDI files directly into the network, we converted them into images. Their note-event structure makes this relatively straightforward.
Working with images felt more familiar, and there were far more examples of convolutional neural networks applied to image data.
GAN structure
Let’s pause. We have talked about using GANs to generate new sounds, but what exactly is a GAN?
GANs were introduced by Goodfellow and colleagues and consist of two models trained together with opposing goals. The generator creates new samples from random input; the discriminator tries to distinguish them from real data. They are often compared to a counterfeiter and a police officer: as one improves the fakes, the other gets better at spotting them.
The structure of a GAN fed with doinus.
During training, the generator tries to create representations of new doinus that the discriminator cannot distinguish from real ones. This competition can improve the generated samples, although fooling the discriminator does not by itself guarantee that a person will find the music interesting.
MiliaGAN, the generator part of the GAN.
Easy peasy, right?
SPOILER: Nothing went as expected.
Round 1: If what one has to say is not better than silence…
We started to feed our monster (well, monsters actually).
We waited until the training converged.
And we freaked out with the resulting doinus.
Yes, this really is a black image: a “shy” MIDI with no notes.
Yes, an empty MIDI. Apparently our GAN decided that silence was better than saying something worse. It went full Simon & Garfunkel and released its own version of The Sound of Silence.
Why?
The images we were trying to create were really sparse, with lots of zeroes and only some ones on the notes being played. The generator initially learnt that by switching all the pixels off, it could trick the dumb discriminator at the beginning. However, during training, at some point, even the dumbest of discriminators was able to detect that a blank image was not a real doinu, which meant that all the efforts made by the generator to produce blank images from noise were now worthless. The generator was not able to adapt fast enough to trick the new discriminator and the training diverged.
To address the sparse images, we took the argmax of each column, turning a 128 × 1024 image into a 1 × 1024 vector. This was possible because the melodies played only one note at a time.
Lesson: simplify the data representation where it makes sense. Make life easier for your neural network.
Round 2: Damn it! Who cares about mixing different doinus?
Initially, we wanted the generator to focus on creating one type of doinu only; the most popular doinu: zortziko txikia. We had only around 180 usable examples of this type. That proved too few for the results we wanted, so we eventually used the whole database. Mixing different types gave us around 2,700 samples: still a small dataset for this experiment, but worth a try.
Yes, the whole database, with different metres, rhymes and all.
It
all went
in.
In addition, we reduced the resolution of the images so that they were less sparse, in order to avoid the problem of the shy gan.
And we reduced the midi resolution even more. We needed to simplify if we wanted to make some kind of progress.
And, surprisingly, the magic happened.
Lessons learned
So, what have we learned after the creation of our little monster, the MiliaGAN?
- We had too little data to train this GAN well. Expanding the dataset helped, although quantity does not replace quality. Few-shot GAN learning was a research direction we wanted to explore.
- Training time matters, provided the process is stable. In our experiments, longer stable training improved some results; longer training is not a guarantee of improvement in every case.
- Simplifying the data representation made a crucial difference for us. In this task, converting sparse matrices into vectors reduced the tendency to generate empty outputs. This is a lesson from our experiment, not a universal rule.
Final remarks
The MiliaGAN project has been a great chance to learn DeepLearning techniques, and Generative Adversarial Networks in particular. It has only been possible thanks to the help from the AI Saturdays team, who created a wonderful environment for learning AI and sharing projects and ideas. Thanks also to our fellow participants for their help and contributions. Without this community, MiliaGAN would not have been possible.
On a personal level, this project has been both challenging and great fun. It combines our culture and our geeky side. It is also the first time that Jon and Kerman have joined forces on such a crazy technology project. It is not our first wild idea together, but that is another story.
Will MiliaGAN revolutionize the world of bertsolarism? We frankly doubt it, but hey, if someone ever asks, we had fun and we learned.
More artificial intelligence!
Saturdays.AI makes AI more accessible through shared learning and projects with a social purpose. Our motto is AI4ALL IS AI4GOOD.
A story from the Saturdays.AI community. Originally published on Nov 22, 2021.
← More community stories
