Lure

Lure is an artistic research project related to artificial intelligence in theatre, music and interactive storytelling. The research is a first step into the possible creation of a full blown opera made with artificial intelligence. A variety of machine learning techniques were explored in combination with musical instruments, voice, narration and the online digital domain. The artistic research investigates questions on the topics of human–machine interaction, artificial intelligence and designing theatrical experiences through generative tools.

The narrative is based on the classic novel Parzival by Wolfram von Eschenbach. Parzival meets three elegant knights, decides to seek King Arthur, and continues a spiritual and physical search for the Grail, looking for the world at the end of the forest where he was raised. In Lure the character of Parzival was used, who is now developed inside a computer, with a similar intentionality. It wants to learn about you and the world, and starts a conversation with you. It tries to express itself by playing with a musician. And it leaves its virtual confinement to explore the world by itself.

With three prototypes three different artistic and technological approaches were studied. Develop empathy with the computational character by discovering how it learns. Experience its inner workings through theatrical expression. And be touched by its musical sensitivity.

The research was presented in a theatrically staged online performance, in which the audience was asked to interact with the character Parzival via a computer and a mobile phone. The presentation was followed up by a Q&A session with the whole team.

Technology

The infrastructure was built up around realtime socket connections, to facilitate the musical dialog between the human drummer and the AI. To overcome the latency, introduced by processing of the data, bars of notes were send over instead of individual hits. The drummer played on a midi drum set and the individual notes were captured by the Max application and sequenced into two bars. The bars were send over to the central Lure server which acted as a broker between the AI model and the human drummer. The forwarded bars were processed by the model into two response bars and send back to the Max application by reverse route. The audio was converted back from midi into hearable audio using a separate sampler for both human drummer and AI.

High level overview of the Lure AI audio architecture based on three primary components: the realtime control server, the AI model running at argmax.ai and a Max midi to audio processor, all connected through realtime socket connections for fast response times.
High level overview of the Lure AI Pix2Pix Search architecture.

Performances

YouTube Streaming Channel / Online / 13-11-2020.
A New Humanity – IE9 Conference / Munich / 11-11-2020.

Press

Review in the Süddeutsche Zeitung titled Unterwegs mit Parzival / 15-11-2020 / Rita Argauer

“the brief moment when computer and human being came together artistically is full of horror and beauty, ambivalence and theatricality. To create something like this through a miserable computer screen is a great achievement.”

Rita Argauer

Awards

Lure was nominated by the Columbia University School of the Arts Digital Storytelling Lab for the Digital Dozen 2021 award – breakthrough in storytelling.

Credits

Lure is a project of Munich composer and theatre maker Mathis Nitschke, Amsterdam interactive artists Luciano Pinna and Klasien van de Zandschulp, scientists from the Munich Volkswagen Group Machine-Learning Research Lab (argmax.ai) Patrick van der Smagt, Djalel Benbouzid, Nutan Chen and Munich dramaturg Elsa Büsing.

With:
Mara Widmann: Narrator
Martina Koppelstetter: Singer
Matthias Lachenmayer: Drummer
Katharina Dobner: Mask and fire design
Dominik Theobald: Digital Tree

Musical contributions by gamut inc, Jörg Hüttner, Martina Koppelstetter, Matthias Lachenmayer, Mathis Nitschke, Alexander Schubert

Parzival’s learning machine is based on the flat latent-variable models explained in argmax.ai/blog/metric/.

Besides custom developed tools the following were used: Google Magenta, Tensorflow Pix2pix, OpenAI GPT-2, Runway, AIVA, Botsquad, Lens Studio, Av3ry

Lure is funded and supported by the Dutch Creative Industries Fund NL, the German Fonds Darstellende Künste, and the Volkswagen Group Machine Learning Research Lab.

Special thanks to Pathos Theater, Mario Klingemann, Memo Akten, Vladimir Viro, Arjan Scherpenisse, Botond Cseke, Milo Schagen, Michael Schwarzfischer, Markus Krause, Tankred Dorst and Wolfram von Eschenbach.