Gil Sperling

A musical performance synthesizing two humans – a dancer and a singer, into one digital avatar.

Created by Gil Sperling. Performed by Jordan Rutter and Stacy Grossfield.

Project Goals:

Create an environment that enables real-time compositing of live motion capture data from multiple performers onto an avatar in digital space. This would allow to work in an improvisational fashion, allowing performers and the director to see the digital results in rehearsal. This would also function as a proving ground for building a live performance, in either remote or theatrical format.
This goal was on the whole achieved. Although there are many technical hurdles and breaking points in the workflow, after some practice we were able to conduct rehearsals while sharing the composited image with the performers, as well as record a documentation of a performance with all the elements.

Enhance the performative capabilities of the opera character by creating a hybrid digital body. Musical performance puts opera singers under extremely demanding conditions, physically and mentally. Directors who have wished to break these limitations often do so at the expense of musical quality (or at least how it is perceived among traditional opera fans), or by having multiple performers for the same character, as in Sasha Waltz’s production of Dido and Aeneas (also by Purcell), where major characters are represented on stage by a singer and a dancer.
This goal was only partly achieved in my view. The facial capture technology I was using did not transfer the full range of expressivity onto the avatar’s face. What we do have is in face a trio – three performers, one digital and two human, creating the scene together. That is why I’m showing the video of the performers as part of the performance.

Implementation:

Body motion capture using Rokoko Smartsuit.
Rokoko Studio software live-streams directly into Unreal Engine.

Facial capture using the Live Link Face app on iPhone X. Live streaming into Unreal Engine over the internet.

Performers:

Stacy Grossfield – dancer
Jordan Rutter – singer

Music:

Aria of the Cold Genius from Henry Purcell’s Opera “King Arthur” (1691).
The aria was familiarized to pop music audiences through Klaus Nomi’s performance of it in 1981.