Interactive art: Digital Puppet on the Virtual Stage

This art project started from my interest in glove puppet art. (See image 01) I did the three experimental stages from 2005 to 2017. Every stage tried different combinations with virtual and physical objects. Over 15 years later some Taiwanese glove puppet troupes have been influenced by this digital puppet project. They adjusted the traditional puppet performance with digital elements. I received art funding from The National Culture and Arts Foundation (NCAF) in 2005 and an EMBA program scholarship from Cambridge Judge Business School sponsor from DIAGEO public limited company in 2006 based on this project.

Image 01: Art Exhibition:  “Vision and Beyond” ,Taiwan, 2005

Summary

Image 02: Digital glove Puppet with usb connector

In 2005, I started a digital art experiment of virtual objects combined with the physical input device based on glove puppetry performing culture in Asia. I created an ” Interactive art: Digital Puppet on the Virtual Stage” digital art project. This project hacked the glove puppet and became an input device that controls a 3D virtual avatar. The virtual stage could be a platform to display dynamic sounds, images, or instant motion changes. The puppeteers could interact with virtual and physical puppets at the same time. (See image 02; the first version Digital puppet prototype.)

The gaps of glove puppetry culture

The first step was to consult with Taiwanese puppeteers to know the puppet’s cultural background. Their attitude toward expanding and continuing Taiwanese puppet art impressed me a lot. Since 19 century, the traditional glove puppet extended to Taiwan from the Mainland Chain building up many local puppet show troupes. The Taiwanese puppet show troupes continue the evolution and innovate it into a unique performing art. The Taiwanese glove puppet was elected to be a representative of Taiwan’s image in 2006 by Internet users. However, the times and language barriers limited the glove puppet performing art to stand on the international stage. Of course, a lot of Taiwanese glove puppet groups are invited to play America, European countries, and Northeastern Asia countries. But most of those are only for cultural visiting performances. No one wants to depth learn this performing art.

Based on the interview with these puppeteers I discovered three difficulties when they try to introduce this unique art to the entire world. 

  1. The limitation of puppetry performing on stage; one puppetry group only plays in one location. They could not perform at multiple locations at the same time. 
  2. The language barrier; The storyboard is based on traditional Chinese fiction stories. The puppetries speak dialect which the audiences are not familiar with.   
  3. The difficulties of puppetry manipulating skill training; The gestures are very unique skills which need practice in many years.  

In reality, the most urgent problem in this puppet performing art is the puppet masters are too old to perform long term performing, and the young generation has no interest in learning and becoming a professional puppeteer. In addition, the government can’t provide good subsidization and performing opportunities, so the puppet show troupes performing works are in special celebrations or be the class of elementary school. When the children grow up, they have been forgotten and lost interest in puppet shows. These statuses become the gap of the continuing performing puppet art.   

The Puppet Art project experiments

For this project, I want to combine traditional glove puppet art with new technology. The puppeteers can perform in different locations at the same time via the Internet and digital puppet. The digital puppets become a more friendly communication tool that makes all ages easy to play. Internet technologies break the confinement of puppet playing places. The monitors can display multiple languages. The project still keeps the original Taiwanese folk puppet music. A beginner can learn to manipulate skills easier by recording data from a puppet master. In this way, the traditional skills can be kept in the world.

In Taiwanese old memories, the puppet show has been performed in a public square since 1956. Then many years later the puppet show was displayed on TV channels. Every day kids expect the program to start at that moment. Today in short of the market needs, the old puppeteers just can teach in an elementary school.

When I realized this problem I started to use the kid’s habits to recall their interests. From the children’s toy market observation that the kids enjoy video games very much such as PSP, Nintendo Switch, VR games, or mobile games. (See image 03) If I use the same concept to create a digital puppet, that would be a good solution.

Image 03: digital gaming for kids and adult
Image 04: the diagram for the remote performance

Although the digital puppet project would be fun and more gameplay into the puppet performing. The spirit/root of glove puppetry art needs to be keeping the puppet master’s gestures and language. Based on this principle, the first experimental stage started in 2005, I created a 3D puppet in virtual space which can be submitted by the internet. When the virtual 3D model date could be transmitted to a different location, that is a way to break the location limitation. (See image 04)

I analyzed the puppet master’s gestures to find the bending points. Then I created the physical glove puppet which embodies six sensors into the glove puppet’s outfit. I could use this special puppet input digital signal to control the virtual 3D puppet character.

In the second experimental stage in 2008, I combined the robotic and storyboard to play a short episode. We analyzed the storyboard to find the mixing points which could create the reaction between the robotic puppet and the virtual 3D puppet. Of course, we kept the puppetry and manipulated gestures from the first experimental stage. I collaborated with the Embedded Systems Laboratory of National Chiao Tung University and created the robot at this stage.

In the third experimental stage in 2017, I focused on making the representation of a virtual stage in real space. I started to measure the stage size when I used the projection system to present the virtual stage. The main feature of the traditional puppet stage is easy assembly and rebuilding in 30~60 mins. The stage size is flexible to fit different performing platforms. Therefore I choose the projection system to resize the virtual images. I recreated the dynamic stage which could change the animation based on the storyline. And the projection system could add the subtitle under the lower part of the stage which is a good way to keep the original culture and can be read in different languages.

1st stage, 2005

2ed stage, 2008

3rd stage, 2017

The puppeteer interview understands the puppetry performing gestures. I assembled the traditional glove puppet and added the digital sensors. 

The testing process shows the digital glove controlling the robot and sending a signal to a virtual 3D puppet character based on the storyboard and sound feedback.

The virtual stage experiment used contemporary art illustration style to decorate the stage. And the additional sound effects enhance the drama scripts.

Feedback

Image 05: three phases of the digital puppet show

“The digital glove puppet in the virtual stage ” art project had three experiments from 2005~2017. The project created 3D virtual puppets which can fly through to different virtual locations. That gave me the opportunity of real-time performance in the multi-location. The virtual stage experiment could display the subtitles on the stage and mix with the fantasy visual and sound effects to attract young audiences. I tried to use technology to mimic the puppeteers’ gestures with sensors and data gloves. Unfortunately, it still couldn’t represent the details of the puppeteers’ hand gestures. I tried evaluating using computer vision technology to track the hand motions. But The camera can’t recognize the overlapping dead stops on the computing system.  

Following those three experiments, I summarize three phases technology, product, and service parts. (See image 05) I already tried from the technology and product sides. The future step is using the service design of puppetry groups and finding out the business model which can share profit with all stakeholders. My final goal is to attract young people to enjoy the glove puppet performing culture. Since 2020 some puppetry groups have received influence and started using digital puppets and virtual stage ideas in their shows. Before 2005 I was the first and only one to present the digital puppet idea in Asia. Now The snowball effect is starting. I hope the glove puppet’s traditional skills can live on for a long time.