Robotic Marionettes
Overview
This project is focused on the creation of an automatic, robotically-actuated puppeteering system. The control of classic, stringed marionettes is an extremely difficult control problem for several reasons. The marionettes themselves have a high number of degrees of freedom, they are underactuated, the system's dynamics exhibit hybrid behavior through impacts and the loss of string tension, and the system itself is very lowly damped which makes it difficult to create dynamic simulations that obey physically realistic energy behaviors over any significant time horizon. When a human puppeteer controls a marionette they inherently comprehend this complexity, and they compensate for it in real-time. The language of choreography only coarsely describes the play, and it is up to the puppeteer to figure out what mechanical inputs will produce the realistic puppet trajectories. They must figure out how to deal with the complex dynamics across a range of frequencies, how to transition from one motion to the next, and how to deal with error between the actual trajectory and the desired trajectory. Ultimately, we aim to produce puppet plays that are generated in an automated fashion from choreography that describes the play in much the same way that real puppeteering works. Essentially we are trying to use puppetry and puppet choreography as a framework for investigating an array of interesting mathematical, engineering and computer science problems.
The first play that we are interested in putting on is a retelling of Pygmalion that was choreographed by Elizabeth Jochum who is now an Assistant Professor at Aalborg University in Denmark. We have already used motion capture technology to capture humans acting out the choreography so that we will be able to quantitatively evaluate our system as it is developed.
Modeling
One important aspect of this project is the creation of a dynamic simulation software package that is capable of simulating a full marionette. We have developed an open source software package called trep that is suitable for simulating a variety of dynamic systems. Trep uses a tree-based structure to represent mechanical systems comprised of interconnected rigid bodies. The tree description is used to create a generic variational integrator applicable to arbitrary systems. The tree structure allows the performance of the simulation to scale favorably with problem size; i.e. a full 25-DOF model of a marionette can be integrated better than real-time. Variational integrators are particularly well-suited to marionettes because they exactly preserve constraints at every time step, and they posses desirable energy and momentum behaviors.
Further leveraging the tree structure, trep is also able to provide exact first and second derivatives of both continuous-time, and discrete-time representations of mechanical systems. This is particularly convenient for control and estimation problems. Trep also features a powerful optimal control algorithm capable of generating dynamically-feasible system trajectories for reference tracking. This feature has successfully gnerated full system trajectories for the marionettes using both simulated references, and experimentally generated references.
Robotic System
Another significant aspect of this project is the creation of a robotic platform capable of actuating the marionettes. The system that we have developed for this purpose consists of a number of magnetically suspended differential drive robots with a series of winches for controlling the lengths of strings that are attached to the marionette. Each robot or "puppeteer" has its own on-board microcontroller that is responsible for implementing closed-loop control of its motors, partaking in a wireless communication network, and using motor odometry to estimate the pose of the robot. The drive wheels of the puppeteers are large ring magnets, and there is a magnetic caster at the front and the back of each puppeteer to provide additional stability. The platform that the puppeteers drive on is a large plastic membrane that is held in tension along all four of its edges; it is then elevated to approximately ten feet off of the ground. A totally passive magnetic vehicle with the same footprint as the robot is placed on the top of the plastic sheet to provide a magnetic surface for the robot to cling to. This "idler" is then towed around by the robot as it drives around on the underside of the tarp. 
A master computer is responsible for determining the desired trajectories for each of the robots, and sending control instructions wirelessly to each robot. The wireless communication is handled by XBee chips.
The entire system is controlled with the Robot Operating System (ROS). Feedback for the vehicle locations and the marionette is provided by a single Microsoft Kinect (visible in the upper-left portion of the image at right).
Where to Learn More
Visit this project's vimeo channel for more videos
Visit the github repository for code related to the project
Visit trep's homepage for trep installation
Related Publications
Trajectory generation for underactuated control of a suspended mass, 2012 IEEE International Conference on Robotics and Automation ICRA, 2012 DOI Google Scholar
Control aesthetics in software architecture for robotic marionettes, American Control Conference (ACC), 2011, IEEE, pp. 3825-3830, 2011 Google Scholar
Scalable variational integrators for constrained mechanical systems in generalized coordinates, Robotics, IEEE Transactions on, vol. 25, IEEE, pp. 1249-1261, 2009 Google Scholar
Dynamic modeling and motion planning for marionettes Rigid bodies articulated by massless strings, Robotics and automation, 2007 IEEE international conference on, IEEE, pp. 330-335, 2007 Google Scholar
Back to top