Facial Motion Capture.

What is it?

Facial Motion Capture is the process of tracking the movements of a person's face using cameras or laser scanners. These movements may then be used to produce computer animation for movies, games, or real-time avatars. Due to the fact that the data gathered is got from the movements of real people, it results in more realistic computer character animation than if the animation was created manually. It is related to body motion capture, but is more challenging due to the higher resolution requirements to detect and track subtle expressions possible from small movements of the markers.

What types of approaches are there to this?

Markerless technologies use the features of the face such as nostrils, the corners of the lips, eyes, wrinkles and then track them. This technology uses active appearance models, principal component analysis, eigen tracking and other techniques to track the desired facial features from frame to frame. This technology is much less cumbersome, and allows greater expression for the actor. These vision based approaches also have the ability to track pupil movement, eyelids, teeth occlusion by the lips and tongue, which are obvious problems in most computer animated features. Typical limitations of vision based approaches are resolution and frame rate, both of which are decreasing as hardware gets better. There is no real software available to the general public using this approach so taking this way to do the project will involve a lot of extra research which realistically can’t be done in the given time frame.

Traditional marker based systems apply up to 350 markers to the actors face and track the marker movement with high resolution cameras. This has been used on movies such as The Polar Express and Beowulf or games like Resident Evil 5 or Call of Duty 5. Next generation systems such as Mova and CaptiveMotion use new approaches to the traditional marker based system and this ends up with higher levels of details. Active Light Emitting Diode Marker technology is currently being used to drive facial animation in real time but this is quiet expensive and not something the general public would consider due to cost, software like Zign Track is something the general public would consider, I shall discuss this software later. If this approach is taken then developing something to be aimed at the same users as Zign Track should be realistically considered.

So what am i actually going to do ?

Basically I have decided to use a marker based approach to the project. I am going to use roughly 13 markers and apply them in the following layout:

Now I am still debating about the camera setup, the movie avatar gave me the idea of creating a head mounted camera at 1 ft 5 " from the face, this will effectively remove the requirements for 3 cameras but it will also be more intrusive in other ways. So it is still up in the air. Now my application will locate the markers, track them and output them in a format called BVH. BVH stands for Biovision Hierarchy, it may be an old format but it is one of few that is supported and it is usable with almost all modelling programs. I am also building in Eye gaze interaction, this will allow me to stop and start the application with my eye. I am doing this to add some complexity to what I am doing and to stop the user ruining their recording with setting themselves up and leaning in to stop a recording. After this side of the project I have the Maya side, here I will learn about creating Maya rigs. I started with building a simple age of empires temple, then a mester chief rendering from halo 1, followed by the high res model of the master chief in halo 3 and I am currently working on UV wrapping and NURBS modelling. When I get this right I shall start working on creating my rig. This rig will be pretty generic because I don't know enough about human facial muscles So I will attempt to produce a normal face with some facial hair and some wrinkles and accurate skin texture. The rig won't be muscle based as I had originally planned.Baking motion capture data onto muscles often causes problems that can be avoided by creating a bridge between groups and using a symetric model with a mirror instead. Once I have the maya rig completed I start work on learning to combine my data with the maya rig. Once this is completed I shall test, package it up and hope all goes well.