Model-based tracking

`Model-based tracking`

Given video footage of a person's face, we developed new techniques to automatically recover the face position and the facial expression from each frame in the video sequence.

A 3D face model is fitted to each frame using a continuous optimization technique. Our model is based on a set of 3D face models that are linearly combined using 3D morphing. Our method has the advantages over previous techniques of fitting directly a realistic 3-dimensional face model and of recovering parameters that can be used directly in an animation system.

We also explored many applications, including performance-driven animation (applying the recovered position and expression of the face to a synthetic character to produce an animation that mimics the input video), relighting the face, varying the camera position, and adding facial ornaments such as tattoos and scars.

Model Fitting

To fit our model to a target face image, we developed an optimization technique whose goal is to compute the parameters yielding a rendering of the model that best ressemble the target image. The discrepancy between the model rendering and the target image is evaluated by using the 2-norm of the difference image.

The error function is mimimized using the Levenberg-Marquardt algorithm.

Here is an example of a video sequence that was tracked using our face model.

The following mpeg animations show some tracking results:

Sequence 1 (trk1.mpg)
Sequence 2 (trk2.mpg)
Sequence 3 (trk3.mpg)
Sequence 4 (trk4.mpg)

Applications

Once we have recovered the position and the geometry of the face at each frame, we can use this information to generate novel animations. We considered two different types of applications. The first one is to apply the recovered parameters to a synthetic face to produce a performance-driven animation. The second set of applications is to alter the original video footage using the recovered 3D geometry. These alterations include changing the lighting, and adding computer graphics elements to decorate the face.

The following mpegs illustrate these applications:

Performance-driven animations:

Exaggeration of facial expression (exg3.mpg)
Change in viewpoint (chv3.mpg)
Transposing animation (fem2.mpg)

Alterations of original footage:

Relighting (rel2.mpg)
Addition of synthetic elements (tat2.mpg)