Model-based tracking

`Model-based tracking`

Given video footage of a person's face, we developed new techniques to
automatically recover the face position and the facial expression from
each frame in the video sequence.
A 3D face model is fitted to each
frame using a continuous optimization technique. Our model is based on
a set of 3D face models that are linearly combined using 3D
morphing. Our method has the advantages over previous techniques of
fitting directly a realistic 3-dimensional face model and of
recovering parameters that can be used directly in an animation
system.

We also explored many applications, including performance-driven
animation (applying the recovered position and expression of the face
to a synthetic character to produce an animation that mimics the input
video), relighting the face, varying the camera position, and adding
facial ornaments such as tattoos and scars.

## Model Fitting

To fit our model to a target face image, we developed an optimization
technique whose goal is to compute the parameters yielding a rendering
of the model that best ressemble the target image. The discrepancy between the model
rendering and the target image is evaluated by using the 2-norm of the difference image.

The error function is mimimized using the Levenberg-Marquardt algorithm.

Here is an example of a video sequence that was tracked using our face model.

The following mpeg animations show some tracking results:

## Applications

Once we have recovered the position and the geometry of the face at
each frame, we can use this information to generate novel
animations. We considered two different types of applications. The
first one is to apply the recovered parameters to a synthetic face to
produce a performance-driven animation. The second set of applications
is to alter the original video footage using the recovered 3D
geometry. These alterations include changing the lighting, and adding
computer graphics elements to decorate the face.
The following mpegs illustrate these applications:

Performance-driven animations:

Alterations of original footage: