Head Tracking 3D with OpenCV

Hello, in this post I want to present a small project developed with OpenCV for a simple head tracking with a normal webcam.
By head tracking I mean that the software tracks the user head movement in the image taken by a camera connected to the computer (could be a webcam or other sources). The tracking is actually a very fast and continuous search for faces. With the acquired position of the tracked head, a 3D render is called and is used to draw on the monitor a ‘fake’ window to give the tracked user the illusion of a proper projection.

The tracking and filtering of the data is done with OpenCV, the rendering with OpenGL/FreeGLUT/GLEW.

The software workflow is as follow:

  1. Initialization, where the Kalman filters used for smoothing the head tracking position, loading the 3D model rendered and the rest of the environment
  2. From this moment on, the 3D loop is activated, and does not communicate with the tracking side at all, except for the “SetCam” function, which specify where the user head is, and update the viewing frustum accordingly.
  3. The tracking loop instead opens the webcam stream, and starts tracking the first face found.
    • When a face is found, a second classification occours in the described area to look for two eyes, and to compute a synthetic distance of the user from the camera.
    • A kalman filter is then applied both to the x-y values, and also to the z value found before
    • Finally, values are communicated to the render for updating the projection matrix.

Here a video of the result:

Download link to the binaries, to try this on your computer (Windows only sorry):

Download binaries

And to download the full visual studio project (VS2012) with all included libraries:

Download sources

In the code I will provide, a lot of simplification and optimization can be done, but for my purpose this was good. For any question please feel free to drop me an email.


One thought on “Head Tracking 3D with OpenCV

  1. Alfonso Generoso Fernández

    Damn, that looks nice. I wonder with this kind of technology is more available. I’ll try to dig into your source (thanks for sharing) and try to apply it into Unreal Engine 4, but I’m kind of affraid, mostly because it’s seems achievable but nobody is doing it, Johnny Lee and his wiimote were in 2007. Anyway, I’ll try to do my best, sure you help a lot.



Leave a Reply

Fill in your details below or click an icon to log in:

WordPress.com Logo

You are commenting using your WordPress.com account. Log Out / Change )

Twitter picture

You are commenting using your Twitter account. Log Out / Change )

Facebook photo

You are commenting using your Facebook account. Log Out / Change )

Google+ photo

You are commenting using your Google+ account. Log Out / Change )

Connecting to %s