VR Headtracking

[JerryScript]

Just an update here. I've only been able to work on this in my spare time, but at least there is good progress, and no need for special lights or other equipment. Current status:

1- head recognition : buggy, first attempted using colorBounds reduction via motion detection, but this doesn't work well for those with long full voluptuous hair, may have to resort to an initialization matrix via a historgram of a scaled down bitmapData (down to 10px), but I really want to avoid histograms

2- facial orientation : buggy, but improving fast. using a modified version of GSkinner's ColorMatrix class in combination with paletteMap, I can easily find the eye height, and with a bit of twiddling of contrast/brightnes I can connect the dots between the eyes

3- noise reduction : not-implemented yet, I'm experimenting with a couple of algorithms to try to make different test sets cancel each other's errors, and to make each test set ignore out-of-bounds data. For those interested in the algorithm I'm trying to adapt, you can read about it here : http://www.mii.lt/informatica/pdf/INFO537.pdf

My current process is as follows:

1- grab webcam image and store in two bitmapData's via draw

2- use difference blend mode and threshold to determine motion area

3- copyPixels of motion area to new bitmapData (this cuts out as much of the image as possible resulting in a smaller image area for further processing)

4- adjust the brightness and contrast for initial color reduction (GSkinner's)

5- paletteMap the results to as few colors as possible (down to 9 or even 6)

6- copyPixels from a narrow band in the center of the motion detection bitmapData at what is assumed to be the forehead height, then use threshold to determine the eye height

6- floodFill the palleteMap bitmapData in the assumed forehead region, then use threshold to determine the facial width

7- copyPixels using the facial width and eye height as a guide to grab the eye region, then use threshold to determine the eye positions

8- draw an ellipse using bitmapFill in a seperate sprite with 2:1 (h:w) proportions based on the floodFill/threshold results to show only the face, then draw a box from eye to eye.

The result is that you can now track the pitch and yaw of the head. I doubt I'll include roll unless absolutely necessary, unless I find a cheap way. Once I work out a good head detection method (regardless of hair), it will be possible to make both fulcrum adjustments, and pitch and yaw, which should be enough for VR with nothing more than a webcam! Mr Malee's Amazing Noggin Turning Control System is not far away!

Attached is an example swf. The text boxes in the lower left are for brightness (left text box) and contrast (right text box). For my system and lighting conditions, I get the best results with brightness adjusted based on conditions (0 to -50), and contrast set as high as possible (up to 100).

Note- this only works with head motion, not full body motion yet! If anyone has some good ideas for how to find the head regardless of how much of the body is showing (and regardless of hair style), I would appreciate your suggestions!