Waxing Lyricial: Flex 2 - Webcam motion tracking part 2

Flex 2 - Webcam motion tracking part 2

Posted At : March 4, 2007 7:44 PM | Posted By : Toby
Related Categories: Flex,Tech,Geek

Woohoo! I have the motion tracking working properly now. I spent ages mucking about with tiles and looping over individual pixels, then ages more trying to understand matrixes and convolution filters. I finally came up with a method where I apply what is basically a blur filter to the snapshot to remove the noise, then grab left and right sides of the video input to track both hands and allow them to control the paddles.

I've now added the paddles, and the track the movement of your hands! Have a look here to see the result.

This seems to work nicely - all you have to do is line up your hand properly by watching the green movment pixels, and you'll be able to contol the paddles properly. What I'm thinking I might do is try to reverse the controls so you're not dealing with a mirror image.

Then all I have to do is add the ball and some behaviour for bouncing and we should be golden :)

Here's the new code - I worked out how to get rid of my dependancy on the panel, so the entire set of code is now in one mxml file. When I have it all working to my satisfaction I will split it out into proper classes etc but for now, here it is:

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="absolute" creationComplete="setup();">

   <mx:Script>
      <![CDATA[
         import mx.core.UIComponent;
         import flash.media.Camera;
         import flash.filters.ConvolutionFilter;

         public var camera : Camera = Camera.getCamera();
         public var video : Video = new Video(camera.width*2, camera.height*2);
         public var pastShot : BitmapData = new BitmapData(video.width,video.height);
         public var currentShot : BitmapData = new BitmapData(video.width,video.height);
         public var displayShot : BitmapData = new BitmapData(video.width,video.height);
         public var displayFrame : UIComponent = new UIComponent();
         public var matrix : Matrix = new Matrix();
         public var shotBitmap : Bitmap;
         public var leftPaddle : Sprite = new Sprite;
         public var rightPaddle : Sprite = new Sprite;
         public var leftPaddleFrame : UIComponent = new UIComponent;
         public var rightPaddleFrame : UIComponent = new UIComponent;

         public function setup() : void {
            video.attachCamera(camera);
            shotBitmap = new Bitmap(displayShot);
            displayFrame.addChild(shotBitmap);
            addChild(displayFrame);

            leftPaddle.graphics.beginFill(0xFFFF0000);
            leftPaddle.graphics.drawRect(0,0,10,40);
            leftPaddleFrame.addChild(leftPaddle);
            addChild(leftPaddleFrame);
            rightPaddle.graphics.beginFill(0xFFFF0000);
            rightPaddle.graphics.drawRect(video.width-10,0,10,40);
            rightPaddleFrame.addChild(rightPaddle);
            addChild(rightPaddleFrame);

            setInterval(snapShot, 40);
         }

         public function snapShot() : void {
            currentShot.draw(video, matrix);

            var workingCopy : BitmapData = currentShot.clone();

            workingCopy.draw(pastShot, matrix, null, "difference");
            workingCopy.threshold(workingCopy,workingCopy.rect,workingCopy.rect.topLeft,">",0xFF333333,0xFF00FF00,0x00FFFFFF,false);
            // set before to the current state, so it's ready for the next call of the function             pastShot = currentShot.clone();
            // clear the display             displayShot.fillRect(displayShot.rect,0xFF000000);

            // draw a rectangle over the center to eliminate the face etc             var maskRect : Rectangle = new Rectangle(displayShot.rect.x+30, displayShot.rect.y, displayShot.rect.width-60, displayShot.rect.height);
            workingCopy.fillRect(maskRect,0xFF000000);

            // this line redisplays the green pixels as a user guide             displayShot.threshold(workingCopy,displayShot.rect,displayShot.rect.topLeft,"==",0xFF00FF00,0xFF00FF00,0x00FFFFFF,false);

            // grab the movement from the left hand area and move the paddle accordingly             var leftBM : BitmapData = new BitmapData(30,video.height);
            leftBM.copyPixels(workingCopy, new Rectangle(0, 0, 30, video.height), new Point(0,0));
            var leftBounds : Rectangle = leftBM.getColorBoundsRect(0xFF00FF00,0xFF00FF00,true);
            if (leftBounds.y > 0) {
               leftPaddle.y = leftBounds.y;
            }
            // grab the movement from the right hand area and move the paddle accordingly             var rightBM : BitmapData = new BitmapData(30,video.height);
            rightBM.copyPixels(workingCopy, new Rectangle(video.width-30, 0, 30, video.height), new Point(0,0));
            var rightBounds : Rectangle = rightBM.getColorBoundsRect(0xFF00FF00,0xFF00FF00,true);
            if (rightBounds.y > 0) {
               rightPaddle.y = rightBounds.y;
            }

            displayShot.draw(reduceNoise(displayShot), matrix);
         }

         public function reduceNoise(srcImg : BitmapData) : BitmapData {
            var ma : Array = [   2, 2, 2,
                           2, 2, 2,
                           2, 2, 2 ]
            var cf : ConvolutionFilter = new ConvolutionFilter(3, 3, ma);
            var outImg : BitmapData = srcImg.clone();
            outImg.applyFilter(srcImg, srcImg.rect, new Point(0,0), cf);
            return outImg;
         }
      ]]>
   </mx:Script>


</mx:Application>

As you'll see looking at the example it is still showing the green movment pixels to help you align your hands. I might see if I can separate this out from the game area to help people line up their hands properly.

Pong is almost there!