Flex 2 - Webcam motion tracking part 3

It's aliiiiive! Finally got it sorted. After a little more fiddling, I have linked the paddles to the movement of each hand (and reversed the image so your right hand controls the right paddle etc) and added a moving ball.

**Edit** - click here for webcam pong! You need a webcam and flash player 9.

The movement of the ball is very simplistic, and there are definitely aspects of the thing that could be improved but I think it's good enough to make the point :)

When I have time (probably next week) I'll see about making it a bit smarter and perhaps try a two player mode via Flash Data Services. Robin Hilliard suggested a nice idea - doing two player via flashcom to allow video stream switching and show half of the game area as half of each video stream :)

In the example here you can see some green pixels scattering behind the paddles when you move your hands. I left this in as a way to help you line up your hands - it works best if you can get into a position where your hands are paralel to your keyboard, palms down, and you can see only the side of your hands being tracked. But experiment and see how you go.

The code is here:

<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="vertical" horizontalAlign="center" creationComplete="setup();">

   <mx:Script>
      <![CDATA[
         import mx.controls.Alert;
         import mx.core.UIComponent;
         import flash.media.Camera;
         import flash.filters.ConvolutionFilter;
   
         public var camera : Camera = Camera.getCamera();
         public var video : Video = new Video(camera.width*2, camera.height*2);
         public var pastShot : BitmapData = new BitmapData(video.width,video.height);
         public var currentShot : BitmapData = new BitmapData(video.width,video.height);
         public var displayShot : BitmapData = new BitmapData(video.width,video.height);
         public var displayFrame : UIComponent = new UIComponent();
         public var matrix : Matrix = new Matrix();
         public var shotBitmap : Bitmap;
         public var leftPaddle : MovieClip = new MovieClip();
         public var rightPaddle : MovieClip = new MovieClip();
         public var ball : MovieClip = new MovieClip();
         public var ballFrame : UIComponent = new UIComponent;
         public var leftPaddleFrame : UIComponent = new UIComponent;
         public var rightPaddleFrame : UIComponent = new UIComponent;
         public var topBounds : MovieClip = new MovieClip();
         public var bottomBounds : MovieClip = new MovieClip();
         public var bottomFrame : UIComponent = new UIComponent;
         public var topFrame : UIComponent = new UIComponent;
         public var rightBounds : MovieClip = new MovieClip();
         public var leftBounds : MovieClip = new MovieClip();
         public var leftFrame : UIComponent = new UIComponent;
         public var rightFrame : UIComponent = new UIComponent;
         public var gamePlayArea : Array = new Array();
         public var gameInterval : Number;
         public var ballInterval : Number;
         public var cdInterval : Number;
         public var started : Boolean = false;
         [Bindable]
         public var countdown_val : Number = 5;
         
         public var speed : Number = 300;
         public var now : Number;
         public var then : Number;
         public var elapsed : Number;
         public var numSec : Number;
         public var ballSpeed : Number = 100;
         public var moveAmount : Number;
         public var lrDir : Number = 1;
         public var udDir : Number = 1;
         
         public function setup() : void {
            // grab video, setup the game screen and paddles etc             setupVideo();
            createGameBoundaries();
            createPaddles();
            createBall();
            // start the video capture             gameInterval = setInterval(snapShot, 40);
            // run the countdown             cdInterval = setInterval(countDown, 1000);
            // get the ball moving             ballInterval = setInterval(setBallMovement, 40);
         }
         
         public function countDown() : void {
            countdown_val--;
            if (countdown_val == 0) {
               countdownLabel.text = "Play!";
               clearInterval(cdInterval);
               started = true;
               now = getTimer();
            }
         }
         
         public function setupVideo() : void {
            video.attachCamera(camera);
            shotBitmap = new Bitmap(displayShot);
            displayFrame.addChild(shotBitmap);
            addChild(displayFrame);
         }
         
         public function createGameBoundaries() : void {
            topBounds.graphics.beginFill(0xFF000000);
            topBounds.graphics.drawRect(0,0,video.width,10);
            topFrame.addChild(topBounds);
            displayFrame.addChild(topFrame);
            bottomBounds.graphics.beginFill(0xFF000000);
            bottomBounds.graphics.drawRect(0,video.height,video.width,10);
            bottomFrame.addChild(bottomBounds);
            displayFrame.addChild(bottomFrame);
            
            rightBounds.graphics.beginFill(0xFF000000);
            rightBounds.graphics.lineStyle(1,0xFF000000);
            rightBounds.graphics.drawRect(0,0,1,video.height);
            rightFrame.addChild(rightBounds);
            displayFrame.addChild(rightFrame);
            leftBounds.graphics.beginFill(0xFF000000);
            leftBounds.graphics.lineStyle(1,0xFF000000);
            leftBounds.graphics.drawRect(video.width,0,1,video.height);
            leftFrame.addChild(leftBounds);
            displayFrame.addChild(leftFrame);
         }
         
         public function createPaddles() : void {
            leftPaddle.graphics.beginFill(0x00FFFFFF);
            leftPaddle.graphics.drawRect(2,0,10,60);
            leftPaddle.height = 60;
            leftPaddle.width = 10;
            leftPaddleFrame.addChild(leftPaddle);
            displayFrame.addChild(leftPaddleFrame);
            
            rightPaddle.graphics.beginFill(0x00FFFFFF);
            rightPaddle.graphics.drawRect(video.width-12,0,10,60);
            rightPaddle.height = 60;
            rightPaddle.width = 10;
            rightPaddleFrame.addChild(rightPaddle);
            displayFrame.addChild(rightPaddleFrame);
         }
         
         public function createBall() : void {
            ball.graphics.beginFill(0x00FF0000);
      ball.graphics.lineStyle(1, 0xFFFF0000);
            ball.graphics.drawCircle(video.width/2, video.height/2, 10);
            ball.graphics.endFill();
            ballFrame.addChild(ball);
            addChild(ballFrame);
         }
         
         public function setBallMovement() : void{
            if (started) {
               elapsed = getTimer() - now;
               now = getTimer();
               numSec = elapsed / 1000;
               moveAmount = ballSpeed * numSec;
               ball.x += lrDir * moveAmount;
               ball.y += udDir * moveAmount;
               if (ball.hitTestObject(rightPaddle) || ball.hitTestObject(leftPaddle)) {
                  lrDir = -lrDir;
               }
               if (ball.hitTestObject(topBounds) || ball.hitTestObject(bottomBounds)) {
                  udDir = -udDir;
               }
               if (ball.hitTestObject(leftBounds) || ball.hitTestObject(rightBounds)) {
                  endGame();
               }
            }   
         }
         
         public function endGame() : void {
            this.removeChild(ballFrame);
            mx.controls.Alert.show("You lose!");
            clearInterval(gameInterval);
            clearInterval(ballInterval);
         }
         
         public function snapShot() : void {
            currentShot.draw(video, matrix);
            
            var workingCopy : BitmapData = currentShot.clone();
            
            workingCopy.draw(pastShot, matrix, null, "difference");
            workingCopy.threshold(workingCopy,workingCopy.rect,workingCopy.rect.topLeft,">",0xFF333333,0xFF00FF00,0x00FFFFFF,false);
            // set before to the current state, so it's ready for the next call of the function             pastShot = currentShot.clone();
            // clear the display             displayShot.fillRect(displayShot.rect,0xFF000000);
            
            // draw a rectangle over the center to eliminate the face etc             var maskRect : Rectangle = new Rectangle(displayShot.rect.x+30, displayShot.rect.y, displayShot.rect.width-60, displayShot.rect.height);
            workingCopy.fillRect(maskRect,0xFF000000);
            
            // this line redisplays the green pixels as a user guide             displayShot.threshold(workingCopy,displayShot.rect,displayShot.rect.topLeft,"==",0xFF00FF00,0xFF00FF00,0x00FFFFFF,false);
            
            // grab the movement from the left hand area and move the paddle accordingly             var leftBM : BitmapData = new BitmapData(30,video.height);
            leftBM.copyPixels(workingCopy, new Rectangle(0, 0, 30, video.height), new Point(0,0));
            var leftBounds : Rectangle = leftBM.getColorBoundsRect(0xFF00FF00,0xFF00FF00,true);
            if (leftBounds.y > 0) {
               rightPaddle.y = leftBounds.y;
            }
            // grab the movement from the right hand area and move the paddle accordingly             var rightBM : BitmapData = new BitmapData(30,video.height);
            rightBM.copyPixels(workingCopy, new Rectangle(video.width-30, 0, 30, video.height), new Point(0,0));
            var rightBounds : Rectangle = rightBM.getColorBoundsRect(0xFF00FF00,0xFF00FF00,true);
            if (rightBounds.y > 0) {
               leftPaddle.y = rightBounds.y;
            }
            
            displayShot.draw(reduceNoise(displayShot), matrix);
         }
         
         public function reduceNoise(srcImg : BitmapData) : BitmapData {
            var ma : Array = [   2, 2, 2,
                           2, 2, 2,
                           2, 2, 2 ]
            var cf : ConvolutionFilter = new ConvolutionFilter(3, 3, ma);
            var outImg : BitmapData = srcImg.clone();
            outImg.applyFilter(srcImg, srcImg.rect, new Point(0,0), cf);
            return outImg;
         }
      ]]>
   </mx:Script>
   
   <mx:Label text="{countdown_val}" fontWeight="bold" fontSize="18" id="countdownLabel" />
   
</mx:Application>

Be gentle, it's my first real flash / flex app :) If anyone reads ths and has suggestions or questions, please post!

Thanks must go to Robin Hilliard and Justin Mclean for answeing my stupid actionscript noob questions, and also to Guy Watson for his excellent web motion tracking tutorial which the core of this is based upon.

I'll go into more detail about the bits inside this in the next few days - I'd like to turn it into a full tutorial as there were a number of challenges in creating it (at least for me!).

Related Blog Entries

TrackBacks
There are no trackbacks for this entry.

Trackback URL for this entry:
http://www.tobytremayne.com/trackback.cfm?20D15335-FCC3-6401-A6B944F3769ABE9A

Comments
Bjorn's Gravatar Great!

Not exactly Minority Report responsiveness but nevertheless good kick off to your flex career. Would be interesting to see how you can further optimise the response to handmovements. Perhaps making the controls 'paddles' smoother and detection more acccurate.

regards,

Bjorn
# Posted By Bjorn | 3/5/07 7:41 PM
Andy's Gravatar Great post, Toby!
If you combine this one with UR earlier, you will create a very nice valuable article! Thanx.

___
http://becomeachef.biz
# Posted By Andy | 8/3/07 12:23 AM