Flex 2 – Webcam motion tracking part 3
If you're new here, you may want to subscribe to my RSS feed. Thanks for visiting!
It’s aliiiiive! Finally got it sorted. After a little more fiddling, I have linked the paddles to the movement of each hand (and reversed the image so your right hand controls the right paddle etc) and added a moving ball.
**Edit** – click here for webcam pong!
You need a webcam and flash player 9.
The movement of the ball is very simplistic, and there are definitely aspects of the thing that could be improved but I think it’s good enough to make the point 
When I have time (probably next week) I’ll see about making it a bit smarter and perhaps try a two player mode via Flash Data Services. Robin Hilliard suggested a nice idea – doing two player via flashcom to allow video stream switching and show half of the game area as half of each video stream
In the example here you can see some green pixels scattering behind the paddles when you move your hands. I left this in as a way to help you line up your hands – it works best if you can get into a position where your hands are paralel to your keyboard, palms down, and you can see only the side of your hands being tracked. But experiment and see how you go.
The code is here:
<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="vertical" horizontalAlign="center" creationComplete="setup();">
<mx:Script>
<![CDATA[
import mx.controls.Alert;
import mx.core.UIComponent;
import flash.media.Camera;
import flash.filters.ConvolutionFilter;
public var camera : Camera = Camera.getCamera();
public var video : Video = new Video(camera.width*2, camera.height*2);
public var pastShot : BitmapData = new BitmapData(video.width,video.height);
public var currentShot : BitmapData = new BitmapData(video.width,video.height);
public var displayShot : BitmapData = new BitmapData(video.width,video.height);
public var displayFrame : UIComponent = new UIComponent();
public var matrix : Matrix = new Matrix();
public var shotBitmap : Bitmap;
public var leftPaddle : MovieClip = new MovieClip();
public var rightPaddle : MovieClip = new MovieClip();
public var ball : MovieClip = new MovieClip();
public var ballFrame : UIComponent = new UIComponent;
public var leftPaddleFrame : UIComponent = new UIComponent;
public var rightPaddleFrame : UIComponent = new UIComponent;
public var topBounds : MovieClip = new MovieClip();
public var bottomBounds : MovieClip = new MovieClip();
public var bottomFrame : UIComponent = new UIComponent;
public var topFrame : UIComponent = new UIComponent;
public var rightBounds : MovieClip = new MovieClip();
public var leftBounds : MovieClip = new MovieClip();
public var leftFrame : UIComponent = new UIComponent;
public var rightFrame : UIComponent = new UIComponent;
public var gamePlayArea : Array = new Array();
public var gameInterval : Number;
public var ballInterval : Number;
public var cdInterval : Number;
public var started : Boolean = false;
[Bindable]
public var countdown_val : Number = 5;
public var speed : Number = 300;
public var now : Number;
public var then : Number;
public var elapsed : Number;
public var numSec : Number;
public var ballSpeed : Number = 100;
public var moveAmount : Number;
public var lrDir : Number = 1;
public var udDir : Number = 1;
public function setup() : void {
// grab video, setup the game screen and paddles etc
setupVideo();
createGameBoundaries();
createPaddles();
createBall();
// start the video capture
gameInterval = setInterval(snapShot, 40);
// run the countdown
cdInterval = setInterval(countDown, 1000);
// get the ball moving
ballInterval = setInterval(setBallMovement, 40);
}
public function countDown() : void {
countdown_val--;
if (countdown_val == 0) {
countdownLabel.text = "Play!";
clearInterval(cdInterval);
started = true;
now = getTimer();
}
}
public function setupVideo() : void {
video.attachCamera(camera);
shotBitmap = new Bitmap(displayShot);
displayFrame.addChild(shotBitmap);
addChild(displayFrame);
}
public function createGameBoundaries() : void {
topBounds.graphics.beginFill(0xFF000000);
topBounds.graphics.drawRect(0,0,video.width,10);
topFrame.addChild(topBounds);
displayFrame.addChild(topFrame);
bottomBounds.graphics.beginFill(0xFF000000);
bottomBounds.graphics.drawRect(0,video.height,video.width,10);
bottomFrame.addChild(bottomBounds);
displayFrame.addChild(bottomFrame);
rightBounds.graphics.beginFill(0xFF000000);
rightBounds.graphics.lineStyle(1,0xFF000000);
rightBounds.graphics.drawRect(0,0,1,video.height);
rightFrame.addChild(rightBounds);
displayFrame.addChild(rightFrame);
leftBounds.graphics.beginFill(0xFF000000);
leftBounds.graphics.lineStyle(1,0xFF000000);
leftBounds.graphics.drawRect(video.width,0,1,video.height);
leftFrame.addChild(leftBounds);
displayFrame.addChild(leftFrame);
}
public function createPaddles() : void {
leftPaddle.graphics.beginFill(0x00FFFFFF);
leftPaddle.graphics.drawRect(2,0,10,60);
leftPaddle.height = 60;
leftPaddle.width = 10;
leftPaddleFrame.addChild(leftPaddle);
displayFrame.addChild(leftPaddleFrame);
rightPaddle.graphics.beginFill(0x00FFFFFF);
rightPaddle.graphics.drawRect(video.width-12,0,10,60);
rightPaddle.height = 60;
rightPaddle.width = 10;
rightPaddleFrame.addChild(rightPaddle);
displayFrame.addChild(rightPaddleFrame);
}
public function createBall() : void {
ball.graphics.beginFill(0x00FF0000);
ball.graphics.lineStyle(1, 0xFFFF0000);
ball.graphics.drawCircle(video.width/2, video.height/2, 10);
ball.graphics.endFill();
ballFrame.addChild(ball);
addChild(ballFrame);
}
public function setBallMovement() : void{
if (started) {
elapsed = getTimer() - now;
now = getTimer();
numSec = elapsed / 1000;
moveAmount = ballSpeed * numSec;
ball.x += lrDir * moveAmount;
ball.y += udDir * moveAmount;
if (ball.hitTestObject(rightPaddle) || ball.hitTestObject(leftPaddle)) {
lrDir = -lrDir;
}
if (ball.hitTestObject(topBounds) || ball.hitTestObject(bottomBounds)) {
udDir = -udDir;
}
if (ball.hitTestObject(leftBounds) || ball.hitTestObject(rightBounds)) {
endGame();
}
}
}
public function endGame() : void {
this.removeChild(ballFrame);
mx.controls.Alert.show("You lose!");
clearInterval(gameInterval);
clearInterval(ballInterval);
}
public function snapShot() : void {
currentShot.draw(video, matrix);
var workingCopy : BitmapData = currentShot.clone();
workingCopy.draw(pastShot, matrix, null, "difference");
workingCopy.threshold(workingCopy,workingCopy.rect,workingCopy.rect.topLeft,">",0xFF333333,0xFF00FF00,0x00FFFFFF,false);
// set before to the current state, so it's ready for the next call of the function
pastShot = currentShot.clone();
// clear the display
displayShot.fillRect(displayShot.rect,0xFF000000);
// draw a rectangle over the center to eliminate the face etc
var maskRect : Rectangle = new Rectangle(displayShot.rect.x+30, displayShot.rect.y, displayShot.rect.width-60, displayShot.rect.height);
workingCopy.fillRect(maskRect,0xFF000000);
// this line redisplays the green pixels as a user guide
displayShot.threshold(workingCopy,displayShot.rect,displayShot.rect.topLeft,"==",0xFF00FF00,0xFF00FF00,0x00FFFFFF,false);
// grab the movement from the left hand area and move the paddle accordingly
var leftBM : BitmapData = new BitmapData(30,video.height);
leftBM.copyPixels(workingCopy, new Rectangle(0, 0, 30, video.height), new Point(0,0));
var leftBounds : Rectangle = leftBM.getColorBoundsRect(0xFF00FF00,0xFF00FF00,true);
if (leftBounds.y > 0) {
rightPaddle.y = leftBounds.y;
}
// grab the movement from the right hand area and move the paddle accordingly
var rightBM : BitmapData = new BitmapData(30,video.height);
rightBM.copyPixels(workingCopy, new Rectangle(video.width-30, 0, 30, video.height), new Point(0,0));
var rightBounds : Rectangle = rightBM.getColorBoundsRect(0xFF00FF00,0xFF00FF00,true);
if (rightBounds.y > 0) {
leftPaddle.y = rightBounds.y;
}
displayShot.draw(reduceNoise(displayShot), matrix);
}
public function reduceNoise(srcImg : BitmapData) : BitmapData {
var ma : Array = [ 2, 2, 2,
2, 2, 2,
2, 2, 2 ]
var cf : ConvolutionFilter = new ConvolutionFilter(3, 3, ma);
var outImg : BitmapData = srcImg.clone();
outImg.applyFilter(srcImg, srcImg.rect, new Point(0,0), cf);
return outImg;
}
]]>
</mx:Script>
<mx:Label text="{countdown_val}" fontWeight="bold" fontSize="18" id="countdownLabel" />
</mx:Application>Be gentle, it’s my first real flash / flex app If anyone reads ths and has suggestions or questions, please post!
Thanks must go to Robin Hilliard and Justin Mclean for answeing my stupid actionscript noob questions, and also to Guy Watson for his excellent web motion tracking tutorial which the core of this is based upon.
I’ll go into more detail about the bits inside this in the next few days – I’d like to turn it into a full tutorial as there were a number of challenges in creating it (at least for me!).
Flex 2 – Webcam motion tracking part 2
Woohoo! I have the motion tracking working properly now. I spent ages mucking about with tiles and looping over individual pixels, then ages more trying to understand matrixes and convolution filters. I finally came up with a method where I apply what is basically a blur filter to the snapshot to remove the noise, then grab left and right sides of the video input to track both hands and allow them to control the paddles.
I’ve now added the paddles, and the track the movement of your hands! Have a look here to see the result.
This seems to work nicely – all you have to do is line up your hand properly by watching the green movment pixels, and you’ll be able to contol the paddles properly. What I’m thinking I might do is try to reverse the controls so you’re not dealing with a mirror image.
Then all I have to do is add the ball and some behaviour for bouncing and we should be golden
Here’s the new code – I worked out how to get rid of my dependancy on the panel, so the entire set of code is now in one mxml file. When I have it all working to my satisfaction I will split it out into proper classes etc but for now, here it is:
<?xml version="1.0" encoding="utf-8"?>
<mx:Application xmlns:mx="http://www.adobe.com/2006/mxml" layout="absolute" creationComplete="setup();">
<mx:Script>
<![CDATA[
import mx.core.UIComponent;
import flash.media.Camera;
import flash.filters.ConvolutionFilter;
public var camera : Camera = Camera.getCamera();
public var video : Video = new Video(camera.width*2, camera.height*2);
public var pastShot : BitmapData = new BitmapData(video.width,video.height);
public var currentShot : BitmapData = new BitmapData(video.width,video.height);
public var displayShot : BitmapData = new BitmapData(video.width,video.height);
public var displayFrame : UIComponent = new UIComponent();
public var matrix : Matrix = new Matrix();
public var shotBitmap : Bitmap;
public var leftPaddle : Sprite = new Sprite;
public var rightPaddle : Sprite = new Sprite;
public var leftPaddleFrame : UIComponent = new UIComponent;
public var rightPaddleFrame : UIComponent = new UIComponent;
public function setup() : void {
video.attachCamera(camera);
shotBitmap = new Bitmap(displayShot);
displayFrame.addChild(shotBitmap);
addChild(displayFrame);
leftPaddle.graphics.beginFill(0xFFFF0000);
leftPaddle.graphics.drawRect(0,0,10,40);
leftPaddleFrame.addChild(leftPaddle);
addChild(leftPaddleFrame);
rightPaddle.graphics.beginFill(0xFFFF0000);
rightPaddle.graphics.drawRect(video.width-10,0,10,40);
rightPaddleFrame.addChild(rightPaddle);
addChild(rightPaddleFrame);
setInterval(snapShot, 40);
}
public function snapShot() : void {
currentShot.draw(video, matrix);
var workingCopy : BitmapData = currentShot.clone();
workingCopy.draw(pastShot, matrix, null, "difference");
workingCopy.threshold(workingCopy,workingCopy.rect,workingCopy.rect.topLeft,">",0xFF333333,0xFF00FF00,0x00FFFFFF,false);
// set before to the current state, so it's ready for the next call of the function
pastShot = currentShot.clone();
// clear the display
displayShot.fillRect(displayShot.rect,0xFF000000);
// draw a rectangle over the center to eliminate the face etc
var maskRect : Rectangle = new Rectangle(displayShot.rect.x+30, displayShot.rect.y, displayShot.rect.width-60, displayShot.rect.height);
workingCopy.fillRect(maskRect,0xFF000000);
// this line redisplays the green pixels as a user guide
displayShot.threshold(workingCopy,displayShot.rect,displayShot.rect.topLeft,"==",0xFF00FF00,0xFF00FF00,0x00FFFFFF,false);
// grab the movement from the left hand area and move the paddle accordingly
var leftBM : BitmapData = new BitmapData(30,video.height);
leftBM.copyPixels(workingCopy, new Rectangle(0, 0, 30, video.height), new Point(0,0));
var leftBounds : Rectangle = leftBM.getColorBoundsRect(0xFF00FF00,0xFF00FF00,true);
if (leftBounds.y > 0) {
leftPaddle.y = leftBounds.y;
}
// grab the movement from the right hand area and move the paddle accordingly
var rightBM : BitmapData = new BitmapData(30,video.height);
rightBM.copyPixels(workingCopy, new Rectangle(video.width-30, 0, 30, video.height), new Point(0,0));
var rightBounds : Rectangle = rightBM.getColorBoundsRect(0xFF00FF00,0xFF00FF00,true);
if (rightBounds.y > 0) {
rightPaddle.y = rightBounds.y;
}
displayShot.draw(reduceNoise(displayShot), matrix);
}
public function reduceNoise(srcImg : BitmapData) : BitmapData {
var ma : Array = [ 2, 2, 2,
2, 2, 2,
2, 2, 2 ]
var cf : ConvolutionFilter = new ConvolutionFilter(3, 3, ma);
var outImg : BitmapData = srcImg.clone();
outImg.applyFilter(srcImg, srcImg.rect, new Point(0,0), cf);
return outImg;
}
]]>
</mx:Script>
</mx:Application>As you’ll see looking at the example it is still showing the green movment pixels to help you align your hands. I might see if I can separate this out from the game area to help people line up their hands properly.
Pong is almost there!
First Flex 2 App – Webcam motion tracking
This is my first exploration into flex. To build the pong game I wanted to make this was the first thing I had to get right. I have copied the ideas and methods from a couple of tutorials I found – specifically the motion tracking tutorial by Guy Watson for flash8, and another about taking snapshots in flex. I had to convert the flash8 code to as3 / flex, and muck about with a few things but the principles remained the same.
I found these tutorials very helpful, and have replicated Guy’s experiment in flex, basically tracking a handful of frames of movement and showing them in a gradually degrading colour – basically a fading motion blur, showing pixels that register movment in green.
It’s not perfect, and you’ll see a lot of noise in it depending on your environment and lighting situation, but if you have a webcam have a look here. It’s probably very simplistic to those who are experienced with actionscript but I thought it was a cool thing to do for a “hello world” app
Amazingly, it’s very very little code to achieve this. I have one actionscript class which is responsible for capturing the webcam stream and putting it in a panel (the left hand panel in the example). The rest is in an mxml file and is pretty small. Here’s the actionscript class:
package components
{
import mx.containers.Panel;
import flash.media.Camera;
import flash.media.Video;
import mx.core.UIComponent;
public class WebcamPanel extends Panel
{
public var video:Video;
public function WebcamPanel(){
super();
insertWebcamVideo();
}
public function insertWebcamVideo():void {
var videoHolder:UIComponent = new UIComponent();
var camera : Camera = Camera.getCamera();
video = new Video(camera.width*2, camera.height*2);
video.attachCamera(camera);
videoHolder.addChild(video);
addChild(videoHolder);
}
}
}I don’t think this is the best way to do it, however I haven’t learned enough yet about placing bitmaps on the screen do do away with it – it’s an easy way to drop the snapshots etc into the display so I haven’t tried to replace it yet. If anyone can add comment on a better way to draw video or bitmaps on the screen please comment!
Here’s the mxml file:
<?xml version="1.0" encoding="utf-8"?>
<mx:Application
xmlns:mx="http://www.adobe.com/2006/mxml"
xmlns:comp="components.*"
layout="horizontal"
creationComplete="startSnapshots()">
<mx:Script>
<![CDATA[
import mx.core.UIComponent;
import flash.display.BitmapData;
public var before : BitmapData = new BitmapData(320,250);
public var out : BitmapData = new BitmapData(320,250);
public var snapStack : Array = new Array;
public var maxFrames : int = 7;
public function startSnapshots() : void {
setInterval(snapshot, 10);
}
public function snapshot() : void {
var shotFrame : UIComponent = new UIComponent();
var now : BitmapData = new BitmapData(vidPanel.width,vidPanel.height);
var shotBitmap : Bitmap = new Bitmap(out);
// clear the child so we don't keep adding more and more (there must be a better way of drawing the bitmap on the screen?)
if (blendPanel.numChildren > 0) {blendPanel.removeChildAt(0); }
// grab the current view
now.draw(vidPanel.video);
// get a copy
var done : BitmapData = now.clone();
// draw over the top of the previous snap, using the difference filter to highlight the changes
done.draw(before, null, null, "difference");
// convert the negative image to show green on white (shows green pixels to trck movement)
done.threshold(done,done.rect,done.rect.topLeft,">",0xFF111111,0xFF00FF00,0x00FFFFFF,false);
// push the snap onto the array
snapStack.push(done.clone());
// if we have too many already then drop the oldest one
if(snapStack.length > maxFrames)
{
snapStack.shift().dispose()
}
var stackHeight : int = snapStack.length;
// set value for colour deterioration (for funky fading effect)
var det : Number = 255/stackHeight;
// set before to the current state, so it's ready for the next call of the function
before = now.clone();
// clear the display
out.fillRect(out.rect,0xFF000000);
// loop through the stack and display a degrading transition between the snaps.
for(var i:int = 0; i < stackHeight; ++i)
{
//determine the current degradation
var g : int = det * i
//copy all the pixels that are green in the current item in the fstack of snapshots and convert them to the degraded green
//add those pixels on top of the bitmap that will displayed to the user
out.threshold(snapStack[i],out.rect,out.rect.topLeft,"==",0xFF00FF00,(255<<24 | 0<<16 | g<<8 | 0),0x00FFFFFF,false);
shotFrame.addChild(shotBitmap);
blendPanel.addChild(shotFrame);
}
}
]]>
</mx:Script>
<comp:WebcamPanel id="vidPanel" width="340" height="280" />
<mx:Panel id="blendPanel" width="340" height="280" />
</mx:Application>Again I’d be very happy to receive any comments on the code and on better ways to do things – I’ve basically looked at example code from flash8 and tried to just learn the relevant methods and convert it to flex.
The next step was to add detection of where exactly the motion is going – up or down, essentially – and be able to make a sprite move accordingly. I’ll post the results of that later on.
Flex 5 day course
I was, as you can see below, intending to write a summary of each day at the course. However I had a business meeting tuesday evening till late, then every night after that I got too caught up in playing with flex to do any blogging
The course was great – we learned a great deal and I’m very much looking forward to doing some real work in the technology. Actionscript 3 is a brilliant language, and the combination of AS3 and MXML is fantastic. Flexbuilder itself has a whole raft of fantastic features and all in all it’s just a brilliant, fun and powerful way to build applications.
Poor Robin Hilliard though – he caught some kind of bug and was feverish on thursday although he struggled bravely through the whole day, but he was very ill on friday and had to have Justin McLean take over for him. Hope you’re feeling better Robin!
I had a great time, and am now working on a couple of flex apps. During the course Robin mentioned a playstation iToy game and it gave me an idea for my first flex app. I’m writing a game of pong where you can move your hands in front of the webcam to control the paddles. It’s working nicely so far, I just have to tweak the motion tracking to be a bit more sensitive – once I have it running I’ll post it on the blog along with source code.
I’ll post a couple of the intermediate steps too – some stuff on general motion tracking etc. It’s a lot of fun