Geeks With Blogs

News
View Szymon Kobalczyk's profile on LinkedIn

Szymon Kobalczyk's Blog A Developer's Notebook

For me one of the coolest new features added in Silverlight 4 is support for capturing video and audio using webcams and microphone. As we’ve seen during Scott Guthrie’s keynote demonstration it is now possible to capture image frames from video stream and apply some interesting effects to it. On top of that we can even process the video and audio streams directly on the client (i.e. inside the browser).

Because Silverlight 4 Beta was already available at PDC I could try the webcam support right away, and so I jumped to refresh my computer vision projects. Unfortunately I never did find time to finish this but I think it’s still worth publishing the code as it is.

One things I was playing with before was the Touchless SDK. It is an open source .NET library for visual object tracking created by Mike Wasserman. In short it identifies objects with specific color characteristics and sends you their position within the image scene enabling to use these objects as input devices. It means you can control your applications without touching the screen or keyboard (and thus it’s called touchless :-)

Original project page is here: http://touchless.codeplex.com/

Touchless SL

Over a year ago I successfully used this library in WPF so now it was quite easy to convert it again to Silverlight 4. You can check it out and download the source code on the demo page. Below I summarize some notes of things I learned about Silverlight 4 API in hopes that Silverlight team can address these issues before final release.

As a starting point I used this excellent summary by Mike Taulty. The main class that handles video processing is the CameraTracker. It extends from VideoSink in order to get access to incoming video samples. Each sample frame is represented by byte array which format is specified by VideoFormat structure. In order to get current VideoFormat you have to override the OnFormatChange method (call to videoSource.VideoCaptureDevice.DesiredFormat will throw exception).

I learned that although VideoFormat specifies the PixelFormatType, currently it is an enum with only two values: Format32bppArgb and Unspecified. It’s WPF counterpart – PixelFormat structure - offers much more including masks and bytes per pixel. And instead of enum we get a static PixelFormats class with number of common formats declared as properties. I hope this will be changed in Silverlight 4 RTM.

Another thing to be aware of is that the video frame buffer is flipped vertically – so you need to read the image lines from bottom to top. I handle this in the GetPixel and SetPixel methods of RawBitmapAdapter helper class:

public RgbColor GetPixel(int x, int y)
{
    int p = _scan0 + (y * _stride) + (x * _bytesPerPixel);
    byte b = _imageData[p];
    byte g = _imageData[p+1];
    byte r = _imageData[p+2];
    return new RgbColor(r, g, b);
}
public void SetPixel(int x, int y, RgbColor color)
{
    int p = _scan0 + (y * _stride) + (x * _bytesPerPixel);
    _imageData[p] = color.Blu;
    _imageData[p+1] = color.Grn;
    _imageData[p+2] = color.Red;
}

In both cases I first calc the offset of pixel in the byte buffer using the scan0 and stride values. Stride is specified in VideoFormat but we have to guess the scan0 value (it specifies the offset of the first line):

if (_stride < 0)
    _scan0 = -_stride * (_height - 1);

Would be helpful the get it in the VideoFormat too. One last comment regarding VideoFormat is I think it’s constructor should be made public.

Before initializing the camera app displays a list of available devices. I tried to bind the collections from my ViewModel however it turns out that calls to GetAvailableVideoCaptureDevices and GetDefaultVideoCaptureDevice return different instances of CaptureDevice so you can’t do something like this:

CaptureDevices = CaptureDeviceConfiguration.GetAvailableVideoCaptureDevices(); 
SelectedCaptureDevice = CaptureDeviceConfiguration.GetDefaultVideoCaptureDevice();

Also I noticed that none of the devices returned by GetAvailableVideoCaptureDevices has IsDefaultDevice flag set, so I had to use FriendlyName to match it.

Because CaptureSource is used in similar way to MediaElement I think it should expose similar API. In particular it would be helpful to get events when it state changes. We should also get some notification when camera is connected or disconnected.

In order to show the markers recognized by Touchless I paint these areas on an overlay. In current version I tried to implement this using a custom MediaStreamSource with double buffering as demonstrated by Pete Brown. It works but you can see the overlay lags significantly in respect to camera video. So I’m going to switch back to WriteableBitmap instead, but I’m not sure how it will perform either. In particular I’m concerned that WriteableBitmap doesn’t have a way to indicate that we are going to begin updating the buffer. In the WPF version you first need to issue Lock to get access to the BackBuffer. I was bit surprised that it’s not required in Silverlight version and I’m concerned how efficient it is without it. 

Altogether the camera API looks very promising and definitively opens many possibilities. However I think the most common scenario people will be trying to implement will be some sort of online IM client, and for this we would also need some generic video codecs for streaming. Right now Silverlight doesn't even have codecs for Jpeg/PNG images so I think it should be added in the first place. In the meantime you can try to use these as an alternative: http://imagetools.codeplex.com/ Updated: Please see this great post by Rene Shulte on saving webcam snapshots to JPEG using FJCore library.

Last thing, unrelated to camera API, is regarding support for commands that was also added in Silverlight 4. I found that the command CanExecute method won’t be required automatically on user input as it happens in WPF so you need to manually send CanExecuteChanged event for each command.

Posted on Saturday, January 2, 2010 6:20 AM Development , Silverlight | Back to top


Comments on this post: Experimenting with Silverlight 4 – using webcam with Touchless SDK

# re: Experimenting with Silverlight 4 – using webcam with Touchless SDK
Requesting Gravatar...
Thanks for sharing such wonderful things....i have been lately following your work...
Left by Sanjay Srikanth Nekkanti on Jan 24, 2010 9:44 AM

# re: Experimenting with Silverlight 4 – using webcam with Touchless SDK
Requesting Gravatar...
Interesting and clever snippet!
Left by mac on Dec 03, 2010 9:45 AM

# re: Experimenting with Silverlight 4 – using webcam with Touchless SDK
Requesting Gravatar...
WOW, the experiment seems quite interesting, thanks for sharing the idea, great post as well...
Left by limo hire leeds on Jan 06, 2011 7:44 AM

# re: Experimenting with Silverlight 4 – using webcam with Touchless SDK
Requesting Gravatar...
I am glad I discovered this blog, I couldnt discover any information on this material prior to. I also run a web site and if you want to ever critical in a little bit of visitor writing for me if attainable feel free to let me know.
Left by jiu jitsu gear on Mar 29, 2011 10:16 AM

# re: Experimenting with Silverlight 4 – using webcam with Touchless SDK
Requesting Gravatar...
Thanks for sharing this experiment with us. I love experiments
Left by tillers FOR SALE on Mar 29, 2011 1:22 PM

# re: Experimenting with Silverlight 4 – using webcam with Touchless SDK
Requesting Gravatar...
I want to take this moment to say that I really love this blog. It has been a good resource of information for me. Thank you so much
Left by Datenrettung Raid on Apr 08, 2011 12:49 AM

# re: Experimenting with Silverlight 4 – using webcam with Touchless SDK
Requesting Gravatar...
I’m happy to have found your very excellent article! I agree with some of your readers and will eagerly look forward to your coming updates. Just saying thanks will not just be adequate
Left by Skiatook Lake on Apr 10, 2011 10:22 PM

# re: Experimenting with Silverlight 4 – using webcam with Touchless SDK
Requesting Gravatar...
I’m happy to have found your very excellent article! I agree with some of your readers and will eagerly look forward to your coming updates. Just saying thanks will not just be adequate +++++++++++++++++
Left by Glass Tile on Apr 15, 2011 5:22 AM

# re: Experimenting with Silverlight 4 – using webcam with Touchless SDK
Requesting Gravatar...
Interesting post and thanks for sharing. Some things in here I have not thought about before.Thanks for making such a cool post which is really very well written.will be referring a lot of friends about this.Keep blogging nice one thank you for sharing
Left by Glass Tile on Apr 15, 2011 5:23 AM

# re: Experimenting with Silverlight 4 – using webcam with Touchless SDK
Requesting Gravatar...
I am very excited that I have found your post because I have been searching for some information about it for almost three hours. You have helped me a lot and reading this your article I have found many new and useful information about this subject.
Left by austin dwi lawyers on Apr 16, 2011 8:27 AM

# re: Experimenting with Silverlight 4 – using webcam with Touchless SDK
Requesting Gravatar...
Thank you very much for sharing this! Looks like a useful snippet however I still avoid Silverlight because it won't work on OS X.
Left by ananda.cer on Jun 20, 2012 4:38 PM

Your comment:
 (will show your gravatar)


Copyright © Szymon Kobalczyk | Powered by: GeeksWithBlogs.net