autocratic for the people.

How Hard Could It Be, Part 2.

(You can find Part 1 here.)

To perform background replacement on an image, one must first perform background subtraction – that is, figure out what portion of the given image is the background, what portion is the foreground, and then separate the two. Let's take a webcam teleconference as the test case, since that's what Apple showed. In this case, the foreground is the person in the conference, and the background is everything else. The problem looks like this:


1. Empty Background


This is the background: the "empty image." What the camera sees when there's no one around.


2. Person Against Background


This is what the camera sees when the person waves at the camera.


3. Background Subtracted (ideal case)


This is what the perfect scenario would look like: the waving hand is pulled out of the image, and the rest is removed. Now we could superimpose this hand over a different image, or video, or whatever.

Looking at the problem like this, an approach immediately presents itself: why not just look at each pixel in the background image and compare it to each pixel in the hand-waving image? If the two pixels are the same, then the pixel must be part of the background. If not, it's gotta be foreground. Right? How hard can it be?

Well unfortunately, if you do that, you get this:



See all the little blue specks? Those are the pixels that didn't change, and so were identified as background. It looks like only very few of the background pixels were actually removed, even before putting anything in the foreground. Even worse, the pixels that are removed change every frame! What happened?

In a word, noise. Noise is the random graininess that you get in images. In digital cameras it's usually the result of electronic interference, a crappy camera sensor, or both, and it's especially noticeable in low quality webcam images such as these. In order to be able to find the background, we're going to need to be able to recognize it even in a noisy image. Fortunately, there's a simple method that works pretty well, and you're most likely already familiar with it (or were, anyway, in stats class). Stay tuned.

posted by avixe @ 9:26 PM