Scivillage.com Casual Discussion Science Forum

Full Version: Behind the magic: Making moving photos a reality (video)
You're currently viewing a stripped down version of our content. View the full version with proper formatting.
The brain has been able to do this via imagination or hallucinations for ages. Now computer programs/devices can automatically accomplish such without tediously incremental, outside help from humans.

https://www.eurekalert.org/pub_releases/...061119.php

EXCERPT: People moving in and out of photographs used to be reserved for the world of Harry Potter. But now computer scientists at the University of Washington have brought that magic to real life. Their algorithm, Photo Wake-Up, can take a person from a 2D photo or a work of art and make them run, walk or jump out of the frame. The system also allows users to view the animation in three dimensions using augmented reality tools. The researchers will be presenting their results June 19 at the Conference on Computer Vision and Pattern Recognition in Long Beach, California. This research first attracted media attention when it was posted in preprint form in December on ArXiv.

[...] To make the magic a reality, Photo Wake-Up starts by identifying a person in an image and making a mask of the body's outline. From there, it matches a 3D template to the subject's body position. Then the algorithm does something surprising: In order to warp the template so that it actually looks like the person in the photo, it projects the 3D person back into 2D.

"It's very hard to manipulate in 3D precisely," said co-author Chung-Yi Weng, a doctoral student in the Allen School. "Maybe you can do it roughly, but any error will be obvious when you animate the character. So we have to find a way to handle things perfectly, and it's easier to do this in 2D."

Photo Wake-Up stores 3D information for each pixel: its distance from the camera or artist and how a person's joints are connected together. Once the template has been warped to match the person's shape, the algorithm pastes on the texture -- the colors from the image. It also generates the back of the person by using information from the image and the 3D template. Then the tool stitches the two sides together to make a 3D person who will be able to turn around.

Once the 3D character is ready to run, the algorithm needs to set up the background so that the character doesn't leave a blank space behind. Photo Wake-Up fills in the hole behind the person by borrowing information from other parts of the image. (MORE)