• Graphic: Christine Daniloff

    Full Screen
  • This image shows the final product of the holographic video setup: a paper crane transmitted from the Kinect camera through a PC into the holographic display.

    Image: James D. Barabas

    Full Screen

3-D TV? How about holographic TV?

Using a single Xbox Kinect and standard graphics chips, MIT researchers demonstrate the highest frame rate yet for streaming holographic video.


Press Contact

Jessica Holmes
Email: newsoffice@mit.edu
Phone: 617-253-2700
MIT News Office

Media Resources

1 images for download

Access Media

Media can only be downloaded from the desktop version of this website.

The Consumer Electronics Show in Las Vegas in January 2010 was abuzz about a slew of prototype 3-D TVs, but if new research from the MIT Media Lab is any indication, holographic TVs could be close behind. At the Society of Photo-Optical Instrumentation Engineers’ (SPIE) Practical Holography conference in San Francisco the weekend of Jan. 23, members of Michael Bove’s Object-Based Media Group presented a new system that can capture visual information using off-the-shelf electronics, send it over the Internet to a holographic display, and update the image at rates approaching those of feature films.

In November, researchers at the University of Arizona made headlines with an experimental holographic-video transmission system that used 16 cameras to capture data and whose display refreshed every two seconds. The new MIT system uses only one data-capture device — the new Kinect camera designed for Microsoft’s Xbox gaming system — and averages about 15 frames per second. Moreover, the MIT researchers didn’t get their hands on a Kinect until the end of December, and only in the week before the conference did they double the system’s frame rate from seven to 15 frames per second. They’re confident that with a little more time, they can boost the rate even higher, to the 24 frames per second of feature films or the 30 frames per second of TV — rates that create the illusion of continuous motion.

The difference between holograms and the type of 3-D images becoming common in movie theaters is frequently overlooked, Bove says. During a screening of, say, the 3-D version of Avatar, viewers on the far-left aisle of the theater see the same image that viewers on the far-right aisle do. That image may have depth, but it’s filmed from a single perspective. As a viewer moves around a hologram, however, his or her perspective on the depicted object changes continuously, just as it would if the object were real.

All the angles

A standard 3-D movie camera captures light bouncing off of an object at two different angles, one for each eye. But in the real world, light bounces off of objects at an infinite number of angles. Holographic video systems use devices that produce so-called diffraction fringes, fine patterns of light and dark that can bend the light passing through them in predictable ways. A dense enough array of fringe patterns, each bending light in a different direction, can simulate the effect of light bouncing off of a three-dimensional object.

The challenge with real-time holographic video is taking video data — in the case of the Kinect, the light intensity of image pixels and, for each of them, a measure of distance from the camera — and, on the fly, converting that data into a set of fringe patterns. Bove and his grad students — James Barabas, David Cranor, Sundeep Jolly and Dan Smalley — have made that challenge even tougher by limiting themselves to off-the-shelf hardware.

“Really, the focus of our work in digital holography — and I think this makes us pretty much unique among the very small community of people in the world even doing holovideo — is that we’re trying to make a consumer product,” Bove says. “So we’ve been saying, ‘How do you make it as cheap as possible — take advantage of hardware and standards and software and everything else that already exists?’ Because that’s the quickest way to bring it to market.”

Using a single Kinect camera and standard graphics chips, MIT researchers demonstrate the highest frame rate yet for streaming holographic video.
Video: Melanie Gonick; additional stills: James D. Barabas

In the group’s lab setup, the Kinect feeds data to an ordinary laptop, which relays it over the Internet. At the receiving end, a PC with three commercial graphics processing units — GPUs — computes the diffraction patterns.

GPUs differ from ordinary computer chips — CPUs — in that their circuitry has been tailored to a cluster of computationally intensive tasks that arise frequently during the processing of large graphics files. Much of the work that went into the new system involved re-describing the problem of computing diffraction patterns in a way that takes advantage of GPUs’ strengths.

Coming attractions

The one component of the researchers’ experimental system that can’t be bought at an electronics store for a couple hundred dollars is the holographic display itself. It’s the result of decades of research that began with MIT’s Stephen Benton, who built the first holographic video display in the late 1980s. (When Benton died in 2003, Bove’s group inherited the holographic-video project.) The current project uses a display known as the Mark-II, a successor to Benton’s original display that both Benton’s and Bove’s groups helped design. But Bove says that his group is developing a new display that is much more compact, produces larger images, and should also be cheaper to manufacture. (Bove and his students reported on an early version of the display at the same SPIE conference four years ago.)

Mark Lucente, director of display products for Zebra Imaging in Austin, Texas, which is commercializing holographic displays for videoconferencing applications, says that his company’s prospective customers are often uncomfortable with the sheer computational intensity of holographic video. “It’s very daunting,” he says. “1.5 gigabytes per second are being generated on the fly.” By demonstrating that off-the-shelf components can keep up with the computational load, Lucente says, Bove’s group is “helping show that it’s within the realm of possibility.” Indeed, he says, “by taking a video game and using it as an input device, [Bove] shows that it’s a hop, skip and a jump away from reality.”

When the Media Lab researchers demonstrate their new technology at the conference in San Francisco, another grad student in Bove’s group, Edwina Portocarrero, sporting a cowled tunic and a wig with side buns, will re-enact the scene from the first Star Wars movie in which a hologram of Princess Leia implores Obi-Wan Kenobi to re-join the battle against the evil empire. The resolution of the real hologram won’t be nearly as high as that of the special-effects hologram in the movie, but as Bove points out, “Princess Leia wasn’t being transmitted in real time. She was stored.”


Topics: Holography, Media Lab, Video

Comments

Why not use the tech from AirCord Labs for visual output as well? I think it would be impressive to see it working. Maybe use 2 Kinects for 360? http://gizmodo.com/5590999/ipad-projector-renders-3d-objects-for-the-naked-eye
Thanks for the comment. AirCord looks really nice. (this is the video I found http://vimeo.com/13389710) We could use something like their beamsplitter arrangement to make our holograms appear to float over physical objects. The AirCord images look good on video, but they still send the same image to both eyes, making them appear flat when viewed in person. Using two Kinects is also a good idea. It would let us fill in backgrounds that aren't captured because they are "hidden" behind our actor.
While the team's efforts to reduce the amount of data needed to transmit, store and display some approximation of holographic video (volumetric or otherwise) are commendable, this article seems to confound holography with depth map extrusions of color video. Techniques for combining real-time color video with depth or disparity maps from time-of-flight cameras such as a Swiss Ranger, in order to create a flawed approximation of a volumetric field, have been reduced to practice by many companies and universities over the previous decade. In fact, most of the major depth camera companies such as Prime Sense, PMD, Canesta, and others provide SDKs which will do this for you semi-automatically, due to the integration of depth and color sensors in the same module. At first I thought the novelty may be in the network transmission of the depth map, but I believe that has existed in military applications and certain large enterprise surveying tools (LiDAR etc.) for some time. But, as I think on it more, perhaps the novelty is in the remapping of the depth extruded color video frames into multiple views that can be rendered using the holography apparatus. This does seem to be an interesting technique, as it may help to solve the bandwidth issues if one were to attempt to transmit all those views. Actually, Philips had been doing this for many years, using their WoWVx format. They would transmit video include color, depth, occlusion, and transparency information. This could then be used by the display to reconstruct multiple views for their automultiscopic displays. Again, not holographic, or volumetric, but you can image that if you were able to sense or synthesize this data, you could adapt that format to work on an actual holographic display. However, it still seems a bit of a stretch to me to compare this technique to the U of Arizona 16-view setup. They are using a different technique with different trade-offs and very different results.
How would the Lytro camera help in the process of creating holographic video (if at all)?
Back to the top