• In these frames of video, a new algorithm amplifies the almost imperceptible change in skin color caused by the pumping of the blood.

    Photo: Michael Rubinstein

    Full Screen

Researchers amplify variations in video, making the invisible visible

New software amplifies changes in successive frames of video that are too subtle for the naked eye.


Press Contact

Sarah McDonnell
Email: s_mcd@mit.edu
Phone: 617-253-8923
MIT News Office

Media Resources

1 images for download

Access Media

Media can only be downloaded from the desktop version of this website.

At this summer's Siggraph — the premier computer-graphics conference — researchers from MIT's Computer Science and Artificial Intelligence Laboratory (CSAIL) will present new software that amplifies variations in successive frames of video that are imperceptible to the naked eye. So, for instance, the software makes it possible to actually "see" someone's pulse, as the skin reddens and pales with the flow of blood, and it can exaggerate tiny motions, making visible the vibrations of individual guitar strings or the breathing of a swaddled infant in a neonatal intensive care unit.

Best of 2012

The system is somewhat akin to the equalizer in a stereo sound system, which boosts some frequencies and cuts others, except that the pertinent frequency is the frequency of color changes in a sequence of video frames, not the frequency of an audio signal. The prototype of the software allows the user to specify the frequency range of interest and the degree of amplification. The software works in real time and displays both the original video and the altered version of the video, with changes magnified.

Although the technique lends itself most naturally to phenomena that recur at regular intervals — such as the beating of a heart, the movement of a vibrating string or the inflation of the lungs — if the range of frequencies is wide enough, the system can amplify changes that occur only once. So, for instance, it could be used to compare different images of the same scene, allowing the user to easily pick out changes that might otherwise go unnoticed. In one set of experiments, the system was able to dramatically amplify the movement of shadows in a street scene photographed only twice, at an interval of about 15 seconds.

Happy accident

The MIT researchers — graduate student Michael Rubinstein, recent alumni Hao-Yu Wu '12, MNG '12 and Eugene Shih SM '01, PhD '10, and professors William Freeman, Fredo Durand and John Guttag — intended the system to amplify color changes, but in their initial experiments, they found that it amplified motion as well. "We started from amplifying color, and we noticed that we'd get this nice effect, that motion is also amplified," Rubinstein says. "So we went back, figured out exactly why that happens, studied it well, and saw how we can incorporate that to do better motion amplification."

Using the system to amplify motion rather than color requires a different kind of filtration, and it works well only if the motions are relatively small. But of course, those are exactly the motions whose amplification would be of interest.

Rubinstein envisions that, among other applications, the system could be used for "contactless monitoring" of hospital patients' vital signs. Boosting one set of frequencies would allow measurement of pulse rates, via subtle changes in skin coloration; boosting another set of frequencies would allow monitoring of breathing. The approach could be particularly useful with infants who are born prematurely or otherwise require early medical attention. "Their bodies are so fragile, you want to attach as few sensors as possible," Rubinstein says.

Similarly, Rubinstein says, the system could be used to augment video baby monitors for the home, so that the respiration of sleeping infants would be clearly visible. A father himself, Rubinstein says that he and his wife equipped their daughter's crib with commercial pressure sensors intended to gauge motion and reassure anxious parents that their children are still breathing. "Those are kind of expensive," Rubinstein says, "and some people really complain about getting false positives with them. So I can really see how this type of technique will be able to work better."

In their paper, the researchers describe experiments in which they began investigating both of these applications. But since they've begun giving talks on the work, Rubinstein says, colleagues have proposed a range of other possible uses, from laparoscopic imaging of internal organs, to long-range-surveillance systems that magnify subtle motions, to contactless lie detection based on pulse rate.

"It's a fantastic result," says Maneesh Agrawala, an associate professor in the electrical engineering and computer science department at the University of California at Berkeley, and director of the department's Visualization Lab. Agrawala points out that Freeman and Durand were part of a team of MIT researchers who made a splash at the 2005 Siggraph with a paper on motion magnification in video. "This approach is both simpler and allows you to see some things that you couldn't see with that old approach," Agrawala says. "The simplicity of the approach makes it something that has the possibility for application in a number of places. I think we'll see a lot of people implementing it because it's fairly straightforward."


Topics: Image Processing, Motion Magnification, Signal processing, Temporal Filtering

Comments

This should make a polygraph more perceptive and subtle.
Motion, as we know, is change of position of an object with reference to Tome and a frame of reference to a previous position/point. If one studies the definition carefully, we would know this statement of Physics commits the fallacy of ad infinitum. Any position for this reference should be preceded by a previous position,which means that we never have a fixed frame of reference ans s a result ,Motion, as we understand now, is not Absolute. Corollary to this fact is that Motion can or may be out there where different scales are applicable. Please read my posts on ‘Time’ and’ Existence of other Universes”
As an example of one of the negative things,the source code is free and can by gotten very easy, so applied to any camera on a battle field were there are troupes in hiding, even if they are wearing there charcoal camouflager suits that stop the enemy using infrared to see there heat signature, now all the enemy need is this free download and they will be able to see any movement how ever small on a battle field ... and that's just one negative thing about this technology, I do see all the positive ones too, but! was just thinking out of the box.
Back to the top