MIT Turned Silent Video of a Chip Bag Into Speech, Through Soundproof Glass

When you talk, the sound doesn't just hit people's ears. It hits everything in the room, and everything in the room shakes a tiny bit. In 2014, researchers at MIT, Microsoft and Adobe showed they could film those shakes and turn them back into sound. One of their test objects was a bag of potato chips. There was no microphone and no audio track, only video.

Here's what they did, how it works, and what it does (and doesn't) mean for your privacy.

What the researchers did

MIT News reported in August 2014 that the team built an algorithm that rebuilds an audio signal from the vibrations of objects in a video. In one experiment, MIT says, they recovered intelligible speech from the vibrations of a potato-chip bag "photographed from 15 feet away through soundproof glass."

The team's paper, "The Visual Microphone: Passive Recovery of Sound from Video," was published in ACM Transactions on Graphics (the SIGGRAPH 2014 proceedings). It gives the details of that test:

• The speech: a person said "Mary had a little lamb ... Welcome to SIGGRAPH!" in a room behind thick soundproof glass.
• The camera: high-speed video at 4 kHz, which means 4,000 frames a second, at 400 by 480 pixels. The bag was lit by natural sunlight.
• The distance: the paper says this test used a 400mm lens "to recover sound from a distance of 3-4 meters." That's about 10 to 13 feet. MIT News gives 15 feet. We've given you both because the two sources don't match.

The chip bag wasn't the only object. MIT says the team also got audio from aluminum foil, the surface of a glass of water and the leaves of a potted plant.

You can watch the result yourself. The researchers' project page links to their demo video, which plays the sound they pulled out of the footage.

How it works

The visual microphone, in four steps

1. Sound shakes things. Sound waves push on objects. A light object like a chip bag moves back and forth with the sound.
2. The shakes are far too small to see. MIT says the motions were about a tenth of a micrometer. In a close-up shot, that's about five thousandths of a pixel.
3. Colors give the motion away. At the edge of an object, a pixel can show a blend of two colors. MIT's example: if a blue edge meets a red background, the edge pixel looks purple. When the edge moves even a sliver, the blend changes a little. Software that tracks those tiny color changes across many frames can measure motion smaller than one pixel.
4. Motion becomes sound. Track the shaking over time, and you have a sound wave. MIT notes that the camera's frame rate has to be higher than the frequency of the sound, so faster cameras can capture higher pitches.

"When sound hits an object, it causes the object to vibrate," lead author Abe Davis told MIT News. "People didn't realize that this information was there."

The paper lists six authors: Abe Davis, Michael Rubinstein, Neal Wadhwa, Gautham J. Mysore, Frédo Durand and William T. Freeman. Rubinstein was at Microsoft Research and Mysore at Adobe Research.

Does it need a special camera?

For clear speech, yes. MIT says some experiments used a high-speed camera at 2,000 to 6,000 frames a second. That's much faster than the 60 frames a second possible with some smartphones.

The paper's lab tests used objects 0.5 to 2 meters from the camera, under photography lamps, at 2,000 to 20,000 frames a second. The sounds were loud: the paper says they ranged from about 80 dB (an actor's stage voice) to 110 dB (a jet engine 100 meters away).

The team also tried an ordinary camera. They used a Pentax K-01 DSLR at 60 frames a second, filming a bag of candy near a loudspeaker playing speech. That camera reads its sensor one row at a time (a "rolling shutter"), so each frame holds slightly different moments. The software used that quirk to pull out more sound than 60 frames a second should allow.

The result was rough. The paper says the speech was "difficult to understand." But MIT says audio like that "may still be good enough to identify the gender of a speaker in a room; the number of speakers; and even, given accurate enough information about the acoustic properties of speakers' voices, their identities."

The paper also says recovering sound from farther away "may require expensive optics with large focal lengths."

Why it matters: what it means for you

It's a lab result, not a street threat. The paper shows what's possible with the right camera, lens, light and a loud sound source. We found no official source saying police or spies use this technique on real cases. MIT News said the work has "obvious applications in law enforcement and forensics." That's a prediction, not a report of real use.

It changes what "soundproof" means. Soundproof glass stops sound from passing through. This method doesn't need sound to pass through. It needs light, which comes through a window just fine.

Video can hold more than you think. The paper notes that knowing how many people are in a room, and their gender, "can be useful in some surveillance scenarios even if intelligible speech cannot be recovered." Even a weak result can leak something.

Our read: For most people, the everyday privacy risks are still the familiar ones: phones, smart speakers and apps with microphone access. But this study is a good reminder that sound leaves traces in places we don't think to look.

Cheat codes: what you can do

These are our general tips. They aren't tested defenses from the paper.

• Close the blinds for private talks. The setup in the paper needs a clear line of sight to an object in the room.
• Lower your voice. The lab sounds ran from a loud stage voice up to jet-engine levels. Loud speech is easier to pick up.
• Think about what's in view of a camera. That includes a video call. Only the high-speed tests produced clear speech, but it's a good habit anyway.
• Check app microphone permissions. On your phone, see which apps can use your mic and turn off the ones that don't need it. That's a far more common way for audio to leak.