The Cutting-Edge Science That Can Turn Everyday Objects, Like a Bag of Chips, Into a Listening Device

For decades we’ve laughed at the per­sis­tent movie and tele­vi­sion cliche of “image enhance,” where­by char­ac­ters — usu­al­ly detec­tives of one kind or anoth­er in pur­suit of a yet-unknown vil­lain — dis­cov­er just the clue they need by way of tech­no­log­i­cal mag­ic that some­how increas­es the amount of detail in a piece of found footage. But now, of course, our age of rapid­ly improv­ing arti­fi­cial intel­li­gence has brought an algo­rithm for that. And not only can such tech­nolo­gies find visu­al data we nev­er thought an image con­tained, they can find son­ic data as well: recov­er­ing the sound, in oth­er words, “record­ed” in osten­si­bly silent video.

“When sound hits an object, it caus­es small vibra­tions of the object’s sur­face,” explains the abstract of “The Visu­al Micro­phone: Pas­sive Recov­ery of Sound from Video,” a paper by Abe Davis, Michael Rubin­stein, Neal Wad­hwa, Gau­tham Mysore, Fre­do Durand, and William T. Free­man. “We show how, using only high-speed video of the object, we can extract those minute vibra­tions and par­tial­ly recov­er the sound that pro­duced them, allow­ing us to turn every­day objects — a glass of water, a pot­ted plant, a box of tis­sues, or a bag of chips — into visu­al micro­phones.” Or a lis­ten­ing device. You can see, and more impres­sive­ly hear, this process in action in the video at the top of the post.

The video just above mag­ni­fies the sound-caused motion of a bag of chips, to give us a sense of what their algo­rithm has to work with when it infers the sound present in the bag’s envi­ron­ment. In a way this all holds up to com­mon sense, giv­en that sound, as we all learn, comes from waves that make oth­er things vibrate, be they our eardrums, our speak­ers — or, as this research reveals, pret­ty much every­thing else as well. Though the bag of chips turned out to work quite well as a record­ing medi­um, some of their oth­er test sub­jects, includ­ing a brick cho­sen specif­i­cal­ly for its lack of sound-cap­tur­ing poten­tial, also did bet­ter than expect­ed.

The hid­den infor­ma­tion poten­tial­ly recov­er­able from video hard­ly stops there, as sug­gest­ed by Rubin­stein’s TED Talk just above. “Of course, sur­veil­lance is the first appli­ca­tion that comes to mind,” he says, to slight­ly ner­vous laugh­ter from the crowd. But “maybe in the future we’ll be able to use it, for exam­ple, to recov­er sound across space, because sound can’t trav­el in space, but light can.” Just one of many sci­en­tif­i­cal­ly noble pos­si­bil­i­ties, for which watch­ing what we say next time we open up a bag of Dori­tos would be, per­haps, a small price to pay.

via Kot­tke

Relat­ed Con­tent:

What Does Sound Look Like?: The Audi­ble Ren­dered Vis­i­ble Through Clever Tech­nol­o­gy

The Geom­e­try of Sound Waves Visu­al­ized

Hear What Music Sounds Like When It’s Cre­at­ed by Syn­the­siz­ers Made with Arti­fi­cial Intel­li­gence

Radiooooo: Dis­cov­er the Musi­cal Time Machine That Lets You Hear What Played on the Radio in Dif­fer­ent Times & Places

Based in Seoul, Col­in Mar­shall writes and broad­casts on cities and cul­ture. His projects include the book The State­less City: a Walk through 21st-Cen­tu­ry Los Ange­les and the video series The City in Cin­e­ma. Fol­low him on Twit­ter at @colinmarshall or on Face­book.