Live data from Hacker News

Extracting audio from visual information

newsoffice.mit.edu

31–40 of 133 posts

Re: Extracting audio from visual information

#31

I am just worried that they are picking up part of information from camera mic. Maybe camera mic output is not totally independent from the video sensor, and is encoded in the final video.

They put the camera behind soundproof glass for one of their tests.

Re: Extracting audio from visual information

#32

I am just worried that they are picking up part of information from camera mic. Maybe camera mic output is not totally independent from the video sensor, and is encoded in the final video.

The subtitle of the article is "Algorithm recovers speech from the vibrations of a potato-chip bag filmed through soundproof glass."

Re: Extracting audio from visual information

#34
post #28

Incredible and terrifying at the same time. If they're doing this kind of stuff right now with consumer cameras, imagine how effective this technology will be in just a few decades. Privacy is fading quickly with the advent of exciting technology like this.

you need a camera that captures at least 2,000 frames per second.

They also took advantage of rolling-shutter sensors running at conventional frame rates to get a "murky" audio signal.

Re: Extracting audio from visual information

#35
post #28

Incredible and terrifying at the same time. If they're doing this kind of stuff right now with consumer cameras, imagine how effective this technology will be in just a few decades. Privacy is fading quickly with the advent of exciting technology like this.

you need a camera that captures at least 2,000 frames per second.

> Because of a quirk in the design of most cameras’ sensors, the researchers were able to infer information about high-frequency vibrations even from video recorded at a standard 60 frames per second.

Re: Extracting audio from visual information

#36

I am just worried that they are picking up part of information from camera mic. Maybe camera mic output is not totally independent from the video sensor, and is encoded in the final video.

They put the camera behind soundproof glass for one of their tests.

I know, but I still wonder about level of soundproof/mechanical vibration and the level of mic and camera sensor isolation VS level of optical vibration. In addition sound can travel many different paths since it is mechanical vibration, and traveling through sound proof glass is not the only path. Anw cool project.

Re: Extracting audio from visual information

#38

Because of a quirk in the design of most cameras’ sensors, the researchers were able to infer information about high-frequency vibrations even from video recorded at a standard 60 frames per second. Can anyone explain what this quirk is?

Rolling shutter effect. http://en.wikipedia.org/wiki/Rolling_shutter

Re: Extracting audio from visual information

#39
post #28

Earlier quoted context omitted.

you need a camera that captures at least 2,000 frames per second.

> Because of a quirk in the design of most cameras’ sensors, the researchers were able to infer information about high-frequency vibrations even from video recorded at a standard 60 frames per second.

The audio from the 60fps video sounds pretty bad though, which I suspect is mostly because of inherent maths/physics limitations rather than anything that software can improve.

Edit: They mention capturing frequencies up to five times higher than the 60Hz frame rate, which would mean a maximum frequency of 300Hz, which would suggest the equivalent of 0.6kHz audio, which is a 73.5th of the audio rate of a CD. I doubt you'd get intelligible speech from current consumer hardware using this technique.

Re: Extracting audio from visual information

#40

I am just worried that they are picking up part of information from camera mic. Maybe camera mic output is not totally independent from the video sensor, and is encoded in the final video.

The camera used in most of these experiments (a Phantom high speed camera) doesn't even have a microphone - so that would be quite impossible.
Post reply on HN