Reminds me of the technique used to uncover "invisible" motion by amplifying the tiny movements found in a video:
http://bits.blogs.nytimes.com/2013/02/27/scientists-uncover-...
Extracting audio from visual information
81–90 of 133 posts
Re: Extracting audio from visual information
#82While it's cool to see this technique applied to consumer video, the general idea has been used in international espionage for a while now. In fact, Léon Theremin (yes, of the musical instrument) invented an early device capable of eavesdropping based on window vibrations: http://en.wikipedia.org/wiki/L%C3%A9on_Theremin#Espionage . I also recall that this is why all of the windows in the White House are fitted with t…
As a kid (in the early 80's!) I accidentally figured this out while "hacking" a sound based "walkie talkie" with a photo cell and a LED pointed at each other through a little telescope, from my house to my neighbors across the bock.
The device worked, but had a bug. a hummmm on the receiver.
For the longest time I could not figure out what was the "hummm", till I noticed it was there even with the transmitter off.. and then, I heard strange voices even with the transmitter being off!
Then I realized the humm was the reflection of incandescent light on the window, and the voices where the people in the room making the window vibrate too..
Ahh.. those where great fun days early on "hacker" life as a hacker. :)
Re: Extracting audio from visual information
#83Earlier quoted context omitted.
The camera used in most of these experiments (a Phantom high speed camera) doesn't even have a microphone - so that would be quite impossible.
Thanks. However, there is still a possibility that mechanical vibration is coming to the video sensor through some path (floor+camera tripod), and affecting the video. After all mechanical vibration is affecting the potato chips. Why is it so impossible to affect video sensor? Especially in the sensitive high speed camera? Let the downvotes begin :)
I fail to see how that would make this any less impressive.
Re: Extracting audio from visual information
#84From watching the video, I get the impression that there is a very large amplitude of the input audio -- taking advantage of the "loud" in loudspeaker.
A loudspeaker is a term for a speaker that you traditionally think of when you think of a speaker:
https://en.wikipedia.org/wiki/Loudspeaker
Re: Extracting audio from visual information
#85This is fun research, but it's basically just a microphone (which works by turning these vibrations into magnetic pulses that generate electric pulses) using a much, much more complex signal processing mechanism (observing photons bouncing off the vibrating surface, translating the digital representation of those photons into a digital simulation of the electric pulses a microphone would generate, and turning that digital simulation into real electric pulses).
Re: Extracting audio from visual information
#86Reminds me of this great hoax / video piece from a few years ago about French scientists decoding voices that were accidentally encoded into vases in ancient Pompeii
http://www.youtube.com/watch?v=ZbpwBTDvXrI
It's in French, but this makes it even funnier.
Re: Extracting audio from visual information
#87Earlier quoted context omitted.
You missed the part where the video's frames per second has to be higher than the audio frequency.
Some cheap video cameras don't shutter the frames (grabbing all pixels at once), they progress thru pixels - leading to a usually-annoying stretchy visual effect. If the sound affects much of the frame with suitable uniformity, you've turned a 60fps camera into a 2MHz "visual" audio sampler (lots of noise, artifacts, & other difficulties aside).
Like in this clip, taken with an iPhone, of guitar strings vibrating?
Re: Extracting audio from visual information
#88While a different technique, this reminds me of how researchers were able to extract keystrokes based on the sound from each key press.
Here is the original paper:
Re: Extracting audio from visual information
#89While a different technique, this reminds me of how researchers were able to extract keystrokes based on the sound from each key press. Here is the original paper: http://rakesh.agrawal-family.com/papers/ssp04kba.pdf
Also reminds me of this: http://cnx.org/content/m13224/latest/?collection=col10380/la...
Re: Extracting audio from visual information
#90Earlier quoted context omitted.
Thankfully most CCTVs are so crappy and so low framerate, I doubt they can get much out of it. If you want to record a particular person through a window, for instance, you can get a laser microphone that catches the vibrations of the glass.
Yeah, I've heard that's what the spies use, although that does require specific effort. What I find more intriguing about this development is how pre-existing footage could be used. While HD CCTV is certainly not popular, I suspect enough has been said in the presence of existing HD video to incriminate a few people :-)
I think it unlikely that pre-existing footage can be used, because HD video is almost always compressed, thus masking the minute vibrations in the pixels. The algorithm described in the article work best for uncompressed video directly from the image sensor, and they can run it in real-time without needing to store the video.