Wouldn't it be really neat to apply this to HD movie sequences, and hear what the sounds on the set and the voices of actors were like pre-production? And how unreal some of the sounds must have turned out with all the visual tweaking that happens in production?
Extracting audio from visual information
41–50 of 133 posts
Re: Extracting audio from visual information
#42Earlier quoted context omitted.
They put the camera behind soundproof glass for one of their tests.
I know, but I still wonder about level of soundproof/mechanical vibration and the level of mic and camera sensor isolation VS level of optical vibration. In addition sound can travel many different paths since it is mechanical vibration, and traveling through sound proof glass is not the only path. Anw cool project.
I'm sure it's such a tiny amount that it isn't really relevant. Easy enough to test. Remove the bag of chips and just film the floor. See how much signal you can derive from that. Not exactly perfect science but gives a quick indication of "signal" that's reaching the system via other means.
Re: Extracting audio from visual information
#43Am I the only one who isn't getting any audio from the video at all? I tried in two different browsers, downloaded the video with youtube-dl and tried to play it with mpv, everything to no avail, it's just a video with no sound.
You probably only downloaded the DASH video, this should work:
$ youtube-dl -f 18 http://newsoffice.mit.edu/2014/algorithm-recovers-speech-fro...
You can use the -F option to list the available formats.
Re: Extracting audio from visual information
#44Earlier quoted context omitted.
> Because of a quirk in the design of most cameras’ sensors, the researchers were able to infer information about high-frequency vibrations even from video recorded at a standard 60 frames per second.
The audio from the 60fps video sounds pretty bad though, which I suspect is mostly because of inherent maths/physics limitations rather than anything that software can improve. Edit: They mention capturing frequencies up to five times higher than the 60Hz frame rate, which would mean a maximum frequency of 300Hz, which would suggest the equivalent of 0.6kHz audio, which is a 73.5th of the audio rate of a CD. I doubt…
There is some small possibility of improvement through software techniques, such as maybe data assimilation, which can use information from surrounding time-frames to improve the measurement. This is assuming that the magnitude of vibrations changes a lot slower than the vibrations themselves, which is usually true, and how most audio compression works. It may be able to clean up the sound a little. However, I would say that the results they have obtained so far are very impressive.
Re: Extracting audio from visual information
#45From watching the video, I get the impression that there is a very large amplitude of the input audio -- taking advantage of the "loud" in loudspeaker.
A good question. I too got that impression from the first example. In the second example (chips bag through glass), the "control" audio had a lot of reverb, which might have been introduced by the phone they acquired it through, but may also suggest that it wasn't just a person talking, but a reproduction through some kind of amplification equipment.
Re: Extracting audio from visual information
#46Binoculars as bugging devices...
Re: Extracting audio from visual information
#47[deleted]
Re: Extracting audio from visual information
#48I am amazed this wasn't funded by DARPA.
I would bet money that this technique is already well known and has been used by certain intelligence agencies.
Re: Extracting audio from visual information
#49Earlier quoted context omitted.
Cameras basically read their sensors one row of pixels at a time. By measuring the distortion of each row, they can detect vibrations higher than the camera's frame rate.
So it's like if 960-row video at 60fps were actually a 57600 rows-per-second video, right? Which they can extract info from because having more rows in a still frame doesn't mean having more information (at least not linearly), i.e. in still frames with no rolling shutter, rows contain redundant vibration already extracted from previous rows. So having a rolling shutter is good for this specific application because i…
What's at play is the sampling theorem. Using instantaneous shutter, considering the speed of sound is very high*, every part of the image is resonating essentially the same sound; the sampling theorem says you can only perfectly reconstruct up to F/2 hertz if you sample at F frames per second. The rolling shutter introduces regular variations at at much higher frequency, allowing much better sampling.
- Actually, doing a quick calculation shows that at 1khz a 1/2 wavelength is just 17cm. I wonder how precise spatial scene/source information has to be to allow this diversity to contribute significantly to the sampling. If you had a planar source and precisely spaced two objects it shouldn't be too hard to increase spectral resolution. The complementary possibilities are also be interesting -- with precisely laid out N objects and a good spectral resolution for each afforded by the shutter you could perhaps resolve the sound into N distinct sources, allowing to determine the origin of the sound; with precisely known source locations you may be able to extract some object location information.
Re: Extracting audio from visual information
#50This is from the movie Eagle Eye, right? The evil computer watches the vibrations in a cup of coffee while someone is speaking.
They also did something like this in the recent remake of "Robocop". Some thugs are in a diner having a discussion, and it's caught on CCTV. Later, the main character goes back and analyzes the video and extracts the audio from it, presumably based on some vibrations somewhere in the scene. They don't say, but I got the impression it was from the window vibrations, similar to the technique of shooting a laser at it and watching the vibrations of the reflected light.