Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Extracting audio from visual information

newsoffice.mit.edu

Post reply on HN
Print view View original

111–120 of 133 posts

Re: Extracting audio from visual information

#111

Post by keerthiko »

anigbrowl wrote: post #100
keerthiko wrote: post #41

Wouldn't it be really neat to apply this to HD movie sequences, and hear what the sounds on the set and the voices of actors were like pre-production? And how unreal some of the sounds must have turned out with all the visual tweaking that happens in production?

I record the dialog for your movies. Aprt from obvious sound effects like Darth Vader voices or so, actors' dialog is changed as little as possible during post production. It's not heavily EQed or comrpessed, because that would mean a corresponding change to the room tone and background noise, which would then fluctuate unnaturally as you went back and forth between the participants in a scene. In scenes with a lot o…

Thanks a lot for that insight. I will assume you work primarily for Hollywood with that description. It is interesting to hear about the behind-the-scenes from film industries in different parts of the world.

In the Indian film industry (maybe not so much Bollywood, but more of the local ones like Tamil, Malayalam, etc) there's a LOT more post being done. Some entire Malayalam films have dubbed voices (for the original movie). Not to mention nearly every song track does not have the singing recorded during the shooting of the dance sequences. So those sets and actor voices I would bet would be completely different.

Re: Extracting audio from visual information

#112

Post by mnw21cam »

T-hawk wrote: post #51
mnw21cam wrote: post #44

Earlier quoted context omitted.

There is some small possibility of improvement through software techniques, such as maybe data assimilation, which can use information from surrounding time-frames to improve the measurement. This is assuming that the magnitude of vibrations changes a lot slower than the vibrations themselves, which is usually true, and how most audio compression works. It may be able to clean up the sound a little. However, I would…

The data comes in faster than 60 fps. A camera sensor doesn't capture the entire frame instantly every 1/60 second. It progressively scans through the frame over some measurable fraction of that 1/60 second. This is that quirk. Suppose the camera scans 720 lines in HD every 1/60 second. Each row is offset in time by 1/43200 second. A rigid object could be slightly offset in space on each line of pixels, indicating th…

> The data comes in faster than 60 fps

Yes, yes, that was completely obvious from the article. We are getting thousands of "measurements" per second.

However, each of those measurements is incredibly inaccurate. Each one is trying to detect the change of colour of 1/200 of the colour range in a single pixel. You may be getting less than a single bit of entropy per measurement.

An advanced signal processing technique will look at the longer-term picture. Sound vibrations are not a random walk - they tend to be a combination of sine wave vibrations, where the rate of change of magnitude of each wavelength is significantly lower than the vibrations themselves. Therefore they are to a certain extent predictable, and this predictability is used by audio compression algorithms. The signal processing algorithm will have to make use of the extremely limited information coming from the measurements, and match up possible sets of varying sine waves that could be causing those measurements. This may be sufficient to reject some of the noise that we could hear on that video, and clean up the sound a bit, but it is quite a hard (and CPU-intensive) processing task.

Re: Extracting audio from visual information

#113

Post by thisjepisje »

keerthiko wrote: post #111
anigbrowl wrote: post #100

Earlier quoted context omitted.

I record the dialog for your movies. Aprt from obvious sound effects like Darth Vader voices or so, actors' dialog is changed as little as possible during post production. It's not heavily EQed or comrpessed, because that would mean a corresponding change to the room tone and background noise, which would then fluctuate unnaturally as you went back and forth between the participants in a scene. In scenes with a lot o…

Thanks a lot for that insight. I will assume you work primarily for Hollywood with that description. It is interesting to hear about the behind-the-scenes from film industries in different parts of the world. In the Indian film industry (maybe not so much Bollywood, but more of the local ones like Tamil, Malayalam, etc) there's a LOT more post being done. Some entire Malayalam films have dubbed voices (for the origin…

Italian films from the 70's used to overdub all the vocals, and of course it wasn't exactly in sync so it looked terrible.

Re: Extracting audio from visual information

#114

Post by chadzawistowski »

arh68 wrote: post #107

Wait, so what would be the best reflector for this type of thing? If I could host the ultimate cocktail party, placing ferns/etc all around the room, could I record every conversation with a single camera? What would those ferns look like? What would they be made of? I'm imagining christmas trees made of cellophane fibers or something.

If you're allowed to place the ferns, why not simply hide microphones in them?

Re: Extracting audio from visual information

#115

Post by JoeAltmaier »

chadzawistowski wrote: post #114
arh68 wrote: post #107

Wait, so what would be the best reflector for this type of thing? If I could host the ultimate cocktail party, placing ferns/etc all around the room, could I record every conversation with a single camera? What would those ferns look like? What would they be made of? I'm imagining christmas trees made of cellophane fibers or something.

If you're allowed to place the ferns, why not simply hide microphones in them?

Because microphones can be detected. When ferns are detected, they are assumed to be innocuous.

Re: Extracting audio from visual information

#116

Post by Sami_Lehtinen »

Joeboy wrote: post #39
pavel_lishin wrote: post #35

Earlier quoted context omitted.

> Because of a quirk in the design of most cameras’ sensors, the researchers were able to infer information about high-frequency vibrations even from video recorded at a standard 60 frames per second.

The audio from the 60fps video sounds pretty bad though, which I suspect is mostly because of inherent maths/physics limitations rather than anything that software can improve. Edit: They mention capturing frequencies up to five times higher than the 60Hz frame rate, which would mean a maximum frequency of 300Hz, which would suggest the equivalent of 0.6kHz audio, which is a 73.5th of the audio rate of a CD. I doubt…

"intelligible speech", are you sure that speech recognition really requires whole frequency range? Often it seems that data can be extracted after all, even if most of it is missing.

Re: Extracting audio from visual information

#117

Post by pessimizer »

If this is something that commodity hardware is now capable of, a total surveillance society is now very cost-effective.

Is the stability required from the camera a dealbreaker when it comes to outdoor mounted cameras in moving air, or would it be pretty easy to algorithmically filter that out? i.e. are winds and drafts predictable enough that they could be removed accurately enough for smaller vibrations to remain?

Re: Extracting audio from visual information

#119

Post by ctdonath »

atomatica wrote: post #87
ctdonath wrote: post #62

Earlier quoted context omitted.

Some cheap video cameras don't shutter the frames (grabbing all pixels at once), they progress thru pixels - leading to a usually-annoying stretchy visual effect. If the sound affects much of the frame with suitable uniformity, you've turned a 60fps camera into a 2MHz "visual" audio sampler (lots of noise, artifacts, & other difficulties aside).

Like in this clip, taken with an iPhone, of guitar strings vibrating? http://youtu.be/TKF6nFzpHBU?t=10s

Exactly. And the term I was looking for is "rolling shutter" (a confident term for an inaction).

Re: Extracting audio from visual information

#120

Post by ctdonath »

DarkIye wrote: post #118

Can somebody executive summary this big lump of wooly garbage?

TL;DR - sound is a physical phenomenon, pushing on objects (more pronounced on wide thin lightweight things like bags). A fast enough video camera with image enhancement can "see" the sound affecting the object, which then can be translated to recreating the sound from the video image.

Not long ago there was a spate of HN articles about apps that could measure your heart rate via the camera (it watches for & measures subtle changes in your skin color which occur during the pulse cycle). This is exactly the same idea, just with a much faster "pulse".

I expect the researchers will next discover the "rolling shutter" (a "that's not a bug, it's a feature!" of cell phone cameras) and discover how to extract the audio info without the need for high-framerate cameras. atomatica found a perfect example: http://youtu.be/TKF6nFzpHBU?t=10s

Post reply on HN