Live data from Hacker News

Extracting audio from visual information

newsoffice.mit.edu

61–70 of 133 posts

Re: Extracting audio from visual information

#61
post #60

Wouldn't it be really neat to apply this to HD movie sequences, and hear what the sounds on the set and the voices of actors were like pre-production? And how unreal some of the sounds must have turned out with all the visual tweaking that happens in production?

You missed the part where the video's frames per second has to be higher than the audio frequency.

You missed the part where they proved that this is not required.

Re: Extracting audio from visual information

#62
post #60

Wouldn't it be really neat to apply this to HD movie sequences, and hear what the sounds on the set and the voices of actors were like pre-production? And how unreal some of the sounds must have turned out with all the visual tweaking that happens in production?

You missed the part where the video's frames per second has to be higher than the audio frequency.

Some cheap video cameras don't shutter the frames (grabbing all pixels at once), they progress thru pixels - leading to a usually-annoying stretchy visual effect. If the sound affects much of the frame with suitable uniformity, you've turned a 60fps camera into a 2MHz "visual" audio sampler (lots of noise, artifacts, & other difficulties aside).

Re: Extracting audio from visual information

#63
post #12

It doesn't seem to mention what kind of camera you'd need to do this at the mentioned distance (15 feet). I'm assuming it's at least a couple thousand bucks, or maybe even so expensive that most professional photographers won't have it, but does anyone actually know? Or did I miss it in the article?

I think this is the cheapest (relatively speaking) high-speed camera: http://edgertronic.com/

There are also some consumer cameras that can capture almost 1,000 fps from a small sensor window. I wonder if those would work.

Re: Extracting audio from visual information

#65

This is amazing. Imagine taking high definition video of a crowd of people. You could pick out objects nearby and hear what individuals are saying. You could nearly produce a 3D auditorium by sampling different points in a video. Couple this with a 3D camera and an Oculus Rift, you could have something incredible.

While I'm not sure how closely this matches up, it sounds like you may at least be interested in exploring these concepts:

http://en.wikipedia.org/wiki/Cocktail_party_effect

http://en.wikipedia.org/wiki/Independent_component_analysis

Edit: Spacing.

Re: Extracting audio from visual information

#66
post #12

It doesn't seem to mention what kind of camera you'd need to do this at the mentioned distance (15 feet). I'm assuming it's at least a couple thousand bucks, or maybe even so expensive that most professional photographers won't have it, but does anyone actually know? Or did I miss it in the article?

The paper says they used a Phantom V10. I'm not knowledgeable about such devices, but various Google results suggest such a camera might top $100,000 depending on the mounts, lenses, and accessories (e.g. data storage and transfer equipment).

Re: Extracting audio from visual information

#69
post #17

This is from the movie Eagle Eye, right? The evil computer watches the vibrations in a cup of coffee while someone is speaking.

lol that is the first thing that I thought as well. I just passed it off as a typical Hollywood trope, but now I'm quite delighted to see it become a reality.

Re: Extracting audio from visual information

#70
post #60

Earlier quoted context omitted.

You missed the part where the video's frames per second has to be higher than the audio frequency.

Some cheap video cameras don't shutter the frames (grabbing all pixels at once), they progress thru pixels - leading to a usually-annoying stretchy visual effect. If the sound affects much of the frame with suitable uniformity, you've turned a 60fps camera into a 2MHz "visual" audio sampler (lots of noise, artifacts, & other difficulties aside).

[deleted]
Post reply on HN