Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Extracting audio from visual information

newsoffice.mit.edu

Post reply on HN
Print view View original

101–110 of 133 posts

Re: Extracting audio from visual information

#101

Post by anigbrowl »

guelo wrote: post #71
iLoch wrote: post #61

Earlier quoted context omitted.

You missed the part where they proved that this is not required.

When Hollywood isn't still using 24fps film cameras they're certainly not using consumer-grade camera sensors.

I know a couple of Hollywood movies that were shot on consumer gear, eg http://www.usa.canon.com/app/html/HDV/XHA1/case_study_06.sht...

For this particular film the rather characteristic look of digital video is appropriate, whereas most productions would rather shoot with much fancier gear. But you'll find plenty DSLRs working on film sets these days, and you will see more and more convergence as high-quality sensors become a commodity. Black Magic have a pocket camera that delivers 13 stops of dynamic range.

Re: Extracting audio from visual information

#102

Post by anigbrowl »

petercooper wrote: post #8

And now think of how much high definition video, CCTV, and other forms of recordings already exist.. and then think about running large collections of pre-existing video through such algorithms, along with the best speech-to-text in the business. You could have a whole new Wikileaks on your hands :-)

You'd be better off learning to lip read.

Re: Extracting audio from visual information

#103

Post by anigbrowl »

Houshalter wrote: post #97

How much vibration does sound actually cause an object. I tried humming near various objects and noticed nothing. I used to be really interested in early mechanical microphones and sound recording, but I couldn't find much information on how they work either.

Gently put your fingers on a (cheap) speaker cone and then play back some music.

Re: Extracting audio from visual information

#105

Post by anigbrowl »

MichailP wrote: post #77
abedavis wrote: post #40

Earlier quoted context omitted.

The camera used in most of these experiments (a Phantom high speed camera) doesn't even have a microphone - so that would be quite impossible.

Thanks. However, there is still a possibility that mechanical vibration is coming to the video sensor through some path (floor+camera tripod), and affecting the video. After all mechanical vibration is affecting the potato chips. Why is it so impossible to affect video sensor? Especially in the sensitive high speed camera? Let the downvotes begin :)

Well, the chip bag weighs a few grams at most. The camera and gear weigh something like 50kg. The higher the video frame rate, the greater the damping you need to prevent unwanted vibration. Notice the tests were performed with a speaker playing back inside the room rather than a person, in order to minimize mechanical vibrations. The sensors are very firmly mounted inside the camera, trust me.

Re: Extracting audio from visual information

#106

Post by Chronic29 »

anigbrowl wrote: post #100
keerthiko wrote: post #41

Wouldn't it be really neat to apply this to HD movie sequences, and hear what the sounds on the set and the voices of actors were like pre-production? And how unreal some of the sounds must have turned out with all the visual tweaking that happens in production?

I record the dialog for your movies. Aprt from obvious sound effects like Darth Vader voices or so, actors' dialog is changed as little as possible during post production. It's not heavily EQed or comrpessed, because that would mean a corresponding change to the room tone and background noise, which would then fluctuate unnaturally as you went back and forth between the participants in a scene. In scenes with a lot o…

Cool story

Re: Extracting audio from visual information

#107

Post by arh68 »

Wait, so what would be the best reflector for this type of thing? If I could host the ultimate cocktail party, placing ferns/etc all around the room, could I record every conversation with a single camera?

What would those ferns look like? What would they be made of? I'm imagining christmas trees made of cellophane fibers or something.

Re: Extracting audio from visual information

#108

Post by ygra »

nl wrote: post #21
randomdrake wrote: post #19

Earlier quoted context omitted.

The video goes into a bit more detail and shows a couple of things that address your question. 1) A lot of the tests were done with a camera that costs thousands of dollars. You can see images of the camera used in one of the experiments. 2) They were able to use consumer-grade cameras to also capture sound. Even frequencies up to 5x higher than the actual 60 FPS of the captured video.

I wonder how well a GoPro would do? They can do 120fps at 720p or 240fps at WVGA.

Not very well, at least for the initial approach with a high-speed camera. With 240 fps you're limited to 240 samples per second. If I'm not terribly mistaken that would limit you to frequencies up to 120 Hz in the reconstruction.

But the GoPro has a rolling shutter as well, so their second approach would be applicable. However, that effectively relies on rows per second and while you have a higher frame rate you have a lower resolution. In the end they could cancel each other out.

Post reply on HN