William T. Freeman is an outstanding vision researcher. His list of publications (http://billf.mit.edu/publications/all) is full of these simple, clever solutions for problems slightly outside the mainstream. I really admire his work.
The Visual Microphone: Passive Recovery of Sound from Video
31–40 of 41 posts
Re: The Visual Microphone: Passive Recovery of Sound from Video
#32Slowly, the creative imaginations coming to life. Eagle Eye....
Re: The Visual Microphone: Passive Recovery of Sound from Video
#33Earlier quoted context omitted.
"It is a rare mind indeed that can render the hitherto non-existent blindingly obvious. The cry 'I could have thought of that' is a very popular and misleading one, for the fact is that they didn't, and a very significant and revealing fact it is too."
It is a very good idea, but there's another problem at hand: It's not novel. I remember hearing about this many years ago, though that was using vibrations in windows to get the sound inside a building. IMO there's nothing to patent here. Unless this is the same group it's just appropriating someone else's work. Besides, random people do think of cool stuff all the time. They just don't normally patent it or start a…
The key difference is that a laser microphone is amplifying the actual vibrations caused by the sounds as they're being produced. This is using silent video and knowledge about the acoustic properties of the depicted objects & environment to simulate/recreate the sound that was not captured. You can't point a laser microphone at the past.
Re: The Visual Microphone: Passive Recovery of Sound from Video
#34Earlier quoted context omitted.
Yes but you have to precisely stagger the start of expositions of each camera and it's hard to do on consumer hardware.
Um, why would you have to precisely stagger? I suspect that you have enough information to actually align the videos after the fact. 10 videos at 250FPS would probably distribute sufficiently.
Would it not be hard to interleave the first frame of these videos given different starting times and angles (ignoring camera movement)? It should be easy if the videos have synchronized timestamps, but that might not always be the case.
Re: The Visual Microphone: Passive Recovery of Sound from Video
#35Earlier quoted context omitted.
"It is a rare mind indeed that can render the hitherto non-existent blindingly obvious. The cry 'I could have thought of that' is a very popular and misleading one, for the fact is that they didn't, and a very significant and revealing fact it is too."
It is a very good idea, but there's another problem at hand: It's not novel. I remember hearing about this many years ago, though that was using vibrations in windows to get the sound inside a building. IMO there's nothing to patent here. Unless this is the same group it's just appropriating someone else's work. Besides, random people do think of cool stuff all the time. They just don't normally patent it or start a…
>vibrations in windows to get the sound inside a building
That is a totally unrelated problem. Recovering sound from a series of still photos compared to recovering sound from vibrations is a completely different issue altogether. Actually executing and doing it is even more different, as you run into all the implementation issues and physical bugaboos. When someone doesn't just have some vague notion, but actually implements a wholly new technique to do something previously impossible, whether it was imagined by others or not, is a patent really that absurd?
Re: The Visual Microphone: Passive Recovery of Sound from Video
#36Well that's pretty scary. I can see one application of this already: casinos in the US are legally prohibited from recording audio on their floors, but have perfectly positioned cameras everywhere. Beyond that, I'm guessing every spy agency on earth will be buying solutions based on this. It would be interesting to know what the genesis of this project was - for example if the NSA or CIA was involved in suggesting to…
This same researcher has been working in this area for awhile, and this is not an unexpected extension of things he's developed previously - like how to take standard video and determine the pulse of any people present (search on 'non-eulerian video magnification'). He's been working in what you can recover from the variations present in video which others have written off as 'noise' for awhile. I wouldn't be surprised if he just stumbled upon the effect sound had while doing other analysis.
I haven't looked, but if the NSA was involved with this, its usually easy to find out. Just look for any grants involved and look up the source of the grant. They don't usually hide their involvement in funding research. Pretty much any study done in the past 10 years about manipulating social graphs was funded by the NSA.
Re: The Visual Microphone: Passive Recovery of Sound from Video
#37Earlier quoted context omitted.
A standard mobile phone today can do 240fps. We're only a few years away from 1000+ for in everyday devices.
Really? Isn't speed limited by the amount of light that can be received in a small lens like on a phone?
The trick is to add the signal of many pixels together, forming a larger effective sample. The output video is very low-resolution and has a low dynamic range after noise removal, but can get hundreds of FPS in decent lighting.
Re: The Visual Microphone: Passive Recovery of Sound from Video
#38Earlier quoted context omitted.
Um, why would you have to precisely stagger? I suspect that you have enough information to actually align the videos after the fact. 10 videos at 250FPS would probably distribute sufficiently.
Would it not be hard to interleave the first frame of these videos given different starting times and angles (ignoring camera movement)? It should be easy if the videos have synchronized timestamps, but that might not always be the case.
Any in-frame motion probably allows you to align to frame after the fact. This is existing technology, and gives you timestamp to frame alignment.
If you are reconstructing sound, you can now fuzz the time alignments to give the maximum signal for the maximum time (non-correlation will damp to random noise quickly). This allows you to pairwise reconstruct time alignments.
At that point, you put them all together and run your detailed analysis.
Now, I didn't say this way EASY. :) Or cheap. Or real-time.
Just that it is possible.
Re: The Visual Microphone: Passive Recovery of Sound from Video
#39Patent pending? The approach should be trivially obvious for anyone familiar with the basics. What happened to the MIT License and liberating knowledge, does MIT not do that anymore?
"It is a rare mind indeed that can render the hitherto non-existent blindingly obvious. The cry 'I could have thought of that' is a very popular and misleading one, for the fact is that they didn't, and a very significant and revealing fact it is too."
I can only work on one thing at a time. Just because I didn't implement this, it only really means that I prioritized implementing something else over this.
Do you believe that implementation has no time-cost?
Also, I don't think your appropriated quote addresses the implications of the MIT License, its history and associated institution, or the liberation of knowledge.
Re: The Visual Microphone: Passive Recovery of Sound from Video
#40Earlier quoted context omitted.
"It is a rare mind indeed that can render the hitherto non-existent blindingly obvious. The cry 'I could have thought of that' is a very popular and misleading one, for the fact is that they didn't, and a very significant and revealing fact it is too."
It is a very good idea, but there's another problem at hand: It's not novel. I remember hearing about this many years ago, though that was using vibrations in windows to get the sound inside a building. IMO there's nothing to patent here. Unless this is the same group it's just appropriating someone else's work. Besides, random people do think of cool stuff all the time. They just don't normally patent it or start a…
I remember the same thing being demonstrated on a mass-market network in the late 90s / early 00's. It was probably the Discovery channel, or similar, and likely a 'spy gadget' type show.
I'd guess someone in the military or intelligence community implemented initial prototypes not long after LASERs became available.