Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Algorithm recovers speech from a potato-chip bag filmed through glass (2014)

news.mit.edu

Post reply on HN
Print view View original

51–60 of 111 posts

Re: Algorithm recovers speech from a potato-chip bag filmed through glass (2014)

#51

Post by im3w1l »

Sharlin wrote: post #38
zeta0134 wrote: post #29

Earlier quoted context omitted.

It's known as the Rolling Shutter effect. It's most commonly observed when attempting to take still imagery of very fast moving objects like rotors on airplanes, but affects pretty much any image sensor that isn't designed specifically to work around it. The following Wikipedia article goes into far more depth, but the basic problem is that an image sensor doesn't sample every pixel instantly, but instead reads them…

And it is why mechanical shutters are still a thing in dedicated cameras. It is still much faster to move two pieces of metal or carbon fiber over the sensor than it is to read out a multi-megapixel CMOS sensor. When shooting video mechanical shutters cannot be used so the rolling shutter effect can cause artifacts such as fast-moving subjects appear slanted.

Borrowing from rendering, it seems like a solution would be for every pixel to have "double buffers", one for the measurement in progress and one for the past measurement. Then you just need to make sure you can read all the past values in less than a frame and that all the buffer swaps happen in sync.

Re: Algorithm recovers speech from a potato-chip bag filmed through glass (2014)

#52

Post by im3w1l »

mirimir wrote: post #18
toomuchtodo wrote: post #13

Earlier quoted context omitted.

Don’t disagree. Scale that to an entire populace though.

Well, windowless bathrooms are not uncommon. And most modern construction has walk-in closets, which typically lack windows. Even so, a source of white noise is always prudent.

And leads to mold and bad air unless you are careful which people aren't

Re: Algorithm recovers speech from a potato-chip bag filmed through glass (2014)

#53

Post by spyder »

an4rchy wrote: post #12

Definitely some interesting applications. Sort of orthogonal but is there a similar way to get sound from seeing someone's lip's move in a video -- would it then be possible to recover audio/conversations from old video only files? Edit: Guess they can read lips via AI https://www.techemergence.com/machine-learning-that-learns-m... Now getting the high quality sound from just video would be amazing.

What about lip-reading with reflected Wi-Fi signals :) :

https://people.eecs.berkeley.edu/~guanhua/paper/WiHear_MobiC...

"WiHear aims to detect human speech by analyzing radio reflections from mouth movements. It requires individual user to train the system extensively, and can recognize only a limited number of words (6 words) with high accuracy."

Re: Algorithm recovers speech from a potato-chip bag filmed through glass (2014)

#54

Post by sbierwagen »

jdietrich wrote: post #40
toomuchtodo wrote: post #5

Earlier quoted context omitted.

With enough processing power, you can listen to anyone as long as you can get an object in frame that’s vibrating with speech (the UK and Chinese CCTV networks comes to mind). Depending on resolution, I’d expect this to be the case for any footage already stored. It’s just a matter of having a distributed computing job kicked off to comb through video data and add the additional audio metadata. Will you only speak of…

>the UK and Chinese CCTV networks comes to mind Minor nitpick, but the UK doesn't have a CCTV network. It has a huge number of privately owned CCTV cameras and a relatively small number of CCTV cameras operated by individual local authorities and police forces. The privately-owned cameras aren't joined up in any useful way and are often of very poor quality; the publicly owned cameras are overwhelmingly used for real…

Run the algo on any reflective objects in the field of view, feed the outputted audio to a transcription NN. Save all high confidence transcribed voice as compressed txt. Save the source video if you pass a threshold of danger keywords. ("bomb", "kill", "prime minister", "didn't pay my tv licence")

Re: Algorithm recovers speech from a potato-chip bag filmed through glass (2014)

#55

Post by JoshuaDavid »

fooker wrote: post #8
gh0zt wrote: post #4

“I’m sure there will be applications that nobody will expect. I think the hallmark of good science is when you do something just because it’s cool and then somebody turns around and uses it for something you never imagined. It’s really nice to have this type of creative stuff.” Thinking atomic bombs :(

>Thinking atomic bombs :( The same ones which have prevented large scale open military conflicts involving superpowers for the last 60+ years? Atom bombs have likely saved more lives than they have taken, if we had conventional wars with modern technology without MAD.

T5

Re: Algorithm recovers speech from a potato-chip bag filmed through glass (2014)

#56

Post by mirimir »

im3w1l wrote: post #52
mirimir wrote: post #18

Earlier quoted context omitted.

Well, windowless bathrooms are not uncommon. And most modern construction has walk-in closets, which typically lack windows. Even so, a source of white noise is always prudent.

And leads to mold and bad air unless you are careful which people aren't

True, but only if you don't ventilate properly. And exhaust fans are also a useful source of white noise. Along with running water.

Re: Algorithm recovers speech from a potato-chip bag filmed through glass (2014)

#57

Post by eigenloss »

zeta0134 wrote: post #29
booleandilemma wrote: post #24

Because of a quirk in the design of most cameras’ sensors, the researchers were able to infer information about high-frequency vibrations even from video recorded at a standard 60 frames per second. That’s a pretty convenient quirk if you ask me.

It's known as the Rolling Shutter effect. It's most commonly observed when attempting to take still imagery of very fast moving objects like rotors on airplanes, but affects pretty much any image sensor that isn't designed specifically to work around it. The following Wikipedia article goes into far more depth, but the basic problem is that an image sensor doesn't sample every pixel instantly, but instead reads them…

A relevant point here is that video compression almost always destroys the signal relied upon in this research.

Re: Algorithm recovers speech from a potato-chip bag filmed through glass (2014)

#58

Post by zeta0134 »

im3w1l wrote: post #51
Sharlin wrote: post #38

Earlier quoted context omitted.

And it is why mechanical shutters are still a thing in dedicated cameras. It is still much faster to move two pieces of metal or carbon fiber over the sensor than it is to read out a multi-megapixel CMOS sensor. When shooting video mechanical shutters cannot be used so the rolling shutter effect can cause artifacts such as fast-moving subjects appear slanted.

Borrowing from rendering, it seems like a solution would be for every pixel to have "double buffers", one for the measurement in progress and one for the past measurement. Then you just need to make sure you can read all the past values in less than a frame and that all the buffer swaps happen in sync.

I believe this is the premise behind image sensors which support global shutter (as opposed to rolling shutter) but other than being aware that this feature might exist, I'm no expert on the subject.

Re: Algorithm recovers speech from a potato-chip bag filmed through glass (2014)

#59

Post by femto »

If interested in this, you might also be interested in the lead author's other research papers

http://abedavis.com/research.html

and his 2016 PhD thesis, which includes the "chip bag" research.

http://abedavis.com/thesis.pdf

I'm interested in this research, from the angle of "Can passive surface measurements tell us what is happening inside the human body?"

Re: Algorithm recovers speech from a potato-chip bag filmed through glass (2014)

#60

Post by fipple »

jwilk wrote: post #45
cma wrote: post #15

At some point it is (controversially) hypothesized that we may be able to pull imprinted recordings off of ancient artifacts: https://en.wikipedia.org/wiki/Archaeoacoustics#Past_interpre...

> [Jones] claimed to have extracted the hum of the potter's wheel from the grooves of a pot, So far so good... > and the word "blue" from an analysis of patch of blue color in a painting. What the hell?

I don’t know about the techniques but the theory isn’t prima facie impossible. A paintbrush can act like a microphone just like anything else, and if the paintbrush (more likely a putty knife or more rigid object) picked up a sound while applying paint, that could manifest in the paint layer.
Post reply on HN