The Sound of Pixels • Hacker News Board

Re: The Sound of Pixels

#11

Post by xtagon » Tue, Sep 25, 2018, 2:27 AM UTC

I wonder if this can segregate vocals from instrumentals in a mix? That would be great for mashups.

Re: The Sound of Pixels

#12

Post by app4soft » Tue, Sep 25, 2018, 3:37 AM UTC

It remind me such apps like PhonoPaper[0], Nature - Oscillator[1] and PixiVisor[2]

[0] http://warmplace.ru/soft/phonopaper

[1] http://warmplace.ru/soft/nosc

[2] http://warmplace.ru/soft/pixivisor/

Re: The Sound of Pixels

#13

Post by tmalsburg2 » Tue, Sep 25, 2018, 3:52 AM UTC

jovial_cavalier wrote: post #8

funkdified wrote: post #7

This is going to be huge for the hard of hearing.

how

Not op but I imagine the idea must be that they can point on the thing they want to year in a noisy environment and software can amplify the sound generated at that location.

Re: The Sound of Pixels

#14

Post by rflrob » Tue, Sep 25, 2018, 4:00 AM UTC

I haven’t tried my hand at any machine learning, but I’m impressed that it could work with only 60 hours of training data. Perhaps the input clips were fairly short, which would increase the total number of videos.

Re: The Sound of Pixels

#15

Post by ggm » Tue, Sep 25, 2018, 4:56 AM UTC

Does anyone remember Gerry Anderson? he designed relays attached to puppets, which made the jaws clack in time to the sound-track being played, while they filmed the puppets for Thunderbirds are go in the 1960s. Look at me ma! my puppet is speaking!

Thus, it only took us 50-odd years to write the reverse-compiler..

Re: The Sound of Pixels

#16

Post by a-dub » Tue, Sep 25, 2018, 5:10 AM UTC

Interesting. I wonder how well a logistic regression that spits out masks would perform in the source separation task.

Also a bit surprising to see that they had to STFT the audio before feeding it into a convnet. I thought half the point of convnets was that they figure out how to do spectral domain representations on their own...

Re: The Sound of Pixels

#17

Post by bscphil » Tue, Sep 25, 2018, 6:33 AM UTC

xtagon wrote: post #11

I wonder if this can segregate vocals from instrumentals in a mix? That would be great for mashups.

Incidentally, due to the way a lot of stereo tracks are mixed, it's often possible to mostly remove the vocal track from a song. I'm more curious if this algorithm could perform the reverse task - playing the vocals only. My intuition is that the results would be poor because of the wide human vocal range and the fact that words need to be discernible, not just notes. But I would love to be proven wrong here.

Re: The Sound of Pixels

#18

Post by black_puppydog » Tue, Sep 25, 2018, 7:03 AM UTC

a-dub wrote: post #16

Interesting. I wonder how well a logistic regression that spits out masks would perform in the source separation task. Also a bit surprising to see that they had to STFT the audio before feeding it into a convnet. I thought half the point of convnets was that they figure out how to do spectral domain representations on their own...

in theory yes, but in practice, giving the network the full information in the right format is crucial to have it train well and quickly.

Re: The Sound of Pixels

#19

Post by black_puppydog » Tue, Sep 25, 2018, 7:03 AM UTC

vernie wrote: post #4

drhodes wrote: post #3

Earlier quoted context omitted.

matching faces with voices in a surveillance situation, which person is talking?

That's actually this paper: https://ai.googleblog.com/2018/04/looking-to-listen-audio-vi...

that's actually a whole subfield, not "this paper".

Re: The Sound of Pixels

#20

Post by rolfvandekrol » Tue, Sep 25, 2018, 12:59 PM UTC

keyle wrote: post #2

Cool stuff. Any real world usage/benefits for this? I can't think of any.

For the sound splitting, it's probably not extremely useful. You can however use this frequency analysis from video to synchronise audio with a video (for a music video where you want to use a studio audio recording with a video clip)