Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster neural networks straight from JPEG (2018)

uber.com

Post reply on HN
Print view View original

31–40 of 108 posts

Re: Faster neural networks straight from JPEG (2018)

#31

Post by sdenton4 »

bob1029 wrote: post #24
kridsdale3 wrote: post #16

Earlier quoted context omitted.

That's kind of the key theory behind why JPEG (and other lossy encodings) work at all. A perfect being would see a JPEG next to a PNG or TIFF and find the first repugnantly error-ridden. But we tend to ignore high-frequency data's specifics most of the time, so it psychologically works. I often wonder though, what do my cat and dog hear when I'm playing compressed music? Does it sounds like a muddy phone call to them…

> Does it sounds like a muddy phone call to them? Likely no. Audio is decidedly less "compressible" in human perceptual terms. The brain is amazingly skilled at detecting time delay and frequency deviations, so this perceptual baseline likely extends (mostly) to your pets. You can fool the eyes a lot more easily. You can take away 50%+ or more of the color information before even a skilled artist will start noticing.

There are real differences in audio perception, though. Frequency range and sensitivity to different frequencies is a big difference in other animals; I would expect cats (who chase rodents, which often have very high pitched or even ultrasonic vocalizations) to be more sensitive to high frequencies than humans, and thus low passed / low sample rate audio could sounds 'bad.'

Another aspect is time resolution. Song birds can have 2-4x the time resolution of human hearing, which helps distinguish sounds in their very fast, complex calls. This may lead to better perception of artifacts in lossy coding schemes, but it's hard to say for sure.

Edit: reference on cat hearing: https://pubmed.ncbi.nlm.nih.gov/4066516

The hearing range of the cat for sounds of 70 dB SPL extends from 48 Hz to 85 kHz, giving it one of the broadest hearing ranges among mammals.

Re: Faster neural networks straight from JPEG (2018)

#33

Post by mochomocha »

I was doing the same thing at Netflix around the same time as a 20% research project. Training GANs end2end directly in JPEG coeffs space (and then rebuild a JPEG from the generated coeffs using libjpeg to get an image). The pitch was that it not only worked, but you could get fast training by representing each JPEG block as a dense + sparse vector (dense for the low DCT coeffs, sparse for the high ones since they're ~all zeros) and using a neural network library with fast ops on sparse data.

Training on pixels is inefficient. Why have your first layers of CNNs relearn what's already smartly encoded in the JPEG bits in the first place before it's blown into a bloated height x width x 3 float matrix?

Re: Faster neural networks straight from JPEG (2018)

#34

Post by corysama »

mochomocha wrote: post #33

I was doing the same thing at Netflix around the same time as a 20% research project. Training GANs end2end directly in JPEG coeffs space (and then rebuild a JPEG from the generated coeffs using libjpeg to get an image). The pitch was that it not only worked, but you could get fast training by representing each JPEG block as a dense + sparse vector (dense for the low DCT coeffs, sparse for the high ones since they're…

As an AI armchair quarterback, I've always held the opinion that the image ML space has a counter-productive bias towards not pre-processing images stemming from a mixture of test purity and academic hubris. "Look what this method can learn from raw data completely independently!" makes for a nice paper. So, they stick with sRGB inputs rather than doing basic classic transforms like converting to YUV420. Everyone learns from the papers, so that's assumed to be the standard practice.

Re: Faster neural networks straight from JPEG (2018)

#35

Post by a-dub »

jdiff wrote: post #23
kridsdale3 wrote: post #16

Earlier quoted context omitted.

That's kind of the key theory behind why JPEG (and other lossy encodings) work at all. A perfect being would see a JPEG next to a PNG or TIFF and find the first repugnantly error-ridden. But we tend to ignore high-frequency data's specifics most of the time, so it psychologically works. I often wonder though, what do my cat and dog hear when I'm playing compressed music? Does it sounds like a muddy phone call to them…

No, audio compression doesn't filter out high frequencies, that's just what computer audio as a whole does. And I don't think there's enough of those high frequency components in what humans typically record for a cat or dog to notice the difference. As far as compression, the tricks that work on us should work on them.

the early xing mp3 codec famously cut everything off above 18khz, but that was out of spec. :)

instead perceptual audio compression typically filters out frequencies that neighbor other frequencies with lots of power. deleting these neighbors is called perceptual masking and to the best of my knowledge, we do not actually know if it works the same way in animal auditory systems.

Re: Faster neural networks straight from JPEG (2018)

#36

Post by johntb86 »

mochomocha wrote: post #33

I was doing the same thing at Netflix around the same time as a 20% research project. Training GANs end2end directly in JPEG coeffs space (and then rebuild a JPEG from the generated coeffs using libjpeg to get an image). The pitch was that it not only worked, but you could get fast training by representing each JPEG block as a dense + sparse vector (dense for the low DCT coeffs, sparse for the high ones since they're…

I would worry that the fixed, non-overlapping block nature of a JPEG would reduce translation invariance - shift an image by 4 pixels and the DCT coefficients may look very different. People have been doing a lot of work to try to reduce the dependence of the image on the actual pixel coordinates - see for example https://research.nvidia.com/publication/2021-12_alias-free-g...

Re: Faster neural networks straight from JPEG (2018)

#37

Post by vajrabum »

ianlevesque wrote: post #27
wpietri wrote: post #20

Earlier quoted context omitted.

That is legitimate, but that's not the point here. The point is Uber's hubris. A hubris very useful in pumping up its stock price ahead of an IPO. If they had quietly planned to license it from the get-go, nobody would have mentioned it.

Uber literally killed people trying to develop their own first. Tesla continues to do so. I wish everyone would license Waymo instead.

Any reason to prefer Waymo over Cruise? I saw an driverless Cruise taxi in SF just the other day. And are they any other competitors still in this space worth watching?

Re: Faster neural networks straight from JPEG (2018)

#39

Post by potatoman22 »

corysama wrote: post #34
mochomocha wrote: post #33

I was doing the same thing at Netflix around the same time as a 20% research project. Training GANs end2end directly in JPEG coeffs space (and then rebuild a JPEG from the generated coeffs using libjpeg to get an image). The pitch was that it not only worked, but you could get fast training by representing each JPEG block as a dense + sparse vector (dense for the low DCT coeffs, sparse for the high ones since they're…

As an AI armchair quarterback, I've always held the opinion that the image ML space has a counter-productive bias towards not pre-processing images stemming from a mixture of test purity and academic hubris. "Look what this method can learn from raw data completely independently!" makes for a nice paper. So, they stick with sRGB inputs rather than doing basic classic transforms like converting to YUV420. Everyone lea…

That's something that surprises me too, given the preprocessing applied to NLP.

Re: Faster neural networks straight from JPEG (2018)

#40

Post by luckystarr »

mochomocha wrote: post #33

I was doing the same thing at Netflix around the same time as a 20% research project. Training GANs end2end directly in JPEG coeffs space (and then rebuild a JPEG from the generated coeffs using libjpeg to get an image). The pitch was that it not only worked, but you could get fast training by representing each JPEG block as a dense + sparse vector (dense for the low DCT coeffs, sparse for the high ones since they're…

I remember I've heard somewhere that our retina encodes the visuals it receives into a compressed signal before forwarding it to the visual cortex. If true, this may actually be how it's done "for real". ;)
Post reply on HN