Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Faster neural networks straight from JPEG (2018)

uber.com

Post reply on HN
Print view View original

61–70 of 108 posts

Re: Faster neural networks straight from JPEG (2018)

#61

Post by whimsicalism »

corysama wrote: post #34
mochomocha wrote: post #33

I was doing the same thing at Netflix around the same time as a 20% research project. Training GANs end2end directly in JPEG coeffs space (and then rebuild a JPEG from the generated coeffs using libjpeg to get an image). The pitch was that it not only worked, but you could get fast training by representing each JPEG block as a dense + sparse vector (dense for the low DCT coeffs, sparse for the high ones since they're…

As an AI armchair quarterback, I've always held the opinion that the image ML space has a counter-productive bias towards not pre-processing images stemming from a mixture of test purity and academic hubris. "Look what this method can learn from raw data completely independently!" makes for a nice paper. So, they stick with sRGB inputs rather than doing basic classic transforms like converting to YUV420. Everyone lea…

In deep ML, people are pretty familiar with the bitter lesson and don't want to waste time on this.

Re: Faster neural networks straight from JPEG (2018)

#62

Post by thomasahle »

potatoman22 wrote: post #39
corysama wrote: post #34

Earlier quoted context omitted.

As an AI armchair quarterback, I've always held the opinion that the image ML space has a counter-productive bias towards not pre-processing images stemming from a mixture of test purity and academic hubris. "Look what this method can learn from raw data completely independently!" makes for a nice paper. So, they stick with sRGB inputs rather than doing basic classic transforms like converting to YUV420. Everyone lea…

That's something that surprises me too, given the preprocessing applied to NLP.

We used to do a lot more preprocessing to NLP, like stemming, removing stop words, or even adding grammar information (NP, VP, etc.). Now we just do basic tokenization. The rest turned out to be irrelevant or even counter productive.

Re: Faster neural networks straight from JPEG (2018)

#63

Post by Llamamoe »

whimsicalism wrote: post #61
corysama wrote: post #34

Earlier quoted context omitted.

As an AI armchair quarterback, I've always held the opinion that the image ML space has a counter-productive bias towards not pre-processing images stemming from a mixture of test purity and academic hubris. "Look what this method can learn from raw data completely independently!" makes for a nice paper. So, they stick with sRGB inputs rather than doing basic classic transforms like converting to YUV420. Everyone lea…

In deep ML, people are pretty familiar with the bitter lesson and don't want to waste time on this.

The bitter lesson is about not trying to encode impossible-to-formalize conceptual knowledge, not avoiding data efficiency and the need to scale the model up to ever higher parameter counts.

If we followed this logic, we'd be training LLMs on character-level UTF-32 and just letting it figure everything out by itself, while needing two orders of magnitude bigger contexts and parameter counts.

Re: Faster neural networks straight from JPEG (2018)

#64

Post by kbelder »

thomasahle wrote: post #62
potatoman22 wrote: post #39

Earlier quoted context omitted.

That's something that surprises me too, given the preprocessing applied to NLP.

We used to do a lot more preprocessing to NLP, like stemming, removing stop words, or even adding grammar information (NP, VP, etc.). Now we just do basic tokenization. The rest turned out to be irrelevant or even counter productive.

But also, that basic tokenization is essential; training it on a raw ascii stream would be much less efficient. There is a sweet spot of processing & abstraction that should be aimed for.

Re: Faster neural networks straight from JPEG (2018)

#65

Post by whimsicalism »

Llamamoe wrote: post #63
whimsicalism wrote: post #61

Earlier quoted context omitted.

In deep ML, people are pretty familiar with the bitter lesson and don't want to waste time on this.

The bitter lesson is about not trying to encode impossible-to-formalize conceptual knowledge, not avoiding data efficiency and the need to scale the model up to ever higher parameter counts. If we followed this logic, we'd be training LLMs on character-level UTF-32 and just letting it figure everything out by itself, while needing two orders of magnitude bigger contexts and parameter counts.

Converting from RGB to YUV is absolutely subject to the bitter lesson because it is trying to generalize from a representation that we have seen works for some classical methods and hard code that knowledge in to the AI which could easily learn (and will anyways) a more useful representation for itself.

> LLMs on character-level UTF-32 and just letting it figure everything out by itself, while needing two orders of magnitude bigger contexts and parameter counts.

This was tried extensively and honestly it is probably still too early to proclaim the demise of this approach. It's also completely different - you're conflating a representation that literally changes the number of forward passes you have to do (ie. the amount of computation - what the bitter lesson is about) vs. one that (at most) would just require stacking on a few layers or so.

A better example for your point (imo) would be audio recognition, where we pre-transform from wave amplitudes into log mel spectrogram for ingestion by the model. I think this will ultimately fall to the bitter lesson as well though.

Also a key difference is that you are proposing going from methods that already work to try to inject more classical knowledge into them. It is oftentimes the case that you'll have an intermediary fusion between deep + classical, but not if you already have working fully deep methods.

Re: Faster neural networks straight from JPEG (2018)

#67

Post by andai »

mhh__ wrote: post #59
andai wrote: post #53

Earlier quoted context omitted.

I was trying to learn how the discrete cosine transform works, so I looked up some code in an open source program. The code said it copied it verbatim from a book from the 90s. I looked up the book and the book said it copied it verbatim from a paper from the 1970s.

The code is irrelevant, in the eyes of the theoretician anyway.

I am theoretically challenged.

Re: Faster neural networks straight from JPEG (2018)

#68

Post by refulgentis »

Llamamoe wrote: post #63
whimsicalism wrote: post #61

Earlier quoted context omitted.

In deep ML, people are pretty familiar with the bitter lesson and don't want to waste time on this.

The bitter lesson is about not trying to encode impossible-to-formalize conceptual knowledge, not avoiding data efficiency and the need to scale the model up to ever higher parameter counts. If we followed this logic, we'd be training LLMs on character-level UTF-32 and just letting it figure everything out by itself, while needing two orders of magnitude bigger contexts and parameter counts.

This will absolutely be the case N doublings of Moore's law from here. Tokens are information loss.

Re: Faster neural networks straight from JPEG (2018)

#69

Post by actionfromafar »

luckystarr wrote: post #40
mochomocha wrote: post #33

I was doing the same thing at Netflix around the same time as a 20% research project. Training GANs end2end directly in JPEG coeffs space (and then rebuild a JPEG from the generated coeffs using libjpeg to get an image). The pitch was that it not only worked, but you could get fast training by representing each JPEG block as a dense + sparse vector (dense for the low DCT coeffs, sparse for the high ones since they're…

I remember I've heard somewhere that our retina encodes the visuals it receives into a compressed signal before forwarding it to the visual cortex. If true, this may actually be how it's done "for real". ;)

The retina does a lot of processing.

Re: Faster neural networks straight from JPEG (2018)

#70

Post by xenadu02 »

sdenton4 wrote: post #31
bob1029 wrote: post #24

Earlier quoted context omitted.

> Does it sounds like a muddy phone call to them? Likely no. Audio is decidedly less "compressible" in human perceptual terms. The brain is amazingly skilled at detecting time delay and frequency deviations, so this perceptual baseline likely extends (mostly) to your pets. You can fool the eyes a lot more easily. You can take away 50%+ or more of the color information before even a skilled artist will start noticing.

There are real differences in audio perception, though. Frequency range and sensitivity to different frequencies is a big difference in other animals; I would expect cats (who chase rodents, which often have very high pitched or even ultrasonic vocalizations) to be more sensitive to high frequencies than humans, and thus low passed / low sample rate audio could sounds 'bad.' Another aspect is time resolution. Song bi…

True but hearing is logarithmic in both volume and frequency domains. Double the power does not equate to anything near double the loudness. Similarly each doubling of frequency is only one octave higher. Hearing up to 80khz doesn't mean hearing 4x more than humans... 10 octaves for humans, 12 octaves for cats. In a musical sense it probably isn't noticeable.
Post reply on HN