Live data from Hacker News

ChatGPT is a blurry JPEG of the web

newyorker.com

301–310 of 317 posts

Re: ChatGPT is a blurry JPEG of the web

#301
post #275

Earlier quoted context omitted.

I don't think "lossy text" is a useful term because it conflates with th*s k*nd *f l*ss* t*xt as well. Lossy compression is designed to be as reversible as it can be to a given threshold. That's not how ChatGPT was either designed or works in practice. There are definitely a lot of mathematical similarities between the two, I won't deny that. Would "partial knowledge compression" be a better term? Partial knowledge o…

To me, "lossy compression" vs "Partial knowledge compression" sounds like six vs a half-dozen. Whatever you call it, I think the author was writing more about how we perceive the results generated from a language-compression model vs an image compression model.

The reason why the author chose the term lossy compression was to make it seem like ChatGPT was nothing but a thing that makes things blurry. Do you see a single mention of ChatGPT being a reliable translator in that article or any sort of distinction made between the different kind of tasks that the model is used for?

So it is nothing like six vs a half dozen, because those mean the same thing, and lossy compression is a bad description of half of what ChatGPT does, which makes it a bad description and not at all equal to another more thoughtful, less emotional, description.

Re: ChatGPT is a blurry JPEG of the web

#302
post #171

Earlier quoted context omitted.

It's not quite that straight-forward, though, in that there are two competing goals: Small size and looking as similar as possible to the original. We're explicitly willing trade accuracy for size. How much depends on the use, but sometimes we're willing to trade away so much quality that the artefacts are plainly visible. And we're willing to trade more accuracy for size when the artefacts doesn't distract . For som…

I don't think you disagree. The point is that JPEG has the constraint: make an image as similar as possible to the source image which not going over x kilobytes. LLMs have no similar constraint, so calling them "compression" is a false analogy; they're not trying to compress information, they're using their dataset to learn general facts about e.g. syntax and culture.

I was really mainly responding to the point of JPEG aiming for indistinguishable. Point being that for a lot of purposes we're fine with, and might even be happier with, very different tradeoffs than those JPEG makes.

Going specifically to AI, we do agree that the lack of constraint means they're not compressors in and of themselves. The training compresses information, but that does not make them compressors. Learning and compressing information is, however, at least in some respects very similar. A key part of the LZW family of compression, for example, is applying heuristics to build a dictionary of bit streams (terms) learned from the input.

AI models can potentially eventually be used at the base of a compression because the models encode a lot of information that can potentially be referenced in space-efficient ways.

E.g. if I have a picture of a sunset, and can find a way of getting Stable Diffusion or similar to generate an image of a sunset that is similar enough from a description smaller than the output image, then I have a compressor and decompressor.

Ignoring the runtime cost and that bringing that down to levels where it'd actually produce a benefit, depending on how close the output it, it may be a totally useless algorithm leading to images that are way too far from the input, or it might turn out pretty good. But the tradeoffs would also be very different from JPEG. For some uses I might be happy with a quite different-looking sunset as long as it's "close enough" and high quality even at very high compression ratios. E.g. "A sunset over the horizon. Photo taken from a beach. A fishing boat in the water" fed to [1] produced a pretty nice sunset. Couple that with a seed to make it deterministic, and I might be happy with that as a compression of an image of a quite different sunset. For other uses I'd much prefer JPEG artefacts and something that is clearly the same sunset. For "real" use of it for compression you'd want someone to research ways of guiding it to produce something much closer to the input (maybe heavily downscaling the original image and using that as the starting point coupled with a description; maybe a set of steps including instructions for infilling etc). I think finding the limits of what you can achieve with trying to use these models to reproduce a specific input with the most minimal possible input would make for fascinating research.

[1] https://huggingface.co/stabilityai/stable-diffusion-2?text=A...

Re: ChatGPT is a blurry JPEG of the web

#304

Earlier quoted context omitted.

What he misses in this analogy is that part of what produces the "blur" is the superimposing of many relevan paragraphs found on the web into one. This mechanism can be very useful, because it could average out errors and give one a less one-sided perspective on a particular issue. It doesn't always work like this, but hopefully it will more and more. Also, even more useful is to do a cluster analysis of the existant…

If anything, the average has far more errors in it. It's a trope on Reddit that experts get downvoted while amateurs who reflect the consensus of other amateurs get upvoted and repeated. Amateurs tend to outnumber experts in real life anyways, having their opinions become more authoritative (because some "AI" repeats it) is probably not a great direction to head in.

But this issue is present equally with Google search, no? What I'm saying is that by smartly aggregating all the different opinions on an issue, a LLM could provide better visibility into our collective mind than we currently have access to by simply scanning the first few results of a search engine query. Let's not forget that we're looking at version 0.0001 on this new technology, so there should be lots of room for growth.

Re: ChatGPT is a blurry JPEG of the web

#305
post #224

Earlier quoted context omitted.

> Yes, hooking one up to our long-standing record-of-everything-in-the-world (google’s knowledge graph) is going to be powerful. This hasn't happened yet and, while I may just lack imagination, despite having a fairly solid understanding of how the latest round of AI works I can't see how it can be done successfully. Until it is in fact done and my lack of imagination is demonstrated, your "going to be powerful" is a…

In my view it’s very simple, which is what makes it so exciting. Summarizing a design doc that I imagine Microsoft and google are both spending millions of dollars of manhours/day working on their versions of: 1. User enters query. 2. LLM augments query if necessary, adding extra terms or clauses. 3. Normal search pipeline returns ranked links, just like it does now. 4. LLM reads the content of the first 100 links an…

I mean, how hard can it be, right?

Re: ChatGPT is a blurry JPEG of the web

#306
post #34

Earlier quoted context omitted.

Except now I can train an AI to do it 24/7...

Except if you want the job done right, you really have to do it yourself...

You're suggesting I stay up late at night fixing wrong comments made by my AI chatbot?

...

Yeah, seems like something I'd do :P

Re: ChatGPT is a blurry JPEG of the web

#307
post #46

Earlier quoted context omitted.

Fabrice Bellard has a project that does precisly this. And does it extremely well, apparently. Previously on HN: https://news.ycombinator.com/item?id=27244004 Apparently it leads the compression of enwik9 ( http://www.mattmahoney.net/dc/text.html ) . Not sure why it isn't eligible for the Hutter Prize, there's some speculations in the previous discussion but I don't know whether they're true.

Thank you! Turns out that GPT does in fact perform lossless compression if you want it to, like in this demo.

The main issue is that most ML frameworks aren't reliably reproducible, and are not designed for such use cases.

Bellard's solution was to code up his own neural network library in C.

Re: ChatGPT is a blurry JPEG of the web

#308
post #105

Earlier quoted context omitted.

I thought the author was uncharacteristically perceptive for a reporter. Yann LeCun or Geoff Hinton couldn't have come up with a better analogy.

The author is not a random reporter but Ted Chiang, a well-known science fiction author. The movie "Arrival" is based on a story by him.

Which explains why this is being promoted:

He paid for an advertisement, wrote this article as that advertisement or had it ghostwritten, and now it's being hyped.

Re: ChatGPT is a blurry JPEG of the web

#309
post #275

Earlier quoted context omitted.

To me, "lossy compression" vs "Partial knowledge compression" sounds like six vs a half-dozen. Whatever you call it, I think the author was writing more about how we perceive the results generated from a language-compression model vs an image compression model.

The reason why the author chose the term lossy compression was to make it seem like ChatGPT was nothing but a thing that makes things blurry. Do you see a single mention of ChatGPT being a reliable translator in that article or any sort of distinction made between the different kind of tasks that the model is used for? So it is nothing like six vs a half dozen, because those mean the same thing, and lossy compression…

I think we're talking about different things, but that said, call it whatever you like - lossy, partial, sub-sampling ...

Re: ChatGPT is a blurry JPEG of the web

#310

Earlier quoted context omitted.

I find it funny people are still using these voice "assistants" despite the frequent frustration. Alexa gets it wrong more than half the time when people I know try to use it, but they seem to still want to believe.

All of these “AI/ML” driven products seem to have a whizz-bang initial release where they seem 80% ready and then never come close to the remaining 5/10/15/20% Hell I used dragon naturally speaking in the late 90s and the stuff now doesn’t even feel 10x better despite billions invested and 10000x the compute. Self driving cars feels similar. Always five year away from mass market. We can tune to be pretty good most o…

Dragon Naturally Speaking was, ironically, more flexible than today's voice tech besides the fact that it wasn't internet-connected (I think?). It's not like you can attempt to write an essay with Alexa or control a browser window with it. What's also funny is how we have this narrative that cloud computing is a necessity for AI, and yet Dragon had NLP that fit on a CD-ROM. Ok, maybe it came on multiple discs... I'm forgetting, but my point still stands.

Most of our advances have been in marketing rather than substance.

The current generation of AI/ML may change that in some way. Dragon Naturally Speaking may have been a thing in the 90s, but I'm pretty sure we didn't have anything close to GPT or Stable Diffusion.

Post reply on HN