Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ChatGPT is a blurry JPEG of the web

newyorker.com

Post reply on HN
Print view View original

31–40 of 317 posts

Re: ChatGPT is a blurry JPEG of the web

#31

Post by Ari_Rahikkala »

> Models like ChatGPT aren’t eligible for the Hutter Prize for a variety of reasons, one of which is that they don’t reconstruct the original text precisely—i.e., they don’t perform lossless compression.

Small nit: The lossiness is not a problem at all. Entropy coding turns an imperfect, lossy predictor into a lossless data compressor, and the better the predictor, the better the compression ratio. All Hutter Prize contestants anywhere near the top use it. The connection at a mathematical level is direct and straightforward enough that "bits per byte" is a common number used in benchmarking language models, despite the fact that they are generally not intended to be used for data compression.

The practical reason why a ChatGPT-based system won't be competing for the Hutter Prize is simply that it's a contest about compressing a 1GB file, and GPT-3's weights are both proprietary and take up hundreds of times more space than that.

Re: ChatGPT is a blurry JPEG of the web

#32

Post by secabeen »

This is a decent summary. I've been thinking about how ChatGPT by it's very nature destroys context and source reputation. When I search for something on the Internet, I get a link to the original content, which I can then evaluate based on my knowledge and the reputation of the original source. Wikipedia is the same, with a big emphasis on citation. ChatGPT and other LLMs destroy that context and knowledge, giving me no tools to evaluate the sources they're using.

Re: ChatGPT is a blurry JPEG of the web

#34

Post by hnfong »

visarga wrote: post #22
ElevenLathe wrote: post #18

Earlier quoted context omitted.

You don't need to correct every wrong thing you read. In fact you will probably feel much better if you don't ever do it at all, or at least take a break for while.

obligatory https://xkcd.com/386/

Except now I can train an AI to do it 24/7...

Re: ChatGPT is a blurry JPEG of the web

#35

Post by iaccountthencom »

most of what goes as "understanding" (where 'our culture' is the agent/actor doing the 'understanding') really is compression of information (abstraction is the form of the compressing)

I thought about this possibility years ago, but as I see more of what neural nets are doing, it makes me more certain I'm onto something (which makes no meaningful difference to me, i.e. being onto what these deep neural models are is useless to me)

in any case, yea sure. neural nets are some kind of lossy compression but nobody thinks about them this way.

and my point is that to create abstract theories which explain lots of things (e.g. physics) is also this kind of 'lossy compression'.

over these theories we say "we understand" stuff, this means we are able to recall things about what the theories are describing, it allows us to reconstruct scenarios and predict the outcomes if/when the scenarios match up.

maybe I'm gearing up to say that 'backpropagation' is a creative action?

shrugs

Re: ChatGPT is a blurry JPEG of the web

#36

Post by dougmwne »

bbor wrote: post #14

Ugh I’m beginning to think I’m going to spend the next 6-12 months commenting “no, large language models aren’t supposed to somehow know everything in the world. No, that’s not what they’re designed for. Yes, hooking one up to our long-standing record-of-everything-in-the-world (google’s knowledge graph) is going to be powerful.” It’s getting to point where I need to consider stop going on HN. This is like when my fa…

I appreciate where you are coming from and I agree that AI is about to go from relative obscurity where just a few geeks were playing around to insane hype. I feel like I’ve spent the last 7 years wondering why no one in the wider world was as impressed as I was, but starting with Stable Diffusion and now ChatGPT, the hype rocket ship has launched. Search TikTok for ChatGPT for all the evidence of that you could ever need.

That said, I still think we are in for a wild ride, even if we go through a hype bubble and pop first. I really don’t think the current crop of Transformer LLMs are the end of the story. Im betting that we are headed towards architectures made up of several different kind of models and AI approaches just like the brain is an apparent concert of specialized regions. You can see that in the new Bing where it’s a combination of a LLM with static training set that can then do up to 3 web searches to build up additional context of fresh data for the prompt, overcoming one of the key disadvantages of a transformer model. The hidden prompt with plain English Asimov's laws are the icing on the cake.

The hype will be insane, but the capabilities are growing quickly and we do not yet seem close to the end of this rich computational ore vein we have hit.

Re: ChatGPT is a blurry JPEG of the web

#37

Post by criddell »

danuker wrote: post #8
mrtksn wrote: post #7

They can always use AI based solutions to unblur the JPEG, like this: https://twitter.com/maxhkw/status/1373063086282739715

"Your Honor, we have evidence Ryan Gosling may have breached our systems."

Would make a good (bad) CSI episode. Enhance the security footage then put out an arrest warrant for Ryan Gosling.

Re: ChatGPT is a blurry JPEG of the web

#38

Post by CPLX »

bbor wrote: post #14

Ugh I’m beginning to think I’m going to spend the next 6-12 months commenting “no, large language models aren’t supposed to somehow know everything in the world. No, that’s not what they’re designed for. Yes, hooking one up to our long-standing record-of-everything-in-the-world (google’s knowledge graph) is going to be powerful.” It’s getting to point where I need to consider stop going on HN. This is like when my fa…

> This is like when my father excitedly told his friends about the coming computer revolution in the 90s and they responded “well it can’t do my dishes or clean the house, they’re just a fad!” Makes me want screaaaaam

Did the computer revolution make your father’s life better?

Serious question.

Re: ChatGPT is a blurry JPEG of the web

#39

Post by didgetmaster »

>For us to have confidence in them, we would need to know that they haven’t been fed propaganda and conspiracy theories—we’d need to know that the jpeg is capturing the right sections of the Web.

But finding the 'right sections of the Web' is a subjective process. This is precisely why many people have lost confidence in the news media. Media outlets (on both sides of the political spectrum) often choose to be hyper-focused on material that supports their narrative while completely ignoring evidence that goes against it.

ChatGPT and any other Large Language Model can suffer from the same 'Garbage-In, Garbage-Out' problem that can infect any other computer system.

Re: ChatGPT is a blurry JPEG of the web

#40

Post by donohoe »

zetazzed wrote: post #17

Damn, I hate to plug products on HN, but I'd say that the New Yorker is the one subscription I've loved maintaining throughout my life. First got it right out of college and appreciate it 20 years later. Everyone is publishing think pieces about ChatGPT - yawn. But only the New Yorker said, hmm, how about if we get frickin' Ted Chiang to write a think piece? (It is predictably very well written.)

Wholly agree.

I worked there many years ago, leading the re-design and re-platform (fun dealing with 90 years of archival content with mixed usage-rights) and paywall implementation (don't hate me, it funds journalism).

When you see how the stories get made and how people work there, well, its just amazing.

Post reply on HN