Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 gets a B on my quantum computing final exam

scottaaronson.blog

Post reply on HN
Print view View original

151–160 of 261 posts

Re: GPT-4 gets a B on my quantum computing final exam

#151

Post by adriand »

dcow wrote: post #143
jen729w wrote: post #25

This is obviously very cool, but at this point — who knows what I’ll say in a year — my concern with these LLMs is that they’re in the uncanny valley. Here’s one passing a very difficult test. Amazing! Now, rely on it to build a nuclear doohickey for a power station or a multi-billion dollar device for CERN or anything really and, well, no. So humans still have to check the output, and now we’re in that situation whe…

I don't think the data supports the fear that AI assisted driving is more or newly dangerous when compared to fully human drivers. Teslas are safer than any other car on the road. Yes they’re newer, but by mile they’re safer. So the fear that “we should be careful” is understandable but ultimately unfounded. We are being careful .

It’s abundantly clear that we are going to expect near-perfect reliability from autonomous vehicles. This isn’t necessarily illogical; they operate in a different context than humans do. We expect humans to make mistakes and we have various ways of dealing with the consequences (eg lawsuits targeted at the responsible individual). The argument from statistics doesn’t appear likely to win the kind of societal approval we need for autonomous vehicles to be accepted.

Re: GPT-4 gets a B on my quantum computing final exam

#152

Post by M4v3R »

mattmanser wrote: post #58
c54 wrote: post #38

Earlier quoted context omitted.

I agree with the point you're making here, but it’s also funny that the description of someone passing a test but not being able to do much without a lot of human supervision is… exactly the description of a human college graduate.

Can anyone answer the chance that example tests of these questions were in its training set? And it's just regurgitating the answers someone else wrote? As I imagine it's a very high chance given how much uni lecturers recycle exam questions. When I was at uni you could just get the last 5 years worth of questions from the library for almost any subject and guess what the questions were probably going to be. Often th…

From the article:

> To the best of my knowledge—and I double-checked—this exam has never before been posted on the public Internet, and could not have appeared in GPT-4’s training data.

Re: GPT-4 gets a B on my quantum computing final exam

#153

Post by redox99 »

jacquesm wrote: post #45
cypress66 wrote: post #8

Earlier quoted context omitted.

Machine translation isn't super human yet. But yes, it's probably only a few years away.

It's superhuman compared to a lot of humans. Especially when you factor in that even the most accomplished individuals rarely have more than five or six languages that they are really good at (exceptions exist) so that makes machine translation superhuman for the vast majority of the human/language pairs.

There is a very clear benchmark here though. Can you use GPT4 translations "as is" for your product[1]? I've used GPT a fair bit for EnglishSpanish (both directions) for my product, and I often have to fix the odd word here and there (but it saves a lot of time as a starting point). And my English is far from perfect, but fortunately GPT's mistakes are orthogonal to mine.

When you don't need to pay a translator, then it's at human/superhuman level.

[1] Assuming you want good translations, of course. There have always been those amusingly bad Google Translate translations of owners that simply did not care.

Re: GPT-4 gets a B on my quantum computing final exam

#154

Post by SanderNL »

davnicwil wrote: post #39

Look, it's impressive yes or yes but is it that surprising that a system like an LLM does well on exams? What is an exam? It's an attempt to test that you have consumed and understood large amounts of information and can apply it to novel (ish) situations, but in a very 'sandboxed' way that is just text-in-text-out. That's literally what these systems are specialists in doing. Is it maybe a bit like saying a driverle…

> It's an attempt to test that you have consumed and understood large amounts of information and can apply it to novel (ish) situations,

So.. that's basically all we need. So what's left? It can't drink a beer?

Economically useful intelligence is not much more than storing and retrieving large amount of contextually relevant information. Sure, humans can do more, like interpretive dancing for example. But that's not really what we are looking for in a desk job.

If we all earned our money by driving time-trials on race tracks I'd be worried yes.

Re: GPT-4 gets a B on my quantum computing final exam

#155

Post by famouswaffles »

fvdessen wrote: post #145
famouswaffles wrote: post #50

Earlier quoted context omitted.

Bilingual/Multilingual LLMs are human level translators more or less. The only way you can think "not on the cusp" is if you haven't actually used GPT-4 for translation. Use it and you'll be set straight pretty quickly.

I was curious so I asked GPT-4 to translate a bit of french literature, here it is, along with the official translation (I'll let you guess which is which) ------- The tale I'm about to unfold commenced with a mysterious handwriting on an envelope. Within the pen strokes that outlined my name and the address of the Fossil Review, a publication I was associated with and where the letter had been forwarded from, there…

assuming the second is GPT. Here is GPT-4's translation when i say, "add a literary flair" to the translation task.

The beginning of all that I am about to recount was an unknown script upon an envelope. Within these strokes of ink that traced my name and the address of the Fossil Review, to which I contributed and from where the letter had been forwarded to me, there swirled a blend of ferocity and gentleness. Behind the questions I posed to myself about the sender and the possible contents of the message, a vague but potent premonition conjured the image of a "cobblestone thrown into a frog pond." And from the depths, the confession rose like a bubble that my life had become rather stagnant of late. Thus, when I opened the letter, I could not discern whether it felt like a revitalizing breath of fresh air or an unwelcome draft.

The same swift and well-connected writing conveyed the message in one fell swoop:

Sir, I have read your article on Mount Analogue. Until now, I believed myself to be the only one convinced of its existence. Today, we are two, tomorrow we shall be ten, perhaps more, and then we can attempt the expedition. We must make contact as soon as possible. Call me at one of the numbers below as soon as you can. I await you.

Either way, the default response is not even close to what you can fully extract out of it, if you simply ask.

Re: GPT-4 gets a B on my quantum computing final exam

#156

Post by rscho »

koboll wrote: post #60
jen729w wrote: post #25

This is obviously very cool, but at this point — who knows what I’ll say in a year — my concern with these LLMs is that they’re in the uncanny valley. Here’s one passing a very difficult test. Amazing! Now, rely on it to build a nuclear doohickey for a power station or a multi-billion dollar device for CERN or anything really and, well, no. So humans still have to check the output, and now we’re in that situation whe…

>No negativity towards AI here. It’s amazing and it’ll change the future. But we need to be careful on the way. Yeah, I suspect a lot of fields will have a similar trajectory to how AI has impacted radiology. It might catch the tumor in 99.9999% of cases, better than any human doctor. But missing a malignant tumor 0.0001% of the time is unacceptable, because it spikes the hospital's malpractice costs. So every single…

> catch the tumor in 99.9999% of cases, better than any human doctor

I don't think results are anywhere close to that in the field. If hospitals could do without radiologists, they would do so immediately. Currently, we are seeing very little technical progress due to applied statistics in the field, and the real cause of that is that tech people don't understand what we do and why we're still very much needed. The problem lies more in information retrieval capabilities than acting on the data itself.

Re: GPT-4 gets a B on my quantum computing final exam

#157

Post by mabbo »

This is the kind of thing that has Google terrified, and rightly so.

Nobody needs a chatbot to pass exams. But what percentage of your Google searches are to answer some obscure question? This thing can answer most basic questions about quantum computing and 10,000 other topics.

In the last 24 hours, I've had ChatGPT solve 3 weird questions about querying PrestoDB, help me get my Bluetooth headphones to work (I needed to press down and hold the power button!) and it gave me some brainstorming ideas around what to do for mother's day (it's coming up soon non-British people).

Before I'd have Googled those things and spent five times longer to solve them- seeing ads on each search I made.

Re: GPT-4 gets a B on my quantum computing final exam

#158

Post by M4v3R »

DC-3 wrote: post #71
famouswaffles wrote: post #50

Earlier quoted context omitted.

Bilingual/Multilingual LLMs are human level translators more or less. The only way you can think "not on the cusp" is if you haven't actually used GPT-4 for translation. Use it and you'll be set straight pretty quickly.

I haven't used GPT-4 for translation so I acknowledge I might be wrong. But GPT-3 was such an irredeemably terrible poet that it made me sceptical that this type of software could ever develop aesthetic taste or artistic vision. Moreover – and I understand this is an uncharitable thing to say, but it is my honest observation – time and again I have noticed the inability of AI cheerleaders to judge literature on its a…

I've used GPT-4 for translation numerous times and it worked astonishingly well, better then Google Translate most of the time. Especially when translating idioms and slang. It's also much better than GPT-3 at poetry - I really suggest you try it before just remaining skeptical about the technology.

Re: GPT-4 gets a B on my quantum computing final exam

#159

Post by timy2shoes »

MengerSponge wrote: post #31
timy2shoes wrote: post #17

Earlier quoted context omitted.

The enrollment at UT Austin is ~50K. No way you can grade that many tests by hand. Source: I went to UT for grad school and was a TA.

It's safe to assume the enrollment of Prof. Aaronson's Quantum Computing course isn't 50k.

Thank you Captain Obvious. The parent comment was about classes at UT in general, not about one specific class.

Re: GPT-4 gets a B on my quantum computing final exam

#160

Post by munchler »

civilized wrote: post #139

> To the best of my knowledge—and I double-checked—this exam has never before been posted on the public Internet, and could not have appeared in GPT-4’s training data. Sure, but you can Google the answers to most of the questions. Personally I've accepted that GPT does learn and apply concepts present in its training data, and all of this would be. Learning is part of intelligence but not the whole thing. (I thought…

> you can Google the answers to most of the questions

1) Students can Google the answers also, but neither students nor GPT-4 are allowed to Google the answers during the test, so it remains a fair comparison.

2) Many of the questions require calculations, which are far less Googleable.

Post reply on HN