Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 gets a B on my quantum computing final exam

scottaaronson.blog

Post reply on HN
Print view View original

231–240 of 261 posts

Re: GPT-4 gets a B on my quantum computing final exam

#231

Post by civilized »

red75prime wrote: post #224
civilized wrote: post #139

> To the best of my knowledge—and I double-checked—this exam has never before been posted on the public Internet, and could not have appeared in GPT-4’s training data. Sure, but you can Google the answers to most of the questions. Personally I've accepted that GPT does learn and apply concepts present in its training data, and all of this would be. Learning is part of intelligence but not the whole thing. (I thought…

> but we are about to stall Maybe if it's a deliberate decision by the researchers. A path to more capabilities is rather clear (recurrence and explicit memory).

No, they won't make any real attempt to slow down. No, they won't do more than make incremental advances. To the extent that large benchmark gains occur, they will be shown to have little practical value.

Re: GPT-4 gets a B on my quantum computing final exam

#232

Post by wizzwizz4 »

redox99 wrote: post #229
wizzwizz4 wrote: post #226

Earlier quoted context omitted.

> But also interesting it's failing to keep the intent of the whole sentence unlike 3.5 It's because it "knows too much". To anthropomorphise a little: its "expectations" of what should be. To anthropomorphise less: GPT-4 is overfitted. GPT-style language models are pretty amazing, but they're not a complete explanation of human language, and can't quite represent it properly. > I'm almost feeling that GPT-4 should b…

(GPT4) In this script from a Doctor Who episode, Clara and the Doctor are having a conversation about the internet. Doctor Who is a British science fiction television series that follows the adventures of the Doctor, a Time Lord from the planet Gallifrey, who travels through time and space in the TARDIS, a time-traveling spaceship. Clara, the Doctor's companion, is trying to access the internet but is unable to find…

The middle three paragraphs are completely wrong. Clara isn't the Doctor's companion in this episode, that's not why the Doctor is confused, the Doctor is giving a year (not a time), Clara doesn't know about “the time difference” (indeed, that is the joke)…

That aside: I was suggesting this as an example of something existing machine translation systems can't translate. The 1207 / 12:07 wordplay could be “understood” by the model (I'm disappointed, albeit not very surprised, that GPT-4 didn't), but producing an adequate translation in a case like this requires actual thought and consideration.

Re: GPT-4 gets a B on my quantum computing final exam

#233

Post by [deleted] »

jen729w wrote: post #25

This is obviously very cool, but at this point — who knows what I’ll say in a year — my concern with these LLMs is that they’re in the uncanny valley. Here’s one passing a very difficult test. Amazing! Now, rely on it to build a nuclear doohickey for a power station or a multi-billion dollar device for CERN or anything really and, well, no. So humans still have to check the output, and now we’re in that situation whe…

[deleted]

Re: GPT-4 gets a B on my quantum computing final exam

#234

Post by reso »

Loeffelmann wrote: post #105
reso wrote: post #78

Earlier quoted context omitted.

I think watching the development of driverless cars in the last 15 years has taught a lot of people to be skeptical of 95% solutions. Sometimes you really need that 100% or the solution is practically useless.

I mean this is clearly not the case with LLM. They create value today even though they are not AGI yet.

LLMs create value but it's not yet clear how much.

Re: GPT-4 gets a B on my quantum computing final exam

#235

Post by redox99 »

wizzwizz4 wrote: post #232
redox99 wrote: post #229

Earlier quoted context omitted.

(GPT4) In this script from a Doctor Who episode, Clara and the Doctor are having a conversation about the internet. Doctor Who is a British science fiction television series that follows the adventures of the Doctor, a Time Lord from the planet Gallifrey, who travels through time and space in the TARDIS, a time-traveling spaceship. Clara, the Doctor's companion, is trying to access the internet but is unable to find…

The middle three paragraphs are completely wrong. Clara isn't the Doctor's companion in this episode, that's not why the Doctor is confused, the Doctor is giving a year (not a time ), Clara doesn't know about “the time difference” (indeed, that is the joke )… That aside: I was suggesting this as an example of something existing machine translation systems can't translate. The 1207 / 12:07 wordplay could be “understoo…

Yes, I also expected GPT4 to get the joke, as I've seen it understand similar jokes.

Re: GPT-4 gets a B on my quantum computing final exam

#236

Post by red75prime »

civilized wrote: post #231
red75prime wrote: post #224

Earlier quoted context omitted.

> but we are about to stall Maybe if it's a deliberate decision by the researchers. A path to more capabilities is rather clear (recurrence and explicit memory).

No, they won't make any real attempt to slow down. No, they won't do more than make incremental advances. To the extent that large benchmark gains occur, they will be shown to have little practical value.

If it'll come to pass, we'll probably have a groundbreaking discovery that the brain violates the physical Church-Turing thesis on our hands. That would be quite unexpected.

Re: GPT-4 gets a B on my quantum computing final exam

#237

Post by xscott »

_cenw wrote: post #127
spaced-out wrote: post #115

Earlier quoted context omitted.

>It might catch the tumor in 99.9999% of cases, better than any human doctor. But missing a malignant tumor 0.0001% of the time is unacceptable, because it spikes the hospital's malpractice costs. So every single scan still has to be reviewed manually by a doctor first, then by the AI as a fallback. I find it hard to believe human doctors miss malignant tumors in less than 1 out of every 10 million cases.

The point is that it may miss a tumor obvious to a human doctor.

I'm pretty sure koboll's point was that by having a doctor in the loop, the hospital can wash their hands of that one person's malpractice suit easy enough. Just fire the doctor, let their individual insurance deal with it, and move on. When the hospital cuts out the middle man, they take on a new level of direct accountability they don't currently have.

Re: GPT-4 gets a B on my quantum computing final exam

#238

Post by emmelaich »

notahacker wrote: post #220
dcow wrote: post #143

Earlier quoted context omitted.

I don't think the data supports the fear that AI assisted driving is more or newly dangerous when compared to fully human drivers. Teslas are safer than any other car on the road. Yes they’re newer, but by mile they’re safer. So the fear that “we should be careful” is understandable but ultimately unfounded. We are being careful .

> Teslas are safer than any other car on the road. This simply isn't true. By the mile they have a worse safety record than other cars in their class (mitigating factors: where they're driven and who drives them). You might be referring to Tesla's marketing statistic that there are fewer accidents per mile involving autopilot - typically engaged in ideal driving circumstances - than when it's switched off, or across…

Where did you get your stats? I could only find crash test and similar ratings, not actual records.

Re: GPT-4 gets a B on my quantum computing final exam

#239

Post by jakelazaroff »

spaced-out wrote: post #213
jakelazaroff wrote: post #193

Earlier quoted context omitted.

"So sorry the AI missed your malignant tumor! On average, it actually performs better than a human doctor. I mean, a human doctor definitely would have caught this one, and yeah, you're going to die, but hopefully the whole average thing makes you feel better!"

Does the opposite work too? What if a human doctor mis-diagnoses me but I can prove in court that an available medical grade AI would have given the correct diagnosis. Could I sue for that?

I don’t really understand why you’d ask this. The point is to save as many lives as possible.

Re: GPT-4 gets a B on my quantum computing final exam

#240

Post by sudosysgen »

M4v3R wrote: post #152
mattmanser wrote: post #58

Earlier quoted context omitted.

Can anyone answer the chance that example tests of these questions were in its training set? And it's just regurgitating the answers someone else wrote? As I imagine it's a very high chance given how much uni lecturers recycle exam questions. When I was at uni you could just get the last 5 years worth of questions from the library for almost any subject and guess what the questions were probably going to be. Often th…

From the article: > To the best of my knowledge—and I double-checked—this exam has never before been posted on the public Internet, and could not have appeared in GPT-4’s training data.

The exam, no, but most of the questions most certainly are. I know this because I've done extremely similar problems for homework and checked my answers online.
Post reply on HN