Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 gets a B on my quantum computing final exam

scottaaronson.blog

Post reply on HN
Print view View original

71–80 of 261 posts

Re: GPT-4 gets a B on my quantum computing final exam

#71

Post by DC-3 »

famouswaffles wrote: post #50
DC-3 wrote: post #21

Earlier quoted context omitted.

I truly pity you for thinking that learning a foreign language is a redundant exercise because of machine translation. And besides, though machines may perform well on menus and tax returns, I hardly think them on the cusp of emitting fine translations of great poems or novels.

Bilingual/Multilingual LLMs are human level translators more or less. The only way you can think "not on the cusp" is if you haven't actually used GPT-4 for translation. Use it and you'll be set straight pretty quickly.

I haven't used GPT-4 for translation so I acknowledge I might be wrong. But GPT-3 was such an irredeemably terrible poet that it made me sceptical that this type of software could ever develop aesthetic taste or artistic vision.

Moreover – and I understand this is an uncharitable thing to say, but it is my honest observation – time and again I have noticed the inability of AI cheerleaders to judge literature on its artistic merits. This doesn't hold universally, but it's common enough that I have resolved to regard such claims with extreme doubt.

Re: GPT-4 gets a B on my quantum computing final exam

#73

Post by theK »

Reading stuff like this, one thing I cannot stop wondering is this:

If Ai can be trusted to do all the trivial tasks and if non trivial tasks require a scaffold of trivial practice, where are we going to keep finding the people qualified enough to actually do the non trivial stuff?

Re: GPT-4 gets a B on my quantum computing final exam

#74

Post by nostoc »

koboll wrote: post #60
jen729w wrote: post #25

This is obviously very cool, but at this point — who knows what I’ll say in a year — my concern with these LLMs is that they’re in the uncanny valley. Here’s one passing a very difficult test. Amazing! Now, rely on it to build a nuclear doohickey for a power station or a multi-billion dollar device for CERN or anything really and, well, no. So humans still have to check the output, and now we’re in that situation whe…

>No negativity towards AI here. It’s amazing and it’ll change the future. But we need to be careful on the way. Yeah, I suspect a lot of fields will have a similar trajectory to how AI has impacted radiology. It might catch the tumor in 99.9999% of cases, better than any human doctor. But missing a malignant tumor 0.0001% of the time is unacceptable, because it spikes the hospital's malpractice costs. So every single…

I suspect AI went that way in radiology not because of the chances of False Negatives, but because radiologist are entrenched in the system and will not yield an insanely lucrative stream of revenue.

Re: GPT-4 gets a B on my quantum computing final exam

#75

Post by ewhanley »

moonchrome wrote: post #26
hgsgm wrote: post #9

Earlier quoted context omitted.

More that most undergrad colleges only teach and test regurgitation because the profs don't care about education.

Have you actually tried using GPT 4 and 3.5 ? I'd say 3.5 could be called a regurgitation machine with a few tricks, GPT 4 seems to apply concepts and feels like showing signs of human level intelligence. If the rate of progress continues for a few years we are collectively fucked.

Yeah, the improvement from 3.5 to 4 is amazing. I'm surprised so many people are so dismissive, and I suspect it's because most of them haven't tried it out. Such are the ever-moving ML/AI goalposts.

Re: GPT-4 gets a B on my quantum computing final exam

#77

Post by qorrect »

famouswaffles wrote: post #29
wizzwizz4 wrote: post #23

Earlier quoted context omitted.

I don't get why people are so optimistic about machine translation. Computers can get explicit meaning across – that's obvious to anyone who understands linear algebra, information theory, and linguistics. But many aspects of translation (puns, tone, cultural context) aren't just about mapping from one vector space to another. A human, no matter how fluently bilingual, would have to think about the problem, and the c…

Have you actually used GPT-4 for translation? Seriously all this talk about only getting explicit meaning across would be easily dispelled in an afternoon if you only bothered to try.

Agreed. A lot of these responses read like they haven't actually tried it yet.

Which is also interesting, I myself actively put off trying it until I eventually gave in. It seems a lot of us are doing the same, maybe its a case of "how good could it actually be?"

Re: GPT-4 gets a B on my quantum computing final exam

#78

Post by reso »

jen729w wrote: post #25

This is obviously very cool, but at this point — who knows what I’ll say in a year — my concern with these LLMs is that they’re in the uncanny valley. Here’s one passing a very difficult test. Amazing! Now, rely on it to build a nuclear doohickey for a power station or a multi-billion dollar device for CERN or anything really and, well, no. So humans still have to check the output, and now we’re in that situation whe…

I think watching the development of driverless cars in the last 15 years has taught a lot of people to be skeptical of 95% solutions. Sometimes you really need that 100% or the solution is practically useless.

Re: GPT-4 gets a B on my quantum computing final exam

#79

Post by scrame »

Isn't this more an indictment of current teaching and testing methods than a praise of LLMs?

Like the old "you can't use a calculator on a test because you won't always have a calculator".

Im old enough that my textbooks still had trig lookup tables when I was a kid. That seems silly now, but it was considered important at the time. I know slightly older people in publishing that used manual photosizers instead of just importing to photoshop.

I remember being told multiple times that VR was going to fix programming because you could just visually move conditional blocks around.

So we have smartphones and scratch and photoshop and whathaveyou, but...

We still need people to draw and write and photograph and sculpt. we still need engineers to use the new models, and we still need mechanics to fix the computers and trains when they break.

If GPT-4 can pass a test, maybe it's a better reference point, or maybe a better study aid. Like taped language lessons vs reading pronunciation diagrams.

You'll never obviate people from the equation that is defined by people,regardless of that singularity that is just around the corner like cold fusion and self driving cars and the cure for the common cold.

Re: GPT-4 gets a B on my quantum computing final exam

#80

Post by koboll »

nostoc wrote: post #74
koboll wrote: post #60

Earlier quoted context omitted.

>No negativity towards AI here. It’s amazing and it’ll change the future. But we need to be careful on the way. Yeah, I suspect a lot of fields will have a similar trajectory to how AI has impacted radiology. It might catch the tumor in 99.9999% of cases, better than any human doctor. But missing a malignant tumor 0.0001% of the time is unacceptable, because it spikes the hospital's malpractice costs. So every single…

I suspect AI went that way in radiology not because of the chances of False Negatives, but because radiologist are entrenched in the system and will not yield an insanely lucrative stream of revenue.

Hospitals would love to fire all radiologists and replace them with software.

They've already done it with outsourcing; a large chunk of what used to be done entirely in-house has been contracted out to remote overseas doctors.

Post reply on HN