Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 gets a B on my quantum computing final exam

scottaaronson.blog

Post reply on HN
Print view View original

191–200 of 261 posts

Re: GPT-4 gets a B on my quantum computing final exam

#191

Post by civilized »

sebzim4500 wrote: post #175
civilized wrote: post #139

> To the best of my knowledge—and I double-checked—this exam has never before been posted on the public Internet, and could not have appeared in GPT-4’s training data. Sure, but you can Google the answers to most of the questions. Personally I've accepted that GPT does learn and apply concepts present in its training data, and all of this would be. Learning is part of intelligence but not the whole thing. (I thought…

I used to agree with you. The paper that made me unsure was "Transformers learn in-context by gradient descent" [1]. Basically, the model learns weights that let it run gradient descent at inference time in order to do in context learning. If transformers can learn this, then I think they can learn almost anything given enough compute/parameters/data. Of course, even if this is true then it's possible that there simp…

Too theoretical to be pivotal. Years ago we had results about how all sorts of learning was reducible to kernel SVM and that drove a big fad for kernel SVM research at NeurIPS and other hot conferences. Nobody cares about SVM anymore. So we don't have a good enough grasp on the theory of AI to predict what is going to be a breakthrough. Only rigorous empirical results convince.

Second, the idea of meta-learning gradient descent, or that some learning algorithms can do it, isn't new or specific to Transformer architectures.

Re: GPT-4 gets a B on my quantum computing final exam

#192

Post by STM32F030R8 »

bigbillheck wrote: post #84
estomagordo wrote: post #68

Earlier quoted context omitted.

Well, if a uni with 5k enrolled can hand grade tests, then obviously UT Austin could, if they had 10x faculty of the first uni.

The more you spend on faculty the less you can spend on sports.

Do you honestly believe UT spends money on its sports? You realize their football team generates BILLIONS a year in sales and TV rights? Football isn’t why UT is overcrowded lol

Re: GPT-4 gets a B on my quantum computing final exam

#193

Post by jakelazaroff »

epups wrote: post #141
_cenw wrote: post #127

Earlier quoted context omitted.

The point is that it may miss a tumor obvious to a human doctor.

As long as on average it performs better that should not be an issue.

"So sorry the AI missed your malignant tumor! On average, it actually performs better than a human doctor. I mean, a human doctor definitely would have caught this one, and yeah, you're going to die, but hopefully the whole average thing makes you feel better!"

Re: GPT-4 gets a B on my quantum computing final exam

#195

Post by numpad0 »

redox99 wrote: post #184
numpad0 wrote: post #169

Earlier quoted context omitted.

Can you try this[0]? I have no access to the -4... Have you actually used GPT-4 for translation? Seriously all this talk about only getting explicit meaning across would be easily dispelled in an afternoon if you only bothered to try. Bing Chat: GPT-4を翻訳に使用したことがありますか？本当に明示的な意味しか伝えられないという話は、試してみれば午後には簡単に反証できます。 (Have you utilized GPT-4 for translations? The story that only really explicit meaning can be conveyed, can…

GPT4 実際にGPT-4を翻訳に使ったことがありますか？本当に、明示的な意味だけを伝えるという話が、試してみるだけで簡単に解決できるなんて、冗談じゃないですか。

WHAT. It's got the second half wrong.

Google: Have you actually used GPT-4 for translation? Really, it's a joke that the story of only conveying explicit meaning can be easily solved by just trying.

DeepL: Have you actually used GPT-4 for translation? Really, it's a joke that all this talk about conveying only explicit meaning can be easily solved by just trying it out.

Mine: Have you actually used GPT-4 for translations? That you can really just, try and easily solve that story that to convey explicit meaning, is such a joke.

Re: GPT-4 gets a B on my quantum computing final exam

#196

Post by InitialLastName »

jacquesm wrote: post #49
H8crilA wrote: post #43

Earlier quoted context omitted.

And the test is actually rather easy. It's an undergraduate test.

It's probably not 'easy' for the average human out there. I would expect >> 90% of humanity to fail that test.

I have highly technical graduate and undergraduate degrees (in CS-adjacent field) and a decade of experience doing software development part time, but no direct experience with quantum computing (or much beyond an undergraduate physics class and an intro CS class in terms of directly related study), and consider myself a better-than-average test-taker. I could get 1D, 1E, 1F and 1T by intuition, 1J by actual knowledge, 1P by tautology, and had roughly no shot at the open-ended questions.

Re: GPT-4 gets a B on my quantum computing final exam

#197

Post by spiderice »

dcow wrote: post #177
adriand wrote: post #151

Earlier quoted context omitted.

It’s abundantly clear that we are going to expect near-perfect reliability from autonomous vehicles. This isn’t necessarily illogical; they operate in a different context than humans do. We expect humans to make mistakes and we have various ways of dealing with the consequences (eg lawsuits targeted at the responsible individual). The argument from statistics doesn’t appear likely to win the kind of societal approval…

Your point about liability is valid, but it’s far from “abundantly clear” that society needs machines to be near perfect as opposed to just significantly better than humans . Solve the liability problem and I would 100% take a machine that performs 30% better than a human or helps a human perform 30% better every time because it means fewer humans die on the road.

> Your point about liability is valid

How is it valid? It's currently a solved problem. The driver of the car is still held liable because they are still, ultimately, driving the car. Tesla's with autopilot seem to make drivers safer, and acting like we don't know who is liable in the event of a crash is just a red herring.

Re: GPT-4 gets a B on my quantum computing final exam

#198

Post by Sharlin »

civilized wrote: post #185
munchler wrote: post #160

Earlier quoted context omitted.

> you can Google the answers to most of the questions 1) Students can Google the answers also, but neither students nor GPT-4 are allowed to Google the answers during the test , so it remains a fair comparison. 2) Many of the questions require calculations, which are far less Googleable.

1) It's not possible to fairly compare human intelligence with something that can memorize gigabytes of text and hold it in non-volatile memory. 2) Months ago, in my earliest interactions with ChatGPT, I asked it to solve math problems. It gave me back stuff with LaTeX formatting. Obviously it had, if not these exact problems, similar templates in its training set. Recently it was shown that GPT is completely incapab…

> It gave me back stuff with LaTeX formatting.

So? It can write LaTeX just like it can write Java or Python or Rust.

Re: GPT-4 gets a B on my quantum computing final exam

#199

Post by bsder »

jen729w wrote: post #25

This is obviously very cool, but at this point — who knows what I’ll say in a year — my concern with these LLMs is that they’re in the uncanny valley. Here’s one passing a very difficult test. Amazing! Now, rely on it to build a nuclear doohickey for a power station or a multi-billion dollar device for CERN or anything really and, well, no. So humans still have to check the output, and now we’re in that situation whe…

It's very easy to stop these AI cold. Make "I don't know" or "Insufficient information" a correct answer--those are not in the training set.

These AI have "Male Answer Syndrome" to the nth degree. They will shamelessly give you an answer even if they have to completely make one up.

Re: GPT-4 gets a B on my quantum computing final exam

#200

Post by simonh »

DC-3 wrote: post #180
fvdessen wrote: post #145

Earlier quoted context omitted.

I was curious so I asked GPT-4 to translate a bit of french literature, here it is, along with the official translation (I'll let you guess which is which) ------- The tale I'm about to unfold commenced with a mysterious handwriting on an envelope. Within the pen strokes that outlined my name and the address of the Fossil Review, a publication I was associated with and where the letter had been forwarded from, there…

The second is profoundly better. The first is unpublishable.

To me the second is dull as ditchwater.
Post reply on HN