Live data from Hacker News

GPT-4 gets a B on my quantum computing final exam

scottaaronson.blog

161–170 of 261 posts

Re: GPT-4 gets a B on my quantum computing final exam

#161

It's impressive on the surface... But give me Google and I too can pass some extremely challenging tests that I actually know nothing about. If someone told you I passed the "bar" exam, you would be impressed. If they then said, he had google and lots of time. You wouldn't be that impressed anymore. The impressive thing here is that AI can read and answer questions... it's not overly impressive that it can use inform…

Passing a bar exam is not possible by using Google and using an hour or two extra..

Re: GPT-4 gets a B on my quantum computing final exam

#162
post #6

When I was growing up in the 2000s, it was required to learn a foreign language in school. I took Spanish but dropped the class after a year, thinking that when I grew up, computers would be able to translate text far better humans could. Would you believe it, transformers were invented ten years later. I wonder if it's worth learning anything anymore.

Learning a second language is a huge investment of time, and at least some money, that will soon have no significant economic upside.

Re: GPT-4 gets a B on my quantum computing final exam

#163
post #85
post #64

Earlier quoted context omitted.

The knee-jerk contrarianism seems to be in vogue right now. Reminds me of some colleagues in 2007 lamenting about the iPhone, "It's not that great; it doesn't even have copy & paste!" Fast forward a few years, and no blackberries in sight.

Oh God, when the iPhone came out people were raging "But it doesn't have a keyboard!" It's so hard for people, including myself, to look at the future potential of a technology. My attitude is to simply keep an open mind nowadays instead of holding very strong opinions about the trajectory of a given technology.

Not everybody happy living in the future. I'm annoyed most of the time I had to use on-screen keyboard on iPhone - it is slow, error prone and auto-correction only makes it worse when I use uncommon terms. I want hardware keyboard like it was on Nokia with Symbian or Blackberry back.

Re: GPT-4 gets a B on my quantum computing final exam

#164

Earlier quoted context omitted.

It’s impressive and surprising. Imagine going back to the year 2021 and telling the good people of HN that in 2023 AI would be as advanced like it is now. Literally no one would have believed you in 2021 if you did.

Exactly everything is obvious in hindsight. No one called this, and certainly no one called it would be so easy to access.

> No one called this, and certainly no one called it would be so easy to access.

Thanks for capturing a point that I couldn't quite articulate myself as to why ChatGPT feels different - the ease of use.

Literally a year ago I was planning to take a python programming course for ML, thinking that a deep understanding of code would be needed to make things work well.

With GPT it's like...I just ask it things in English and it'll do it?

The ease of access to LMMs is groundbreaking the way the simplicity of IOS launched the modern smartphone era. Even babies could use an Iphone.

And now any child old enough to type sentences can use ChatGPT.

Re: GPT-4 gets a B on my quantum computing final exam

#165
post #53

Earlier quoted context omitted.

Machine translation isn't super human yet. But yes, it's probably only a few years away.

Doubt it … GPTs speak any language they know natively, but if asked for translations they seem unable to deal with sentence structures and logics that exists in the source but not allowed in the target language. When that happens they fail to recognize gibberishness of that.

Can you provide an example? Because from my experience it's the exact opposite - GPT-4 can handle translation, especially when there are complex sentences and context that needs to be kept across sentences, way better than Google Translate currently can.

Re: GPT-4 gets a B on my quantum computing final exam

#166

Earlier quoted context omitted.

Bilingual/Multilingual LLMs are human level translators more or less. The only way you can think "not on the cusp" is if you haven't actually used GPT-4 for translation. Use it and you'll be set straight pretty quickly.

I was curious so I asked GPT-4 to translate a bit of french literature, here it is, along with the official translation (I'll let you guess which is which) ------- The tale I'm about to unfold commenced with a mysterious handwriting on an envelope. Within the pen strokes that outlined my name and the address of the Fossil Review, a publication I was associated with and where the letter had been forwarded from, there…

The second is clearly the human, as ChatGPT will not be so daring as not to translate absolutely everything it can including "Revue de Fossiles".

The human writing flows better for the most part, although I like the second paragraph ChatGPT wrote, esp "presently we are a pair".

But in terms of being a functional translation, ChatGPT is fully adequate. I have used it a lot for this purpose, from many languages, and never found it to be less than accurate. You can also tweak the tone of voice and many other things with simple English requests. This puts it generations ahead of existing tools like Google Translate, and imho puts it into the class of technologies that are close enough to perfect that they will be hard to ever replace.

Re: GPT-4 gets a B on my quantum computing final exam

#167
post #101

Earlier quoted context omitted.

So who will then certify them?

Certify them? What for?

Important work. It is a typical quality control architecture in the modern world to have an independent party evaluate whether one can perform an important task or not. Typical examples would be SE vs PE licensing on structural engineering or the various bar exams.

Re: GPT-4 gets a B on my quantum computing final exam

#168

Earlier quoted context omitted.

Can anyone answer the chance that example tests of these questions were in its training set? And it's just regurgitating the answers someone else wrote? As I imagine it's a very high chance given how much uni lecturers recycle exam questions. When I was at uni you could just get the last 5 years worth of questions from the library for almost any subject and guess what the questions were probably going to be. Often th…

Yes, the vast majority of these questions are standard known problems it definitely already saw with a slightly different formulation.

You can try phrasing the question in a way that it wouldn't be phrased but would still demonstrate understanding of concept.

I remember Yann LeCun gave an interview and he came up with some random question like "If I'm holding a peace of paper with both of my hands above the desk and I release one what would happen". His point was that since the LLM doesn't have a world model it wouldn't be able to answer these trivial intuitive questions unless it saw something similar in the training set. And then the interviewer tried it and it failed. That was 3.5. I've tried many variation of that class of problem with 4 and it seems to generalize basic physics concepts quite well. So maybe 4 learned basic physics ? Why couldn't it learn QM theory as well ?

Re: GPT-4 gets a B on my quantum computing final exam

#169
post #97

Earlier quoted context omitted.

I don't get why people are so optimistic about machine translation. Computers can get explicit meaning across – that's obvious to anyone who understands linear algebra, information theory, and linguistics. But many aspects of translation (puns, tone, cultural context) aren't just about mapping from one vector space to another. A human, no matter how fluently bilingual, would have to think about the problem, and the c…

>no machine translation system currently in existence could translate the 逆転裁判 games to Ace Attorney games Maybe it's already in the training set, but GPT-4 does give that exact translation. I've found that GPT-4 is exceptionally good at translating idioms and other big picture translation issues. Where it occasionally makes mistakes is with small grammatical and word order issues that previous tools do tend to get r…

Can you try this[0]? I have no access to the -4...

  Have you actually used GPT-4 for translation? Seriously all this talk about only getting explicit meaning across would be easily dispelled in an afternoon if you only bothered to try. 
Bing Chat:

  GPT-4を翻訳に使用したことがありますか?本当に明示的な意味しか伝えられないという話は、試してみれば午後には簡単に反証できます。
  (Have you utilized GPT-4 for translations? The story that only really explicit meaning can be conveyed, can be easily disproved by afternoon if tried.) 
Google:

  実際にGPT-4を翻訳に使ったことはありますか? 真剣に、明示的な意味だけを理解することについてのこのすべての話は、あなたが試してみるだけなら、午後には簡単に払拭されるでしょう.
  (Have you actually used GPT-4 for translation? Seriously, This stories of all about understanding solely explicit meanings are, if it is only for you to try, will be easily swept away by afternoon.)
DeepL:

  実際にGPT-4を使って翻訳したことがあるのですか?明示的な意味しか伝わらないという話は、やってみようと思えば、午後には簡単に払拭されるはずです。
  (Do you have experience of actually translating using GPT-4? The story that only explicit meaning is conveyed, if so desired, can be easily swept away by afternoon)
If I'd do it:

  GPT-4を翻訳に使ったことがあって言ってる? 真面目に言って、表層的な意味しか取れないとかないって暇な時にやってみれば分かると思うんだけど。
  (Are you saying having used GPT-4 for translation? Seriously speaking, I think that it only gets superficial meaning isn't [true] if [you] would try [it] when [you'd] have time.)
0: https://news.ycombinator.com/item?id=35530380

Re: GPT-4 gets a B on my quantum computing final exam

#170
post #60
post #25

This is obviously very cool, but at this point — who knows what I’ll say in a year — my concern with these LLMs is that they’re in the uncanny valley. Here’s one passing a very difficult test. Amazing! Now, rely on it to build a nuclear doohickey for a power station or a multi-billion dollar device for CERN or anything really and, well, no. So humans still have to check the output, and now we’re in that situation whe…

>No negativity towards AI here. It’s amazing and it’ll change the future. But we need to be careful on the way. Yeah, I suspect a lot of fields will have a similar trajectory to how AI has impacted radiology. It might catch the tumor in 99.9999% of cases, better than any human doctor. But missing a malignant tumor 0.0001% of the time is unacceptable, because it spikes the hospital's malpractice costs. So every single…

Image recognition and statistics is already being used as a first pass for pathologists in full force today. It’s weird to pretend like this is some new uncharted frontier for medicine and/or that insurance doesn't know how to handle it…
Post reply on HN