Live data from Hacker News

GPT-4 gets a B on my quantum computing final exam

scottaaronson.blog

21–30 of 261 posts

Re: GPT-4 gets a B on my quantum computing final exam

#21
post #6

When I was growing up in the 2000s, it was required to learn a foreign language in school. I took Spanish but dropped the class after a year, thinking that when I grew up, computers would be able to translate text far better humans could. Would you believe it, transformers were invented ten years later. I wonder if it's worth learning anything anymore.

I truly pity you for thinking that learning a foreign language is a redundant exercise because of machine translation.

And besides, though machines may perform well on menus and tax returns, I hardly think them on the cusp of emitting fine translations of great poems or novels.

Re: GPT-4 gets a B on my quantum computing final exam

#22
I tried the first one on GPT-3.5-turbo and it got the answer wrong haha. Goes to show you how much better GPT-4 is...

GPT-3.5-TURBO:

---------------------------

True.

A pure state of n qubits is described by an n-dimensional complex unit vector, also known as a quantum state vector. In Dirac notation, a pure state of n qubits can be represented as:

|ψ⟩ = a_0|0⟩ + a_1|1⟩ + ... + a_(2^n-1)|2^n-1⟩

where a_i are complex numbers and |i⟩ represents the i-th computational basis state. The normalization condition |a_0|^2 + |a_1|^2 + ... + |a_(2^n-1)|^2 = 1 must also be satisfied.

(hallucination?)

---------------------------

GPT-4:

---------------------------

False. A pure state of n qubits is described by a 2^n-dimensional complex unit vector. In a quantum system, each qubit can be in a superposition of two basis states, so for n qubits, there are 2^n possible basis states in the overall Hilbert space.

---------------------------

Interesting to note that with GPT-3.5-turbo, a single followup chain-of-reasoning prompt ("Let's think step by step", i.e. no introduction of a priming prompt like "Are you sure?" or "Why are you wrong?") helps the model identify the correct answer:

GPT-3.5-TURBO:

---------------------------

Sure, let's break it down step by step:

    Qubits: A qubit is the fundamental unit of quantum information. It is a quantum analogue to the classical bit. While a classical bit can take one of two values, 0 or 1, a qubit can be in a superposition of both 0 and 1 states simultaneously.

    Basis states: For a single qubit, there are two basis states, often denoted as |0> and |1>. A qubit can be in any linear combination (superposition) of these basis states, represented as α|0> + β|1>, where α and β are complex numbers, and the sum of the magnitudes squared (|α|^2 + |β|^2) equals 1.

    Multiple qubits: When you have a system of n qubits, each qubit can be in a superposition of its basis states. The overall state of the n-qubit system is described by the tensor product of the individual qubit states. The number of basis states in the combined system is equal to the product of the number of basis states of each individual qubit.

    n-dimensional vector: For n classical bits, you would indeed need an n-dimensional vector to describe their state. However, with n qubits in a quantum system, the number of basis states grows exponentially, as opposed to linearly in the classical case.

    2^n-dimensional complex unit vector: As we mentioned earlier, each qubit can be in a superposition of two basis states. Therefore, for n qubits, there are 2^n possible basis states. The overall state of an n-qubit system is represented by a 2^n-dimensional complex unit vector, with each component corresponding to the probability amplitude of one of the 2^n basis states.
In summary, a pure state of n qubits is described by a 2^n-dimensional complex unit vector, not an n-dimensional one.

---------------------------

Re: GPT-4 gets a B on my quantum computing final exam

#23
post #6

When I was growing up in the 2000s, it was required to learn a foreign language in school. I took Spanish but dropped the class after a year, thinking that when I grew up, computers would be able to translate text far better humans could. Would you believe it, transformers were invented ten years later. I wonder if it's worth learning anything anymore.

Machine translation isn't super human yet. But yes, it's probably only a few years away.

I don't get why people are so optimistic about machine translation. Computers can get explicit meaning across – that's obvious to anyone who understands linear algebra, information theory, and linguistics. But many aspects of translation (puns, tone, cultural context) aren't just about mapping from one vector space to another. A human, no matter how fluently bilingual, would have to think about the problem, and the current systems don't think.

If we keep working on them, these systems will likely get better and better at low-level translation (including translating idioms), but no machine translation system currently in existence could translate the 逆転裁判 games to Ace Attorney games. Perhaps computers can do it – I don't see a theoretical reason they shouldn't be able to – but it would take a fundamentally different approach.

Re: GPT-4 gets a B on my quantum computing final exam

#24
post #11

Earlier quoted context omitted.

Machine translation isn't super human yet. But yes, it's probably only a few years away.

It might not be super human in the sense of beating the C2 standard, but it sure knows more German than I do. And Spanish, and Greek, and Esperanto, and… well, everything. But I've been really trying with that German.

I moved to Japan recently so have been trying to learn the language. Im atrociously bad and dream of the day when I can read a sentence without getting tripped up on kanji, grammar, etc. I mentioned this to a coworker who has been studying for several years and she said she just uses Google translate for everything. Even with her pretty good japanese, translate is better and faster than she is for doing things like going through forms, letters, etc. We may reach a point where computers are good enough that they beat all but native speakers, maybe.

Re: GPT-4 gets a B on my quantum computing final exam

#25
This is obviously very cool, but at this point — who knows what I’ll say in a year — my concern with these LLMs is that they’re in the uncanny valley.

Here’s one passing a very difficult test. Amazing! Now, rely on it to build a nuclear doohickey for a power station or a multi-billion dollar device for CERN or anything really and, well, no.

So humans still have to check the output, and now we’re in that situation where humans driving a Tesla on autopilot who are supposed to be 100% aware of the road aren’t, because they get lazy and doze off, and now the car crashes and whoops.

No negativity towards AI here. It’s amazing and it’ll change the future. But we need to be careful on the way.

Re: GPT-4 gets a B on my quantum computing final exam

#26
post #9
post #5

This is pretty incredible. I guess the thing that we have to keep in mind is that we really can't compete with an AI that understands natural language this well and also has the entirety of the internet as its reference materials. Maybe we need to accept that or rework our definition of what makes humans special

More that most undergrad colleges only teach and test regurgitation because the profs don't care about education.

Have you actually tried using GPT 4 and 3.5 ? I'd say 3.5 could be called a regurgitation machine with a few tricks, GPT 4 seems to apply concepts and feels like showing signs of human level intelligence. If the rate of progress continues for a few years we are collectively fucked.

Re: GPT-4 gets a B on my quantum computing final exam

#27
post #6

When I was growing up in the 2000s, it was required to learn a foreign language in school. I took Spanish but dropped the class after a year, thinking that when I grew up, computers would be able to translate text far better humans could. Would you believe it, transformers were invented ten years later. I wonder if it's worth learning anything anymore.

Machine translation isn't super human yet. But yes, it's probably only a few years away.

Google Translate works great for having conversations with spanish speakers right now. Don't know how well it works for other languages.

Re: GPT-4 gets a B on my quantum computing final exam

#28
post #6

When I was growing up in the 2000s, it was required to learn a foreign language in school. I took Spanish but dropped the class after a year, thinking that when I grew up, computers would be able to translate text far better humans could. Would you believe it, transformers were invented ten years later. I wonder if it's worth learning anything anymore.

Even if machines could translate perfectly they wouldn't make you able to speak to people on the street in a foreign country. I mean, in X years maybe we'll have "real time" devices but they'd be extremely clunky unless they were working from brainwaves or something.

Re: GPT-4 gets a B on my quantum computing final exam

#29

Earlier quoted context omitted.

Machine translation isn't super human yet. But yes, it's probably only a few years away.

I don't get why people are so optimistic about machine translation. Computers can get explicit meaning across – that's obvious to anyone who understands linear algebra, information theory, and linguistics. But many aspects of translation (puns, tone, cultural context) aren't just about mapping from one vector space to another. A human, no matter how fluently bilingual, would have to think about the problem, and the c…

Have you actually used GPT-4 for translation?

Seriously all this talk about only getting explicit meaning across would be easily dispelled in an afternoon if you only bothered to try.

Re: GPT-4 gets a B on my quantum computing final exam

#30
post #25

This is obviously very cool, but at this point — who knows what I’ll say in a year — my concern with these LLMs is that they’re in the uncanny valley. Here’s one passing a very difficult test. Amazing! Now, rely on it to build a nuclear doohickey for a power station or a multi-billion dollar device for CERN or anything really and, well, no. So humans still have to check the output, and now we’re in that situation whe…

Also today humans need to check the output, code reviews etc.

Maybe in the near future humans can just focus on the reviewing and testing part.

Post reply on HN