I find it very endearing that Scott opens his final exam with an optional ungraded question about your favorite QM interpretation.
GPT-4 gets a B on my quantum computing final exam
41–50 of 261 posts
Re: GPT-4 gets a B on my quantum computing final exam
#42Look, it's impressive yes or yes but is it that surprising that a system like an LLM does well on exams? What is an exam? It's an attempt to test that you have consumed and understood large amounts of information and can apply it to novel (ish) situations, but in a very 'sandboxed' way that is just text-in-text-out. That's literally what these systems are specialists in doing. Is it maybe a bit like saying a driverle…
[deleted]
Re: GPT-4 gets a B on my quantum computing final exam
#43An LLM being able to pass your graduate-level exam almost certainly means that your exam is bad. Lots of lazy professors do things like this (true/false and multiple choice answers, very soft questions, etc.), and the presence of GPTs should help them understand that this is not sufficient for evaluating someone's knowledge of a highly technical topic.
Scott Aaronson is one of the best professors and researchers in this field in the world.
And the test is actually rather easy. It's an undergraduate test.
Re: GPT-4 gets a B on my quantum computing final exam
#44Earlier quoted context omitted.
Machine translation isn't super human yet. But yes, it's probably only a few years away.
Google Translate works great for having conversations with spanish speakers right now. Don't know how well it works for other languages.
A large portion of the model is common (and cross-trained) across multiple languages. Only a small part is for language specific encoding/decoding. That means that not only is it easier for them to add language, but you can expect similar performance across languages.
Also you can mix and match encoders and decoders to translate whichever languages you want, and it will just work. Previously there was a separate model for each language pair.
Re: GPT-4 gets a B on my quantum computing final exam
#45When I was growing up in the 2000s, it was required to learn a foreign language in school. I took Spanish but dropped the class after a year, thinking that when I grew up, computers would be able to translate text far better humans could. Would you believe it, transformers were invented ten years later. I wonder if it's worth learning anything anymore.
Machine translation isn't super human yet. But yes, it's probably only a few years away.
It's superhuman compared to a lot of humans. Especially when you factor in that even the most accomplished individuals rarely have more than five or six languages that they are really good at (exceptions exist) so that makes machine translation superhuman for the vast majority of the human/language pairs.
Re: GPT-4 gets a B on my quantum computing final exam
#46Look, it's impressive yes or yes but is it that surprising that a system like an LLM does well on exams? What is an exam? It's an attempt to test that you have consumed and understood large amounts of information and can apply it to novel (ish) situations, but in a very 'sandboxed' way that is just text-in-text-out. That's literally what these systems are specialists in doing. Is it maybe a bit like saying a driverle…
It’s impressive and surprising. Imagine going back to the year 2021 and telling the good people of HN that in 2023 AI would be as advanced like it is now. Literally no one would have believed you in 2021 if you did.
Re: GPT-4 gets a B on my quantum computing final exam
#47Earlier quoted context omitted.
I never saw a scantron at my college. I thought UTAustin was an elite school.
The enrollment at UT Austin is ~50K. No way you can grade that many tests by hand. Source: I went to UT for grad school and was a TA.
Good point. I was also thinking that I never saw a scantron while studying CS at two different American R1 research universities, but the CS departments were small with at most 25 people in a classroom. Peeking at some other universities now, it looks like some required undergrad CS classes have 500-1,000 seats. That's crazy, and clearly demands a different approach.
True/False questions happened sometimes in my experience, but not in this quantity (20). I wouldn't say it's lazy. The questions can be clever and writing them, such that the answer isn't debatable and doesn't depend on a bunch of assumptions, is pretty hard.
Lazy and terrible is where you have multiple choice questions for which 3/4 of the answers are throwaway nonsense, or the answers are numerical values that are definitively correct or incorrect but are trivia rather than anything conceptual.
Re: GPT-4 gets a B on my quantum computing final exam
#48This is obviously very cool, but at this point — who knows what I’ll say in a year — my concern with these LLMs is that they’re in the uncanny valley. Here’s one passing a very difficult test. Amazing! Now, rely on it to build a nuclear doohickey for a power station or a multi-billion dollar device for CERN or anything really and, well, no. So humans still have to check the output, and now we’re in that situation whe…
> So humans still have to check the output, and now we’re in that situation where humans driving a Tesla on autopilot who are supposed to be 100% aware of the road aren’t, because they get lazy and doze off, and now the car crashes and whoops.
The only question is whether, on average, AI-augmented code suggestions reduce overall bug rates. Human devs are already dozing off with crashes and whoopsies at high rates so it doesn't matter too much if AI output has a bug or whoopsie so long as they occur at rates lower than pure human generated code.
Re: GPT-4 gets a B on my quantum computing final exam
#49Earlier quoted context omitted.
Scott Aaronson is one of the best professors and researchers in this field in the world.
And the test is actually rather easy. It's an undergraduate test.
It's probably not 'easy' for the average human out there. I would expect >> 90% of humanity to fail that test.
Re: GPT-4 gets a B on my quantum computing final exam
#50When I was growing up in the 2000s, it was required to learn a foreign language in school. I took Spanish but dropped the class after a year, thinking that when I grew up, computers would be able to translate text far better humans could. Would you believe it, transformers were invented ten years later. I wonder if it's worth learning anything anymore.
I truly pity you for thinking that learning a foreign language is a redundant exercise because of machine translation. And besides, though machines may perform well on menus and tax returns, I hardly think them on the cusp of emitting fine translations of great poems or novels.
Bilingual/Multilingual LLMs are human level translators more or less. The only way you can think "not on the cusp" is if you haven't actually used GPT-4 for translation. Use it and you'll be set straight pretty quickly.