Live data from Hacker News

GPT-4 could pass bar exam, AI researchers say

the-decoder.com

121–130 of 167 posts

Re: GPT-4 could pass bar exam, AI researchers say

#121
post #61

Earlier quoted context omitted.

We were perhaps a bit too enamored with the idea that it was intellect that made us unique, and thus knowledge workers would be the last to be replaced. Pouring our brains out by the Petabytes for neural networks to pick them up made the economics just work for an AI industrial revolution to start from there.

No, we were enamored with the idea that intelligence was well distributed between people, as if following Descartes' massive incipit "Good sense must be the best distributed thing in the world, given that nobody seems to be asking for more". Inability to recognize intelligence is and will be devastating.

> Inability to recognize intelligence is and will be devastating.

It's a pop-culture quote from a movie that was no masterpiece, I know, but "I, Robot" presented in two sentences an argument for having more sober expectations on what machine intelligence could be capable of, and of our own

> Detective Del Spooner: "Can a robot write a symphony? Can a robot turn a… canvas into a beautiful masterpiece?"

> Sonny: "Can you?"

We're discrediting the capabilities of current machine learning models for being unable of producing the thoughts that many, many people are unable to either.

Alright, so the models are not at the level that us HN philosopher kings hold ourselves to be, and they won't be Senior Architects of distributed systems or what have you very soon, but what does it say about Average Joe, slightly-above-Average Joe, and their economic prospects? Specially since in the West and much of the developing world, we were taking solace in the idea that a service economy comprised of knowledge workers would provide plenty of opportunities on a political and economic landscape where manufacture was gone, or had never arrived.

Re: GPT-4 could pass bar exam, AI researchers say

#122
post #16

Earlier quoted context omitted.

The next frontier for GPT-esque technologies is building one that is capable of saying "I don't know". GPT as it stands now is essentially incapable of it. (The cases of that you see in the current ChatGPT preview are, as near as I can tell, all rules-based overlays run by OpenAI for various reasons. When it declines to comment, and then more-or-less scolds you for even asking, you got caught before even getting to t…

You can get a measurable improvement by prompting GPT specifically with an instruction to say "I don't know" if it's unsure. It'll still go off the rails sometimes. More important would be a model that cites hard facts.

"You can get a measurable improvement by prompting GPT specifically with an instruction to say "I don't know" if it's unsure."

That won't work. It's easy to get the model to say "I don't know" with the correct prompt, but since the model doesn't even have "knowing" in it, it's just outputting "I don't know" based on a random roll of the probability of its training text having someone said "I don't know". The text "I don't know" won't actually correspond to whether the model knows something or not.

And while we can get into a lengthy and philosophical debate about what it takes to "know" something, my previous paragraph is fairly robust to any sensible definition of "knowing". Write your favorite definition of "knowing" something, then look at the architecture of what GPT actually is on the inside, and tell me if it can actually "know" something based on that architecture. You can of course write the more-or-less begging the question "knowing is a matter of producing correct text when prompted about some fact", but I would have numerous questions around applying that definition of "knowing" to anything other than GPT, or what it means when GPT confidently confabulates something. Don't forget to write your definition and do your analysis in the context not just of GPT outputting the correct capital of Oregon when prompted, but the way it will confidently discuss all sorts of things that don't exist. Your definition should be able to account for some sort of difference between confidently outputting correct data and the way it will equally confidently output complete fiction, and indicate some manner in which GPT has some sort of state difference that indicates it is somehow "aware" of when it is doing one or the other. Because I would say if it can't "tell" if it's confidently emitting facts or confidently emitting fiction that there is a very important and real sense it doesn't really "know" the facts, either. (And I absolutely would apply that standard to humans without question; if you can't tell if you're making stuff up or not, you don't know whatever it is you're talking about.)

Re: GPT-4 could pass bar exam, AI researchers say

#123
post #74

Earlier quoted context omitted.

It will be exactly the opposite. You are missing what doctors are mostly doing. Caring for elderly people. They will still need human communication, attention and care, the expertise can come from something like ChatGPT.

You do realize that a large part of of patient interaction is from nurses and physician's assistants and not doctors, right?

I realize that this highly depends on the country and its medical system.

I live in a country where there is universal healthcare and you can just book an appointment with any doctor of any kind without going through any gatekeeping.

I think the placebo effect is at work here: While I don't doubt any nurse could handle 95% of the cases a general practitioner has to face here every day, elderly patients want that expert opinion from the guy they value highly and trust in so much.

Re: GPT-4 could pass bar exam, AI researchers say

#124

So, how new knowledge would be created? GPT has no reasoning capability. So, as time goes on, information massive(s) will be filled with GPT-X made up answers. It means GPT-X+1 will be trained on GPT-X generated data. So, without reasoning, how this thing will work in perspective?

I wouldn't assume that future versions are going to work the same way past versions did.

Re: GPT-4 could pass bar exam, AI researchers say

#125

I feel like I can now see the event horizon of commoditized intelligence. No idea what society (is "society" even the right word? Who knows) is going to look like on the other side of it, but it is going to be wildly different. Perhaps a brief period where everyone is using an AI to do their job, uh, I mean, assist their work, but beyond that it's unknowable. Moreover, this looks like it is going to be happening soon…

GPT has no reasoning ability, it has billions of parameters that make it pretend it has it, purely going off of previously digested material. As long as it comes across some reasoning process that have not been seen before in the training wordset, which can be as easy as a middle school math question, it fails. Because it has no ability to extrapolate logic. If it manages to pass Bar test, that says more about the Ba…

[deleted]

Re: GPT-4 could pass bar exam, AI researchers say

#126
post #61

Earlier quoted context omitted.

No, we were enamored with the idea that intelligence was well distributed between people, as if following Descartes' massive incipit "Good sense must be the best distributed thing in the world, given that nobody seems to be asking for more". Inability to recognize intelligence is and will be devastating.

> Inability to recognize intelligence is and will be devastating. It's a pop-culture quote from a movie that was no masterpiece, I know, but "I, Robot" presented in two sentences an argument for having more sober expectations on what machine intelligence could be capable of, and of our own > Detective Del Spooner: "Can a robot write a symphony? Can a robot turn a… canvas into a beautiful masterpiece?" > Sonny: "Can y…

"Learn to code" isn't such valuable advice for the Average Joe anymore, is it?

Re: GPT-4 could pass bar exam, AI researchers say

#127

Earlier quoted context omitted.

GPT has no reasoning ability, it has billions of parameters that make it pretend it has it, purely going off of previously digested material. As long as it comes across some reasoning process that have not been seen before in the training wordset, which can be as easy as a middle school math question, it fails. Because it has no ability to extrapolate logic. If it manages to pass Bar test, that says more about the Ba…

You are implying either: * Understanding complex language does not require logic/reasoning, * There are infinitely many forms of logic/reasoning or at least more than those existing in a vast training set. Neither of which is likely true. What do you think of the Minerva system, which can solve multi-step quantitative reasoning questions better than many competent students and most adults? https://ai.googleblog.com/2…

The LSAT tests formal logic. Some of it is complicated. Much less of it is required for the practice of law.

Src: scored 99.8th percentile on LSAT, tutored it, now working at major law firm

Re: GPT-4 could pass bar exam, AI researchers say

#128

I feel like I can now see the event horizon of commoditized intelligence. No idea what society (is "society" even the right word? Who knows) is going to look like on the other side of it, but it is going to be wildly different. Perhaps a brief period where everyone is using an AI to do their job, uh, I mean, assist their work, but beyond that it's unknowable. Moreover, this looks like it is going to be happening soon…

GPT has no reasoning ability, it has billions of parameters that make it pretend it has it, purely going off of previously digested material. As long as it comes across some reasoning process that have not been seen before in the training wordset, which can be as easy as a middle school math question, it fails. Because it has no ability to extrapolate logic. If it manages to pass Bar test, that says more about the Ba…

I've already been using GPT and ChatGPT to much success for my work.

Yes, it doesn't have reasoning ability, but being able to manage knowledge and information in the way that these models can is still an amazing feat.

Re: GPT-4 could pass bar exam, AI researchers say

#130

Earlier quoted context omitted.

I feel a bit like this with the whole firestorm around AI artwork as well— it's been a big wakeup call to people who have been creating using technology-assisted workflows for decades, but still felt in their gut that they were bringing something unique to the table and were therefore "safe" from being completely automated away. That hitting the button for magic eraser or magic lasso or magic color correction was som…

Hard to tell, other knowledge workers and people in creative industries were already squeezed, designers for instance have had a tough time for a very long time. Will things change, politically, because now marketers and Software developers join those ranks, for instance? Programming was an outlet, if not a gold rush, for many people as the basic technical skills to create Software with the already sophisticated tool…

Yeah, well, and even looking to the immediate subject of the article... like, whether your lawyer is going to become a bot in ten years, a huge amount of what used to be part of the legal practice has already been automated away in terms of the research side, nevermind specialized firms that just crank through bog-standard family-law or property-transfer cases by plugging the relevant details into an Excel template.

Basically it's the same story as everywhere else, where technological augmentation has already created a huge squeeze, and now suddenly even the senior people are wondering if the writing is on the wall for them too.

Post reply on HN