Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 could pass bar exam, AI researchers say

the-decoder.com

Post reply on HN
Print view View original

91–100 of 167 posts

Re: GPT-4 could pass bar exam, AI researchers say

#91

Post by lukko »

fwsgonzo wrote: post #18
preommr wrote: post #4

How soon before this qualifies as a public defender? Gonna put this on my dystopia bingo.

I knew we were going to replace 9/10 doctors and 9/10 lawyers the same day I got to try ChatGPT. It's just a matter of time - whoever does it properly first. I am talking about the first line of defense here, like a chat bot. Courtrooms will probably still work the same way for a long time. It's not like most lawyers or doctors are great. Most are completely average - which is fine. Not everyone wants to read the lat…

Hmm, I think this misunderstands what doctors do... It is not about making elaborate or obscure diagnoses, passing exams or applying the latest research given that one is always working in a resource-limited system. It is about treating illness in the context of a person, their beliefs and their lifestyle, with sensitivity and compassion. The doctor-patient relationship is complex and very human, and doctors in some form will be involved even if they are at some point supported by AI.

Re: GPT-4 could pass bar exam, AI researchers say

#92

Post by morsecodist »

I am sorry but this title is click bait. These researchers ran GPT-3.5 on only the multiple choice sections of the Bar and it passed 2/7 sections. Is this really impressive? Absolutely. But the only element of the article that is about GPT-4 potentially passing the Bar is one paragraph near the end:

> According to the researchers, the history of large language model development strongly suggests that such models could soon pass all categories of the MBE portion of the Bar Exam. Based on anecdotal evidence related to GPT-4 and LAION’s Bloom family of models, the researchers believe this could happen within the next 18 months.

GPT-4 could potentially pass the Bar, it could potentially do a lot of things. But by their own admission the researchers have no hard evidence for this.

Re: GPT-4 could pass bar exam, AI researchers say

#93

Post by mdp2021 »

flatline wrote: post #82
Aperocky wrote: post #53

Earlier quoted context omitted.

GPT has no reasoning ability, it has billions of parameters that make it pretend it has it, purely going off of previously digested material. As long as it comes across some reasoning process that have not been seen before in the training wordset, which can be as easy as a middle school math question, it fails. Because it has no ability to extrapolate logic. If it manages to pass Bar test, that says more about the Ba…

It does have some ability to extrapolate to new problems, provided its training corpus has reasonably close coverage. It is not going to be making new scientific discoveries or insights but then neither are most people. With a sufficiently large training set I think these models can achieve human parity for a subset of language generation tasks , and be effectively of human intelligence. They nearly already have. It…

> a subset of language generation tasks ... if the outcome is the same

Which tasks? The output of some crafter with a limited number of modules, and of somebody who can assess the output, cannot be the same - unless you would have accepted the output of a mentally crippled entity in the first place.

Re: GPT-4 could pass bar exam, AI researchers say

#94

Post by Aperocky »

sebzim4500 wrote: post #66
Aperocky wrote: post #53

Earlier quoted context omitted.

GPT has no reasoning ability, it has billions of parameters that make it pretend it has it, purely going off of previously digested material. As long as it comes across some reasoning process that have not been seen before in the training wordset, which can be as easy as a middle school math question, it fails. Because it has no ability to extrapolate logic. If it manages to pass Bar test, that says more about the Ba…

>As long as it comes across some reasoning process that have not been seen before in the training wordset, which can be as easy as a middle school math question Is this true even if you tell it to show its working? In my experience that drastically improves its ability to do math problems.

At the end of the day it's still a language prediction model.

Which means whatever apparent logic you're getting out of it is from text that it has learned. not reasoning embedded within those text, but the actual text itself.

Re: GPT-4 could pass bar exam, AI researchers say

#95

Post by evouga »

microtherion wrote: post #24

IF it could (I wouldn't know one way or the other), I'd consider that a damning indictment of the Bar Exam failing to test for sentience, rather than evidence of GPT-4 having attained the same.

Right. Would we be impressed if a layman could pass the bar, given infinite time and access to the entire Internet (including the copious amount of bar exam study guides and worked example problems)? If not why are we impressed that a language model trained on that data can?

Meanwhile when I ask ChatGPT which of six numbers are odd, it confidently reports a mix of even numbers, odd numbers, and letters.

This is a fun milestone but the angst above about the “end of commoditized intelligence” etc. is unwarranted.

Re: GPT-4 could pass bar exam, AI researchers say

#96

Post by Iwan-Zotow »

So, how new knowledge would be created?

GPT has no reasoning capability. So, as time goes on, information massive(s) will be filled with GPT-X made up answers. It means GPT-X+1 will be trained on GPT-X generated data. So, without reasoning, how this thing will work in perspective?

Re: GPT-4 could pass bar exam, AI researchers say

#97

Post by allochthon »

czzr wrote: post #67

There’s a gap between passing the bar exam and actually practicing law - I’m pretty certain that I (someone with no legal training whatsoever) could pass the bar exam if you gave me unlimited access to the internet and a couple of additional hours to write the test. However, I don’t think that would make me an effective lawyer. Ultimately standardised tests are proxy measurements of legal ability - it’s easy to see h…

> I do expect that even unreliable versions will be very useful tools for practicing lawyers, though.

Agreed. It's like being able to call up a map on Google Maps for an area that you're already familiar with. The map can help you remember things about the area and terrain that you might not have recalled right away. A kind of cognitive aid.

Re: GPT-4 could pass bar exam, AI researchers say

#98

Post by morsecodist »

softwaredoug wrote: post #75

The Bar Exam is multiple choice, right? This isn't grading some freeform essay or generating arbitrary legal opinion. It's answering from a limited set of answers. IMO it's cool, but not THAT shocking given what we've seen from ChatGPT? Especially given GPT 3.5 is only 17% below human test takers?

From the article it looks like there are multiple choice and written sections but they only ran the model on the multiple choice portion.

Re: GPT-4 could pass bar exam, AI researchers say

#99

Post by evouga »

kuschku wrote: post #89
Aperocky wrote: post #53

Earlier quoted context omitted.

GPT has no reasoning ability, it has billions of parameters that make it pretend it has it, purely going off of previously digested material. As long as it comes across some reasoning process that have not been seen before in the training wordset, which can be as easy as a middle school math question, it fails. Because it has no ability to extrapolate logic. If it manages to pass Bar test, that says more about the Ba…

Most jobs today don't need novel reasoning. This is the equivalent of the steam machine for intelligence. During the industrialization, machines did not replace all jobs, but they replaced or changed most jobs. The same will happen here. A typical office job will have a few hours a week of actual, intensive thought. The vast majority of time will be spent doing simple, repetitive work. This work can be automated, or…

> You could even have it write unit tests based on the JIRA ticket description.

This is a wonderful point: writing unit tests is exactly the kind of mind-numbing tedium that I'm super excited to automate away.

Re: GPT-4 could pass bar exam, AI researchers say

#100

Post by Aperocky »

ragebol wrote: post #70
ok123456 wrote: post #55

Earlier quoted context omitted.

Most people's reasoning ability functions at this level.

Which should be worrying. It's also why most people don't pass (nor try) the bar exam.

Not really, this demonstrated how bar exam don't require a large amount of reasoning, but lots of recitations.

Humans are bad at reciting things, but a lot better (compared to GPT) at reasoning.

Post reply on HN