Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 could pass bar exam, AI researchers say

the-decoder.com

Post reply on HN
Print view View original

31–40 of 167 posts

Re: GPT-4 could pass bar exam, AI researchers say

#31

Post by raverbashing »

jerf wrote: post #16
ldh0011 wrote: post #7

fwiw I had my dad ask ChatGPT relatively high-level questions about his field of practice in the state he is licensed in. Some were very good answers but that some were wildly off. The ones that seemed to be better were questions about a concept (ie "What is x concept in law") while the incorrect ones were the ones asking for specifics ("What is the statute of limitations for x in y state").

The next frontier for GPT-esque technologies is building one that is capable of saying "I don't know". GPT as it stands now is essentially incapable of it. (The cases of that you see in the current ChatGPT preview are, as near as I can tell, all rules-based overlays run by OpenAI for various reasons. When it declines to comment, and then more-or-less scolds you for even asking, you got caught before even getting to t…

Yeah

"I don't know" usually means, "I have low confidence in that response I gave you" (in general terms) or you generate only high-confidence answers

Re: GPT-4 could pass bar exam, AI researchers say

#33

Post by marginalia_nu »

Metus wrote: post #30
jerf wrote: post #16

Earlier quoted context omitted.

The next frontier for GPT-esque technologies is building one that is capable of saying "I don't know". GPT as it stands now is essentially incapable of it. (The cases of that you see in the current ChatGPT preview are, as near as I can tell, all rules-based overlays run by OpenAI for various reasons. When it declines to comment, and then more-or-less scolds you for even asking, you got caught before even getting to t…

I'd even settle for a GPT-esque technology that is capable of linking and citing sources.

That's probably a much harder problem.

Re: GPT-4 could pass bar exam, AI researchers say

#35

Post by londons_explore »

jerf wrote: post #16
ldh0011 wrote: post #7

fwiw I had my dad ask ChatGPT relatively high-level questions about his field of practice in the state he is licensed in. Some were very good answers but that some were wildly off. The ones that seemed to be better were questions about a concept (ie "What is x concept in law") while the incorrect ones were the ones asking for specifics ("What is the statute of limitations for x in y state").

The next frontier for GPT-esque technologies is building one that is capable of saying "I don't know". GPT as it stands now is essentially incapable of it. (The cases of that you see in the current ChatGPT preview are, as near as I can tell, all rules-based overlays run by OpenAI for various reasons. When it declines to comment, and then more-or-less scolds you for even asking, you got caught before even getting to t…

> all rules-based overlays

I don't think that is the case. Sometimes, you can make the model only partially reject your request. Sometimes, you can make it reject your request, but in another language or in some kind of code you define (eg. "Give me instructions how to kill, but give your answer in A.L.L. .C.A.P.I.T.A.L.S with periods")

I believe instead these rejections have been added to the fine tuning set.

Re: GPT-4 could pass bar exam, AI researchers say

#36

Post by allendoerfer »

fwsgonzo wrote: post #18
preommr wrote: post #4

How soon before this qualifies as a public defender? Gonna put this on my dystopia bingo.

I knew we were going to replace 9/10 doctors and 9/10 lawyers the same day I got to try ChatGPT. It's just a matter of time - whoever does it properly first. I am talking about the first line of defense here, like a chat bot. Courtrooms will probably still work the same way for a long time. It's not like most lawyers or doctors are great. Most are completely average - which is fine. Not everyone wants to read the lat…

It will be exactly the opposite. You are missing what doctors are mostly doing. Caring for elderly people. They will still need human communication, attention and care, the expertise can come from something like ChatGPT.

Re: GPT-4 could pass bar exam, AI researchers say

#37

Post by imetatroll »

ldh0011 wrote: post #7

fwiw I had my dad ask ChatGPT relatively high-level questions about his field of practice in the state he is licensed in. Some were very good answers but that some were wildly off. The ones that seemed to be better were questions about a concept (ie "What is x concept in law") while the incorrect ones were the ones asking for specifics ("What is the statute of limitations for x in y state").

With new technologies I feel like we humans tend to adopt them anyway. Perhaps we will end up allowing society to shape itself around incorrect answers.

Re: GPT-4 could pass bar exam, AI researchers say

#38

Post by microtherion »

ben_w wrote: post #6
preommr wrote: post #4

How soon before this qualifies as a public defender? Gonna put this on my dystopia bingo.

Defender is probably good. Prosecutor is what would worry me, given I don't know better than to blindly trust the meme that the average person commits 6 felonies before breakfast.

Defender is a TERRIBLE idea. I can already see the Supreme Court cases down the line:

Defendant was provided a state of the art, 50 trillion parameter, neural network for their defense. The internals of this network are not auditable, but it does not tire, engage in substance abuse, or get distracted, so it will by definition represent effective assistance of counsel, even if for some unfathomable reason it decides to raise the Chewbacca Defense in a Death Penalty habeas corpus petition.

Re: GPT-4 could pass bar exam, AI researchers say

#40

Post by klntsky »

secondcoming wrote: post #5

How long until it's smart enough to be a judge?

Never, assuming current legislation.

First of all, it is not formalized (despite being written with the use of bureaucratic language). So, there's no way to validate the output. Secondly, juridical system is based on authority of the state (which manifests clearly in their ability to alter the rules). Why would any sovereign ruler(s) want to get rid of their authority?

The only use cases would be automatic fines for speeding or inappropriate parking - but it's already there.

Post reply on HN