Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 could pass bar exam, AI researchers say

the-decoder.com

Post reply on HN
Print view View original

11–20 of 167 posts

Re: GPT-4 could pass bar exam, AI researchers say

#11

Post by mdp2021 »

secondcoming wrote: post #5

How long until it's smart enough to be a judge?

...After sharpness and judgement will be implemented.

--

Incidentally: there is an interesting video interview to Noam Chomsky and Gary Marcus on limits of current attempts at https://www.youtube.com/watch?v=PBdZi_JtV4c

...And Gary Marcus saying just before 7:00 that "something is missing" (understatement): ontology.

Gray Marcus: «...and these systems fall apart left and right».

Nice summary from Gary Marcus: «What they do is, they perpetuate past data - they don't really understand the world».

Re: GPT-4 could pass bar exam, AI researchers say

#12

Post by DisjointedHunt »

ldh0011 wrote: post #7

fwiw I had my dad ask ChatGPT relatively high-level questions about his field of practice in the state he is licensed in. Some were very good answers but that some were wildly off. The ones that seemed to be better were questions about a concept (ie "What is x concept in law") while the incorrect ones were the ones asking for specifics ("What is the statute of limitations for x in y state").

Because it doesn't presently have memory or look things up in a table or the internet.

You will notice that both are very easy fixes that computers have perfected in retrieval over the past 5 or so decades.

Re: GPT-4 could pass bar exam, AI researchers say

#13

Post by fnordpiglet »

ldh0011 wrote: post #7

fwiw I had my dad ask ChatGPT relatively high-level questions about his field of practice in the state he is licensed in. Some were very good answers but that some were wildly off. The ones that seemed to be better were questions about a concept (ie "What is x concept in law") while the incorrect ones were the ones asking for specifics ("What is the statute of limitations for x in y state").

I’m by far a layman in this respect but I feel like it’s the difference between conceptualizing and information retrieval. Further it feels like IR is a well researched area and by allowing the conceptualizing part access to a modern IR system would allow it to form searches, pull the IR results, sift them, and summarize them.

Re: GPT-4 could pass bar exam, AI researchers say

#16

Post by jerf »

ldh0011 wrote: post #7

fwiw I had my dad ask ChatGPT relatively high-level questions about his field of practice in the state he is licensed in. Some were very good answers but that some were wildly off. The ones that seemed to be better were questions about a concept (ie "What is x concept in law") while the incorrect ones were the ones asking for specifics ("What is the statute of limitations for x in y state").

The next frontier for GPT-esque technologies is building one that is capable of saying "I don't know". GPT as it stands now is essentially incapable of it.

(The cases of that you see in the current ChatGPT preview are, as near as I can tell, all rules-based overlays run by OpenAI for various reasons. When it declines to comment, and then more-or-less scolds you for even asking, you got caught before even getting to the model itself.)

Re: GPT-4 could pass bar exam, AI researchers say

#17

Post by laci37 »

ldh0011 wrote: post #7

fwiw I had my dad ask ChatGPT relatively high-level questions about his field of practice in the state he is licensed in. Some were very good answers but that some were wildly off. The ones that seemed to be better were questions about a concept (ie "What is x concept in law") while the incorrect ones were the ones asking for specifics ("What is the statute of limitations for x in y state").

I got the same feeling asking ChatGPT about some basic logic and maths concepts. IMO GPT can find the relevant training data to regurgitate, but i don't think it connects concepts.

Re: GPT-4 could pass bar exam, AI researchers say

#18

Post by fwsgonzo »

preommr wrote: post #4

How soon before this qualifies as a public defender? Gonna put this on my dystopia bingo.

I knew we were going to replace 9/10 doctors and 9/10 lawyers the same day I got to try ChatGPT. It's just a matter of time - whoever does it properly first. I am talking about the first line of defense here, like a chat bot. Courtrooms will probably still work the same way for a long time.

It's not like most lawyers or doctors are great. Most are completely average - which is fine. Not everyone wants to read the latest research, and instead just go home and "turn off" after work. That said, most people would like to visit a doctor who keeps up with information, and doesn't tell you to do mental exercises when you have IBS.

The trend continues just like before. Less accountants, less bank tellers, less store clerks. We no longer have 10 people assembling that globus with painted glue. I do wonder what the ratio of workers to machines is now?

Re: GPT-4 could pass bar exam, AI researchers say

#19

Post by HDThoreaun »

preommr wrote: post #4

How soon before this qualifies as a public defender? Gonna put this on my dystopia bingo.

I could easily see myself preferring an AI public defender to an overworked lawyer with 50 other cases in the next 2 weeks. What's dystopian is the current situation.

Re: GPT-4 could pass bar exam, AI researchers say

#20

Post by sdenton4 »

moneywoes wrote: post #10

Wonder how data biases will surface

The fun part here is that most humans in the legal profession carry pretty extreme biases, judges included... The hope for legal ai is that you could progressively improve the biases, instead of waiting for N years for a bad judge to retire same maaaaybe get replaced by someone better.
Post reply on HN