GPT-4 could pass bar exam, AI researchers say • Hacker News Board

Re: GPT-4 could pass bar exam, AI researchers say

#21

Post by nopinsight » Tue, Jan 03, 2023, 3:08 PM UTC

Related: "Large Language Models Encode Clinical Knowledge" https://arxiv.org/abs/2212.13138

"On the MedQA dataset consisting of USMLE style questions with 4 options, our Flan-PaLM 540B model achieved a multiple-choice question (MCQ) accuracy of 67.6%..."

"The percentages of correctly answered items required to pass varies by Step and from form to form within each Step. However, examinees typically must answer approximately 60 percent of items correctly to achieve a passing score." -- https://www.usmle.org/bulletin-information/scoring-and-score...

.

It seems like the models in the paper could pass USMLE already.

Some tests suggest that Med-PaLM is close to human clinicians in many aspects, incl reasoning (Figures 6-7). Other tests show that Med-PaLM still returns inappropriate/incorrect results much more often than clinicians do, however (Figure 8).

Re: GPT-4 could pass bar exam, AI researchers say

#22

Post by jacquesm » Tue, Jan 03, 2023, 3:09 PM UTC

DisjointedHunt wrote: post #12

ldh0011 wrote: post #7

fwiw I had my dad ask ChatGPT relatively high-level questions about his field of practice in the state he is licensed in. Some were very good answers but that some were wildly off. The ones that seemed to be better were questions about a concept (ie "What is x concept in law") while the incorrect ones were the ones asking for specifics ("What is the statute of limitations for x in y state").

Because it doesn't presently have memory or look things up in a table or the internet. You will notice that both are very easy fixes that computers have perfected in retrieval over the past 5 or so decades.

Just stick Google's pre-search tools in front of the current version and it would solve a large chunk of those problems. The right tool for the job, essentially. After all, you wouldn't ask your English professor to solve a math problem either.

Re: GPT-4 could pass bar exam, AI researchers say

#23

Post by Aardwolf » Tue, Jan 03, 2023, 3:10 PM UTC

I envisioned a cocktail shaking robot, but apparently Bar Exam is an exam for US lawyers

Re: GPT-4 could pass bar exam, AI researchers say

#24

Post by microtherion » Tue, Jan 03, 2023, 3:10 PM UTC

IF it could (I wouldn't know one way or the other), I'd consider that a damning indictment of the Bar Exam failing to test for sentience, rather than evidence of GPT-4 having attained the same.

Re: GPT-4 could pass bar exam, AI researchers say

#25

Post by Workaccount2 » Tue, Jan 03, 2023, 3:11 PM UTC

I feel like I can now see the event horizon of commoditized intelligence. No idea what society (is "society" even the right word? Who knows) is going to look like on the other side of it, but it is going to be wildly different. Perhaps a brief period where everyone is using an AI to do their job, uh, I mean, assist their work, but beyond that it's unknowable.

Moreover, this looks like it is going to be happening sooner rather than later.

Re: GPT-4 could pass bar exam, AI researchers say

#26

Post by jacquesm » Tue, Jan 03, 2023, 3:12 PM UTC

ben_w wrote: post #6

preommr wrote: post #4

How soon before this qualifies as a public defender? Gonna put this on my dystopia bingo.

Defender is probably good. Prosecutor is what would worry me, given I don't know better than to blindly trust the meme that the average person commits 6 felonies before breakfast.

And if the cost of prosecution falls then more and more of those 6 felonies will end up prosecuted. The same happened with speed cameras, initially it was to reduce accidents, now it is just another income stream (which I'm sure still reduces accidents, but that's no longer the main reason they are out there).

Re: GPT-4 could pass bar exam, AI researchers say

#27

Post by SV_BubbleTime » Tue, Jan 03, 2023, 3:12 PM UTC

moneywoes wrote: post #10

Wonder how data biases will surface

This is what I found immediately interesting about ChatGPT.

I asked about controversial topics. Its answers didn’t seem like biases that were programmed in, but rather it took traditional media and gave it more weight than what turned out to be the truth only accepted much later on and still against a media retelling.

I lost a lot of faith in it knowing it was more CNN than careful deliberating AI.

Re: GPT-4 could pass bar exam, AI researchers say

#28

Post by charcircuit » Tue, Jan 03, 2023, 3:13 PM UTC

How do they know GPT-4 will be enough to let it pass? Is there even a big enough difference in the training data for it to improve in the areas it was struggling with?

Re: GPT-4 could pass bar exam, AI researchers say

#29

Post by izacus » Tue, Jan 03, 2023, 3:13 PM UTC

laci37 wrote: post #17

ldh0011 wrote: post #7

fwiw I had my dad ask ChatGPT relatively high-level questions about his field of practice in the state he is licensed in. Some were very good answers but that some were wildly off. The ones that seemed to be better were questions about a concept (ie "What is x concept in law") while the incorrect ones were the ones asking for specifics ("What is the statute of limitations for x in y state").

I got the same feeling asking ChatGPT about some basic logic and maths concepts. IMO GPT can find the relevant training data to regurgitate, but i don't think it connects concepts.

I mean, it's a bullshit generator. It'll grab whatever it find in training set that kinda fits the topic and make sure it hits the word count - like a lazy student before deadline.

And that's also the result - sometimes it hits something good. Sometimes it spews up utter crock and it doesn't have any notion or understanding of the difference.

However, it does look good to the lazy and uninformed and it'll soon render judgemenets about your livelihood in the future. The same type of people who thought putting an AI in control of Teslas and copyright enforcement on YouTube will put this thing in control of your health and punishment very soon as well.

Re: GPT-4 could pass bar exam, AI researchers say

#30

Post by Metus » Tue, Jan 03, 2023, 3:14 PM UTC

jerf wrote: post #16

ldh0011 wrote: post #7

fwiw I had my dad ask ChatGPT relatively high-level questions about his field of practice in the state he is licensed in. Some were very good answers but that some were wildly off. The ones that seemed to be better were questions about a concept (ie "What is x concept in law") while the incorrect ones were the ones asking for specifics ("What is the statute of limitations for x in y state").

The next frontier for GPT-esque technologies is building one that is capable of saying "I don't know". GPT as it stands now is essentially incapable of it. (The cases of that you see in the current ChatGPT preview are, as near as I can tell, all rules-based overlays run by OpenAI for various reasons. When it declines to comment, and then more-or-less scolds you for even asking, you got caught before even getting to t…

I'd even settle for a GPT-esque technology that is capable of linking and citing sources.