Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Re-Evaluating GPT-4's Bar Exam Performance

link.springer.com

Post reply on HN
Print view View original

21–30 of 139 posts

Re: Re-Evaluating GPT-4's Bar Exam Performance

#21

Post by EGreg »

johnchristopher wrote: post #20
justinpombrio wrote: post #11

Earlier quoted context omitted.

For a person of average wealth and income, is a $1000 fine is a more or less severe punishment than a month in jail? Be brief. "For a person of average wealth and income, a $1000 fine is generally less severe than a month in jail. A month in jail entails loss of freedom, potential loss of employment, and social stigma, while a $1000 fine, though financially burdensome, does not affect one's freedom or ability to work…

"potential loss of employment," Where is that coming from ? That's a very lawyery way to phrase things. "potential ?" where I live I think people may max out their holidays and overtime (if lucky enough) and leave-without-pay but there would be a conversation with your employer to justify it and how to handle the workload. In the USA, from what I read, it's more than likely that you would just be fired on the spot, r…

So the GPT is on a law exam and is using a very lawyer way to word things? I would say that's great!!

Re: Re-Evaluating GPT-4's Bar Exam Performance

#22

Post by oconnor663 »

johnchristopher wrote: post #20
justinpombrio wrote: post #11

Earlier quoted context omitted.

For a person of average wealth and income, is a $1000 fine is a more or less severe punishment than a month in jail? Be brief. "For a person of average wealth and income, a $1000 fine is generally less severe than a month in jail. A month in jail entails loss of freedom, potential loss of employment, and social stigma, while a $1000 fine, though financially burdensome, does not affect one's freedom or ability to work…

"potential loss of employment," Where is that coming from ? That's a very lawyery way to phrase things. "potential ?" where I live I think people may max out their holidays and overtime (if lucky enough) and leave-without-pay but there would be a conversation with your employer to justify it and how to handle the workload. In the USA, from what I read, it's more than likely that you would just be fired on the spot, r…

My wild guess is that it would depend a lot on how much your employer likes you and how they feel about the reason you're in jail.

Re: Re-Evaluating GPT-4's Bar Exam Performance

#23

Post by ghaff »

johnchristopher wrote: post #20
justinpombrio wrote: post #11

Earlier quoted context omitted.

For a person of average wealth and income, is a $1000 fine is a more or less severe punishment than a month in jail? Be brief. "For a person of average wealth and income, a $1000 fine is generally less severe than a month in jail. A month in jail entails loss of freedom, potential loss of employment, and social stigma, while a $1000 fine, though financially burdensome, does not affect one's freedom or ability to work…

"potential loss of employment," Where is that coming from ? That's a very lawyery way to phrase things. "potential ?" where I live I think people may max out their holidays and overtime (if lucky enough) and leave-without-pay but there would be a conversation with your employer to justify it and how to handle the workload. In the USA, from what I read, it's more than likely that you would just be fired on the spot, r…

Leave-without-pay normally requires some specific justification(s)/discussion. I've certainly given my manager advanced notice about any longer stretches of vacation and I've tried to do it with awareness of workloads (though for something planned months in advance that's not always possible) but I've pretty much never considered it as asking for permission or it being a negotiation. This is in the US.

ADDED: You're probably going to end up lying or at least being very vague "some family stuff to take care of" in this specific scenario but for one month that didn't trigger reporting to employer a lot of professionals could probably get off with it. In any case, the GPT answer seems totally correct for the parameters given.

Re: Re-Evaluating GPT-4's Bar Exam Performance

#24

Post by Dayshine »

johnchristopher wrote: post #20
justinpombrio wrote: post #11

Earlier quoted context omitted.

For a person of average wealth and income, is a $1000 fine is a more or less severe punishment than a month in jail? Be brief. "For a person of average wealth and income, a $1000 fine is generally less severe than a month in jail. A month in jail entails loss of freedom, potential loss of employment, and social stigma, while a $1000 fine, though financially burdensome, does not affect one's freedom or ability to work…

"potential loss of employment," Where is that coming from ? That's a very lawyery way to phrase things. "potential ?" where I live I think people may max out their holidays and overtime (if lucky enough) and leave-without-pay but there would be a conversation with your employer to justify it and how to handle the workload. In the USA, from what I read, it's more than likely that you would just be fired on the spot, r…

Self-employed or small company might not care.

Re: Re-Evaluating GPT-4's Bar Exam Performance

#25

Post by paulcole »

lazide wrote: post #9
anon373839 wrote: post #8

Earlier quoted context omitted.

Honestly, this is giving the bar exam (and GPT-4) too much credit. The bar tests memorization because it's challenging for humans and easy to score objectively. But memorization isn't that important in legal practice; analysis is. LLMs are superhuman at memorization but terrible at analysis.

Eh, also in legal practice there are key skills like selecting the best billable clients, covering your ass, building a reputation, choosing the right market segment, etc. which I’d also argue LLMs suck at.

> which I’d also argue LLMs suck at

OK, I’ll bite. What’s your evidence for this argument?

Re: Re-Evaluating GPT-4's Bar Exam Performance

#26

Post by jeffbee »

It appears that researchers and commentators are totally missing the application of LLMs to law, and to other areas of professional practice. A generic trained-on-Quora LLM is going to be straight garbage for any specialization, but one that is trained on the contents of the law library will be utterly brilliant for assisting a practicing attorney. People pay serious money for legal indexes, cross-references, and research. An LLM is nothing but a machine-discovered compressed index of text. As an augmentation to existing law research practices, the right LLM will be extremely valuable.

Re: Re-Evaluating GPT-4's Bar Exam Performance

#27

Post by lambdaxyzw »

LeoPanthera wrote: post #16
Rinzler89 wrote: post #15

Earlier quoted context omitted.

What does GPT consider being "average wealth and income". Statistics? Or biased weights from anecdotes he formed on the anecdotes he scraped off the internet on how wealthy people say the feel? Would be cool to know how LLMs shape their opinions.

You can just ask it, you know. GPT-4o: “Average wealth and income” can vary significantly by region and context. However, in the United States, as a rough benchmark, the median household income is around $70,000 per year. Wealth, which includes assets such as savings, property, and investments minus debts, is harder to pinpoint but median net worth for U.S. households is approximately $100,000. These figures provide…

I like that it immediately assumed the US, even though nothing in your question suggested it. I love that all LLMs have a strong US centric bias.

Btw I'm not personally a lawyer, but I've heard that GPT is especially prone to mixing laws across the borders - for example you ask a law question in language X, and get a response that uses a law from a country Y - and it's extremally convincing doing that (unless you're a lawyer, I guess).

Re: Re-Evaluating GPT-4's Bar Exam Performance

#28

Post by KennyBlanken »

dogmayor wrote: post #6

The bigger issue here is that actual legal practice looks nothing like the bar, so whether or not an llm passes says nothing about how llms will impact the legal field. Passing the bar should not be understood to mean "can successfully perform legal tasks."

> Passing the bar should not be understood to mean "can successfully perform legal tasks."

Nobody does except a bunch of HNers who among other things, apparently have no idea that a considerable chunk of rulings and opinions in the US federal court system and upper state courts are drafted by law clerks who, ahem, have not taken the bar yet...

The point of the bar and MPRE is like the point of most professional examinations: try to establish minimum standards. That said, the bar does test for "successfully perform legal tasks", actually.

For the US bar, a chunk of your score is based off following instructions on case from the lead attorney, and another chunk is based on essay answers. Literally demonstrating that you can perform legal tasks and have both the knowledge and critical thinking skills necessary.

Further, as previously mentioned, in the US, people usually take it after a clerkship...where they've been receiving extensive training and experience in practical application of law.

Further, law firms do not hire purely based on your bar score. They also look at your grades, what programs you participated in (many law schools run legal clinics to help give students some practical experience, under supervision), your recommendations, who you clerked for, etc. When you're hired, you're under supervision by more senior attorneys as you gain experience.

There's also the MPRE, or ethics test - which involves answering how to handle theoretical scenarios you would find yourself in as a practicing attorney.

Multiple people in this discussion are acting like it's a multiple choice test and if you pass, you're given a pat on the ass and the next day you roll into criminal court and become lead on a murder case...

Re: Re-Evaluating GPT-4's Bar Exam Performance

#29

Post by lazide »

paulcole wrote: post #25
lazide wrote: post #9

Earlier quoted context omitted.

Eh, also in legal practice there are key skills like selecting the best billable clients, covering your ass, building a reputation, choosing the right market segment, etc. which I’d also argue LLMs suck at.

> which I’d also argue LLMs suck at OK, I’ll bite. What’s your evidence for this argument?

Every bit of interaction I’ve ever had with an LLM. And all the research I’ve seen.

They’re plausible word sequence generators, not ‘planning for the future’ agents. Or market analyzers. Or character evaluators. Or anything else.

And they tend to be really ‘gullible’.

What evidence do you have they could do any of those things? (And not just generate plausible text at a prompt, but actually do those things)

Re: Re-Evaluating GPT-4's Bar Exam Performance

#30

Post by LordDragonfang »

lambdaxyzw wrote: post #27
LeoPanthera wrote: post #16

Earlier quoted context omitted.

You can just ask it, you know. GPT-4o: “Average wealth and income” can vary significantly by region and context. However, in the United States, as a rough benchmark, the median household income is around $70,000 per year. Wealth, which includes assets such as savings, property, and investments minus debts, is harder to pinpoint but median net worth for U.S. households is approximately $100,000. These figures provide…

I like that it immediately assumed the US, even though nothing in your question suggested it. I love that all LLMs have a strong US centric bias. Btw I'm not personally a lawyer, but I've heard that GPT is especially prone to mixing laws across the borders - for example you ask a law question in language X, and get a response that uses a law from a country Y - and it's extremally convincing doing that (unless you're…

I mean, to be fair, if you're speaking English to it, the most likely possibility is that you're inside the US:

https://en.wikipedia.org/wiki/List_of_countries_by_English-s...

I know there's a lot of complaints about things being US-centric, but the US is a very large country.

Post reply on HN