Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Re-Evaluating GPT-4's Bar Exam Performance

link.springer.com

Post reply on HN
Print view View original

131–139 of 139 posts

Re: Re-Evaluating GPT-4's Bar Exam Performance

#131

Post by mordymoop »

Zenzero wrote: post #117
mordymoop wrote: post #66

Earlier quoted context omitted.

On what topics you understand well does GOT-4o or Claude Opus produce garbage?

Anything deeper than surface level in medicine. Try getting it to properly select crystalloids with proper additives for a patient with a given history and lab results and watch in horror as it confidently gives instructions that would kill the patient. What is even more irritating is that I had gpt4 debate me on things that it was completely wrong about and it was only when I responded with a stern rebuke that it hi…

To be fair, I have not found MDs to be particularly reliable for answering basic questions about medicine either.

Re: Re-Evaluating GPT-4's Bar Exam Performance

#132

Post by mensetmanusman »

Zenzero wrote: post #129
mensetmanusman wrote: post #122

Earlier quoted context omitted.

LLMs are not good at answering expert level questions at the forefront of human knowledge.

Unfortunately it would be considered basic medicine in this case.

Is it basic but not documented? Basic to me means the first google search result is generally correct.

Re: Re-Evaluating GPT-4's Bar Exam Performance

#133

Post by lazide »

mrguyorama wrote: post #128
lazide wrote: post #111

Earlier quoted context omitted.

Like the chatbot wouldn’t squeeze you 10x harder. At least a human CEO has to worry about being arrested or someone setting their house on fire.

Steve Jobs didn't even worry about cancer enough to save his life. Why the fuck do you think he would have an even remote understanding that squeezing people could result in consequences for him?

I don’t think you’re making the point you think you’re making here.

Re: Re-Evaluating GPT-4's Bar Exam Performance

#134

Post by Zenzero »

mensetmanusman wrote: post #132
Zenzero wrote: post #129

Earlier quoted context omitted.

Unfortunately it would be considered basic medicine in this case.

Is it basic but not documented? Basic to me means the first google search result is generally correct.

That's not how medicine operates.

Medical problems are highly contextual, so you are not going to get much valuable information at the level of what a doctor is thinking from the first page of Google. That doesn't mean it isn't a simple within our area of expertise.

Re: Re-Evaluating GPT-4's Bar Exam Performance

#135

Post by Zenzero »

mordymoop wrote: post #131
Zenzero wrote: post #117

Earlier quoted context omitted.

Anything deeper than surface level in medicine. Try getting it to properly select crystalloids with proper additives for a patient with a given history and lab results and watch in horror as it confidently gives instructions that would kill the patient. What is even more irritating is that I had gpt4 debate me on things that it was completely wrong about and it was only when I responded with a stern rebuke that it hi…

To be fair, I have not found MDs to be particularly reliable for answering basic questions about medicine either.

OK. I can't speak for what you've experienced. I can only offer what I see from LLMs given what I know.

Re: Re-Evaluating GPT-4's Bar Exam Performance

#136

Post by thehoneybadger »

euroderf wrote: post #34
thehoneybadger wrote: post #32

It is difficult to comment without sounding obnoxious, but having taken the bar exam, I found the exam simple. Surprisingly simple. I think it was the single most over hyped experience of my life. I was fed all this insecurity and walked into the convention center expecting to participate in the biggest intellectual challenge in my life. Instead, it was endless multiple choice questions and a couple contrived scenari…

> It may also be surprising to some to understand that legal writing is prized for its degree of formalism. It aims to remove all connotation from a message so as to minimize misunderstanding, much like clean code. > The more your argument is structured, akin to a computer program, the better. You certainly make legal writing sound like a flavor of technical writing. Simplicity, clarity, structure. Is this an accurat…

Yes, that is an accurate comparison.

Re: Re-Evaluating GPT-4's Bar Exam Performance

#137

Post by keefle »

fnordpiglet wrote: post #43

Scoring 96 percentile among humans taking the exam without moving goal posts would have been science fiction two years ago. Now it’s suddenly not good enough and the fact a computer program can score decent among passing lawyers and first time test takers is something to sneer at. The fact I can talk to the computer and it responds to me idiomatically and understands my semantic intent well enough to be nearly indist…

The profusion of jaded nerds, although saddening at times, seems to be pushing science forward. I have a feeling that a prolonged sense of "Awe" can hinder progression at times, and the lack of it is usually a sign of the adaptability of a group (how quick new developments are normalized?)

Re: Re-Evaluating GPT-4's Bar Exam Performance

#138

Post by randomtoast »

Jensson wrote: post #116
randomtoast wrote: post #110

Earlier quoted context omitted.

I hope you don't hold a teaching position at a university then.

I did, the growth students have from first to second year is enormous. Everyone know freshmen produce garbage answers, that is why they are freshmen and not out doing work, they are there to learn not to produce answers. If freshmen answers were good enough people wouldn't bother hiring college grads, just hire dropouts and high school grads. > I hope you don't hold a teaching position at a university then. You think…

> You think teachers shouldn't have growth mindset for students? I think students can grow from producing garbage answers to good answers, that is what they are there for.

I think many students including freshman have interesting and sometimes thought provoking ideas. And they come up with creative solutions, which is based on their previous experience in life. I would never describe that as garbage.

Re: Re-Evaluating GPT-4's Bar Exam Performance

#139

Post by mensetmanusman »

Zenzero wrote: post #134
mensetmanusman wrote: post #132

Earlier quoted context omitted.

Is it basic but not documented? Basic to me means the first google search result is generally correct.

That's not how medicine operates. Medical problems are highly contextual, so you are not going to get much valuable information at the level of what a doctor is thinking from the first page of Google. That doesn't mean it isn't a simple within our area of expertise.

In my area of expertise, a well formulated google search can result in a page 1 full of academic articles on the general topic, but there isn’t necessarily consensus. This might be a case of the curse of knowledge :)
Post reply on HN