Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Re-Evaluating GPT-4's Bar Exam Performance

link.springer.com

Post reply on HN
Print view View original

111–120 of 139 posts

Re: Re-Evaluating GPT-4's Bar Exam Performance

#111

Post by lazide »

pojzon wrote: post #103
threeseed wrote: post #75

Earlier quoted context omitted.

Phoebe Moore who that quote was attributed to has never been a CEO or even worked at a non-academic organisation. So much of a what a CEO does is fostering culture, hiring people and setting a unique vision for the company. Imagine thinking people would be inspired to work for a chatbot. Hilariously ridiculous.

If that chatbot had Steve Jobs voice ? I dunno, I would probably prefer to work under that chatbot than my current CEO that only tries to squize as much as possible out of ppl already working for him.

Like the chatbot wouldn’t squeeze you 10x harder.

At least a human CEO has to worry about being arrested or someone setting their house on fire.

Re: Re-Evaluating GPT-4's Bar Exam Performance

#112

Post by ghaff »

djmips wrote: post #89
LamaOfRuin wrote: post #80

Earlier quoted context omitted.

I would classify all of those as "non-traditional" learning techniques, unless you actually mean using a textbook while taking a class with a human teacher. Well written textbooks are consumable on their own for some people, but most are not written for that.

That's a good observation about textbooks and helps explain why I had difficulties trying to teach myself topics from a textbook alone!

A lot just aren't very good but they also tend to make assumptions about prior knowledge in line with what would be typical prerequisites for a class and some degree of guidance.

Re: Re-Evaluating GPT-4's Bar Exam Performance

#113

Post by ghaff »

johnchristopher wrote: post #96
ghaff wrote: post #23

Earlier quoted context omitted.

Leave-without-pay normally requires some specific justification(s)/discussion. I've certainly given my manager advanced notice about any longer stretches of vacation and I've tried to do it with awareness of workloads (though for something planned months in advance that's not always possible) but I've pretty much never considered it as asking for permission or it being a negotiation. This is in the US. ADDED: You're…

> ADDED: You're probably going to end up lying or at least being very vague "some family stuff to take care of" in this specific scenario but for one month that didn't trigger reporting to employer a lot of professionals could probably get off with it. In any case, the GPT answer seems totally correct for the parameters given. Where I live that is subject to cancellation of the work contact, you can't lie about why y…

Maybe you're talking about something else but for standard PTO/vacation that I'm owed it would seem absurd if I had to justify how I was going to spend my time off. It's none of your business.

Re: Re-Evaluating GPT-4's Bar Exam Performance

#115

Post by ptx »

munchler wrote: post #57
d0mine wrote: post #54

Earlier quoted context omitted.

On any topic that I understand well, LLM output is garbage: it requires more energy to fix it than to solve the original problem to begin with. Are we sure these exams are not present in the training data? (ability to recall information is not impressive for a computer) Still I'm terrible at many many tasks e.g., drawing from description and the models widen significantly types of problems that I can even try (where…

> On any topic that I understand well, LLM output is garbage: it requires more energy to fix it than to solve the original problem to begin with. That's probably true, which is why human most knowledge workers aren't going away any time soon. That said, I have better luck with a different approach: I use LLM's to learn things that I don't already understand well. This forces me to actively understand and validate the…

I've had teachers who didn't understand the subject they were teaching. It's not a good experience and replicating that seems like a terrible idea.

Re: Re-Evaluating GPT-4's Bar Exam Performance

#116

Post by Jensson »

randomtoast wrote: post #110
Jensson wrote: post #108

Earlier quoted context omitted.

> At worst, it's more like a freshman-level answer That is garbage.

I hope you don't hold a teaching position at a university then.

I did, the growth students have from first to second year is enormous. Everyone know freshmen produce garbage answers, that is why they are freshmen and not out doing work, they are there to learn not to produce answers. If freshmen answers were good enough people wouldn't bother hiring college grads, just hire dropouts and high school grads.

> I hope you don't hold a teaching position at a university then.

You think teachers shouldn't have growth mindset for students? I think students can grow from producing garbage answers to good answers, that is what they are there for. An LLM however doesn't grow, so while such students are worth teaching even though they produce garbage answers the LLM isn't.

Re: Re-Evaluating GPT-4's Bar Exam Performance

#117

Post by Zenzero »

mordymoop wrote: post #66
d0mine wrote: post #54

Earlier quoted context omitted.

On any topic that I understand well, LLM output is garbage: it requires more energy to fix it than to solve the original problem to begin with. Are we sure these exams are not present in the training data? (ability to recall information is not impressive for a computer) Still I'm terrible at many many tasks e.g., drawing from description and the models widen significantly types of problems that I can even try (where…

On what topics you understand well does GOT-4o or Claude Opus produce garbage?

Anything deeper than surface level in medicine.

Try getting it to properly select crystalloids with proper additives for a patient with a given history and lab results and watch in horror as it confidently gives instructions that would kill the patient.

What is even more irritating is that I had gpt4 debate me on things that it was completely wrong about and it was only when I responded with a stern rebuke that it hit me with the usual "Apologies for the misunderstanding..."

Re: Re-Evaluating GPT-4's Bar Exam Performance

#118

Post by johnchristopher »

ghaff wrote: post #113
johnchristopher wrote: post #96

Earlier quoted context omitted.

> ADDED: You're probably going to end up lying or at least being very vague "some family stuff to take care of" in this specific scenario but for one month that didn't trigger reporting to employer a lot of professionals could probably get off with it. In any case, the GPT answer seems totally correct for the parameters given. Where I live that is subject to cancellation of the work contact, you can't lie about why y…

Maybe you're talking about something else but for standard PTO/vacation that I'm owed it would seem absurd if I had to justify how I was going to spend my time off. It's none of your business.

Obviously not talking about standard PTO/vacation.

Where I live you must inform your employer unless you are lucky enough to have enough PTO/va. to spend your PTO/vacation in prison and hide it from your employer. But even then I don't think that will work out because there are administrative stuff related to social welfare you have to comply to and at some point it will be on your employer's radar anyway (why is that guy exempt from social welfare taxes for that specific month ? and why did the police asked me to confirm he was working here ? etc.).

BUT in practice, until recently, if imprisonment is less than 3 years then you won't spend a day in prison (unless you are deemed too dangerous). But then you have an electronic bracelet and other obligations that will at some point alert your employer.

I read now that prison time of any length will have to be spent in prison (no more less than 3 years or 2 years or 18 months pass). But for prison time less than 18 months you can have a bracelet on the first day.

All that to say I don't think it's manageable and possible to hide prison time from your employer, even if you have enough PTO/vac. days to cover for it.

Re: Re-Evaluating GPT-4's Bar Exam Performance

#119

Post by burnished »

ptx wrote: post #115
munchler wrote: post #57

Earlier quoted context omitted.

> On any topic that I understand well, LLM output is garbage: it requires more energy to fix it than to solve the original problem to begin with. That's probably true, which is why human most knowledge workers aren't going away any time soon. That said, I have better luck with a different approach: I use LLM's to learn things that I don't already understand well. This forces me to actively understand and validate the…

I've had teachers who didn't understand the subject they were teaching. It's not a good experience and replicating that seems like a terrible idea.

A key advantage is that LLMs dont have emotional states that need to be managed.

Re: Re-Evaluating GPT-4's Bar Exam Performance

#120

Post by Digory »

speedgoose wrote: post #46
Digory wrote: post #7

They originally scored against a test usually taken by people who failed the bar. So, GPT-4 scores closer to the bottom of people who pass the bar the first time. In other words, it matches the people who cull the rules from texts already written, but who cannot apply it imaginatively.

> In other words, it matches the people who cull the rules from texts already written, but who cannot apply it imaginatively. Where did you find that in the article?

If you can recite the black letter law, you've got a good chance of passing the bar. The higher essay scores usually require creative arguments about resolving competing rules and policies.

It's easier to extract the formal statement of the rule against perpetuities from a reddit corpus, than to apply the rule to an artificially complex fact pattern in an essay question.

Post reply on HN