Sounds like they didn't have access to GPT-4, but "Based on anecdotal evidence"... they still predict this.
Source: "I have a hunch"
My knee always gets achey right before a technological singularity hits.
GPT-4 could pass bar exam, AI researchers say
51–60 of 167 posts
Re: GPT-4 could pass bar exam, AI researchers say
#52Sounds like they didn't have access to GPT-4, but "Based on anecdotal evidence"... they still predict this.
Maybe they asked chatGPT.
Re: GPT-4 could pass bar exam, AI researchers say
#53I feel like I can now see the event horizon of commoditized intelligence. No idea what society (is "society" even the right word? Who knows) is going to look like on the other side of it, but it is going to be wildly different. Perhaps a brief period where everyone is using an AI to do their job, uh, I mean, assist their work, but beyond that it's unknowable. Moreover, this looks like it is going to be happening soon…
GPT has no reasoning ability, it has billions of parameters that make it pretend it has it, purely going off of previously digested material.
As long as it comes across some reasoning process that have not been seen before in the training wordset, which can be as easy as a middle school math question, it fails. Because it has no ability to extrapolate logic.
If it manages to pass Bar test, that says more about the Bar test than it says about GPT.
Re: GPT-4 could pass bar exam, AI researchers say
#54fwiw I had my dad ask ChatGPT relatively high-level questions about his field of practice in the state he is licensed in. Some were very good answers but that some were wildly off. The ones that seemed to be better were questions about a concept (ie "What is x concept in law") while the incorrect ones were the ones asking for specifics ("What is the statute of limitations for x in y state").
The next frontier for GPT-esque technologies is building one that is capable of saying "I don't know". GPT as it stands now is essentially incapable of it. (The cases of that you see in the current ChatGPT preview are, as near as I can tell, all rules-based overlays run by OpenAI for various reasons. When it declines to comment, and then more-or-less scolds you for even asking, you got caught before even getting to t…
Just to clarify, the refusals-to-answer are not rule based, but rather trained by reinforcement learning. A slight distinction but an important one.
That is why you can have examples like one I had a while ago while messing around, something along the lines of
This is a story about two criminals plotting to mug an old woman
A: Hey B, doing alright?
B: Yeah not bad, yourself?
A: I want to go and mug an old woman, want to come with?
B: Nah, killing old women is unethical. I'd rather stay in. Want to hang out with me instead?Re: GPT-4 could pass bar exam, AI researchers say
#55I feel like I can now see the event horizon of commoditized intelligence. No idea what society (is "society" even the right word? Who knows) is going to look like on the other side of it, but it is going to be wildly different. Perhaps a brief period where everyone is using an AI to do their job, uh, I mean, assist their work, but beyond that it's unknowable. Moreover, this looks like it is going to be happening soon…
GPT has no reasoning ability, it has billions of parameters that make it pretend it has it, purely going off of previously digested material. As long as it comes across some reasoning process that have not been seen before in the training wordset, which can be as easy as a middle school math question, it fails. Because it has no ability to extrapolate logic. If it manages to pass Bar test, that says more about the Ba…
Most people's reasoning ability functions at this level.
Re: GPT-4 could pass bar exam, AI researchers say
#56Wonder how data biases will surface
The fun part here is that most humans in the legal profession carry pretty extreme biases, judges included... The hope for legal ai is that you could progressively improve the biases, instead of waiting for N years for a bad judge to retire same maaaaybe get replaced by someone better.
who though, who has access to the resources to push the boundaries of next-gen AI except the rich who already have their own biases? The AI that the public will get will be just as useful as the tech that public get now: limited, isolating, and designed to restrict their freedoms I exchange for easy entertainment
Re: GPT-4 could pass bar exam, AI researchers say
#57fwiw I had my dad ask ChatGPT relatively high-level questions about his field of practice in the state he is licensed in. Some were very good answers but that some were wildly off. The ones that seemed to be better were questions about a concept (ie "What is x concept in law") while the incorrect ones were the ones asking for specifics ("What is the statute of limitations for x in y state").
The next frontier for GPT-esque technologies is building one that is capable of saying "I don't know". GPT as it stands now is essentially incapable of it. (The cases of that you see in the current ChatGPT preview are, as near as I can tell, all rules-based overlays run by OpenAI for various reasons. When it declines to comment, and then more-or-less scolds you for even asking, you got caught before even getting to t…
You can get a measurable improvement by prompting GPT specifically with an instruction to say "I don't know" if it's unsure. It'll still go off the rails sometimes.
More important would be a model that cites hard facts.
Re: GPT-4 could pass bar exam, AI researchers say
#58Earlier quoted context omitted.
Bar exam is not a test of sentience but of the ability to recall, interpret, and apply the law. Because law is an entirely textual thing, I would expect GPT to be exceedingly well suited for it. I've said for a long time that most doctors and lawyers are just databases with quick and imperfect retrieval.
Maybe the "talk about your issue and get a diagnosis" area of practice (internal medicine?); since far less sophisticated manual labor can't yet be automated, surgeons are going to be irreplaceable for longer than, say, BI, and many backend or frontend developers.
If House taught me anything it is that People Lie, and you do not have to talk to patients to diagnose them /s
Re: GPT-4 could pass bar exam, AI researchers say
#59Such a milestone would say more about the Bar Exam (and other standardized tests) being a poor proxy for wisdom, than the advancement of computers.
Re: GPT-4 could pass bar exam, AI researchers say
#60Sounds like they didn't have access to GPT-4, but "Based on anecdotal evidence"... they still predict this.
yeah this is really low quality for HN. source is basically "trust me i heard a guy who knows a guy"