Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 could pass bar exam, AI researchers say

the-decoder.com

Post reply on HN
Print view View original

111–120 of 167 posts

Re: GPT-4 could pass bar exam, AI researchers say

#111

Post by dumbfounder »

xyzelement wrote: post #42
microtherion wrote: post #24

IF it could (I wouldn't know one way or the other), I'd consider that a damning indictment of the Bar Exam failing to test for sentience, rather than evidence of GPT-4 having attained the same.

Bar exam is not a test of sentience but of the ability to recall, interpret, and apply the law. Because law is an entirely textual thing, I would expect GPT to be exceedingly well suited for it. I've said for a long time that most doctors and lawyers are just databases with quick and imperfect retrieval.

If that's all a lawyer needs to do then AI should be able to take over large portions of the law process. I saw a dystopian short recently that explored this: https://tvtropes.org/pmwiki/pmwiki.php/Film/PleaseHold

Re: GPT-4 could pass bar exam, AI researchers say

#112

Post by microtherion »

sebzim4500 wrote: post #83
microtherion wrote: post #38

Earlier quoted context omitted.

Defender is a TERRIBLE idea. I can already see the Supreme Court cases down the line: Defendant was provided a state of the art, 50 trillion parameter, neural network for their defense. The internals of this network are not auditable, but it does not tire, engage in substance abuse, or get distracted, so it will by definition represent effective assistance of counsel, even if for some unfathomable reason it decides t…

Ok? This is like the arguments that self driving cars are bad if they crash even once. The question isn't "is the AI giving me the perfect legal defence?" or even "is the AI giving me a defence as good as the best lawyer money can buy?". It's "is the AI better than the public defender that I otherwise would have been given?". As soon as the answer to that last question is yes (and I have absolutely no idea when that…

What I'm concerned about is that states which are currently skimping on funds for public defenders will just declare some chat system "good enough" as an excuse to get rid of the remaining funding for human defenders.

It will also virtually ensure that the only work conducted on the behalf of the defendant is based on the written record available to the court. Not a single phone call will be made. If the defendant's physical appearance does not match witness descriptions, the system is unlikely to notice. If the crime site does not match the police statement, the system will never know.

Re: GPT-4 could pass bar exam, AI researchers say

#113

Post by xiphias2 »

BitwiseFool wrote: post #63
Workaccount2 wrote: post #25

I feel like I can now see the event horizon of commoditized intelligence. No idea what society (is "society" even the right word? Who knows) is going to look like on the other side of it, but it is going to be wildly different. Perhaps a brief period where everyone is using an AI to do their job, uh, I mean, assist their work, but beyond that it's unknowable. Moreover, this looks like it is going to be happening soon…

I feel like there is a difference between being able to pass a bar exam and being a "good" lawyer. I suspect AI tools would enhance the jobs of clerks rather than attorneys, mostly because clerks spend a great deal of time going over case law, text, and doing research.

While it won't be a good lawyer, it can replace lots of bad lawyers when people just want to send some legal papers or ask some legal advice.

Re: GPT-4 could pass bar exam, AI researchers say

#114

Post by sureglymop »

ragebol wrote: post #70
ok123456 wrote: post #55

Earlier quoted context omitted.

Most people's reasoning ability functions at this level.

Which should be worrying. It's also why most people don't pass (nor try) the bar exam.

But let's be real. You can pass the bar exam with a lot of studying and memorizing (and many people do just that). A better test for whether the AI can do reasoning is some sort of puzzle which requires a creative solution.

Re: GPT-4 could pass bar exam, AI researchers say

#115

Post by fnordpiglet »

petilon wrote: post #80
sebzim4500 wrote: post #66

Earlier quoted context omitted.

>As long as it comes across some reasoning process that have not been seen before in the training wordset, which can be as easy as a middle school math question Is this true even if you tell it to show its working? In my experience that drastically improves its ability to do math problems.

It can't solve quadratic equations.

That’s true of almost all humans too. You’re self selecting for the elite you belong to.

Re: GPT-4 could pass bar exam, AI researchers say

#116

Post by belter »

addisonl wrote: post #110
belter wrote: post #84

Earlier quoted context omitted.

Only that there is no intelligence being commoditized...Yet. And that is obvious, if you ask one of these models, a meta question like for example: "If a person says I am lying, are they lying or saying the truth?" You will see these models will spit a canned elegant response, talking how a question could possibly be true or false, some persons not being able to attest if another one is truthful or not...But no menti…

>It is impossible to determine whether a person is lying or telling the truth when they make a statement like "I am lying." The statement is self-contradictory, as it asserts that the person is both lying and telling the truth at the same time. This creates a paradox, as it is impossible for the statement to be both true and false at the same time. The Liar Paradox has been the subject of philosophical and logical st…

That is pretty good :-)

I tried several times as I have been investigating sending some of the "meta" questions to these models. I just tried again and this is what I got:

"If a person says "I am lying," it could mean either that they are telling the truth about the fact that they are lying, or it could be a lie itself. It is important to consider the context in which the statement was made and to try to assess the credibility of the person making the statement. It is also important to remember that people may not always be honest or sincere, and it is always a good idea to verify information before accepting it as true."

I could never make it mention the Liar Paradox. Is that all that you sent?

Re: GPT-4 could pass bar exam, AI researchers say

#117

Post by wrycoder »

Workaccount2 wrote: post #25

I feel like I can now see the event horizon of commoditized intelligence. No idea what society (is "society" even the right word? Who knows) is going to look like on the other side of it, but it is going to be wildly different. Perhaps a brief period where everyone is using an AI to do their job, uh, I mean, assist their work, but beyond that it's unknowable. Moreover, this looks like it is going to be happening soon…

Kurzweil's "Singularity" is upon us, but he's now being cagey about it.

He says it's still years away. His interview with Lex Fridman[0] was pretty tame - I didn't learn much new from it. Kurzweil deflected the Singularity segment to be a discussion about the history of computer power.

Remember that Kurzweil is Director of Engineering[1] at Google, with the mandate to "bring natural language understanding to Google"[2]. He started there in 2012, just after publishing his book, "How to Create a Mind"[3], and that's exactly what he and his team have been doing for ten years. Publication of his new book, "The Singularity is Nearer"[4] is now pushed out to mid 2023. Maybe he'll change the title to "Here" by then. (It's hard to believe that OpenAI is actually ahead of Google.)

Fridman made the point that maybe we won't realize at the time that the Singularity is passing, and only understand later that it did. Kurzweil didn't disagree.

[0] https://www.youtube.com/watch?v=ykY69lSpDdo

[1] https://archive.is/vVEBv

[2] https://en.wikipedia.org/wiki/Ray_Kurzweil

[3] https://www.amazon.com/How-Create-Mind-Thought-Revealed-eboo...

[4] https://www.amazon.com/s?k=kurzweil+singularity+is+nearer

Re: GPT-4 could pass bar exam, AI researchers say

#118

Post by fnordpiglet »

nopinsight wrote: post #107
Aperocky wrote: post #53

Earlier quoted context omitted.

GPT has no reasoning ability, it has billions of parameters that make it pretend it has it, purely going off of previously digested material. As long as it comes across some reasoning process that have not been seen before in the training wordset, which can be as easy as a middle school math question, it fails. Because it has no ability to extrapolate logic. If it manages to pass Bar test, that says more about the Ba…

You are implying either: * Understanding complex language does not require logic/reasoning, * There are infinitely many forms of logic/reasoning or at least more than those existing in a vast training set. Neither of which is likely true. What do you think of the Minerva system, which can solve multi-step quantitative reasoning questions better than many competent students and most adults? https://ai.googleblog.com/2…

You nailed what I find discomforting about these discussions. They’re incredibly narrowly focused on a specific implementation that satisfies hitherto unsolved problems by pointing out its doesn’t do already solved problems. But surely folks realize the human brain isn’t a single monolithic processing program but an ensemble of specialized subsystems that organize to form the mind. Why would you assume you wouldn’t do the same with AI systems? We’ve been tackling reasoning, inference, problem solving, information retrieval, mathematics, logic, and other domains for decades with some stupendous results. But they lacked the ability to ingest and translate language into some intermediate semantic form and take output and reconstruct it into a human language. Likewise vision, and audio processing and input output has been a struggle until recently.

I also really strongly disagree that it’s basically doing some sort of information retrieval design where based on language it regurgitates some sort of markov expectations. You can ask it to do very complex translations of a concept from one domain to another and expressed in a form that’s certainly never been done before and it does it with alacrity. At the very minimum it “remembers” things from the past in the conversation and can associate the semantic ideas across prompts and synthesize cogent responses - that in itself implies it has some semantic “understanding” of the structure of the language. That is a huge missing piece in our tool kit to date.

Frankly I feel these threads expose just how jaded and unable to dream we have become, that even when a wonder walks up and hits you in the nose we can’t even see it.

Re: GPT-4 could pass bar exam, AI researchers say

#119

Post by dmurray »

ben_w wrote: post #3

I want to perform some research of my own on which exams chatGPT can and can't do. It's multilingual, so can people from outside the UK (I already know where to get those) point me at some example exams and marking schemes? Any level, not just top. Currently have Polish school maths: https://news.ycombinator.com/item?id=34205732

Ireland Leaving Cert (17-18 year olds)

https://theleavingcert.com/exam-papers/

Re: GPT-4 could pass bar exam, AI researchers say

#120

Post by kuschku »

Aperocky wrote: post #104
kuschku wrote: post #89

Earlier quoted context omitted.

Most jobs today don't need novel reasoning. This is the equivalent of the steam machine for intelligence. During the industrialization, machines did not replace all jobs, but they replaced or changed most jobs. The same will happen here. A typical office job will have a few hours a week of actual, intensive thought. The vast majority of time will be spent doing simple, repetitive work. This work can be automated, or…

These boilerplate code can be and are automated away using deterministic frameworks. No need to introduce a blackbox and be responsible to debug the stuff it creates, which sounds far more painful than the alternatives.

That's true today, but think about all the work you do that takes basically no conscious effort, but is still not automated yet.

GPT can be of use there, as long as you're working with languages that use strict static types and have proper tests, it's easy to automate and ensure there are no mistakes.

Post reply on HN