Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 could pass bar exam, AI researchers say

the-decoder.com

Post reply on HN
Print view View original

101–110 of 167 posts

Re: GPT-4 could pass bar exam, AI researchers say

#101

Post by forgetfulness »

saint_fiasco wrote: post #46
Metus wrote: post #30

Earlier quoted context omitted.

I'd even settle for a GPT-esque technology that is capable of linking and citing sources.

YouChat is a chatbot that tries to do just that. I asked it what is going on in Peru and it gave a good answer including a citation: >In Peru, a political crisis has been unfolding over the past few months, with the ousting of former President Pedro Castillo over his refusal to step down [1]. Protests have been held in response to Castillo's ouster, and they have been met with a strong police response. Additionally,…

It's not the explanation of a political scientist whose column you'd prize reading, but it's better than most online commentary that humans would produce; for instance, it just takes for granted that he was being asked to step down and refused, and skipped over his unconstitutional attempt to dissolve Congress, but it makes an attempt to present facts.

So, it's at the level of a person of average intelligence and a bit over superficial investment in what's being asked about.

I lack the knowledge now to tell if it will stall at this level, but that's nothing to sneeze at for something whose labor comes for free and tirelessly, and may keep improving.

Re: GPT-4 could pass bar exam, AI researchers say

#103

Post by mikepurvis »

forgetfulness wrote: post #45
Workaccount2 wrote: post #25

I feel like I can now see the event horizon of commoditized intelligence. No idea what society (is "society" even the right word? Who knows) is going to look like on the other side of it, but it is going to be wildly different. Perhaps a brief period where everyone is using an AI to do their job, uh, I mean, assist their work, but beyond that it's unknowable. Moreover, this looks like it is going to be happening soon…

We were perhaps a bit too enamored with the idea that it was intellect that made us unique, and thus knowledge workers would be the last to be replaced. Pouring our brains out by the Petabytes for neural networks to pick them up made the economics just work for an AI industrial revolution to start from there.

I feel a bit like this with the whole firestorm around AI artwork as well— it's been a big wakeup call to people who have been creating using technology-assisted workflows for decades, but still felt in their gut that they were bringing something unique to the table and were therefore "safe" from being completely automated away. That hitting the button for magic eraser or magic lasso or magic color correction was someone okay in a way that the AI itself sitting in the driver's seat was not.

Now that's been reduced to pointing out minor flaws that the next generation of AI artists will trivially resolve, and sharing memes beseeching other humans to participate in a boycott.

There's real pain and angst there, and I don't want to be callous about it with a comparison to buggy-whip manufacturers or something. But I wish the participants in these types of discussions were able to zoom out a bit and see that there's a larger societal issue here around automation, and that the real solution is going to be rethinking the basic economics of how we distribute wealth in a time of extraordinary machine-driven productivity— productivity that is no longer just about assembly lines and primary industries, but now also includes an increasing bite out of realms previously classified as "knowledge work".

Re: GPT-4 could pass bar exam, AI researchers say

#104

Post by Aperocky »

kuschku wrote: post #89
Aperocky wrote: post #53

Earlier quoted context omitted.

GPT has no reasoning ability, it has billions of parameters that make it pretend it has it, purely going off of previously digested material. As long as it comes across some reasoning process that have not been seen before in the training wordset, which can be as easy as a middle school math question, it fails. Because it has no ability to extrapolate logic. If it manages to pass Bar test, that says more about the Ba…

Most jobs today don't need novel reasoning. This is the equivalent of the steam machine for intelligence. During the industrialization, machines did not replace all jobs, but they replaced or changed most jobs. The same will happen here. A typical office job will have a few hours a week of actual, intensive thought. The vast majority of time will be spent doing simple, repetitive work. This work can be automated, or…

These boilerplate code can be and are automated away using deterministic frameworks. No need to introduce a blackbox and be responsible to debug the stuff it creates, which sounds far more painful than the alternatives.

Re: GPT-4 could pass bar exam, AI researchers say

#105

Post by ben_w »

microtherion wrote: post #38
ben_w wrote: post #6

Earlier quoted context omitted.

Defender is probably good. Prosecutor is what would worry me, given I don't know better than to blindly trust the meme that the average person commits 6 felonies before breakfast.

Defender is a TERRIBLE idea. I can already see the Supreme Court cases down the line: Defendant was provided a state of the art, 50 trillion parameter, neural network for their defense. The internals of this network are not auditable, but it does not tire, engage in substance abuse, or get distracted, so it will by definition represent effective assistance of counsel, even if for some unfathomable reason it decides t…

If you're worried about it being deployed too soon (like the issues we see with certain self driving systems), then I agree.

I'm assuming the case where it's actually good rather than merely better than me (I'm not a lawyer, so a low… bar… to pass).

Re: GPT-4 could pass bar exam, AI researchers say

#106

Post by lukko »

nopinsight wrote: post #21

Related: "Large Language Models Encode Clinical Knowledge" https://arxiv.org/abs/2212.13138 "On the MedQA dataset consisting of USMLE style questions with 4 options, our Flan-PaLM 540B model achieved a multiple-choice question (MCQ) accuracy of 67.6%..." "The percentages of correctly answered items required to pass varies by Step and from form to form within each Step. However, examinees typically must answer approxi…

I'm kind of surprised the model doesn't score higher as there is clear pattern to questions + answers and there would a huge amount of training data for USMLE. But as stated elsewhere, there is an enormous gap between passing exams and treating real patients as a doctor. It's rarely about making obscure diagnoses found in exam questions, but about managing illness in the context of a patient and their lifestyle, with many very human aspects - difficult communication, ethics & assessing family dynamics. Written exams are just to assess whether a medical student has the minimum required knowledge to practice, but also there are lots of practical exams and communication scenarios required too. It may well be the same for lawyers - passing the bar does not really relate to actual day-to-day practice.

Re: GPT-4 could pass bar exam, AI researchers say

#107

Post by nopinsight »

Aperocky wrote: post #53
Workaccount2 wrote: post #25

I feel like I can now see the event horizon of commoditized intelligence. No idea what society (is "society" even the right word? Who knows) is going to look like on the other side of it, but it is going to be wildly different. Perhaps a brief period where everyone is using an AI to do their job, uh, I mean, assist their work, but beyond that it's unknowable. Moreover, this looks like it is going to be happening soon…

GPT has no reasoning ability, it has billions of parameters that make it pretend it has it, purely going off of previously digested material. As long as it comes across some reasoning process that have not been seen before in the training wordset, which can be as easy as a middle school math question, it fails. Because it has no ability to extrapolate logic. If it manages to pass Bar test, that says more about the Ba…

You are implying either:

* Understanding complex language does not require logic/reasoning,

* There are infinitely many forms of logic/reasoning or at least more than those existing in a vast training set.

Neither of which is likely true.

What do you think of the Minerva system, which can solve multi-step quantitative reasoning questions better than many competent students and most adults?

https://ai.googleblog.com/2022/06/minerva-solving-quantitati...

Note: If you look at LSAT test samples, many questions are tests of complex logical reasoning, a requisite for legal professions.

Re: GPT-4 could pass bar exam, AI researchers say

#108

Post by forgetfulness »

mikepurvis wrote: post #103
forgetfulness wrote: post #45

Earlier quoted context omitted.

We were perhaps a bit too enamored with the idea that it was intellect that made us unique, and thus knowledge workers would be the last to be replaced. Pouring our brains out by the Petabytes for neural networks to pick them up made the economics just work for an AI industrial revolution to start from there.

I feel a bit like this with the whole firestorm around AI artwork as well— it's been a big wakeup call to people who have been creating using technology-assisted workflows for decades, but still felt in their gut that they were bringing something unique to the table and were therefore "safe" from being completely automated away. That hitting the button for magic eraser or magic lasso or magic color correction was som…

Hard to tell, other knowledge workers and people in creative industries were already squeezed, designers for instance have had a tough time for a very long time. Will things change, politically, because now marketers and Software developers join those ranks, for instance?

Programming was an outlet, if not a gold rush, for many people as the basic technical skills to create Software with the already sophisticated tooling available today presented an economic opportunity, but if "describe your problem, get crappy app" becomes viable, it may squeeze the market for junior developers.

For as long as it has existed, Software has been subject to the Jevons Paradox [1], and every advancement in making its development cheaper and its supply more abundant has only made it so more activities become powered by Software and Software developers, but it's hard to tell how this will impact the job market, especially if Software was absorbing people who didn't find more opportunities in the broader service sector.

1. https://en.wikipedia.org/wiki/Jevons_paradox

Re: GPT-4 could pass bar exam, AI researchers say

#109

Post by sdenton4 »

tetris11 wrote: post #56
sdenton4 wrote: post #20

Earlier quoted context omitted.

The fun part here is that most humans in the legal profession carry pretty extreme biases, judges included... The hope for legal ai is that you could progressively improve the biases, instead of waiting for N years for a bad judge to retire same maaaaybe get replaced by someone better.

who though, who has access to the resources to push the boundaries of next-gen AI except the rich who already have their own biases? The AI that the public will get will be just as useful as the tech that public get now: limited, isolating, and designed to restrict their freedoms I exchange for easy entertainment

I'm confident that these things will get easier. It is approximately ten thousand times easier to train a decent classifier in 2023 than it was in 2013... We're also now living in a world with foundation models and fine tuning, which makes it /very/ possible to improve and specialize publicly released models. We see a lot of that with stable diffusion already.

Re: GPT-4 could pass bar exam, AI researchers say

#110

Post by addisonl »

belter wrote: post #84
Workaccount2 wrote: post #25

I feel like I can now see the event horizon of commoditized intelligence. No idea what society (is "society" even the right word? Who knows) is going to look like on the other side of it, but it is going to be wildly different. Perhaps a brief period where everyone is using an AI to do their job, uh, I mean, assist their work, but beyond that it's unknowable. Moreover, this looks like it is going to be happening soon…

Only that there is no intelligence being commoditized...Yet. And that is obvious, if you ask one of these models, a meta question like for example: "If a person says I am lying, are they lying or saying the truth?" You will see these models will spit a canned elegant response, talking how a question could possibly be true or false, some persons not being able to attest if another one is truthful or not...But no menti…

>It is impossible to determine whether a person is lying or telling the truth when they make a statement like "I am lying." The statement is self-contradictory, as it asserts that the person is both lying and telling the truth at the same time. This creates a paradox, as it is impossible for the statement to be both true and false at the same time. The Liar Paradox has been the subject of philosophical and logical study for centuries, and there is no universally agreed upon resolution to it.

ChatGPT's response to me asking "If a person says I am lying, are they lying or saying the truth?"

Post reply on HN