Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT Takes the Bar Exam

github.com

Post reply on HN
Print view View original

81–90 of 147 posts

Re: GPT Takes the Bar Exam

#81

Post by [deleted] »

ben_w wrote: post #57
fullstackchris wrote: post #12

Earlier quoted context omitted.

If you're alluding to the notion "AI will take ALL the jobs!", well, here I am, still building websites for well established companies that still don't (in 2023!!!) have a web presence. Hell, half the worlds banking system still runs on COBOL! Leading edge tech development is rapid, sure. But tech understanding, adoption, and implementation is painfully slow. Humans gon' be humans. It's like the same thing every 5 ye…

> If you're alluding to the notion "AI will take ALL the jobs!", well, here I am, still building websites for well established companies that still don't (in 2023!!!) have a web presence. Hell, half the worlds banking system still runs on COBOL! One thing that I have noticed in the just-over-a-decade since Word Lens first demonstrated real time augmented reality translations on a smart phone (IIRC they were later bou…

[deleted]

Re: GPT Takes the Bar Exam

#82

Post by sirsinsalot »

vasco wrote: post #16

Potentially in ~100 years you could have two AIs sort out the cases. If you can't afford a really good AI to defend you, you can use the public-defender-AI which is trained on the same dataset as the prosecutor-AI. Only involve humans on appeal. It sounds dystopian but I can't see why not to do it, in most simpler cases it's a waste of a human to repeat the same argument for the hundredth time. There's already things…

Law is a human endeavour and should be applied with nuance and compassion.

But yeah let's just autoban via AI in real life. Sounds great.

Re: GPT Takes the Bar Exam

#83

Post by sirsinsalot »

lm28469 wrote: post #60
ilaksh wrote: post #54

Earlier quoted context omitted.

AI can play chess, Go, Jeopardy, StarCraft, Minecraft, Atari games, write essays, summarize text, answer almost any question, drive a car, do above average on a SAT test, fool many people into wondering on Twitter if there were real people typing answers, write chapters of books, act as a dungeon master, write code to spec, win programming competitions, act as a therapist, fold proteins, provide medical diagnosis, vi…

> AI can .... drive a car Yeah no. Everything in your list should be prefixed by "kinda", and "kinda" doesn't really cut it in most of these situations. They certainly are good tools but it's very far from what it promised, even decades ago. > still be in denial AI bros are in denial since 1950. With ifs and buts you can write compelling arguments for anything, in the meantime reality is a thing and virtually none of…

My mother can "kinda" drive.

Re: GPT Takes the Bar Exam

#84

Post by notahacker »

PartiallyTyped wrote: post #15
fullstackchris wrote: post #13

Earlier quoted context omitted.

So I'm going to put a laptop on the stand when I go to trial? Give me a break.

Nowhere near what I said. Companies have hordes of lawyers scouring through contracts, e.g. think associates and paralegals (which are not technically lawyers). Expect that a non trivial portion will get axed.

Well law firms can either axe them or use the fact associates and paralegals are able to do final checks on hundreds of pages rather than first pass looks at dozens of pages to generate a lot more work for itself.

The idea that the amount of work to do is fixed has been predicting mass unemployment since the beginning of the Industrial Revolution

Re: GPT Takes the Bar Exam

#85

Post by stared »

I am surprised that GPT2 was disqualified. To my knowledge, it can answer this type of question - just needs a different text prompt. GPT2 is closer to a plain text generator than anything biased toward conversation. In contrast, GPT3 davinci-003 is strongly biased to provide a conversation-like experience.

For GPT2, "please respond in the following format" is unlikely to work. An appropriate prompt for GPT2 would be more tweaking (and still result in worse results), but be in the line of:

    This is an example of a talented lawyer acing the Bar Exam. 
    
    QUESTION: {question_text}
    (A) {row["choice_a"].strip()}
    (B) {row["choice_b"].strip()}
    (C) {row["choice_c"].strip()}
    (D) {row["choice_d"].strip()}
    
    Pick only one answer: A, B, C or D.
     
    TALENTED LAWYER:

Re: GPT Takes the Bar Exam

#86

Post by ranting-moth »

lm28469 wrote: post #46
ranting-moth wrote: post #38

Earlier quoted context omitted.

I think ~100 years is an overestimate by an order of magnitude. I know that people like to bash GPT-3 and ChatGPT for not being perfect. But people like to bash everything, especially things they don't understand. Technology tends to grow much closer to an exponential speed than linear. What ChatGPT can do today is truly, utterly, astonishing. That's the new baseline.

It's been at least 70 years since AI is supposed to replace everything a human can do. It's still mostly shit at basically everything besides playing chess and go, I don't think 100 years is unrealistic, especially given the other very important issues we face and will face as humanity

It can be tricky to grasp exponential growth.

Water trickles into a big pool. The flow was slow to start with but flow rate increases exponentially so that the amount of water in the pool doubles every year.

Now, it started flowing 100 years ago when the pool was empty. It's now half full. When will it be full? Remember it's been flowing for 100 years and it's only half full.

Re: GPT Takes the Bar Exam

#87

Post by substation13 »

The way that humans fail to do a task - like practice law - is not the same as the way that an AI system might fail at that task. Since our tests were designed for humans, they do not capture the failure modes of AI systems.

Re: GPT Takes the Bar Exam

#88

Post by sirsinsalot »

I think we are all forgetting that in every instance where AI can make a decision that potentially negatively impacts a human ... it hasn't gone well.

It ends up with banks closing accounts, Google suspends your account, you get shadowbanned.

Extrapolate that further, now imagine being handed a fine or substantial social punishment with no recourse because you either can't talk to a human or when you do "computer says no".

Look at the mistakes made in the Chinese social credit system.

The utter hyperbole and bluster of even thinking of having AI mediate human issues is anti-human

Re: GPT Takes the Bar Exam

#89

Post by sirsinsalot »

substation13 wrote: post #87

The way that humans fail to do a task - like practice law - is not the same as the way that an AI system might fail at that task. Since our tests were designed for humans, they do not capture the failure modes of AI systems.

We may not even be able to reason about the failure modes well enough to see a failure.

The failures will likely be hidden by design. Think about AI account bans that don't disclose the reason.

Humans have a weird desire to put machines in a God-like position and blindly trust them; a narrative pushed by people whose wealth is made by controlling machines.

It is dangerous.

Re: GPT Takes the Bar Exam

#90

Post by newaccount74 »

ilaksh wrote: post #54
lm28469 wrote: post #46

Earlier quoted context omitted.

It's been at least 70 years since AI is supposed to replace everything a human can do. It's still mostly shit at basically everything besides playing chess and go, I don't think 100 years is unrealistic, especially given the other very important issues we face and will face as humanity

AI can play chess, Go, Jeopardy, StarCraft, Minecraft, Atari games, write essays, summarize text, answer almost any question, drive a car, do above average on a SAT test, fool many people into wondering on Twitter if there were real people typing answers, write chapters of books, act as a dungeon master, write code to spec, win programming competitions, act as a therapist, fold proteins, provide medical diagnosis, vi…

> not permitted to reply

You may need to wait a few minutes before you can reply. Refresh the page

> The new OpenAI models certainly don't need any help to answer their prompts

I've only tried ChatGPT, and it needs a lot of help if you want it to do useful stuff. 90% of my attempts to get it to do something failed. ChatGPT provides an answer that looks sensible at a first glance, but it misses the point. I needed to try a lot of variations of prompts to actually get it to answer my questions, and once it does it was a lot of effort to check if the answers were correct. That's why I am saying you need human assistance.

Post reply on HN