Sounds like they didn't have access to GPT-4, but "Based on anecdotal evidence"... they still predict this.
Source: "I have a hunch"
GPT-4 could pass bar exam, AI researchers say
41–50 of 167 posts
Re: GPT-4 could pass bar exam, AI researchers say
#42IF it could (I wouldn't know one way or the other), I'd consider that a damning indictment of the Bar Exam failing to test for sentience, rather than evidence of GPT-4 having attained the same.
Bar exam is not a test of sentience but of the ability to recall, interpret, and apply the law. Because law is an entirely textual thing, I would expect GPT to be exceedingly well suited for it.
I've said for a long time that most doctors and lawyers are just databases with quick and imperfect retrieval.
Re: GPT-4 could pass bar exam, AI researchers say
#43How soon before this qualifies as a public defender? Gonna put this on my dystopia bingo.
I knew we were going to replace 9/10 doctors and 9/10 lawyers the same day I got to try ChatGPT. It's just a matter of time - whoever does it properly first. I am talking about the first line of defense here, like a chat bot. Courtrooms will probably still work the same way for a long time. It's not like most lawyers or doctors are great. Most are completely average - which is fine. Not everyone wants to read the lat…
Just recently a friend of mine tried an online service that does psychological counseling (I don't want to name names, but if you listen to podcasts on Spotify you've probably heard their ads.) She showed me the transcript of her one and only session with a supposedly human "counselor" and compared to ChatGPT it was like schizophrenic word salad nonsense. I can absolutely imagine that niche being filled by an AI.
Re: GPT-4 could pass bar exam, AI researchers say
#44Earlier quoted context omitted.
I got the same feeling asking ChatGPT about some basic logic and maths concepts. IMO GPT can find the relevant training data to regurgitate, but i don't think it connects concepts.
I mean, it's a bullshit generator. It'll grab whatever it find in training set that kinda fits the topic and make sure it hits the word count - like a lazy student before deadline. And that's also the result - sometimes it hits something good. Sometimes it spews up utter crock and it doesn't have any notion or understanding of the difference. However, it does look good to the lazy and uninformed and it'll soon render…
Maybe then we’ll actually have some kind of quality to treatment. I’ve seen numerous doctors over the years for chronic health conditions and vast majority of them don’t really listen and can’t keep your whole history in their head while also trying to hear the new stuff. They are over worked with far too many patients.
Re: GPT-4 could pass bar exam, AI researchers say
#45I feel like I can now see the event horizon of commoditized intelligence. No idea what society (is "society" even the right word? Who knows) is going to look like on the other side of it, but it is going to be wildly different. Perhaps a brief period where everyone is using an AI to do their job, uh, I mean, assist their work, but beyond that it's unknowable. Moreover, this looks like it is going to be happening soon…
We were perhaps a bit too enamored with the idea that it was intellect that made us unique, and thus knowledge workers would be the last to be replaced. Pouring our brains out by the Petabytes for neural networks to pick them up made the economics just work for an AI industrial revolution to start from there.
Re: GPT-4 could pass bar exam, AI researchers say
#46Earlier quoted context omitted.
The next frontier for GPT-esque technologies is building one that is capable of saying "I don't know". GPT as it stands now is essentially incapable of it. (The cases of that you see in the current ChatGPT preview are, as near as I can tell, all rules-based overlays run by OpenAI for various reasons. When it declines to comment, and then more-or-less scolds you for even asking, you got caught before even getting to t…
I'd even settle for a GPT-esque technology that is capable of linking and citing sources.
YouChat is a chatbot that tries to do just that. I asked it what is going on in Peru and it gave a good answer including a citation:
>In Peru, a political crisis has been unfolding over the past few months, with the ousting of former President Pedro Castillo over his refusal to step down [1]. Protests have been held in response to Castillo's ouster, and they have been met with a strong police response. Additionally, truckers and some farm groups are planning to go on strike on Monday to demand measures to alleviate their economic hardship. Peru is also facing an economic downturn, with many businesses facing closure due to the crisis.
The citation link was https://www.reuters.com/world/americas/what-happens-perus-fo...
Some details are wrong, it says something will happen on Monday but it does not realize that's supposed to be relative to the publication of the cited article. But it did correctly summarize what the source says.
Re: GPT-4 could pass bar exam, AI researchers say
#47IF it could (I wouldn't know one way or the other), I'd consider that a damning indictment of the Bar Exam failing to test for sentience, rather than evidence of GPT-4 having attained the same.
Bar exam is not a test of sentience but of the ability to recall, interpret, and apply the law. Because law is an entirely textual thing, I would expect GPT to be exceedingly well suited for it. I've said for a long time that most doctors and lawyers are just databases with quick and imperfect retrieval.
Maybe the "talk about your issue and get a diagnosis" area of practice (internal medicine?); since far less sophisticated manual labor can't yet be automated, surgeons are going to be irreplaceable for longer than, say, BI, and many backend or frontend developers.
Re: GPT-4 could pass bar exam, AI researchers say
#48Wonder how data biases will surface
This is what I found immediately interesting about ChatGPT. I asked about controversial topics. Its answers didn’t seem like biases that were programmed in, but rather it took traditional media and gave it more weight than what turned out to be the truth only accepted much later on and still against a media retelling. I lost a lot of faith in it knowing it was more CNN than careful deliberating AI.
It's well documented that controversial topics are subject to varying degrees of censorship and prompt editing/modification/appending. What you think may be a response in alignment with corporate media may in fact be corporations disallowing you from obtaining actual responses through various means that are being tested now. We can't know unless we have open source access to the unmodified model.
Re: GPT-4 could pass bar exam, AI researchers say
#49Earlier quoted context omitted.
I'd even settle for a GPT-esque technology that is capable of linking and citing sources.
That's probably a much harder problem.
Getting it to cite a source is easy: https://news.ycombinator.com/item?id=34016435
Getting it to cite one that actually exists, ah, now that's a hard problem. Given how slimmed down the tech currently is, even if one can hypothesize some mechanism for having the system keep track of where it got certain ideas (and it is not at all obvious to me how to encode into an otherwise notoriously opaque neural net where ideas came from, given that we can't even point at an "idea" or "concept" or "fact" in a neural net at all), it is hard to imagine it wouldn't take so many additional resources that we'd have to trim the model size down to tiny fractions of what it is now.
For all the people going "wow" at the current state of GPT, I wouldn't be surprised that in 20 years it's actually seen as a dead end. I'm also impressed, but at the same time, I'm seeing the limitations it has for practical use. The hypotheses about why pure neural net approaches are going to be too problematic to use are basically coming true. AI models that can't give human-comprehensible reasons for their conclusions, including attestation of sources, are too dangerous to use. They're just black boxes, and for all you know someone's got their finger on the scale of the black boxes. OpenAI is already doing that, quite visibly, and even if you are comfortable with their reasons for doing so today, you should conclude from the fact they basically immediately stuck their fingers on the scale that you aren't getting some super AI to answer your questions, but a manifestation of some particular group of human's answer to your questions. But... I can already get that! I don't need to pay OpenAI to AI-wash their answers.
Re: GPT-4 could pass bar exam, AI researchers say
#50How soon before this qualifies as a public defender? Gonna put this on my dystopia bingo.
I knew we were going to replace 9/10 doctors and 9/10 lawyers the same day I got to try ChatGPT. It's just a matter of time - whoever does it properly first. I am talking about the first line of defense here, like a chat bot. Courtrooms will probably still work the same way for a long time. It's not like most lawyers or doctors are great. Most are completely average - which is fine. Not everyone wants to read the lat…
-- discussing this with a young bar tender a few weeks ago - showed him GTP3 - he asked it some basic legal and medical questions - got pretty freaked out - said - i guess those jobs will go away - i thought so to - he asked me what is a good safe job - sat thought about it for a few minutes - thinking what would i really want to pay a human for no matter what - realized the answer was right in front of me - make my drink - tell me some gossip - listen to a rant - flirt a little - want that with a human - i told him - he smiled - remember wasn't so long ago we still had these(1) --
https://api.time.com/wp-content/uploads/2015/08/phones1.jpeg (1)
