Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

GPT-4 could pass bar exam, AI researchers say

the-decoder.com

Post reply on HN
Print view View original

151–160 of 167 posts

Re: GPT-4 could pass bar exam, AI researchers say

#151

Post by wrycoder »

jerf wrote: post #16
ldh0011 wrote: post #7

fwiw I had my dad ask ChatGPT relatively high-level questions about his field of practice in the state he is licensed in. Some were very good answers but that some were wildly off. The ones that seemed to be better were questions about a concept (ie "What is x concept in law") while the incorrect ones were the ones asking for specifics ("What is the statute of limitations for x in y state").

The next frontier for GPT-esque technologies is building one that is capable of saying "I don't know". GPT as it stands now is essentially incapable of it. (The cases of that you see in the current ChatGPT preview are, as near as I can tell, all rules-based overlays run by OpenAI for various reasons. When it declines to comment, and then more-or-less scolds you for even asking, you got caught before even getting to t…

I asked ChatGPT to give me the name of a Victorian novel I'd lost track of. I gave it a plot summary of the first third of the book.

ChatGPT said it was unable to come up with an answer, because it was not connected to the internet. It gave me a number of suggestions on how I could research the question myself.

Re: GPT-4 could pass bar exam, AI researchers say

#152

Post by joshspankit »

kuschku wrote: post #89
Aperocky wrote: post #53

Earlier quoted context omitted.

GPT has no reasoning ability, it has billions of parameters that make it pretend it has it, purely going off of previously digested material. As long as it comes across some reasoning process that have not been seen before in the training wordset, which can be as easy as a middle school math question, it fails. Because it has no ability to extrapolate logic. If it manages to pass Bar test, that says more about the Ba…

Most jobs today don't need novel reasoning. This is the equivalent of the steam machine for intelligence. During the industrialization, machines did not replace all jobs, but they replaced or changed most jobs. The same will happen here. A typical office job will have a few hours a week of actual, intensive thought. The vast majority of time will be spent doing simple, repetitive work. This work can be automated, or…

> Most jobs today don't need novel reasoning. This is the equivalent of the steam machine for intelligence.

Like the point above; that says more about the work.

It’s going to be really interesting how the middle-class narrative pushes back on AI revealing how little work is actually done during office hours.

Re: GPT-4 could pass bar exam, AI researchers say

#153

Post by joshspankit »

etangent wrote: post #69
ok123456 wrote: post #55

Earlier quoted context omitted.

Most people's reasoning ability functions at this level.

I would argue that it does not matter. The AI could even be "smarter" on pure IQ/reasoning, but in terms of practical reasoning that humans need that depends on exposure to real world, the AIs will still take decades to catch up. The radiology AIs are technically more accurate than radiologists on any sufficiently large dataset, and yet they still have not replaced radiologists (or even are anywhere close to).

There’s an interesting sideroad here: A lot of medical jobs are there because of legal culpability: If someone fails the blame can be placed on them and not on the hospital who hired them. It might be a long time before they get to have that same protection with AI (probably miles and miles of contracts negotiated with a 3rd party who owns and operates the AI that’s used).

Re: GPT-4 could pass bar exam, AI researchers say

#154

Post by ben_w »

w1nst0nsm1th wrote: post #143

AI seems to be the next financial buzzword, after crypto, gig economy, CDO, dotcom, and so on. I have seen a video a few days ago saying we are coming out of data era and entering the 'Knowledge Era' thank to AI where knowledge is following a logarythmic path. A 'revolution', a 'paradigm shift', and other bubblebabble. Who was telling that ? A 30 years old startup CEO wearing... a t-shirt and a jeans... You see the p…

> I'm not an AI specialist, but for what I know, current AI are nothing more than fine tuned statistic algorythm.

Yes, but Google Page Rank is just fancy matrix multiplication and worth a fortune, while the finely tuned statistics that is machine learning (specifically chatGPT) knows more about CSS and German than I do:

~~~~

Me: Erzählen Sie bitte wie ich kann ein div tag mit schwarzes hintergrund, dreihundert px hoch und 24 pixels wide machen mit css

chatGPT: Um ein div-Tag mit einem schwarzen Hintergrund, 300px hoch und 24px breit zu erstellen, könntest du den folgenden CSS-Code verwenden:

div { background-color: black; height: 300px; width: 24px; }

Um den CSS-Code anzuwenden, kannst du ihn entweder in einem style-Tag innerhalb deines HTML-Dokuments oder in einer externen CSS-Datei platzieren und dann per link-Tag in deinem HTML-Dokument einbinden.

Beispiel:

div { background-color: black; height: 300px; width: 24px; }

~~~~

The actual HTML and CSS it gave me is multiple lines and sensibly indented, don't know a convenient way to mark a block as pre-formatted. Note that chatGPT understood me correctly even though I forgot the German for "wide" and switched to English for one word only.

(I do know more CSS than is in this example; I used chatGPT over the weekend to update my website, and it solved two problems that I didn't know pure CSS could even do, but that conversation is too big to bother putting into a comment here).

Re: GPT-4 could pass bar exam, AI researchers say

#155

Post by coderoffortune »

curriculum wrote: post #147
evouga wrote: post #95

Earlier quoted context omitted.

Right. Would we be impressed if a layman could pass the bar, given infinite time and access to the entire Internet (including the copious amount of bar exam study guides and worked example problems)? If not why are we impressed that a language model trained on that data can? Meanwhile when I ask ChatGPT which of six numbers are odd, it confidently reports a mix of even numbers, odd numbers, and letters. This is a fun…

Along the same lines, asking > How many words are in the sentence "This is a test of artificial intelligence"? yields an answer of: > There are 8 words in the sentence "This is a test of artificial intelligence." (There are 7).

My guess is that AI omitted 'a' because this is essentially how natural language processing works. Perhaps it cannot see 'a' because the input has been stripped of 'a' or 'the', and so on.

Re: GPT-4 could pass bar exam, AI researchers say

#156

Post by w1nst0nsm1th »

ben_w wrote: post #154
w1nst0nsm1th wrote: post #143

AI seems to be the next financial buzzword, after crypto, gig economy, CDO, dotcom, and so on. I have seen a video a few days ago saying we are coming out of data era and entering the 'Knowledge Era' thank to AI where knowledge is following a logarythmic path. A 'revolution', a 'paradigm shift', and other bubblebabble. Who was telling that ? A 30 years old startup CEO wearing... a t-shirt and a jeans... You see the p…

> I'm not an AI specialist, but for what I know, current AI are nothing more than fine tuned statistic algorythm. Yes, but Google Page Rank is just fancy matrix multiplication and worth a fortune, while the finely tuned statistics that is machine learning (specifically chatGPT) knows more about CSS and German than I do: ~~~~ Me: Erzählen Sie bitte wie ich kann ein div tag mit schwarzes hintergrund, dreihundert px hoc…

I asked it what was the xbrl taxinomy tag on us gaap for change in executive management of a SEC registered company in SEC filling and the answer doesn't fit compared to the whole xbrl taxinomy published on the SEC website. It also answered me 2 different kind of SEC form for it. It also gave me the correct url for the us gaap xbrl taxinomy on sec website.

That being said, both xbrl.org and the SEC document for us gaap xbrl reporting (an xml document) are kind of greedy about providing a documentation for what the tags actually cover. xbrl.org provide no documentation at all and advise an xbrl.org membership for developers, And the SEC document provided the tags but no information of what the tags cover.

The answer from chatGPT seems to about 'labels', used in xbrl document to describe xbrl taxinomy tag in different contexts, for example 'income in miami store'. But a change in a top executive position, like for 'CFO', once again required in SEC filling, shouldn't be subject to various arbitrary kind of label, because then the whole thing make no sense. If you call a 'cat', a 'little domestic pet'...

I searched google for the tag or label provided by chatGPT and google provided zilch. I searched the document provided by the SEC website, zilch again.

So either the code for the SEC form is wrong, either the tag or the label is wrong... or I don't know what else.

It seems, according to comments and posts from HN, that chatGPT can give good approximative answer, but fails without any notice once you ask for details.

According to an article published on HN a few days ago, 'chatGPT hallucinate facts'.

Re: GPT-4 could pass bar exam, AI researchers say

#157

Post by ben_w »

w1nst0nsm1th wrote: post #156
ben_w wrote: post #154

Earlier quoted context omitted.

> I'm not an AI specialist, but for what I know, current AI are nothing more than fine tuned statistic algorythm. Yes, but Google Page Rank is just fancy matrix multiplication and worth a fortune, while the finely tuned statistics that is machine learning (specifically chatGPT) knows more about CSS and German than I do: ~~~~ Me: Erzählen Sie bitte wie ich kann ein div tag mit schwarzes hintergrund, dreihundert px hoc…

I asked it what was the xbrl taxinomy tag on us gaap for change in executive management of a SEC registered company in SEC filling and the answer doesn't fit compared to the whole xbrl taxinomy published on the SEC website. It also answered me 2 different kind of SEC form for it. It also gave me the correct url for the us gaap xbrl taxinomy on sec website. That being said, both xbrl.org and the SEC document for us ga…

It absolutely does indeed hallucinate[0] on occasion.

Despite how remarkable and useful it already is, don't make the mistake of putting it unsupervised in charge of anything, as it's going to mess up at least as often as a self driving car.

[0] or whatever we want to call the behaviour; also seen it called BSing (because it doesn't really know what truth is) and "mansplaining as a service"

Re: GPT-4 could pass bar exam, AI researchers say

#158

Post by Aperocky »

nopinsight wrote: post #107
Aperocky wrote: post #53

Earlier quoted context omitted.

GPT has no reasoning ability, it has billions of parameters that make it pretend it has it, purely going off of previously digested material. As long as it comes across some reasoning process that have not been seen before in the training wordset, which can be as easy as a middle school math question, it fails. Because it has no ability to extrapolate logic. If it manages to pass Bar test, that says more about the Ba…

You are implying either: * Understanding complex language does not require logic/reasoning, * There are infinitely many forms of logic/reasoning or at least more than those existing in a vast training set. Neither of which is likely true. What do you think of the Minerva system, which can solve multi-step quantitative reasoning questions better than many competent students and most adults? https://ai.googleblog.com/2…

> Understanding complex language does not require logic/reasoning

The key is understanding. It does not need to, it has already seen the question asked in a 100 different ways, it also seen the answer to all of those. It just rephrases those answer via a neural network and that happen to pass the bar test.

> There are infinitely many forms of logic/reasoning or at least more than those existing in a vast training set.

More importantly, differences between forms are subtle and cannot be understood, that's why ChatGPT confidently give wrong answers on stackoverflow: https://meta.stackoverflow.com/questions/421831/temporary-po...

Re: GPT-4 could pass bar exam, AI researchers say

#159

Post by Aperocky »

fnordpiglet wrote: post #118
nopinsight wrote: post #107

Earlier quoted context omitted.

You are implying either: * Understanding complex language does not require logic/reasoning, * There are infinitely many forms of logic/reasoning or at least more than those existing in a vast training set. Neither of which is likely true. What do you think of the Minerva system, which can solve multi-step quantitative reasoning questions better than many competent students and most adults? https://ai.googleblog.com/2…

You nailed what I find discomforting about these discussions. They’re incredibly narrowly focused on a specific implementation that satisfies hitherto unsolved problems by pointing out its doesn’t do already solved problems. But surely folks realize the human brain isn’t a single monolithic processing program but an ensemble of specialized subsystems that organize to form the mind. Why would you assume you wouldn’t d…

Language prediction model is not a closely guarded secret, I suggest looking into academic papers about what they are and maybe even see/do some implementation yourself.

There are no magic, it is just a more complicated transpose, created by training over perhaps 10% of all available text on the internet.

It does have a lot of use, for one I think it would probably put grammarly out of business, and maybe even do some work for law firms.

Re: GPT-4 could pass bar exam, AI researchers say

#160

Post by ss108 »

nopinsight wrote: post #107
Aperocky wrote: post #53

Earlier quoted context omitted.

GPT has no reasoning ability, it has billions of parameters that make it pretend it has it, purely going off of previously digested material. As long as it comes across some reasoning process that have not been seen before in the training wordset, which can be as easy as a middle school math question, it fails. Because it has no ability to extrapolate logic. If it manages to pass Bar test, that says more about the Ba…

You are implying either: * Understanding complex language does not require logic/reasoning, * There are infinitely many forms of logic/reasoning or at least more than those existing in a vast training set. Neither of which is likely true. What do you think of the Minerva system, which can solve multi-step quantitative reasoning questions better than many competent students and most adults? https://ai.googleblog.com/2…

Also just adding to my earlier reply (can't edit), none of it is "complex" relative to the complexity of some of the concepts in computer science or more brainy parts of complicated professional software development.
Post reply on HN