51–60 of 66 posts
Earlier quoted context omitted.
As soon as the companies behind these systems stop marketing them as do-anything machines, I will stop judging them on their ability to do everything. The ChatGPT input field still says ‘Ask anything’, and that is what I shall do.
You can ask me anything. I don’t see that as a promise that I am infallible.
__________________
Answers: $1
Thoughtful Answers: $5
Correct Answers: $50
Dumb Looks are Free
Supposedly, they validated it upon release by showing each task to at most nine people and only keeping the ones that at least two people got correct in two tries. But still, they have had to subsequently fix more than a dozen of them.
>Last week, the ARC Prize team released an updated test, called ARC-AGI-2, and it appears to have sent the AIs back to the drawing board. The full o3 model has not yet been tested, but a version of o1 dropped from 32 percent on the original puzzles to just 3 percent on the new version, and a “mini” version of o3 currently available to the public dropped from roughly 30 percent to below 2 percent. (An OpenAI spokesper…
I tried my own test recently: "Write a history of the Greek language but reverse it, so that one would need to read it from right to left and bottom to top." ChatGPT wrote the history and showed absolutely no awareness, let alone, "understanding" of the second half of the prompt.
>Last week, the ARC Prize team released an updated test, called ARC-AGI-2, and it appears to have sent the AIs back to the drawing board. The full o3 model has not yet been tested, but a version of o1 dropped from 32 percent on the original puzzles to just 3 percent on the new version, and a “mini” version of o3 currently available to the public dropped from roughly 30 percent to below 2 percent. (An OpenAI spokesper…
I tried my own test recently: "Write a history of the Greek language but reverse it, so that one would need to read it from right to left and bottom to top." ChatGPT wrote the history and showed absolutely no awareness, let alone, "understanding" of the second half of the prompt.
Earlier quoted context omitted.
I tried my own test recently: "Write a history of the Greek language but reverse it, so that one would need to read it from right to left and bottom to top." ChatGPT wrote the history and showed absolutely no awareness, let alone, "understanding" of the second half of the prompt.
Try playing a game of Hangman with ChatGPT. It's hilarious.
Edit: scratch that, it thought there was a six letter word starting with "trs" and then changed its mind to "tre" when I guessed "e." Hilarious.
>Last week, the ARC Prize team released an updated test, called ARC-AGI-2, and it appears to have sent the AIs back to the drawing board. The full o3 model has not yet been tested, but a version of o1 dropped from 32 percent on the original puzzles to just 3 percent on the new version, and a “mini” version of o3 currently available to the public dropped from roughly 30 percent to below 2 percent. (An OpenAI spokesper…
I tried my own test recently: "Write a history of the Greek language but reverse it, so that one would need to read it from right to left and bottom to top." ChatGPT wrote the history and showed absolutely no awareness, let alone, "understanding" of the second half of the prompt.
Earlier quoted context omitted.
>> If you want to saturate any model today give it a string and a grammar and ask it to generate the string from the grammar. I'm not sure I understand what that means - could you explain please?
I'm not GP, but here goes: LLMs are token-based, which are words or word fragments; they have limited ability to work on a letter-by-letter basis. They can't reliably count letters in a sentence, for example. "give it a string and a grammar and ask it to generate the string from the grammar" can't be done by inference alone because of this: they would generate tokens that don't match the grammar. But you can use a gr…
Changing my tests from the strings I was interested in to four or more letter common words _did_ improve the ability of reasoning LLMs to get the right answer, at the cost of the context exploding to thousands of tokens.
Unfortunately I can't tell you by how much because the couple of dozen tests I did after reading your post ate my $50 I keep in an account for these types of things.
The following question ate through 8k thinking tokens to get the right answer in Claude3.7 Sonnet Extended:
---
Given the following grammar:
::=
::= Rome | Paris | London | end_path
::= | end_route
::= end_company |
Rome Paris Rome end_path Rome London end_path end_company
---
Incidentally it got the right answer no less than 4 times in the thinking token stream. I'd not seen this model act like this before.
Earlier quoted context omitted.
Just copied your prompt and it handled it just fine.
?siht ekil kool rewsna eht diD Edit: realized just now that my summary of the 'test' failed to specify the request fully: the letters need to be reversed, too. Maybe I'm just bad with AI tools, because I didn't even get a response that 'this like looked' (i.e. reversed the order of the words).
It might work in an agent system where it can make and execute code to solve problems.
Earlier quoted context omitted.
> that’s good enough as far as I’m concerned But in that case, why an LLM. If we want Question-Answer machines to be reliable, they must have the skills which include "counting" just as a basic example.
The purpose of the LLM would be to translate natural language into computer language, not to do the calculation itself.
Earlier quoted context omitted.
I tried my own test recently: "Write a history of the Greek language but reverse it, so that one would need to read it from right to left and bottom to top." ChatGPT wrote the history and showed absolutely no awareness, let alone, "understanding" of the second half of the prompt.
As much I think AI is overhyped too, that is a prime use case that would be better solved by passing the text to a tool, rather than jam a complex transformations like that into its latent space.