21–30 of 118 posts
Why are all the comments here so negative... this is a good thing, turbo has less memorization but keeps the same reasoning ability. That's excellent and a relief.
Earlier quoted context omitted.
I mean sure, it memorized some of the answers. I'm not denying that. Clearly, it didn't memorize all of them.
When people say "oh look how amazing, it can solve programming problems!" when in fact it has only seen the models CHEAT, is an enormous problem. For cases where finding the answer it's perfectly fine, but it's not fine for claims that it can code. There's a huge difference.
Earlier quoted context omitted.
I mean sure, it memorized some of the answers. I'm not denying that. Clearly, it didn't memorize all of them.
When people say "oh look how amazing, it can solve programming problems!" when in fact it has only seen the models CHEAT, is an enormous problem. For cases where finding the answer it's perfectly fine, but it's not fine for claims that it can code. There's a huge difference.
It can code and it has memorized some coding questions are not mutually exclusive.
Aider has had an Exercism benchmarking suite for quite some time. Interestingly, my benchmark results of GPT 4 Turbo show an opposite result: the new gpt-4-1106-preview did significantly better on the first try than the March and June models. https://aider.chat/docs/benchmarks-1106.html Aider benchmarks against the 133 Exercism python exercises, not js exercises that mentat's benchmark uses. So this is not an apples-…
Wouldn't this actually be exactly proof that the model has improved over its predecessor by having to solve the problem itself rather than rely on memory?
What use is a model that memorizes the answers to all the benchmarks (see the 7b models on open llm leaderboard for more info on that).
For my NLP pipelines, I batch n-articles together to process (extract fields from) in one prompt (final output is something like this {"1":[{}], "2": [{},{}]...}) in one message. Compute-wise it's inefficient but OpenAI charges by the token so it doesn't matter. It's very reliable on gpt-4 8k.
I was also pretty happy with the results on 4-turbo initially but it seems that once you go past 30k-ish tokens in context (needs way more testing), it shits itself. The indexes don't match anymore and n_final_output is different from n_articles.
Still, great model and even if the limits are lower in practice I suspect I'll get good use out of it.
Edit: With better prompting, it feels stable at n=42, ~42000 prompt tokens.
I've definitely had instances where 4 memorized a common puzzle and failed a subtly altered variant but then got the variant after changing variable names or otherwise making it look different from what it would have memorized.
> We designed a test for this theory: we reran the benchmarks without showing the models the instructions to each exercise. Instead, we just told them that they were Exercism exercises, and gave them the exercise names and function stubs. This summarizes all my skepticism agains the AI field. Pretty clear that they aren't solving the problem, they have them memorized.
For example the function stubs I can find are "value_of_card()" in exercise "Black Jack", or "generate_seat_letters()" in exercise "Plane Tickets". I think I could guess those without seeing the rest of the question.
Earlier quoted context omitted.
I mean sure, it memorized some of the answers. I'm not denying that. Clearly, it didn't memorize all of them.
When people say "oh look how amazing, it can solve programming problems!" when in fact it has only seen the models CHEAT, is an enormous problem. For cases where finding the answer it's perfectly fine, but it's not fine for claims that it can code. There's a huge difference.
Earlier quoted context omitted.
I mean sure, it memorized some of the answers. I'm not denying that. Clearly, it didn't memorize all of them.
When people say "oh look how amazing, it can solve programming problems!" when in fact it has only seen the models CHEAT, is an enormous problem. For cases where finding the answer it's perfectly fine, but it's not fine for claims that it can code. There's a huge difference.