Super looking forward to this.
I tried Codestral and nothing came close. Not even slightly. It was the only LLM that consistently put out code for me that was runnable and idiomatic.
Large Enough
461–470 of 512 posts
Re: Large Enough
#462Earlier quoted context omitted.
To help keep track of the race, I put together a simple dashboard to visualize model/provider leaders in capability, throughput, and cost. Hope someone finds it useful! Google Sheet: https://docs.google.com/spreadsheets/d/1foc98Jtbi0-GUsNySddv...
Not my site, but check out https://artificialanalysis.ai
Familiar! The Artificial Analysis Index is the metric models are sorted by in my sheet. But their data and presentation has some gaps.
I made this sheet to get a glanceable landscape view comparing the three key dimensions I care about, and fill in the missing evals. AA only lists scores for a few increasingly-dated and problematic evals benchmarks. Not just my opinion, none of their listed metrics are in HuggingFace Leaderboard 2 (June 2024).
That said I love the AA Index score because it provides a single normalized score that blends vibe-check qual (chatbot elo) with widely reported quant (MMLU, MT Bench). I wish it composed more contemporary evals, but don't have the rigor/attention to make my own score and am not aware of a better substitute.
Re: Large Enough
#463Links to chat with models that released this week: Large 2 - https://chat.mistral.ai/chat Llama 3.1 405b - https://www.llama2.ai/ I just tested Mistral Large 2 and Llama 3.1 405b on 5 prompts from my Claude history. I'd rank as: 1. Sonnet 3.5 2. Large 2 and Llama 405b (similar, no clear winner between the two) If you're using Claude, stick with it. My Claude wishlist: 1. Smarter (yes, it's the most intelligent, and y…
All 3 models you ranked cannot get "how many r's are in strawberry?" correct. They all claim 2 r's unless you press them. With all the training data I'm surprised none of them fixed this yet.
While strawberry can be attributed to tokenization here are some other basic stuff I’ve seen language models fail at:
1. Play tic tac toe such that you never lose
2. Which is bigger 9.11 or 9.9
3. 4 digit multiplication even with CoT prompting
Re: Large Enough
#464Earlier quoted context omitted.
Does not seem work universally. Just tested a few with this prompt "create a javascript function to count any letter in any word. Run this function for the letter "r" and the word "strawberry" and print the count" ChatGPT-4o => Output is 3. Passed Claude3.5 => Output is 2. Failed. Told it the count is wrong. It apologised and then fixed the issue in the code. Output is now 3. Useless if the human does not spot the er…
Your function returns 3, and I don't see how it can return 2.
I did not run the code myself. The code block and console log I have pasted is verbatim copy from Claude3.5
Re: Large Enough
#465Earlier quoted context omitted.
But then why does it stick to its guns on other questions but not this one?
I haven't played with this model, but rarely do I find working w/ Claude or GPT-4 for that to be the case. If you say it's incorrect, it will give you another answer instead of insisting on correctness.
Wait what? You haven’t used 4o and you confidently described how it works?
Re: Large Enough
#466Earlier quoted context omitted.
>If you insert characters to breaks the tokens down, it find the correct result: how many r's are in "s"t"r"a"w"b"e"r"r"y" ? The issue is that humans don't talk like this. I don't ask someone how many r's there are in strawberry by spelling out strawberry, I just say the word.
Humans also constantly make mistakes that are due to proximity in their internal representation. "Could of"/"Should of" comes to mind: the letters "of" have a large edit distance from "'ve", but their pronunciation is very similar. Especially native speakers are prone to the mistake as they grew up learning english as illiterate children, from sounds only, compared to how most people learning english as second langua…
Yeah, but for most people, it would be because the don't know how to spell "Philippines" at all. Confoundingly, LLMs know exactly how to spell Strawberry and still get this wrong.
Re: Large Enough
#467Earlier quoted context omitted.
I stopped my ChatGPT subscription and subscribed instead to Claude, it's simply much better. But, it's hard to tell how much better day to day beyond my main use cases of coding. It is more that I felt ChatGPT felt degraded than Claude were much better. The hedonic treadmill runs deep.
Claude’s license is too insane, you can’t use it for anything that competes with the everything thing. Not sure what folks who accept Anthropic license are thinking after they read the terms. Seems they didn’t read the terms, and they aren’t thinking? (Wouldn’t you want outputs you could use to compete with intelligence??? What are you thinking after you read their terms?)
If it really is as you say then that sounds like it won't hold up when challenged in court but IANAL...
Re: Large Enough
#468Earlier quoted context omitted.
On the other hand explain to me how you are able to read the word “spotvoxilhapentosh”.
I think that humans indeed identify words as a whole and do not read letter by letter. However, this implies you need to know the word to begin with. I can write "asdf" and you might be oblivious to what I mean. I can mention "adsf" to a JavaScript developer and he will immediately think of the tool versioning tool. Because context and familiarity is important.
I believe it's a bit more nuanced than that. Short ubiquitous words like "and" or "the" we instantly recognize at a glance, but long unfamiliar or rarer words we read from the beginning, one syllable or letter at a time, until pattern recognition from memory kicks in. All unconsciously, unless the word is so odd, out of place, mispelled, or unknown that it comes to conscious awareness and interrupts our reading.
Re: Large Enough
#469Earlier quoted context omitted.
I stopped my ChatGPT subscription and subscribed instead to Claude, it's simply much better. But, it's hard to tell how much better day to day beyond my main use cases of coding. It is more that I felt ChatGPT felt degraded than Claude were much better. The hedonic treadmill runs deep.
GPT-4 was probably as good as Claude Sonnet 3.5 at its outset, but OpenAI ran it into the ground with whatever they’re doing to save on inference costs, otherwise scale, align it, or add dumb product features.
> OpenAI ran it into the ground with whatever they’re doing to save on inference costs, otherwise scale, align it, or add dumb product features.
They googlified it. (Yandex isn't better at google because it improved. It's better because it stayed mostly the same.)
My recommendation to disrupting industry leaders now is becoming good enough and then simply wait until the leader self-implodes.
Re: Large Enough
#470Earlier quoted context omitted.
Your function returns 3, and I don't see how it can return 2.
I did not run the code myself. The code block and console log I have pasted is verbatim copy from Claude3.5
Claude isn't actually running console.log() it produced correct code.
This prompt "please write a javascript function that takes a string and a letter and iterates over the characters in a string and counts the occurrences of the letter"
Produced a correct function given both chatGPT4o and claude3.5 for me.