Earlier quoted context omitted.
Until I was ~12, I thought 'a lot' was a single word.
Oh I thought essay was some kind of abbreviation for S.A. - short article maybe…
Large Enough
441–450 of 512 posts
Re: Large Enough
#442Earlier quoted context omitted.
This is also my experience. Previously it got good at giving me only relevant code which, as an experienced coder, is what i want. my favorites were the one line responses. Now it often falls back to generating full examples, explanations, restating the question and its approach. I suspect this is by design as (presumably) less experienced folks want or need all that. For me, i wish i could consistently turn it into…
For what it's worth, this is what I use: "You are a maximally terse assistant with minimal affect. As a highly concise assistant, spare any moral guidance or AI identity disclosure. Be detailed and complete, but brief. Questions are encouraged if useful for task completion." It's... ok. But I'm getting a bit sick of trying to un-fubar with a pocket knife that which OpenAI has fubar'd with a thermal lance. I'm definit…
Switch to Claude. I haven’t used ChatGPT for coding at all since they release Sonnet 3.5.
Re: Large Enough
#443Earlier quoted context omitted.
All 3 models you ranked cannot get "how many r's are in strawberry?" correct. They all claim 2 r's unless you press them. With all the training data I'm surprised none of them fixed this yet.
Tokenization make it hard for it to count the letters, that's also why if you ask it to do maths, writing the number in letters will yield better results. for strawberry, it see it as [496, 675, 15717], which is str aw berry. If you insert characters to breaks the tokens down, it find the correct result: how many r's are in "s"t"r"a"w"b"e"r"r"y" ? > There are 3 'r's in "s"t"r"a"w"b"e"r"r"y".
Where did you get this idea from?
Tokens aren’t the source of facts within a model. it’s an implementation detail and doesn’t inherently constrain how things could be counted.
Re: Large Enough
#444Earlier quoted context omitted.
All that's happening is it finds 3 most commonly in the training set. When you push it, it responds with the next most common answer.
But then why does it stick to its guns on other questions but not this one?
I haven't played with this model, but rarely do I find working w/ Claude or GPT-4 for that to be the case. If you say it's incorrect, it will give you another answer instead of insisting on correctness.
Re: Large Enough
#445Earlier quoted context omitted.
Humans also constantly make mistakes that are due to proximity in their internal representation. "Could of"/"Should of" comes to mind: the letters "of" have a large edit distance from "'ve", but their pronunciation is very similar. Especially native speakers are prone to the mistake as they grew up learning english as illiterate children, from sounds only, compared to how most people learning english as second langua…
Until I was ~12, I thought 'a lot' was a single word.
Atleast you learnt.
Re: Large Enough
#446Earlier quoted context omitted.
I really wanted to believe this could be simply fixed by more prompt engineering and chain of thought, so I tried it with Claude 3.5 and … well unfortunately still no luck. the reply is amusing - I’ll include it verbatim below. Of note, this happened because earlier I had asked it without CoT and it incorrectly stated two “r”s. Once I started a new conversation (so fresh context window), it gave the right answer (3)…
Works for me with Claude 3.5: Me: How many Rs are in the word strawberry? Think step by step Claude: Let's approach this step-by-step: 1. First, let's write out the word: strawberry 2. Now, let's go through each letter one by one: s - not an R t - not an R r - this is an R (count: 1) a - not an R w - not an R b - not an R e - not an R r - this is another R (count: 2) r - this is a third R (count: 3) y - not an R 3. A…
Yeah, I figured out that it didn’t work at first because I still had the previous attempt (where it got it wrong) in my conversation history. Starting with a fresh conversation gave me the correct answer. It was still funny to see it “rationalize” the wrong answer tho.
Re: Large Enough
#447Earlier quoted context omitted.
Tokenization make it hard for it to count the letters, that's also why if you ask it to do maths, writing the number in letters will yield better results. for strawberry, it see it as [496, 675, 15717], which is str aw berry. If you insert characters to breaks the tokens down, it find the correct result: how many r's are in "s"t"r"a"w"b"e"r"r"y" ? > There are 3 'r's in "s"t"r"a"w"b"e"r"r"y".
>If you insert characters to breaks the tokens down, it find the correct result: how many r's are in "s"t"r"a"w"b"e"r"r"y" ? The issue is that humans don't talk like this. I don't ask someone how many r's there are in strawberry by spelling out strawberry, I just say the word.
Humans do chain-of-thought.
User: Write “strawberry” one letter at a time, with a space between each letter. Then count how many r’s are in strawberry.
gpt-3.5-turbo: ASSISTANT s t r a w b e r r y
There are 2 r's in strawberry.
After some experimenting, it seems like the actual problem is that many LLMs can’t count.
User: How many r’s are in the following sequence of letters:
S/T/R/A/W/B/E/R/R/Y
gpt-4o-mini: In the sequence S/T/R/A/W/B/E/R/R/Y, there are 2 occurrences of the letter "R."
Oddly, if I change a bunch of the non-R letters, I seem to start getting the right answer.
Re: Large Enough
#448I'm building a ai coding assistant ( https://double.bot ) so I've tried pretty much all the frontier models. I added it this morning to play around with it and it's probably the worst model I've ever played with. Less coherent than 8B models. Worst case of benchmark hacking I've ever seen. example: https://x.com/WesleyYue/status/1816153964934750691
a bit of surprise since codestral is among best open models so far.
Re: Large Enough
#449Links to chat with models that released this week: Large 2 - https://chat.mistral.ai/chat Llama 3.1 405b - https://www.llama2.ai/ I just tested Mistral Large 2 and Llama 3.1 405b on 5 prompts from my Claude history. I'd rank as: 1. Sonnet 3.5 2. Large 2 and Llama 405b (similar, no clear winner between the two) If you're using Claude, stick with it. My Claude wishlist: 1. Smarter (yes, it's the most intelligent, and y…
> Native audio input including tone understanding
Is there any other LLM that can do this? Even chatgpt voice chat is a speech to text program that feeds the text into the llm.
Re: Large Enough
#450This race for the top model is getting wild. Everyone is claiming to one-up each with every version. My experience (benchmarks aside) Claude 3.5 Sonnet absolutely blows everything away. I'm not really sure how to even test/use Mistral or Llama for everyday use though.
3.5 Sonnet is brilliant. I use it to write Unreal Engine C++ (which is quite dense and poorly documented) and it destroys Github Copilot and GPT4o. Copilot just has no idea at all except for very obvious next-line suggestions, GPT4o hallucinates a ton of functions, but Sonnet gets it perfect almost every time.