Does any one of the top models have access to the internet and googling things? I want an LLM to look things up and do casual research for me when I’m lazy.
Large Enough
331–340 of 512 posts
Re: Large Enough
#332This race for the top model is getting wild. Everyone is claiming to one-up each with every version. My experience (benchmarks aside) Claude 3.5 Sonnet absolutely blows everything away. I'm not really sure how to even test/use Mistral or Llama for everyday use though.
I don't get it. My husband also swears by Clause Sonnet 3.5, but every time I use it, the output is considerably worse than GPT-4o
Just don't listen to anecdata, and use objective metrics instead: https://chat.lmsys.org/?leaderboard
Re: Large Enough
#333Links to chat with models that released this week: Large 2 - https://chat.mistral.ai/chat Llama 3.1 405b - https://www.llama2.ai/ I just tested Mistral Large 2 and Llama 3.1 405b on 5 prompts from my Claude history. I'd rank as: 1. Sonnet 3.5 2. Large 2 and Llama 405b (similar, no clear winner between the two) If you're using Claude, stick with it. My Claude wishlist: 1. Smarter (yes, it's the most intelligent, and y…
All 3 models you ranked cannot get "how many r's are in strawberry?" correct. They all claim 2 r's unless you press them. With all the training data I'm surprised none of them fixed this yet.
How many "r"s are in [496, 675, 15717]?
Re: Large Enough
#334Sharing PyLLMs [1] reasoning benchmark results for some of the recent models. Surprised by nemo (speed/quality) and mistral large is actually pretty good (but painfully slow).
AnthropicProvider('claude-3-haiku-20240307') Median Latency: 1.61 | Aggregated speed: 122.50 | Accuracy: 44.44%
MistralProvider('open-mistral-nemo') Median Latency: 1.37 | Aggregated speed: 100.37 | Accuracy: 51.85%
OpenAIProvider('gpt-4o-mini') Median Latency: 2.13 | Aggregated speed: 67.59 | Accuracy: 59.26%
MistralProvider('mistral-large-latest') Median Latency: 10.18 | Aggregated speed: 18.64 | Accuracy: 62.96%
AnthropicProvider('claude-3-5-sonnet-20240620') Median Latency: 3.61 | Aggregated speed: 59.70 | Accuracy: 62.96%
OpenAIProvider('gpt-4o') Median Latency: 3.25 | Aggregated speed: 53.75 | Accuracy: 74.07% |
Re: Large Enough
#335Earlier quoted context omitted.
Testing models on their tokenization has always struck me as kinda odd. Like, that has nothing to do with their intelligence.
Call me when models understand when to convert the token into actual letters and count them. Can’t claim they’re more than word calculators before that.
The model communicates in a language, but our letters are not necessary for such and in fact not part of the english language. You could write english using per word pictographs and it would still be the same english&the same information/message. It's like asking you if there is a '5' in 256 but you read binary.
Re: Large Enough
#336Does any one of the top models have access to the internet and googling things? I want an LLM to look things up and do casual research for me when I’m lazy.
I'd suggest using Perplexity.
Re: Large Enough
#337Earlier quoted context omitted.
Language diversity means access to more training data, and you might also hope that by learning the same concept in multiple languages it does a better job of learning the underlying concept independent of the phrase structure... At least from a distance it seems like training a multilingual state of the art model might well be easier than a monolingual one.
Multiple input and output processes in different languages has zero effect on associative learning and creative formulation in my estimations. We've already done studies that show there is no correlation between human intelligence and knowing multiple languages, after having to put up with decades of "Americans le dumb because..." and this is no different. The amount of discourse on a single topic has a limited degre…
Humans also don't learn by reading the entire internet... assuming human psych studies apply to LLMs at all is just wrong.
Re: Large Enough
#338Earlier quoted context omitted.
It is. Strawberry is one token in many tokenziers. The model doesn't have a concept that there are letters there.
If I ask an LLM to generate new words for some concept or category, it can do that. How do the new words form, if not from joining letters?
Not letters, but tokens. Think that it's translating everything to/from Chinese.
Re: Large Enough
#339Earlier quoted context omitted.
It doesn’t test “on tokenization” though. What happens when an answer is generated is few abstraction levels deeper than tokens. A “thinking” “slice” of an llm is completely unaware of tokens as an immediate part of its reasoning. The question just shows lack of systemic knowledge about strawberry as a word (which isn’t surprising, tbh).
It is. Strawberry is one token in many tokenziers. The model doesn't have a concept that there are letters there.
If I show you a strawberry and ask how many r’s are in the name of this fruit, you can tell me, because one of the things you know about strawberries is how to spell their name.
Very large language models also “know” how to spell the word associated with the strawberry token, which you can test by asking them to spell the word one letter at a time. If you ask the model to spell the word and count the R’s while it goes, it can do the task. So the failure to do it when asked directly (how many r’s are in strawberry) is pointing to a real weakness in reasoning, where one forward pass of the transformer is not sufficient to retrieve the spelling and also count the R’s.
Re: Large Enough
#340Earlier quoted context omitted.
It’s not impressive that one has to go to that length though.
To be fair, I just asked a real person and had to go to even greater lengths: Me: How many "r"s are in strawberry? Them: What? Me: How many times does the letter "r" appear in the word "strawberry"? Them: Is this some kind of trick question? Me: No. Just literally, can you count the "r"s? Them: Uh, one, two, three. Is that right? Me: Yeah. Them: Why are you asking me this?
You need to prime the other person with a system prompt that makes them compliant and obedient.