171–180 of 512 posts
One thing that `exponentialists` forget is that each step also requires exponentially more energy and resources.
These companies full of brilliant engineers are throwing millions of dollars in training costs to produce SOTA models that are... "on par with GPT-4o and Claude Opus"? And then the next 2.23% bump will cost another XX million? It seems increasingly apparent that we are reaching the limits of throwing more data at more GPUs; that an ARC prize level breakthrough is needed to move the needle any farther at this point.
I think GPT5 will be the signal of whether or not we have hit a plateau. The space is still rapidly developing, and while large model gains are getting harder to pick apart, there have been enormous gains in the capabilities of light weight models.
Earlier quoted context omitted.
It’s not impressive that one has to go to that length though.
You can always find something to be unimpressed by I suppose, but the fact that this was fixable with plain english is impressive enough to me.
*) that is, except sometimes by making adjustments to the system prompt
Earlier quoted context omitted.
When using a prompt that involves thinking first, all three get it correct. "Count how many rs are in the word strawberry. First, list each letter and indicate whether it's an r and tally as you go, and then give a count at the end." Llama 405b: correct Mistral Large 2: correct Claude 3.5 Sonnet: correct
It’s not impressive that one has to go to that length though.
I still prefer ChatGPT-4o and use Claude if I have issues but never does any better
Claude Sonnet 3.5 outperforms GPT-4o by a significant margin on every one of my use cases.
What do you use it for?
The models are converging slowly. In the end, it will come down to the user experience and the "personality." I have been enjoying the new Claude Sonnet. It feels sharper than the others, even though it is not the highest-scoring one. One thing that `exponentialists` forget is that each step also requires exponentially more energy and resources.
Earlier quoted context omitted.
You'd think so, but 3.5-turbo was multilingual from the get go and benefitted massively from it. If you want to position yourself as a global leader, then excluding 95% of the world who aren't English native speakers seems like a bad idea.
Yeah clearly, OpenAI is rocketing forward and beyond.
I mean more on a model performance level though. It's been shown that something trained in one language trains the model to be able to output it in any other language it knows. There's quality human data being left on the table otherwise. Besides, translation is one of the few tasks that language models are by far the best at if trained properly, so why not do something you can sell as a main feature?
Earlier quoted context omitted.
I suspect this is why OpenAI is going more in the direction of optimising for price / latency / whatever with 4o-mini and whatnot. Presumably they found out long before the rest of us did that models can't really get all that much better than what we're approaching now, and once you're there the only thing you can compete on is how many parameters it takes and how cheaply you can serve that to users.
> the only thing you can compete on is how many parameters it takes and how cheaply you can serve that to users. The problem with this strategy is that it's really tough to compete with open models in this space over the long run. If you look at OpenAI's homepage right now they're trying to promote "ChatGPT on your desktop", so it's clear even they realize that most people are looking for a local product. But once ag…
OpenAI did a good move with making GPTo mini so dirty cheap that it's faster and cheaper to run than LLama 3.1 70B. Most consumers will interact with LLM via some apps using LLM API, Web Panel on desktop or native mobile app for the same reason most people use GMail etc. instead of native email client. Setting up IMAP, POP etc is for most people out of reach the same like installing Ollama + Docker + OpenWebUI
App developers are not gonna bet on local LLM only as long they are not mainstream and preinstalled on 50%+ devices.
"Mistral Large 2 is equipped with enhanced function calling and retrieval skills and has undergone training to proficiently execute both parallel and sequential function calls, enabling it to serve as the power engine of complex business applications." Why does the chart below say the "Function Calling" accuracy is about 50%? Does that mean it fails half the time with complex operations?
Relatedly, what does "parallel" function calling mean in this context?
Location and population of Paris, France
{
"role": "assistant",
"content": "",
"tool_calls": [
{
"function": {
"name": "get_city_coordinates",
"arguments": "{\"city\": \"Paris\"}"
}
}, {
"function": {
"name": "get_city_population",
"arguments": "{\"city\": \"Paris\"}"
}
}
]
}
Earlier quoted context omitted.
Agreed on everything, but calling the marvel movies slop…I think that word has gone too far.
Not all Marvel films are slop. But, as a fan who comes from a family of fans and someone who has watched almost all of them; lets be real. That particular film, really and most of them, contain copious amounts of what is absolutely slop . I don't know if the utility is worse than an LLM that is SOTA for 2 months that no one even bothers switching to however - at least the marvel slop is being used for entertainment b…
I thought that was the allure, something that's camp funny and an easy watch.
I have only watched a few of them so I am not fully familiar?