Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Enough

mistral.ai

Post reply on HN
Print view View original

71–80 of 512 posts

Re: Large Enough

#71

Post by nathanasmith »

__jl__ wrote: post #34
happyhardcore wrote: post #17

Earlier quoted context omitted.

I suspect this is why OpenAI is going more in the direction of optimising for price / latency / whatever with 4o-mini and whatnot. Presumably they found out long before the rest of us did that models can't really get all that much better than what we're approaching now, and once you're there the only thing you can compete on is how many parameters it takes and how cheaply you can serve that to users.

Meta just claimed the opposite in their Llama 3.1 paper. Look at the conclusion. They say that their experience indicates significant gains for the next iteration of models. The current crop of benchmarks might not reflect these gains, by the way.

They also said in the paper that 405B was only trained to "compute-optimal" unlike the smaller models that were trained well past that point indicating the larger model still had some runway so had they continued it would have kept getting stronger.

Re: Large Enough

#73

Post by rkwz »

tikkun wrote: post #5

Links to chat with models that released this week: Large 2 - https://chat.mistral.ai/chat Llama 3.1 405b - https://www.llama2.ai/ I just tested Mistral Large 2 and Llama 3.1 405b on 5 prompts from my Claude history. I'd rank as: 1. Sonnet 3.5 2. Large 2 and Llama 405b (similar, no clear winner between the two) If you're using Claude, stick with it. My Claude wishlist: 1. Smarter (yes, it's the most intelligent, and y…

> Longer context window (1M+)

What's your use case for this? Uploading multiple documents/books?

Re: Large Enough

#74

Post by imtringued »

__jl__ wrote: post #34
happyhardcore wrote: post #17

Earlier quoted context omitted.

I suspect this is why OpenAI is going more in the direction of optimising for price / latency / whatever with 4o-mini and whatnot. Presumably they found out long before the rest of us did that models can't really get all that much better than what we're approaching now, and once you're there the only thing you can compete on is how many parameters it takes and how cheaply you can serve that to users.

Meta just claimed the opposite in their Llama 3.1 paper. Look at the conclusion. They say that their experience indicates significant gains for the next iteration of models. The current crop of benchmarks might not reflect these gains, by the way.

LLMs are reaching saturation on even some of the latest benchmarks and yet I am still a little disappointed by how they perform in practice.

They are by no means bad, but I am now mostly interested in long context competency. We need benchmarks that force the LLM to complete multiple tasks simultaneously in one super long session.

Re: Large Enough

#75

Post by skybrian »

Liquix wrote: post #10

These companies full of brilliant engineers are throwing millions of dollars in training costs to produce SOTA models that are... "on par with GPT-4o and Claude Opus"? And then the next 2.23% bump will cost another XX million? It seems increasingly apparent that we are reaching the limits of throwing more data at more GPUs; that an ARC prize level breakthrough is needed to move the needle any farther at this point.

I think it’s impressive that they’re doing it on a single (large) node. Costs matter. Efficiency improvements like this will probably increase capabilities eventually.

I’m also optimistic about building better (rather than bigger) datasets to train on.

Re: Large Enough

#76

Post by tikkun »

rkwz wrote: post #73
tikkun wrote: post #5

Links to chat with models that released this week: Large 2 - https://chat.mistral.ai/chat Llama 3.1 405b - https://www.llama2.ai/ I just tested Mistral Large 2 and Llama 3.1 405b on 5 prompts from my Claude history. I'd rank as: 1. Sonnet 3.5 2. Large 2 and Llama 405b (similar, no clear winner between the two) If you're using Claude, stick with it. My Claude wishlist: 1. Smarter (yes, it's the most intelligent, and y…

> Longer context window (1M+) What's your use case for this? Uploading multiple documents/books?

Correct

Re: Large Enough

#77

Post by rvnx »

chipdart wrote: post #56
Workaccount2 wrote: post #35

Earlier quoted context omitted.

I think GPT5 will be the signal of whether or not we have hit a plateau. The space is still rapidly developing, and while large model gains are getting harder to pick apart, there have been enormous gains in the capabilities of light weight models.

> I think GPT5 will be the signal of whether or not we have hit a plateau. I think GPT5 will tell if OpenAI hit a plateau. Sam Altman has been quoted as claiming "GPT-3 had the intelligence of a toddler, GPT-4 was more similar to a smart high-schooler, and that the next generation will look to have PhD-level intelligence (in certain tasks)" Notice the high degree of upselling based on vague claims of performance, and…

PhD level-of-task-execution sounds like the LLM will debate whether the task is ethical instead of actually doing it

Re: Large Enough

#78

Post by splwjs »

__jl__ wrote: post #34
happyhardcore wrote: post #17

Earlier quoted context omitted.

I suspect this is why OpenAI is going more in the direction of optimising for price / latency / whatever with 4o-mini and whatnot. Presumably they found out long before the rest of us did that models can't really get all that much better than what we're approaching now, and once you're there the only thing you can compete on is how many parameters it takes and how cheaply you can serve that to users.

Meta just claimed the opposite in their Llama 3.1 paper. Look at the conclusion. They say that their experience indicates significant gains for the next iteration of models. The current crop of benchmarks might not reflect these gains, by the way.

I sell widgets. I promise the incalculable power of widgets has yet to be unleashed on the world, but it is tremendous and awesome and we should all be very afraid of widgets taking over the world because I can't see how they won't.

Anyway here's the sales page. the widget subscription is so premium you won't even miss the subscription fee.

Re: Large Enough

#79

Post by ketzo »

rkwz wrote: post #73
tikkun wrote: post #5

Links to chat with models that released this week: Large 2 - https://chat.mistral.ai/chat Llama 3.1 405b - https://www.llama2.ai/ I just tested Mistral Large 2 and Llama 3.1 405b on 5 prompts from my Claude history. I'd rank as: 1. Sonnet 3.5 2. Large 2 and Llama 405b (similar, no clear winner between the two) If you're using Claude, stick with it. My Claude wishlist: 1. Smarter (yes, it's the most intelligent, and y…

> Longer context window (1M+) What's your use case for this? Uploading multiple documents/books?

Uploading large codebases is particularly useful.

Re: Large Enough

#80

Post by Tostino »

romeros wrote: post #63
chipdart wrote: post #41

Earlier quoted context omitted.

> It seems increasingly apparent that we are reaching the limits of throwing more data at more GPUs; I think you're just seeing the "make it work" stage of the combo "first make it work, then make it fast". Time to market is critical, as you can attest by the fact you framed the situation as "on par with GPT-4o and Claude Opus". You're seeing huge investments because being the first to get a working model stands to b…

ChatGPT is like Google now. It is the default. Even if Claude becomes as good as ChatGPT or even slightly better it won't make me switch. It has to be like a lot better. Way better. It feels like ChatGPT won the time to market war already.

Eh, with the degradation of coding performance in ChatGPT I made the switch. Seems much better to work with on problems, and I have to do way less hand holding to get good results.

I'll switch again soon as something better is out.

Post reply on HN