Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Enough

mistral.ai

Post reply on HN
Print view View original

341–350 of 512 posts

Re: Large Enough

#341

Post by impossiblefork »

lolinder wrote: post #129
Liquix wrote: post #10

These companies full of brilliant engineers are throwing millions of dollars in training costs to produce SOTA models that are... "on par with GPT-4o and Claude Opus"? And then the next 2.23% bump will cost another XX million? It seems increasingly apparent that we are reaching the limits of throwing more data at more GPUs; that an ARC prize level breakthrough is needed to move the needle any farther at this point.

> It seems increasingly apparent that we are reaching the limits of throwing more data at more GPUs Yes. This is exactly why I'm skeptical of AI doomerism/saviorism. Too many people have been looking at the pace of LLM development over the last two (2) years, modeled it as an exponential growth function, and come to the conclusion that AGI is inevitable in the next ${1-5} years and we're headed for ${(dys|u)topia}. B…

Notice though, that all these improvements have been with pretty basic transformer models that output all their tokens-- no internal thoughts, no search, no architecture improvements and things are only fed through them once.

But we could add internal thoughts-- we could make the model generate tokens that aren't part of its output but are there for it to better figure out its next token. This was tried QuietSTAR.

Hochreiter is also active with alternative models, and there's all the microchip design companies, Groq, Etched, etc. trying to speed up models and reduce model running cost.

Therefore, I think there's room for very great improvements. They may not come right away, but there are so many obvious paths to improve things that I think it's unreasonable to think progress has stalled. Also, presumably GPT-5 isn't far away.

Re: Large Enough

#342

Post by microtonal »

TiredOfLife wrote: post #321
microtonal wrote: post #302

Earlier quoted context omitted.

The problem is that the model never gets to see individual letters. The tokenizers used by these models break up the input in pieces. Even though the smallest pieces/units are bytes in most encodings (e.g. BBPE), the tokenizer will cut up most of the input in much larger units, because the vocabulary will contain fragments of words or even whole words. For example, if we tokenize Welcome to Hacker News, I hope you li…

The thing is, how the tokenizing work is about as relevant to the person asking the question as name of the cat of the delivery guy who delivered the GPU that the llm runs on.

How the tokenizer works explains why a model can’t answer the question, what the name of the cat is doesn’t explain anything.

This is Hacker News, we are usually interested in how things work.

Re: Large Enough

#343

Post by logicchains »

RyanAdamas wrote: post #315
gpm wrote: post #193

Earlier quoted context omitted.

Language diversity means access to more training data, and you might also hope that by learning the same concept in multiple languages it does a better job of learning the underlying concept independent of the phrase structure... At least from a distance it seems like training a multilingual state of the art model might well be easier than a monolingual one.

Multiple input and output processes in different languages has zero effect on associative learning and creative formulation in my estimations. We've already done studies that show there is no correlation between human intelligence and knowing multiple languages, after having to put up with decades of "Americans le dumb because..." and this is no different. The amount of discourse on a single topic has a limited degre…

Google mentioned this in one of their papers, they found for large enough models including more languages did indeed lead to an overall increase in performance.

Re: Large Enough

#344

Post by novok »

logicchains wrote: post #304
novok wrote: post #297

I kind of wonder why a lot of these places don't give "amateur" sized models anymore at around the 18B & 30B parameter sizes that you can run on a single 3090 or M2 Max at reasonable speeds and RAM requirements? It's all 7B, 70B, 400B sizing nowadays.

Because you can just quantise the 70B model to 3-4 bits and it'll perform better than a 30B model but be a similar size.

A 70B 4bit model does not fit in a 24GB VRAM card, 30B models are the sweet spot for that size of card at 20GB, with 4GB left for the system to still function.

Re: Large Enough

#346

Post by PhilippGille »

lostmsu wrote: post #332
skerit wrote: post #198

Earlier quoted context omitted.

I don't get it. My husband also swears by Clause Sonnet 3.5, but every time I use it, the output is considerably worse than GPT-4o

Just don't listen to anecdata, and use objective metrics instead: https://chat.lmsys.org/?leaderboard

You might also want to look into other benchmarks: https://old.reddit.com/r/LocalLLaMA/comments/1ean2i6/the_fin...

Re: Large Enough

#347

Post by freediver »

coder543 wrote: post #238
freediver wrote: post #126

Earlier quoted context omitted.

That would make each API call cost at least $3 ($3 is price per million input tokens). And if you have a 10 message interaction you are looking at $30+ for the interaction. Is that what you would expect?

Gemini 1.5 Pro charges $0.35/million tokens up to the first million tokens or $0.70/million tokens for prompts longer than one million tokens, and it supports a multi-million token context window. Substantially cheaper than $3/million, but I guess Anthropic’s prices are higher.

It is also much worse.

Re: Large Enough

#348

Post by stavros »

SirMaster wrote: post #268
generalizations wrote: post #221

Earlier quoted context omitted.

Testing models on their tokenization has always struck me as kinda odd. Like, that has nothing to do with their intelligence.

How is a layman supposed to even know that it's testing on that? All they know is it's a large language model. It's not unreasonable they should expect it to be good at things having to do with language, like how many letters are in a word. Seems to me like a legit question for a young child to answer or even ask.

> How is a layman supposed to even know that it's testing on that?

They're not, but laymen shouldn't think that the LLM tests they come up with have much value.

Re: Large Enough

#349

Post by brandall10 »

paulcole wrote: post #254
ChikkaChiChi wrote: post #46

Earlier quoted context omitted.

4o will get the answer right on the first go if you ask it "Search the Internet to determine how many R's are in strawberry?" which I find fascinating

I didn't even need to do that. 4o got it right straight away with just: "how many r's are in strawberry?" The funny thing is, I replied, "Are you sure?" and got back, "I apologize for the mistake. There are actually two 'r's in the word strawberry."

All that's happening is it finds 3 most commonly in the training set. When you push it, it responds with the next most common answer.

Re: Large Enough

#350

Post by coder543 »

freediver wrote: post #347
coder543 wrote: post #238

Earlier quoted context omitted.

Gemini 1.5 Pro charges $0.35/million tokens up to the first million tokens or $0.70/million tokens for prompts longer than one million tokens, and it supports a multi-million token context window. Substantially cheaper than $3/million, but I guess Anthropic’s prices are higher.

It is also much worse.

Is it, though? In my limited tests, Gemini 1.5 Pro (through the API) is very good at tasks involving long context comprehension.

Google's user-facing implementations of Gemini are pretty consistently bad when I try them out, so I understand why people might have a bad impression about the underlying Gemini models.

Post reply on HN