Large Enough

Post by TiredOfLife » Wed, Jul 24, 2024, 7:46 PM UTC

microtonal wrote: post #302

VincentEvans wrote: post #279

Earlier quoted context omitted.

I don’t know anything about LLMs beyond using ChatGPT and Copilot… but unless because of this lack of knowledge I am misinterpreting your reply - it sounds as if you are excusing the model giving a completely wrong answer to a question that anyone intelligent enough to learn alphabet can answer correctly.

The problem is that the model never gets to see individual letters. The tokenizers used by these models break up the input in pieces. Even though the smallest pieces/units are bytes in most encodings (e.g. BBPE), the tokenizer will cut up most of the input in much larger units, because the vocabulary will contain fragments of words or even whole words. For example, if we tokenize Welcome to Hacker News, I hope you li…

Post by swalsh » Wed, Jul 24, 2024, 7:47 PM UTC

bugglebeetle wrote: post #12

satvikpendem wrote: post #7

Earlier quoted context omitted.

I stopped my ChatGPT subscription and subscribed instead to Claude, it's simply much better. But, it's hard to tell how much better day to day beyond my main use cases of coding. It is more that I felt ChatGPT felt degraded than Claude were much better. The hedonic treadmill runs deep.

GPT-4 was probably as good as Claude Sonnet 3.5 at its outset, but OpenAI ran it into the ground with whatever they’re doing to save on inference costs, otherwise scale, align it, or add dumb product features.

Post by dvansoye » Wed, Jul 24, 2024, 7:49 PM UTC

lolinder wrote: post #129

Liquix wrote: post #10

These companies full of brilliant engineers are throwing millions of dollars in training costs to produce SOTA models that are... "on par with GPT-4o and Claude Opus"? And then the next 2.23% bump will cost another XX million? It seems increasingly apparent that we are reaching the limits of throwing more data at more GPUs; that an ARC prize level breakthrough is needed to move the needle any farther at this point.

> It seems increasingly apparent that we are reaching the limits of throwing more data at more GPUs Yes. This is exactly why I'm skeptical of AI doomerism/saviorism. Too many people have been looking at the pace of LLM development over the last two (2) years, modeled it as an exponential growth function, and come to the conclusion that AGI is inevitable in the next ${1-5} years and we're headed for ${(dys|u)topia}. B…

Post by baq » Wed, Jul 24, 2024, 7:54 PM UTC

antonvs wrote: post #322

baq wrote: post #275

Earlier quoted context omitted.

Call me when models understand when to convert the token into actual letters and count them. Can’t claim they’re more than word calculators before that.

That's misleading. When you read and comprehend text, you don't read it letter by letter, unless you have a severe reading disability. Your ability to comprehend text works more like an LLM. Essentially, you can compare the human brain to a multi-model or modular system. There are layers or modules involved in most complex tasks. When reading, you recognize multiple letters at a time[ ], and those letters are essenti…

Post by read_if_gay_ » Wed, Jul 24, 2024, 7:58 PM UTC

swalsh wrote: post #323

bugglebeetle wrote: post #12

Earlier quoted context omitted.

GPT-4 was probably as good as Claude Sonnet 3.5 at its outset, but OpenAI ran it into the ground with whatever they’re doing to save on inference costs, otherwise scale, align it, or add dumb product features.

GPT-4 degraded significantly, but you probably have some rosey glasses on. Sonnet is signifcantly better.