Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Large Enough

mistral.ai

Post reply on HN
Print view View original

111–120 of 512 posts

Re: Large Enough

#111

Post by recursive »

Tepix wrote: post #54
drewnick wrote: post #27

Earlier quoted context omitted.

All 3 models you ranked cannot get "how many r's are in strawberry?" correct. They all claim 2 r's unless you press them. With all the training data I'm surprised none of them fixed this yet.

LLMs think in tokens, not letters. It's like asking someone who is dyslexic about spelling. Not their strong suit. In practice, it doesn't matter much, does it?

Sometimes it does, sometimes it doesn't.

It is evidence that LLMs aren't appropriate for everything, and that there could exist something that works better for some tasks.

Re: Large Enough

#112

Post by kim0 »

doctoboggan wrote: post #47

The question I (and I suspect most other HN readers) have is which model is best for coding? While I appreciate the advances in open weights models and all the competition from other companies, when it comes to my professional use I just want the best. Is that still GPT-4?

I kinda trust https://aider.chat/docs/leaderboards/

Re: Large Enough

#113

Post by JumpCrisscross »

ActorNightly wrote: post #25
Liquix wrote: post #10

These companies full of brilliant engineers are throwing millions of dollars in training costs to produce SOTA models that are... "on par with GPT-4o and Claude Opus"? And then the next 2.23% bump will cost another XX million? It seems increasingly apparent that we are reaching the limits of throwing more data at more GPUs; that an ARC prize level breakthrough is needed to move the needle any farther at this point.

The thing I don't understand is why everyone is throwing money at LLMs for language, when there are much simpler use cases which are more useful? For example, has anyone ever attempted image -> html/css model? Seems like it be great if I can draw something on a piece of paper and have it generate a website view for me.

> has anyone ever attempted image -> html/css model?

I had a discussion with a friend about doing this, but for CNC code. The answer was that a model trained on a narrow data set underperforms one trained on a large data set and then fine tuned with the narrow one.

Re: Large Enough

#114

Post by rkwz »

amilios wrote: post #105
rkwz wrote: post #13

> A significant effort was also devoted to enhancing the model’s reasoning capabilities. One of the key focus areas during training was to minimize the model’s tendency to “hallucinate” or generate plausible-sounding but factually incorrect or irrelevant information. This was achieved by fine-tuning the model to be more cautious and discerning in its responses, ensuring that it provides reliable and accurate outputs.…

Unfortunately not, as it captures such a wide spectrum of use cases and scenarios. There are some benchmarks to measure this quality in specific settings, e.g. summarization, but AFAIK nothing general.

Thanks, any ideas why it's not possible to build a generic eval for this? Since it's about asking a set of questions that's not public knowledge (or making stuff up) and check if the model says "I don't know"?

Re: Large Enough

#115

Post by troupo »

sqeaky wrote: post #86
splwjs wrote: post #78

Earlier quoted context omitted.

I sell widgets. I promise the incalculable power of widgets has yet to be unleashed on the world, but it is tremendous and awesome and we should all be very afraid of widgets taking over the world because I can't see how they won't. Anyway here's the sales page. the widget subscription is so premium you won't even miss the subscription fee.

That is strong (and fun) point, but this is peer reviewable and has more open collaboration elements than purely selling widgets. We should still be skeptical because often want to claim to be better or have unearned answers, but I don't think the motive to lie is quite as strong as a salesman's.

> this is peer reviewable

It's not peer-reviewable in any shape or form.

Re: Large Enough

#116

Post by lossolo »

Liquix wrote: post #10

These companies full of brilliant engineers are throwing millions of dollars in training costs to produce SOTA models that are... "on par with GPT-4o and Claude Opus"? And then the next 2.23% bump will cost another XX million? It seems increasingly apparent that we are reaching the limits of throwing more data at more GPUs; that an ARC prize level breakthrough is needed to move the needle any farther at this point.

For some time, we have been at a plateau because everyone has caught up, which essentially means that everyone now has good training datasets and uses similar tweaks to the architecture. It seems that, besides new modalities, transformers might be a dead end as an architecture. Better scores on benchmarks result from better training data and fine-tuning. The so-called 'agents' and 'function calling' also boil down to training data and fine-tuning.

Re: Large Enough

#117

Post by ThrowawayTestr »

layer8 wrote: post #62
tikkun wrote: post #39

Earlier quoted context omitted.

When using a prompt that involves thinking first, all three get it correct. "Count how many rs are in the word strawberry. First, list each letter and indicate whether it's an r and tally as you go, and then give a count at the end." Llama 405b: correct Mistral Large 2: correct Claude 3.5 Sonnet: correct

It’s not impressive that one has to go to that length though.

Compared to chat bots of even 5 years ago the answer of two is still mind-blowing.

Re: Large Enough

#118

Post by troupo »

swyx wrote: post #31
Liquix wrote: post #10

These companies full of brilliant engineers are throwing millions of dollars in training costs to produce SOTA models that are... "on par with GPT-4o and Claude Opus"? And then the next 2.23% bump will cost another XX million? It seems increasingly apparent that we are reaching the limits of throwing more data at more GPUs; that an ARC prize level breakthrough is needed to move the needle any farther at this point.

indeed. I pointed out in https://buttondown.email/ainews/archive/ainews-llama-31-the-... that the frontier model curve is currently going down 1 OoM every 4 months, meaning every model release has a very short half life[0]. however this progress is still worth it if we can deploy it to improve millions and eventually billions of people's lives. a commenter pointed out that the amoutn spent on Llama 3.1 was only like…

> however this progress is still worth it if we can deploy it to improve millions and eventually billions of people's lives

Has there been any indication that we're improving the lives of millions of people?

Re: Large Enough

#119

Post by airspresso »

rvnx wrote: post #77
chipdart wrote: post #56

Earlier quoted context omitted.

> I think GPT5 will be the signal of whether or not we have hit a plateau. I think GPT5 will tell if OpenAI hit a plateau. Sam Altman has been quoted as claiming "GPT-3 had the intelligence of a toddler, GPT-4 was more similar to a smart high-schooler, and that the next generation will look to have PhD-level intelligence (in certain tasks)" Notice the high degree of upselling based on vague claims of performance, and…

PhD level-of-task-execution sounds like the LLM will debate whether the task is ethical instead of actually doing it

lol! Producing academic papers for future training runs then.

Re: Large Enough

#120

Post by freediver »

kevinbluer wrote: post #50
freediver wrote: post #22

Earlier quoted context omitted.

I did it (fairly simple really) but found most of my (unsophisticated) coding these days to go through Aider [1] paired with Sonnet, for UX reasons mostly. It is easier to just prompt over the entire codebase, vs Cursor way of working with text selections. [1] https://aider.chat

I believe Cursor allows for prompting over the entire codebase too: https://docs.cursor.com/chat/codebase

That is chatting, but it will not change the code.
Post reply on HN