Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small Models Have Arrived

calv.info

Post reply on HN
Print view View original

201–210 of 370 posts

Re: Small Models Have Arrived

#201

Post by shimman »

applfanboysbgon wrote: post #98
jimmaswell wrote: post #91

Earlier quoted context omitted.

This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always perfor…

This idea has not failed to pan out at all. I work for a startup that is exactly what GP described, and am set for life because of how wildly successful it is. Notably, we are successful, in a genuine sense of the word: we bootstrapped from running tiny models to larger and larger models on our own slowly improving fleet of GPUs, and now have millions in revenue without a single dime of outside investment. Conversely…

Are you willing to say the same of the startup or are you still stealth?

Re: Small Models Have Arrived

#202

Post by theendisney »

swatcoder wrote: post #77
NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

Yes. The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words. The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of res…

At work i only had early copilot which was hysterically bad at everything. As i wanted it to do the same task repeatedly and could spot wrong results instantly i kept evolving a prompt that attempted to correct all ways it found to do it wrong. It kept inventing new ways to get it wrong until it eventually got it right 90% of the time. My theory is that an avanced model that has no issues with a task could do the prompt enginering much better than i ever could. You could for example run x different queries that all do the same thing and compare the results y times. If there are >1 correct results and the wrong versions are all unique you should be able to drill down to a valid result with even a truly shit model running on a potato. Basically what humans do.

Re: Small Models Have Arrived

#203

Post by shimman »

oceanplexian wrote: post #100
zamadatix wrote: post #72

Earlier quoted context omitted.

I got a great deal on ~72 TB of NVMe right before storage prices shot up, doesn't make it any less ridiculous that I have it or any more relevant to people talking about building a NAS now. 99% of people, even in tech, do not have the stupid amounts of hardware people like us hobby on.

Most people in the US have a car, and the average new car is $40,000. Hell where I live a middle class consumer will spend double that on a Boat or an RV and think nothing of it. These aren’t elite tech workers. It’s not unfathomable that if a personal, generally intelligent local AI provides enough utility and doesn’t require you to tweak CLI flags millions of Americans would want one.

There are many payday loan operators and those willing to sell predatory loans to those workers you mention buying boats or RVs. I've yet to see a payday loan open up in SF to help tech workers buy hardware.

Re: Small Models Have Arrived

#204

Post by kakugawa »

a13n wrote: post #199

Regarding the Pareto frontier and related benchmarks, I have a hard time taking anything seriously that claims that Opus is anywhere near the intelligence of Fable. Are there any benchmarks that haven't just been benchmaxxed that more accurately represent actual usage?

FrontierCode is prob the closest. [1] It's closed source (so no direct benchmaxxing), and it was calibrated by 20+ open source maintainers. It shows Opus 5 (medium), beating out the other reasoning levels by a large margin. i.e. Opus 5 w/ higher reasoning levels actually reduces performance. [2]

However, you'll have to gauge for yourself how closely their tasks resemble your tasks.

1/ https://cognition.com/blog/frontier-code

2/ https://cognition.com/frontiercode

Re: Small Models Have Arrived

#205

Post by 9cb14c1ec0 »

jermaustin1 wrote: post #85
9cb14c1ec0 wrote: post #65

Earlier quoted context omitted.

Yes, the tunnel vision around local models on this site is crazy. The percentage of people in the world who can afford the hardware is extremely low.

I don't think there is tunnel vision. I'm just saying that I have a couple 3090s I invested in a handful of years ago, and they are still going strong today as multiple GPU-needing technologies emerged. I'm not saying everyone has to run local LLMs, because the APIs are in a race to the bottom, and my $10 of OpenRouter credits I bought months ago is down to $8.94 because most models give you MILLIONS of tokens for a…

> I'm just saying that I have a couple 3090s

This is tunnel vision. The percentage of people who could afford the hardware you could at the time you back it so vanishingly small. I do not know a single non-tech person who has multiple graphics cards in a single computer.

Re: Small Models Have Arrived

#206

Post by michael0church »

giraffe_lady wrote: post #32
michael0church wrote: post #14

It makes sense that we’ll see “room at the bottom” strategies. Currently, large parameter counts seem to be slush funds of world knowledge, language skills (because language’s nuances and open vocabulary make it high-dimensional), and reasoning primitives, the general belief being that the latter takes up the least space in the model. There are many applications where world knowledge is unnecessary or even a negative…

Everyone wants this to be it but over and over we discover that the bigger a model is the better it is at all tasks, even ones far outside the domain it was optimized for. IE claude fable is better at writing both code and prose than smaller code- and prose-specific models. The way vision and language models converge into the same geometric space should be extremely alarming for the "you don't need global knowledge f…

Fable 5 is actually a lousy writer. Opus 4.6 is the best for writing and prose assessment. Gemini 3 is smarter at reading comprehension but tends to be more unstable in judgement.

Re: Small Models Have Arrived

#207

Post by TwelveEyes »

iainmerrick wrote: post #191
nickysielicki wrote: post #168

Earlier quoted context omitted.

The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. There’s no evidence at this point that this is true.

I think you have it backwards. The common mistake is to think “maybe if we use a blend of raw data and hand-crafted heuristics, we’ll get the best of both worlds!” But the bitter lesson says no, beyond a certain point it’s better just to use the data. Thinking that an LLM might be able to improve on purely “big data” machine learning seems to me to be the same incorrect idea. Its “intelligence” is no more useful than…

Also, training it on chess books is literally training it on human knowledge, and not the actual game, which is exactly what the bitter lesson says not to do.

Re: Small Models Have Arrived

#208

Post by Dylan16807 »

iainmerrick wrote: post #191
nickysielicki wrote: post #168

Earlier quoted context omitted.

The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. There’s no evidence at this point that this is true.

I think you have it backwards. The common mistake is to think “maybe if we use a blend of raw data and hand-crafted heuristics, we’ll get the best of both worlds!” But the bitter lesson says no, beyond a certain point it’s better just to use the data. Thinking that an LLM might be able to improve on purely “big data” machine learning seems to me to be the same incorrect idea. Its “intelligence” is no more useful than…

> I think you have it backwards.

> maybe if we use a blend of raw data and hand-crafted heuristics

I don't follow. They're suggesting giving raw chess data to the LLM, no heuristics involved.

Re: Small Models Have Arrived

#209

Post by edot »

nickysielicki wrote: post #128
jimmaswell wrote: post #91

Earlier quoted context omitted.

This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always perfor…

The Bitter Lesson is very popular right now. It seems true right now. It’s having its moment right now. That doesn’t actually mean it’s axiomatically true. Commenter below gets it absolutely correct: stockfish, which runs on your 5 year old phone, is dramatically better at chess than Fable. Like, so much better that it’s not even remotely comparable. The theory of the Bitter Lesson, and it’s only a theory, is that LL…

Yes, but an LLM will just call stockfish if it needs to play chess … sure if you arbitrarily constrain an LLM to use no tools it’ll suck at chess. But no one is using LLMs in isolation. Even consumer-grade, bone-stock ChatGPT has tools.

Re: Small Models Have Arrived

#210

Post by vatsachak »

noodletheworld wrote: post #193
teiferer wrote: post #151

A friend of mine told me earlier today that they had a discussion at work (a coding shop) about "downgrading" to luna from sol for cost reasons and that many were quite unhappy about this because they didn't want inferior tech to be forced upon them. Do they have a point? Is sol actually worth the extra cost? Especially if you ramp up the effort level?

Is sol better? Yes. Categorically. Anyone who tells you otherwise and that luna is “just as good” does not know what they are talking about. Going from sol to luna is a downgrade. It is not a question, it is a fact. > Is sol actually worth the extra cost? Is a question only you can answer, because it has no generic answer. Right now, for me, being able to use sol is worth the cost, but using it all the time is not. I…

Idk... Luna is great if you generate specs before implementation.

Sure a Lexus is better than a used Prius, until you include price

Post reply on HN