Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small Models Have Arrived

calv.info

Post reply on HN
Print view View original

291–300 of 377 posts

Re: Small Models Have Arrived

#291

Post by TeMPOraL »

rmunn wrote: post #255
9cb14c1ec0 wrote: post #205

Earlier quoted context omitted.

> I'm just saying that I have a couple 3090s This is tunnel vision. The percentage of people who could afford the hardware you could at the time you back it so vanishingly small. I do not know a single non-tech person who has multiple graphics cards in a single computer.

And right now the demand for GPU is far outpacing the supply, even with factories at full production, which is keeping prices high and out of reach of most people. But unless something happens to shut down the factories (not impossible, but hasn't happened yet), eventually production will catch up to demand and prices will return to sane-ish levels. Won't happen this year, almost certainly not next year... but I woul…

Newer cards aimed at consumer market are not capable of being used for local models the way 3090s are. That's on purpose: this capability is now used to price-differentiate between "normies playing games" and "companies in data center business".

Re: Small Models Have Arrived

#292

Post by deaux »

giraffe_lady wrote: post #32
michael0church wrote: post #14

It makes sense that we’ll see “room at the bottom” strategies. Currently, large parameter counts seem to be slush funds of world knowledge, language skills (because language’s nuances and open vocabulary make it high-dimensional), and reasoning primitives, the general belief being that the latter takes up the least space in the model. There are many applications where world knowledge is unnecessary or even a negative…

Everyone wants this to be it but over and over we discover that the bigger a model is the better it is at all tasks, even ones far outside the domain it was optimized for. IE claude fable is better at writing both code and prose than smaller code- and prose-specific models. The way vision and language models converge into the same geometric space should be extremely alarming for the "you don't need global knowledge f…

> claude fable is better at writing both code and prose than smaller code- and prose-specific models.

There's a million ways to define "better at writing prose", and it's only better at some of them. It's worse at many others.

Re: Small Models Have Arrived

#293

Post by drxzcl »

kridsdale1 wrote: post #198
NickNaraghi wrote: post #13

> Across his various startups, Peter has seen two kinds of work: > 1. the "IQ 180" work. some mad scientist genius type comes up with some crazy solution you've never thought of. > 2. the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts. Interesting comp to pg's Maker's Schedule, Manager's Schedule https://www.paulgraham.com/makersschedule.html I'm curious about…

I conceive of this as Protoss vs Zerg. I’ve had a lot of success in my career following a Zerg strategy.

We call it Ravenclaw vs Hufflepuff. I guess it's a generation thing.

Re: Small Models Have Arrived

#294

Post by deaux »

pseudosavant wrote: post #154
teiferer wrote: post #151

A friend of mine told me earlier today that they had a discussion at work (a coding shop) about "downgrading" to luna from sol for cost reasons and that many were quite unhappy about this because they didn't want inferior tech to be forced upon them. Do they have a point? Is sol actually worth the extra cost? Especially if you ramp up the effort level?

It is a good question. Luna is definitely a very capable model. Much more capable than the top SOTA models from 12 months ago. It definitely isn't at the same level as Sol, but you get 20x the tokens for the cost, and it has a much faster tokens/second rate. If this is a cost conscious company where I'm going to get a fairly limited amount of Sol, or a nearly unlimited amount of Luna, I'm probably choosing Luna.

> Much more capable than the top SOTA models from 12 months

Confidently wrong here. It's absolutely not across the board "more capable" than GPT 5 or Opus 4.1 or even Gemini 2.5 Pro. It's potentially better at certain specific tasks, mostly agentic coding implementation work. I.e. tool calling and usage of bash. It's worse at a large range of other tasks.

Re: Small Models Have Arrived

#295

Post by edg5000 »

swiftcoder wrote: post #3

I find it quite funny all these folks who are addicted to chasing frontier models, only just noticing that small models became "good enough" for most tasks. Those of us without fable-sized expense accounts noticed this quite a while back

Sol is 100 EUR/m and I prefer it over Fable, and you get practically unlimited usage unless you go really overboard with it. So I don't get the cost argument. Would be better to own hardware, but with the shortages that's not very attractive atm.

Re: Small Models Have Arrived

#296

Post by jillesvangurp »

We've been testing and developing a few openai based applications. Relatively simple stuff. I topped up my api credit with 10$ a few months ago. I just checked, we burned something like 61 cents over this period. About 126 requests over the last month. That's just us testing but it shows me that this stuff is relatively cheap. The reason we don't have a lot of requests is that most of this stuff pretty much works on the first try. This is not hard to build.

I think AI is an underused capability in applications. People come up with all these big plans to build agents. But even a simple button to "calculate a color palette that takes into account these symbolic names the user entered" is a simple prompt that comes back quickly and doesn't require a lot of reasoning. We use that with Luna and we give it some more context.

Image based stuff is more expensive but still quite reasonable. We do some things with Sol that work on images. It's not fast but the result is worth waiting for. These are features that save the user time and feel a bit magical. They just click a button, wait 20-30 seconds and then magic happens. Beats them clicking around for half an hour.

Local model hosting would force me into higher cost and overhead for less intelligence. That's only a good tradeoff if you are really going to use lots of tokens or you need this for other reasons (e.g. data security). But otherwise you just end up paying for underutilized GPU capacity.

Re: Small Models Have Arrived

#297

Post by magimas »

jillesvangurp wrote: post #296

We've been testing and developing a few openai based applications. Relatively simple stuff. I topped up my api credit with 10$ a few months ago. I just checked, we burned something like 61 cents over this period. About 126 requests over the last month. That's just us testing but it shows me that this stuff is relatively cheap. The reason we don't have a lot of requests is that most of this stuff pretty much works on…

I think in general these smaller applications of AI are seriously overlooked even though they will actually add lots of economic value for companies and lots of quality-of-life features for users.

Re: Small Models Have Arrived

#298

Post by paganel »

> Most of the "human tokens" at companies today are spent this way

It sucks that we got to this point when it comes to our civilization. At least guys like this one here can be happy with their AI-sourced money.

Re: Small Models Have Arrived

#299

Post by miki123211 »

theendisney wrote: post #202
swatcoder wrote: post #77

Earlier quoted context omitted.

Yes. The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words. The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of res…

At work i only had early copilot which was hysterically bad at everything. As i wanted it to do the same task repeatedly and could spot wrong results instantly i kept evolving a prompt that attempted to correct all ways it found to do it wrong. It kept inventing new ways to get it wrong until it eventually got it right 90% of the time. My theory is that an avanced model that has no issues with a task could do the pro…

Yes! As long as you have some criteria to judge the final answer, you can do a kind of "prompt-side RLVR", where you have the model generate prompt changes, try a bunch of different prompts and see which ones improve the results.

You don't necessarily need a bigger model to do this.

Re: Small Models Have Arrived

#300

Post by miki123211 »

jimmaswell wrote: post #91
swatcoder wrote: post #77

Earlier quoted context omitted.

Yes. The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words. The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of res…

This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always perfor…

I think large-scale systems are optimal when the number of tasks is relatively bounded and costs don't matter.

When you're doing something "at scale", you want smaller models, weaker heuristics and rules written in code, as they're much cheaper, much faster, much more predictable and much easier to reason about.

If your task is "tell me which pages on the internet are most relevant to this search query", running Claude Fable to evaluate every single page would be ridiculous. You may run something like it at the end, to synthesize information from a small result set for example, as that's a "human-sized" task.

Post reply on HN