Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small Models Have Arrived

calv.info

Post reply on HN
Print view View original

301–310 of 372 posts

Re: Small Models Have Arrived

#301

Post by EagnaIonat »

giraffe_lady wrote: post #32
michael0church wrote: post #14

It makes sense that we’ll see “room at the bottom” strategies. Currently, large parameter counts seem to be slush funds of world knowledge, language skills (because language’s nuances and open vocabulary make it high-dimensional), and reasoning primitives, the general belief being that the latter takes up the least space in the model. There are many applications where world knowledge is unnecessary or even a negative…

Everyone wants this to be it but over and over we discover that the bigger a model is the better it is at all tasks, even ones far outside the domain it was optimized for. IE claude fable is better at writing both code and prose than smaller code- and prose-specific models. The way vision and language models converge into the same geometric space should be extremely alarming for the "you don't need global knowledge f…

> Everyone wants this to be it but over and over we discover that the bigger a model is the better it is at all tasks,

Try using a LLM model for RAG embeddings and get back to us on that.

Re: Small Models Have Arrived

#302

Post by pornel »

abustamam wrote: post #259

As someone who uses a lot of opus, I'm curious what changes to my workflow I need to make to make small models work. My workflow currently is for a large feature I'd explain the feature and how I envision it technically and run the grill me skill where it'd ask me questions that I may not have thought of, then it'll write a spec and execute from there. It spawns an orchestrator so supposedly its supposed to use haiku…

Opus is good at deciding what to do beyond literal interpretation of the prompt (like checking assumptions before coding). Smaller models need to be told not only what to do, but how to behave.

oh-my-pi's Advisor feature can fill a lot of this gap (a smaller model told to reflect on its own output catches when it's doing a rush job and skips instructions).

Re: Small Models Have Arrived

#303

Post by smartbit »

theendisney wrote: post #202
swatcoder wrote: post #77

Earlier quoted context omitted.

Yes. The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words. The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of res…

At work i only had early copilot which was hysterically bad at everything. As i wanted it to do the same task repeatedly and could spot wrong results instantly i kept evolving a prompt that attempted to correct all ways it found to do it wrong. It kept inventing new ways to get it wrong until it eventually got it right 90% of the time. My theory is that an avanced model that has no issues with a task could do the pro…

Trying until you find something that doesn’t fail is the model used throughout evolution & engineering, think of bridges collapsed, airplanes falling from the sky or exploded steam locomotives.

In engineering you learn from these mistakes and try never making them again. Do we want to go through this evolution every time we solve a software issue? Just because we can with an unlimited number of cheap tokens? I think not, I’d rather use the knowledge build up that also knows about the edge cases forgotten to test. Or better, use multiple models that evaluate each other, as Entropic describes it in their recent report https://news.ycombinator.com/item?id=49316271

> We expect that agents coordinating in the wild will act in higher variance ways than we see here, because they’ll have different backgrounds and therefore different contexts. They also, presumably, won’t all be Claudes.

Re: Small Models Have Arrived

#304

Post by 0xdeadfa11 »

U4E4 wrote: post #267
2001zhaozhao wrote: post #127

A dream of mine is to be able to host a LLM-powered video game that I can host on a home server running a decent mid-range GPU like the RTX 5060, and the LLM is fast and intelligent enough to make for a fun game experience for a few dozen concurrent players. People can ask for features and they just get made and added to the game on the fly for the lobby to enjoy. The hosting costs would be manageable enough that I d…

I’m doing this now. Homebrew DnD engine and mobile app chat interface. Rich 3d sim and haptics for the dice rolls. But otherwise no fancy graphics rendering. The juice is in the imaginative generate group story telling. Local real time STT on self hosted video chat, to convert the banter and side talk into next player action dice rolls. Qwen on an m3 ultra 96gb handles NPC, dm, bot party members, arc, consistency, hi…

I'm very interested in this. There already are a few commercial and open source solutions but none are of my liking.

Are you going to publish yours somewhere? I'd love to play Cyberpunk 2020 with AI

Re: Small Models Have Arrived

#306

Post by rmunn »

TeMPOraL wrote: post #291
rmunn wrote: post #255

Earlier quoted context omitted.

And right now the demand for GPU is far outpacing the supply, even with factories at full production, which is keeping prices high and out of reach of most people. But unless something happens to shut down the factories (not impossible, but hasn't happened yet), eventually production will catch up to demand and prices will return to sane-ish levels. Won't happen this year, almost certainly not next year... but I woul…

Newer cards aimed at consumer market are not capable of being used for local models the way 3090s are. That's on purpose: this capability is now used to price-differentiate between "normies playing games" and "companies in data center business".

For now. That won't last forever. Yes, it'll take quite some time to work through the current production backlog, which is why I'm predicting five years, not one or two. But the trajectory has always been "new video card comes out, game devs push the limits of what it can do, gamers buy new card so the hot new game can run faster, rinse and repeat". And that includes wanting more VRAM so the game can load more of the scene at once, load higher-res textures, etc.

Which means it's inevitable that eventually, even the consumer game market will be buying GPUs with 32 or 64 GB of RAM. And there are decent models that will run at that size. Even the "normies playing games" market, as you call it, will end up with the capacity to run local models. It'll take a few more years than it would have if the data-center companies weren't trying to buy up all the GPUs, but it's not like gamers are going to stop wanting to play games. So in the long run, Anthropic et al are still going to have to figure out how to deal with competition from local models that run on your gaming video card. Which won't ever be at parity with the models that take terabytes of VRAM to run, but are very rapidly approaching "good enough for what most people want to do".

Re: Small Models Have Arrived

#307

Post by incrudible »

Where are these 180 IQ models? Even Fable needs guidance so it actually produces output I consider on par with what I could have written myself in a domain I am familiar with, and I am certainly not at the frontier in those domains.

Re: Small Models Have Arrived

#308

Post by WASDx »

jbjbjbjb wrote: post #9
swiftcoder wrote: post #3

I find it quite funny all these folks who are addicted to chasing frontier models, only just noticing that small models became "good enough" for most tasks. Those of us without fable-sized expense accounts noticed this quite a while back

I’ve been playing around with Luna, Terra and Sol and for the type of work I’ve been doing lately I actually think Sol is just a likely to trip up as Luna. Examples were Sol over assuming, persisting in the wrong direction, over engineering a little script to do some exploration of api. They can all be fixed but it’s a waste of tokens, I rather have Luna do it because course correction on small pieces of work is chea…

[dead]

Re: Small Models Have Arrived

#309

Post by maniflames »

dearilos wrote: post #265
andsoitis wrote: post #226

> One thing a few investors I've talked with have mentioned: "It's weird we're not seeing more consumer AI companies. Why is that?" What would consumer AI company even be? The frontier labs have declared they will eat everything and they have a head start. Best bet would to be a contrarian and build products and services that people actually want or need. Fine to be AI powered or augmented, but consumer companies do…

> What would consumer AI company even be? Something that helps homeowners build anything on their property, for example https://myhomecat.com

What purpose does AI serve here apart from lowering costs by generating a plan?

Especially if I need to ‘consult an expert’ isn’t this just a customer company using AI?

Re: Small Models Have Arrived

#310

Post by 59nadir »

theendisney wrote: post #202
swatcoder wrote: post #77

Earlier quoted context omitted.

Yes. The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words. The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of res…

At work i only had early copilot which was hysterically bad at everything. As i wanted it to do the same task repeatedly and could spot wrong results instantly i kept evolving a prompt that attempted to correct all ways it found to do it wrong. It kept inventing new ways to get it wrong until it eventually got it right 90% of the time. My theory is that an avanced model that has no issues with a task could do the pro…

Early Copilot was tab-complete in editors and was honestly the best version of LLM-assisted development I've used, because it was intentionally small in scope, trivial to verify output from at a glance and easy to opt in and opt out of. I have yet to see anything more useful in terms of code generation; very small-scale code generation (think function-level) comes close, but is a lot more tedious.

For analysis, bug hunting, overview and some porting work to popular languages and so on I think the current SotA is fantastic, but they're still very disappointing for code generation past function or small module level.

Post reply on HN