Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small Models Have Arrived

calv.info

Post reply on HN
Print view View original

241–250 of 377 posts

Re: Small Models Have Arrived

#241

Post by charv »

NickNaraghi wrote: post #13

> Across his various startups, Peter has seen two kinds of work: > 1. the "IQ 180" work. some mad scientist genius type comes up with some crazy solution you've never thought of. > 2. the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts. Interesting comp to pg's Maker's Schedule, Manager's Schedule https://www.paulgraham.com/makersschedule.html I'm curious about…

> July 2009

> Thanks to Sam Altman, … for reading drafts of this.

Oh how times have changed!

Re: Small Models Have Arrived

#242

Post by senectus1 »

not sure how this is really about model size.

I'm keenly interested in seeing super small (like 10's or 100's of mb) special purpose models starting to come into their own.

that might actually be transformational globally not just in rich countries.

Re: Small Models Have Arrived

#243

Post by BobbyTables2 »

mikepurvis wrote: post #172
nickysielicki wrote: post #128

Earlier quoted context omitted.

The Bitter Lesson is very popular right now. It seems true right now. It’s having its moment right now. That doesn’t actually mean it’s axiomatically true. Commenter below gets it absolutely correct: stockfish, which runs on your 5 year old phone, is dramatically better at chess than Fable. Like, so much better that it’s not even remotely comparable. The theory of the Bitter Lesson, and it’s only a theory, is that LL…

But isn't that really just about giving "front end" models more access to specialized tool libraries, which include models tuned to specific tasks? Like the first model says ah, we're being asked to code something, oh and we've been provided with some example code, let me invoke a tool call to my model the recognizes many languages, that model says that we're looking at ocaml. Okay, I better pass this off to my ocaml…

It’s kinda funny that your last paragraph is basically describing why sparse files, sparse matrices, etc. are used in other contexts.

It really is absurd to ask programming questions to a model also trained about the lifecycle of a fruit fly.

Instead of building small models from scratch, we train an enormous model and use ridiculous amounts of GPU memory. In the end, the whole thing is shoved into RAM because we don’t know where the useful parts are…

We certainly would know where they were if they were just in smaller models in the first place!

Re: Small Models Have Arrived

#245

Post by LoveMistral »

casper14 wrote: post #61
LoveMistral wrote: post #21

Earlier quoted context omitted.

Same. Mistral 7b has been more than I ever needed for text for years now. Unless you must 1-shot with no harness it’s the same amount of power, maybe more because the big “good” models make too many assumptions and tend to become rigid. Mistral 7b can do anything, and it’s basically instant even on an M3

What are some limitations you have found with using a smaller model like that?

As a developer, none related to the model. It’s faster, lighter, makes less assumptions.

For a casual consumer, it is no doubt a worse chat buddy. Knows less historical facts à la Wikipedia.

But I’m not really using LLMs for that kind of entertainment and I don’t rely on them for fact-checking anyway. It’s almost worse to rely on a smoother talker for something it can’t possibly know.

For feeding in thousands of daily updates and getting it to predict the next one based on a crude list of tasks, it’s great. For completing code files in my style, also great. It can also handle most small customer service issues and refunds related to my apps on App Store, the back and forth to collect data from users.

For a booking engine I run with a partner, it is great at confirming bookings and following up. It handles cancellations which are about 40% of confirmed bookings due to the nature of the business. I used to feel like I was wasting my time with them - there is no way we can charge for it. Having basically a smart bot do it makes everyone happier (even the customer who knows they aren’t hurting anyone’s feelings).

For invoices, I prefill notes and things based on previous ones. It’s fine. I don’t need a massive model that takes 30s to reply etc. it’s way overkill. Maybe worse than overkill - off in a direction I don’t need.

Re: Small Models Have Arrived

#246

Post by shostack »

andsoitis wrote: post #226

> One thing a few investors I've talked with have mentioned: "It's weird we're not seeing more consumer AI companies. Why is that?" What would consumer AI company even be? The frontier labs have declared they will eat everything and they have a head start. Best bet would to be a contrarian and build products and services that people actually want or need. Fine to be AI powered or augmented, but consumer companies do…

Yeah, I think if we reframed it as "why is there no consumer AWS?" It would make more sense.

AI is a utility that can abstract code to such a high level it is indiscernible from natural language.

Re: Small Models Have Arrived

#247

Post by catoc »

antihipocrat wrote: post #213
nickysielicki wrote: post #168

Earlier quoted context omitted.

The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. There’s no evidence at this point that this is true.

Maybe a future frontier LLM could approach the problem by first building its own stockfish, then applying the subsequent results

Maybe a future LLM after that could approach the problem by first simulating a human brain, then learning from the ‘human’ gameplay.

Just kidding of course

Re: Small Models Have Arrived

#248

Post by daquisu »

bartleeanderson wrote: post #158

Nobody is questioning what is meant by small? When you started mentioning frontier models that don't run locally, I just go TB;DR "Too big, didn't read"

I read the full post. It is more about cheap models with good enough performance rather than small models. "Local" was not mentioned at all

Re: Small Models Have Arrived

#249

Post by nonethewiser »

andsoitis wrote: post #226

> One thing a few investors I've talked with have mentioned: "It's weird we're not seeing more consumer AI companies. Why is that?" What would consumer AI company even be? The frontier labs have declared they will eat everything and they have a head start. Best bet would to be a contrarian and build products and services that people actually want or need. Fine to be AI powered or augmented, but consumer companies do…

There is a thin line. Lots of AI powered things are just workflows you could implement with claude code and some skills. Or even just a prompt on the web chat. But I think these big technological swings sometimes take a long time to shake out. Society is still adapting to the internet.

I think there is more opportunity for consumer application of AI. Im surprised we dont see more models in game. Small models that do more purpose driven things not "this npc is just a chatbot with a personality in its system prompt." I mean we literally have systems in games called "AI" and we dont think these much more powerful AIs have a place?

Re: Small Models Have Arrived

#250

Post by butterisgood »

I don’t care about the cost if the results are weak. I did a bunch of work with Sol only to find Opus 5 spot a bunch of bugs.

And it was correct. The results weren’t as good as they should have been for Sol. How am I going to trust Luna?

Post reply on HN