Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small Models Have Arrived

calv.info

Post reply on HN
Print view View original

261–270 of 377 posts

Re: Small Models Have Arrived

#261

Post by abustamam »

bunderbunder wrote: post #174
teiferer wrote: post #151

A friend of mine told me earlier today that they had a discussion at work (a coding shop) about "downgrading" to luna from sol for cost reasons and that many were quite unhappy about this because they didn't want inferior tech to be forced upon them. Do they have a point? Is sol actually worth the extra cost? Especially if you ramp up the effort level?

I don’t love the “forced upon them” framing; if that’s really how people are thinking about it then maybe they should pause and reflect for a moment yhat it isn’t their money being spent. Amd the default isn’t always having the latest and greatest, it’s not paying for anything at all. Now, if the debate is really about which option is more cost effective, then we could easily run an A/B test to find out. Though TBH m…

I recently just had fable run itself into a loop. Im sure it would have stopped itself at some point (perhaps when i ran our of tokens?) But I still stopped it early when I noticed it trying to get something to work when the solution was in a file in a sibling repo.

Re: Small Models Have Arrived

#264

Post by Computer0 »

abustamam wrote: post #259

As someone who uses a lot of opus, I'm curious what changes to my workflow I need to make to make small models work. My workflow currently is for a large feature I'd explain the feature and how I envision it technically and run the grill me skill where it'd ask me questions that I may not have thought of, then it'll write a spec and execute from there. It spawns an orchestrator so supposedly its supposed to use haiku…

I’d imagine it’s more focused on individual methods and how they connect

Re: Small Models Have Arrived

#265

Post by dearilos »

andsoitis wrote: post #226

> One thing a few investors I've talked with have mentioned: "It's weird we're not seeing more consumer AI companies. Why is that?" What would consumer AI company even be? The frontier labs have declared they will eat everything and they have a head start. Best bet would to be a contrarian and build products and services that people actually want or need. Fine to be AI powered or augmented, but consumer companies do…

> What would consumer AI company even be?

Something that helps homeowners build anything on their property, for example https://myhomecat.com

Re: Small Models Have Arrived

#266

Post by jurgenburgen »

antihipocrat wrote: post #213
nickysielicki wrote: post #168

Earlier quoted context omitted.

The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. There’s no evidence at this point that this is true.

Maybe a future frontier LLM could approach the problem by first building its own stockfish, then applying the subsequent results

Or maybe an LLM could just tool call stockfish and doesn’t need to have more than a basic understanding of chess. The bitter lesson seems extraordinarily wasteful on the compute side.

Re: Small Models Have Arrived

#267

Post by U4E4 »

2001zhaozhao wrote: post #127

A dream of mine is to be able to host a LLM-powered video game that I can host on a home server running a decent mid-range GPU like the RTX 5060, and the LLM is fast and intelligent enough to make for a fun game experience for a few dozen concurrent players. People can ask for features and they just get made and added to the game on the fly for the lobby to enjoy. The hosting costs would be manageable enough that I d…

I’m doing this now. Homebrew DnD engine and mobile app chat interface. Rich 3d sim and haptics for the dice rolls. But otherwise no fancy graphics rendering. The juice is in the imaginative generate group story telling. Local real time STT on self hosted video chat, to convert the banter and side talk into next player action dice rolls. Qwen on an m3 ultra 96gb handles NPC, dm, bot party members, arc, consistency, history, memory, adventure beats. There’s even a mechanic for bards to compose an end-of-adventure ballad that compresses the shenanigans into verse.

Bootstrapped it with fable, opus or sol 5.6 when I had extra usage to burn over the last month. Playtesting it solo as both DM and PC was fun too. Character consistency and originality is remarkably compelling.

Upshot is that for the gaming hours that my friends have available, the social interpersonal gaming experience competes well for their AAA gaming time. There’s even some good async passive campaign continuation patterns that run in our signal thread to keep things going between sync sessions.

And players have requested features directly in the session transcript that have been direct implemented.

Re: Small Models Have Arrived

#268

Post by tobiasu »

viscousviolin wrote: post #79
NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

If someone has an old GPU laying around, say a GTX 1080 with 8 GB of memory, would that be enough to get a (small?) local model running?

Sure. The fastest small coding model is probably Mellum2 12B-A2.5 by Jetbrains. It matches or beats all Qwen models in this class.

Can even run on a notebook CPU and comes in Base (best for FIM), Instruct and Thinking variants. mradermacher has imatrix quants for people who can't run it at Q8.

IQ4 should fit, but even if it doesn't, llama.cpp has options to partially offload models to system memory.

Re: Small Models Have Arrived

#270

Post by Someone »

Animats wrote: post #143
nickysielicki wrote: post #128

Earlier quoted context omitted.

The Bitter Lesson is very popular right now. It seems true right now. It’s having its moment right now. That doesn’t actually mean it’s axiomatically true. Commenter below gets it absolutely correct: stockfish, which runs on your 5 year old phone, is dramatically better at chess than Fable. Like, so much better that it’s not even remotely comparable. The theory of the Bitter Lesson, and it’s only a theory, is that LL…

Good point. Dumb AIs are needed for customer service. Most of that industry is still at "press 1 for sales, 2 for billing..." and needs something that will run locally on a 1U server.

> Most of that industry is still at "press 1 for sales, 2 for billing..." and needs something that will run locally on a 1U server.

Needs? Customers want something that immediately answers their question/solves their problem, but that’s far away, even ignoring the “run locally on a 1U server” and that that may not be in the company’s interest.

For many companies, that support line is a cost center, not a PR mechanism.

Also “Press 1 for sales, 2 for billing...” has the big advantage that it handles all accents, speech impediments, etc.

Long term I think a solution where a user’s agent trained on their voice, running on their phone communicates with the support agent of a company is where we will end up, and support phone lines will cease to exist.

Post reply on HN