Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small Models Have Arrived

calv.info

Post reply on HN
Print view View original

231–240 of 374 posts

Re: Small Models Have Arrived

#231

Post by inigyou »

MrDrMcCoy wrote: post #220
cjkaminski wrote: post #179

Earlier quoted context omitted.

Yes, and the technology to improve the interface you described is already available to run hundreds of concurrent instances on a 1U server. The barrier to entry is getting the people who manage those systems to care enough to implement something better.

Fact. My company's largest partner is CoreWeave, and convincing leadership that we could run it ourselves on partner discounted hardware for a lot less money has gone nowhere.

Maybe you need to walk into the office with a $1000 server running a hundred instances of whatever your code does.

Re: Small Models Have Arrived

#232

Post by mumbo_rmj »

spl757 wrote: post #40
zatkin wrote: post #12

Maybe I'm being super reductive here, but operating small models at the core of your business kind of moves the needle from making external API calls (against frontier models) to running internal API calls (against your locally-run models). It seems like if we want local models to take off, it will need to become easier to run local models for cheap. I'm thinking like reducing the barrier of entry for running "local…

I only run local models and I don't give them access to much externally. I don't do anything serious with it, but it comes in handy and I know that they can do so much more. I'm on a meager RTX 3060 12GB and a GTX 1660 Ti with 6GB for some extra vram space. When I first started playing with local models, I was really impressed with what I was able to achieve locally. That's great, but the thing that worries me is tha…

My money would be on ‘for security purposes we just can’t let these get into the wrong hands’ and then the lobbyist go to work.

Re: Small Models Have Arrived

#233

Post by inigyou »

CamperBob2 wrote: post #123
applfanboysbgon wrote: post #107

Earlier quoted context omitted.

This is a misunderstanding of either the bitter lesson or what was being claimed, on multiple accounts. Firstly, the bitter lesson is merely about human expertise-tuned algorithms vs. throwing raw compute at a domain. But, notably, it is still domain-specific. No matter how much compute you throw at training an LLM, it is never going to beat a Chess engine at Chess. If you give a Chess engine 1,000,000 compute units…

I don't know much about chess engines; do they still use hand-tuned algorithms, or are they more like AlphaZero, where they learn through self-play to beat any/all possible human contenders? I don't believe DeepBlue was automated to that extent, but it may have been. In the latter case, the chess example would tend to support the Bitter Lesson, rather than refute it. I would also be VERY slow to claim that general-pu…

AFAIK Stockfish still remains a traditional (some-variant-of-) minimax-search engine, but now uses a microscopic neural network to evaluate the positions at the maximum search depth rather than heuristics. I expect any pure ML engine can learn to exploit any fixed heuristics, but now Stockfish is trying to balance both strategies, which still heavily ends up on the side of raw search depth.

Re: Small Models Have Arrived

#234

Post by inigyou »

layer8 wrote: post #76
9cb14c1ec0 wrote: post #65

Earlier quoted context omitted.

Yes, the tunnel vision around local models on this site is crazy. The percentage of people in the world who can afford the hardware is extremely low.

It seems roughly similar to the pricing level of personal computers in the early eighties (i.e. IBM PC and Apple Macintosh). I’d expect prices to come down significantly over the next few years. Not so much in the next year or two, but after that.

That's in nominal dollars. However, inflation since then has been about a factor of ten to fifty and it hasn't trickled down at all.

Re: Small Models Have Arrived

#235

Post by leoh »

kridsdale1 wrote: post #198
NickNaraghi wrote: post #13

> Across his various startups, Peter has seen two kinds of work: > 1. the "IQ 180" work. some mad scientist genius type comes up with some crazy solution you've never thought of. > 2. the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts. Interesting comp to pg's Maker's Schedule, Manager's Schedule https://www.paulgraham.com/makersschedule.html I'm curious about…

I conceive of this as Protoss vs Zerg. I’ve had a lot of success in my career following a Zerg strategy.

Zerg rush

Re: Small Models Have Arrived

#236

Post by trucks-refinish »

viscousviolin wrote: post #79
NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

If someone has an old GPU laying around, say a GTX 1080 with 8 GB of memory, would that be enough to get a (small?) local model running?

I'm able to run qwen 3.6 35b a3b with my rtx 2060 super which also has 8gb of vram. It's a very capable model, not as smart as a frontier model but it is still very useful.

I use an 6bit quant and get around 20 tokens per second.

Re: Small Models Have Arrived

#237

Post by ImaCake »

TJTorola wrote: post #35
LPisGood wrote: post #16

Earlier quoted context omitted.

Small amounts of world knowledge seems like it would inherently be tied to more hallucinations.

Perhaps we'll get to a point where believing any un-sourced information from an LLM will feel crazy. I don't want my model to know more than it needs to perform logic and use tools. Once it is capable of using tools I would much rather it looked up information or sourced it from existing context rather than just divine it from it's weights.

The problem is it needs world knowledge to know what to lookup. This puts a floor on how little it can know while being able to look up what it doesn't know. Maybe its better if it knows a lot but has a good instinct for verifying that.

Re: Small Models Have Arrived

#238

Post by mchusma »

jbjbjbjb wrote: post #9
swiftcoder wrote: post #3

I find it quite funny all these folks who are addicted to chasing frontier models, only just noticing that small models became "good enough" for most tasks. Those of us without fable-sized expense accounts noticed this quite a while back

I’ve been playing around with Luna, Terra and Sol and for the type of work I’ve been doing lately I actually think Sol is just a likely to trip up as Luna. Examples were Sol over assuming, persisting in the wrong direction, over engineering a little script to do some exploration of api. They can all be fixed but it’s a waste of tokens, I rather have Luna do it because course correction on small pieces of work is chea…

I have an agentic workflow and Luna just always gets stuck, SOL and grok 4.6 don’t. I like Luna in theory I just find not much practical work for it yet in coding type work.

Now I think Luna is plenty good for many applications inside a very good harness/scaffold. And I think there are a lot of those usecases. So I think these small models are really good for application developers.

But for entrepreneurial knowledge work all of my work still benefits a lot from more intelligence.

Re: Small Models Have Arrived

#239

Post by lqstuart »

jimmaswell wrote: post #91
swatcoder wrote: post #77

Earlier quoted context omitted.

Yes. The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words. The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of res…

This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always perfor…

It’s not hard to fine tune a 4B parameter model to outperform GPT-5.x or Claude on a domain-specific task. You don’t even need GRPO or whatever the flavor of the month RL technique is either. I suspect what you’re saying is that massive frontier models are the best for doing big slow tasks on human time horizons—and I emphatically agree—but keep in mind LLMs are useful for quite a bit more than just multi-turn chatbots; and in those instances, throwing infinite compute and latency budgets at a problem is generally not a great way to find the “best” solution :)

Re: Small Models Have Arrived

#240

Post by ImaCake »

pseudosavant wrote: post #154
teiferer wrote: post #151

A friend of mine told me earlier today that they had a discussion at work (a coding shop) about "downgrading" to luna from sol for cost reasons and that many were quite unhappy about this because they didn't want inferior tech to be forced upon them. Do they have a point? Is sol actually worth the extra cost? Especially if you ramp up the effort level?

It is a good question. Luna is definitely a very capable model. Much more capable than the top SOTA models from 12 months ago. It definitely isn't at the same level as Sol, but you get 20x the tokens for the cost, and it has a much faster tokens/second rate. If this is a cost conscious company where I'm going to get a fairly limited amount of Sol, or a nearly unlimited amount of Luna, I'm probably choosing Luna.

Luna on xhigh unlocks gh copilot for me. Sol, even discounted, is too expensive to use and is much slower. Luna on the other hand seems to be so cheap you don't have to think about cost.
Post reply on HN