Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small Models Have Arrived

calv.info

Post reply on HN
Print view View original

91–100 of 376 posts

Re: Small Models Have Arrived

#91

Post by jimmaswell »

swatcoder wrote: post #77
NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

Yes. The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words. The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of res…

This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always performed the best at all tasks. Suggesting otherwise has become an extraordinary claim requiring extraordinary evidence.

http://www.incompleteideas.net/IncIdeas/BitterLesson.html

Recent comment touching on this in relation to LLM's in more depth: https://news.ycombinator.com/item?id=49322695#49323341

Re: Small Models Have Arrived

#92

Post by dominotw »

NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

ppl keep talking about the supposed unexplored and untapped "model overhang" but very few things in the world are where you can write elaborate test criteria to before using ai.

A sales person sending a prospect email doesnt have a way to write a test harness for it. Yet these tasks dominate what humans do compared to writing a crud app . otherwise anthropic wouldnt have trillions dollar valuation

Re: Small Models Have Arrived

#93

Post by zahlman »

9cb14c1ec0 wrote: post #65
thayne wrote: post #49

Earlier quoted context omitted.

A single, used 3090 costs more than I have ever spent on a computer.

Yes, the tunnel vision around local models on this site is crazy. The percentage of people in the world who can afford the hardware is extremely low.

I mean, I'm not rushing out to buy that kind of hardware myself, but it is a matter of perspective. People commonly spend an order of magnitude more on a car, and that's just the sticker price.

Re: Small Models Have Arrived

#94

Post by dominotw »

swiftcoder wrote: post #3

I find it quite funny all these folks who are addicted to chasing frontier models, only just noticing that small models became "good enough" for most tasks. Those of us without fable-sized expense accounts noticed this quite a while back

no they are not good enough for "most" tasks

Re: Small Models Have Arrived

#95

Post by Gecko4072 »

xnx wrote: post #83
jermaustin1 wrote: post #42

Earlier quoted context omitted.

Having multiple 6 year old cards doesn't seem like it's that big of burden for local LLMs. I get that a lot of people don't have them. And a single one can be VERY performant. And the smaller models like a 7B can run on much smaller hardware like a mid-range [3|4|5]060. My entire AI Dev Box cost $4500 in parts. 128GB RAM, i7-10700, 1TB and 2TB SSD, and 2x 3090s. Today's prices and inflation have definitely made that…

> 2x 3090s You could sell those and have enough money to pay for hosted inference for years.

But after all those years you’d still have 2 3090s, which are now about 6 years old and still holding value.

Re: Small Models Have Arrived

#96

Post by swiftcoder »

kccqzy wrote: post #47
swiftcoder wrote: post #3

I find it quite funny all these folks who are addicted to chasing frontier models, only just noticing that small models became "good enough" for most tasks. Those of us without fable-sized expense accounts noticed this quite a while back

> for most tasks The word “most” is doing a lot of work here. On a percentage basis perhaps most tasks a typical SWE needs to do when they aren’t in meetings or writing docs are just glorified autocomplete. But that’s boring and that’s why people don’t usually talk about it. People are addicted to chasing frontier models because they all have memories of spending a week on a deeply challenging algorithm problem or ev…

> crazy complicated algorithms they cannot implement themselves

I'm not sure I know very many engineers who would fall in this bucket. Or do you mean the business types who suddenly think AI can replace all the engineers?

Re: Small Models Have Arrived

#97

Post by Foobar8568 »

zahlman wrote: post #90
eqmvii wrote: post #84

Earlier quoted context omitted.

I see it in a slightly opposite way: even the good models are relatively cheap, and so I worry what we might miss by spending too much time playing with the Sonnets of the world when the Opuses are still objectively a bargain for the power they bring.

> when the Opuses are still objectively a bargain for the power they bring. The cost isn't just what you're billed. There are security, privacy etc. concerns.

I know companies that are using github, even using public repo, and request their teams to not use SOTA models, but are ok with local models. Just stupid policy.

Re: Small Models Have Arrived

#98

Post by applfanboysbgon »

jimmaswell wrote: post #91
swatcoder wrote: post #77

Earlier quoted context omitted.

Yes. The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words. The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of res…

This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always perfor…

This idea has not failed to pan out at all. I work for a startup that is exactly what GP described, and am set for life because of how wildly successful it is. Notably, we are successful, in a genuine sense of the word: we bootstrapped from running tiny models to larger and larger models on our own slowly improving fleet of GPUs, and now have millions in revenue without a single dime of outside investment. Conversely, you cannot call taking on ~1 trillion in debt and purchase commitments to scale "success". OpenAI and Anthropic are underwater financially. To be precise, they're in the Mariana Trench.

Re: Small Models Have Arrived

#100

Post by oceanplexian »

zamadatix wrote: post #72
jermaustin1 wrote: post #42

Earlier quoted context omitted.

Having multiple 6 year old cards doesn't seem like it's that big of burden for local LLMs. I get that a lot of people don't have them. And a single one can be VERY performant. And the smaller models like a 7B can run on much smaller hardware like a mid-range [3|4|5]060. My entire AI Dev Box cost $4500 in parts. 128GB RAM, i7-10700, 1TB and 2TB SSD, and 2x 3090s. Today's prices and inflation have definitely made that…

I got a great deal on ~72 TB of NVMe right before storage prices shot up, doesn't make it any less ridiculous that I have it or any more relevant to people talking about building a NAS now. 99% of people, even in tech, do not have the stupid amounts of hardware people like us hobby on.

Most people in the US have a car, and the average new car is $40,000. Hell where I live a middle class consumer will spend double that on a Boat or an RV and think nothing of it. These aren’t elite tech workers.

It’s not unfathomable that if a personal, generally intelligent local AI provides enough utility and doesn’t require you to tweak CLI flags millions of Americans would want one.

Post reply on HN