Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Small Models Have Arrived

calv.info

Post reply on HN
Print view View original

111–120 of 373 posts

Re: Small Models Have Arrived

#111

Post by HDThoreaun »

wild_egg wrote: post #102
applfanboysbgon wrote: post #98

Earlier quoted context omitted.

This idea has not failed to pan out at all. I work for a startup that is exactly what GP described, and am set for life because of how wildly successful it is. Notably, we are successful, in a genuine sense of the word: we bootstrapped from running tiny models to larger and larger models on our own slowly improving fleet of GPUs, and now have millions in revenue without a single dime of outside investment. Conversely…

Wait, you actually found a viable counter to The Bitter Lesson? Please say more

The issue is that GP is misusing the bitter lesson. Yes, search + learn tends to be more effective than human rules based strategies, but that's not what's being considered here. The original claim is effectively that AGI isn't needed for most tasks and more value can be created by using search + learn to solve specific problems instead of applying general models to every problem. Then GP commented a non sequitur

Re: Small Models Have Arrived

#112

Post by z3t4 »

applfanboysbgon wrote: post #98
jimmaswell wrote: post #91

Earlier quoted context omitted.

This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always perfor…

This idea has not failed to pan out at all. I work for a startup that is exactly what GP described, and am set for life because of how wildly successful it is. Notably, we are successful, in a genuine sense of the word: we bootstrapped from running tiny models to larger and larger models on our own slowly improving fleet of GPUs, and now have millions in revenue without a single dime of outside investment. Conversely…

Do you have a website?

Re: Small Models Have Arrived

#113

Post by irishcoffee »

xnx wrote: post #83
jermaustin1 wrote: post #42

Earlier quoted context omitted.

Having multiple 6 year old cards doesn't seem like it's that big of burden for local LLMs. I get that a lot of people don't have them. And a single one can be VERY performant. And the smaller models like a 7B can run on much smaller hardware like a mid-range [3|4|5]060. My entire AI Dev Box cost $4500 in parts. 128GB RAM, i7-10700, 1TB and 2TB SSD, and 2x 3090s. Today's prices and inflation have definitely made that…

> 2x 3090s You could sell those and have enough money to pay for hosted inference for years.

I keep seeing this comment. This is _hacker news_ where, back in the day, people just hacked on things, because it was a hobby. They weren't "moneymaxxing" or desperately trying to be as insanely efficient as possible. They hacked on stuff with a can of surge at 3am because it was fun.

Your comment is like a meta comment of "LLMs are generating everything, after a while the ouroboros will eat itself. (Which I agree with)" If people aren't hacking on this shit just because, you have completely conceded control of software to a handful of sociopaths, and open source software is dead.

Re: Small Models Have Arrived

#114

Post by kccqzy »

swiftcoder wrote: post #96
kccqzy wrote: post #47

Earlier quoted context omitted.

> for most tasks The word “most” is doing a lot of work here. On a percentage basis perhaps most tasks a typical SWE needs to do when they aren’t in meetings or writing docs are just glorified autocomplete. But that’s boring and that’s why people don’t usually talk about it. People are addicted to chasing frontier models because they all have memories of spending a week on a deeply challenging algorithm problem or ev…

> crazy complicated algorithms they cannot implement themselves I'm not sure I know very many engineers who would fall in this bucket. Or do you mean the business types who suddenly think AI can replace all the engineers?

It probably depends on the background and the company. For example if one works at a startup that happens to use technology, it’s unlikely to happen because SWEs just translate business rules to code. But if one works at the place where the technology itself is the focus, then yes most people will fall in that bucket.

In fact I noticed that this is the one place where people discussing AI on HN tend to talk past each other. On the one hand people are talking about supreme intelligence like designing new algorithms (on the same vein as finding counter examples for the Jacobian conjecture) and on the other hand people are just satisfied using AI to automate a few quotidian tasks that hitherto couldn’t be automated.

Re: Small Models Have Arrived

#115

Post by riazrizvi »

dgellow wrote: post #70
riazrizvi wrote: post #63

Earlier quoted context omitted.

I think there's something subtle about language and ambiguity that means they aren't designed to become superintelligent autonomous machines. They're value is as information repositories that actual intelligent autonomous machines (us) mine and string together.

Yes LLMs are a beautiful way to compact knowledge. It would be such a cool technology to develop and worked with if it wasn’t linked to such a toxic industry

I think you're just observing ppl in one of these rare instances where enough of them come together because they are motivated. 'Toxic' is the clamoring sound of a crowded room where what gets through to your ears are just the most annoying snippets of incomplete conversations. I dare you to hang out with any actual people here, understand their viewpoint and listen to what they actually have to say in person, within the context of watching them do it.

Re: Small Models Have Arrived

#116

Post by Zigurd »

I recently had some relevant experience: for a couple of months now I've been experimenting with on device models to summarize feeds in a Bluesky client I am developing. The feature extracts topic areas, categorizes posts, and creates a summary under each topic.

At first the results were hot garbage, and progress was slow. I hooked up the settings to download models from Hugging Face conveniently, so I could run experiments faster, and I massaged the prompts a bit. Last week this feature made a qualitative jump from science experiment to something I'd actually use.

The fact that all runs on the device means I've got no variable costs associated with adding this to what will be, at best, a pretty low revenue product. I've tested it on trailing edge devices like an M1 Mac and a Pixel 8, and performance is very tolerable.

The key is I'm not asking for open ended answers to open ended problems. When it proves to be useful it's not going to get less useful or more expensive.

There are vast domains of uses for LLM models with similar characteristics and likely similar results.

Re: Small Models Have Arrived

#118

Post by bmitc »

HoldOnAMinute wrote: post #103
swatcoder wrote: post #77

Earlier quoted context omitted.

Yes. The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words. The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of res…

Someone will eventually figure out how to package it all into a single, cheap chip

That you can then write text to program and make applications with.

Re: Small Models Have Arrived

#119

Post by CamperBob2 »

jimmaswell wrote: post #91
swatcoder wrote: post #77

Earlier quoted context omitted.

Yes. The infancy phase of this technology is represented by the pursuit of making wildly grand, wildly expensive, all-purpose models that somehow discern a user's full accurate intent from a lazy, underdeveloped, vague idea that they ambiguously and poorly express in a couple dozen words. The adolescence will arrive as those outsized and ill-considered ambitions collapse and we instead see a cambrian explosion of res…

This idea has failed to pan out time and time again - people have an instinct that hand-crafted finely-tuned specialized AI systems must be optimal, but throwing more scale and compute to something more generally smart always wins out. It's especially palpable just looking at the last few years of LLM's: a frontier model with all the world knowledge you can stuff in it and every tool at its disposal has always perfor…

Suggesting otherwise has become an extraordinary claim requiring extraordinary evidence.

VibeThinker 3B constitutes extraordinary evidence, IMO. The first such evidence I've seen myself. Very small model, very low literacy, almost no world knowledge, but it is as good at math and logical reasoning as models a hundred times larger.

The Bitter Lesson is a valid and trenchant observation about how about we got here, but I think it's a mistake to assume it tells us very much about where we're going. Too much has changed recently and is still doing so.

Re: Small Models Have Arrived

#120

Post by keeda »

NitpickLawyer wrote: post #10

> But I also think the demand for "fast/cheap/good-enough" models is just about to take off. There's a sort of "revelation" I had in ~early '24 when I used a 7B local model with a library called Guidance (initially out of MS, then the team moved) to create a flow where the model would receive pseudocode for tests, first write the tests, and once I approved then started writing code until the tests passed. This was be…

Yep, I've been having excellent experiences with the models even from the 2023 era. They required a lot of "holding it right" (mostly: being very precise in what went into the context) but their raw coding capabilities were astonishingly good even then.

However, back then I was getting the AI to write individual functions or classes or a test suite. I was decomposing the larger task into smaller tasks, delegating some of them to the AI, reviewing the results and composing the codebase from those. I was also essentially the harness.

Today the models can write and test and deploy an entire project. In terms of the code quality, I actually don't think today's frontier models would have written it much better than the 2023 models did. So in terms of raw coding capabilities i.e. converting a high-level specification into working code, I think we hit the peak way back in 2024 itself.

What has changed is the AI has learned how to do the task I was doing (besides being the "harness"!), which was the mid-to-higher level "engineering" aspects like decomposing a task, specifying it to a reasonable level, reviewing the outputs, and course correcting as needed.

I'm not sure if that is something the AI labs explicitly focused on during training (which may be why Meta is having its highly paid engineers do annotation work), or an emergent property of "better reasoning" (which I believe Dario implied in a podcast), or some mix of both.

But the fact remains that even the weaker models are more capable than we realize, and many being open weights, are here to stay.

Post reply on HN