100 t/s of crap is still crap.
Small Models Have Arrived
251–260 of 372 posts
Re: Small Models Have Arrived
#252It's kinda fun that people get to experience what the 8-bit era was like!
Re: Small Models Have Arrived
#253Luna is definitely "good enough" for so, so many use cases. And that price point...
Goes to show that open source = cheap, closed source = expensive is wrong-headed. Closed source is going to beat open source on price point due to scale (try buying a few GPUs to run your own model on...) and labs innovating across the full inference stack.
That leaves the value prop of open source as privacy / customizability.
Re: Small Models Have Arrived
#254Earlier quoted context omitted.
Fact. My company's largest partner is CoreWeave, and convincing leadership that we could run it ourselves on partner discounted hardware for a lot less money has gone nowhere.
Maybe you need to walk into the office with a $1000 server running a hundred instances of whatever your code does.
We got rid of all our offices, whole company is remote now.
Re: Small Models Have Arrived
#255Earlier quoted context omitted.
I don't think there is tunnel vision. I'm just saying that I have a couple 3090s I invested in a handful of years ago, and they are still going strong today as multiple GPU-needing technologies emerged. I'm not saying everyone has to run local LLMs, because the APIs are in a race to the bottom, and my $10 of OpenRouter credits I bought months ago is down to $8.94 because most models give you MILLIONS of tokens for a…
> I'm just saying that I have a couple 3090s This is tunnel vision. The percentage of people who could afford the hardware you could at the time you back it so vanishingly small. I do not know a single non-tech person who has multiple graphics cards in a single computer.
And right now the demand for GPU is far outpacing the supply, even with factories at full production, which is keeping prices high and out of reach of most people. But unless something happens to shut down the factories (not impossible, but hasn't happened yet), eventually production will catch up to demand and prices will return to sane-ish levels. Won't happen this year, almost certainly not next year... but I would be shocked if the current high prices were to persist for a decade. Eventually the percentage of people who can afford that hardware will grow to be a decent chunk of the computer-owning population. And they'll be following READMEs written by the early adopters, for installing open-source harnesses to work with open-weight models.
My personal expectation is closer to 5 years than 10, which is why I wouldn't touch Anthropic or OpenAI stock with a ten-foot pole, personally, no matter how high their theoretical valuation is. Because their business model is doomed in the long run.
Re: Small Models Have Arrived
#256> Across his various startups, Peter has seen two kinds of work: > 1. the "IQ 180" work. some mad scientist genius type comes up with some crazy solution you've never thought of. > 2. the "token spewer" work. being ultra responsive, pushing the ball forward across dozens of different fronts. Interesting comp to pg's Maker's Schedule, Manager's Schedule https://www.paulgraham.com/makersschedule.html I'm curious about…
> July 2009 > Thanks to Sam Altman, … for reading drafts of this. Oh how times have changed!
Have they? Some say he's still reading those drafts, along with everything else on the Internet.
Re: Small Models Have Arrived
#257Earlier quoted context omitted.
Can you elaborate on this?
The model doesn't have to be smart if all it's doing is pushing a few different buttons. I don't have to be an automotive engineer to start my car and put it in drive.
I mean - how specifically are you using knowledge graphs?
Re: Small Models Have Arrived
#258> One thing a few investors I've talked with have mentioned: "It's weird we're not seeing more consumer AI companies. Why is that?" What would consumer AI company even be? The frontier labs have declared they will eat everything and they have a head start. Best bet would to be a contrarian and build products and services that people actually want or need. Fine to be AI powered or augmented, but consumer companies do…
There is a thin line. Lots of AI powered things are just workflows you could implement with claude code and some skills. Or even just a prompt on the web chat. But I think these big technological swings sometimes take a long time to shake out. Society is still adapting to the internet. I think there is more opportunity for consumer application of AI. Im surprised we dont see more models in game. Small models that do…
I don't know much about game development but given how intensive some AI models can be i feel like adding LLMs to games can have performance issues, whereas in-game AI for computer controlled characters works in-enginer.
Re: Small Models Have Arrived
#259As someone who uses a lot of opus, I'm curious what changes to my workflow I need to make to make small models work. My workflow currently is for a large feature I'd explain the feature and how I envision it technically and run the grill me skill where it'd ask me questions that I may not have thought of, then it'll write a spec and execute from there. It spawns an orchestrator so supposedly its supposed to use haiku for mechanical things but im not confident that works.
How much of this can be replaced with small models, and what changes to my workflow do i need to make?
Re: Small Models Have Arrived
#260Earlier quoted context omitted.
This seems really backwards. The Bitter Lesson is all about large data-based approaches vs hand-crafted ones, it doesn't say anything about language models not trained specifically for chess. I can't find the comment you're referring to, but the latest versions of stockfish are based on neural networks trained on millions of games, so if anything the Bitter Lesson turned out true here.
The conclusion of the bitter lesson would be that a large language model trained on chess commentary as well as being trained on millions of chess games would outperform stockfish which is only trained on millions of chess games. There’s no evidence at this point that this is true.
Chess is a brute force search problem. Humans are not good at chess, even a small computer can beat Magnus Carlsen.
It would be better to compare models at how well they can write the code for chess engines, otherwise it's just saying that Fable is not a good CPU emulator, which is obvious.