21–30 of 48 posts
Training is the expensive part here. It seems much more likely that the training of these models slows down drastically and is written off as a sunk cost, a few companies continue running inference on years-old models, and the free versions go away.
This is addressed in the very first sentence of the article that you obviously didn't read.
Never used a LLM or anything explicitly.
Got annoyed when I had to deal with AI chatbots as front-line customer service - although that only happened once or twice in the last couple of months.
So basically, keep doing what I'm doing.
I like AI for specifically targeted applications: - e.g. 100,000+ AI "eyeballs" vs. a few 100 for diagonstic imaging, working out whether there's something to worry about or not. I hate the idea of generalised AI, LLM's etc.
Lowering the bar to enable 'creative output' from non-creative individuals just fucks up the world, because natural talent is replaced by unnatural talent, especially in (late) capitalism, where money is worth more than human experience to those few control-freak managers.
I'm old. I even earnt enough to buy a house with lawn over 4 years ago during my (pre-AI) career as a Software Developer. Get off my damn lawn.
Missing Option 3) hardware and software continue to evolve and AI becomes cost efficient at the same price and eventually even lower
I mean, we're not even up to the "Model T" era of AI development and more like in the 'coach-built' phase where every individual instance needs a bunch of custom work and tuning. Just wait until they get them down to where every Teddy Ruxpin has a full LLM running on a few AA batteries and then see where the market lands.
I always imagine these AI discussion in the context of a bunch of horses discussing these 'horseless carriages' circa 1900...
> it's the running costs of these major AI services that are also astronomical There's wildly different reports about whether the cost of just inference (not the training) is expensive or not... Sam Altman has said “We’re profitable on inference. If we didn’t pay for training, we’d be a very profitable company.” But a lot of folks are convinced that inference prices are currently being propped up by burning through i…
Any individual Sunday service is nearly cost free if we don't calculate in the 100+ years it took to build the church...
> it's the running costs of these major AI services that are also astronomical There's wildly different reports about whether the cost of just inference (not the training) is expensive or not... Sam Altman has said “We’re profitable on inference. If we didn’t pay for training, we’d be a very profitable company.” But a lot of folks are convinced that inference prices are currently being propped up by burning through i…
>> Sam Altman has said “We’re profitable on inference. If we didn’t pay for training, we’d be a very profitable company.” Any individual Sunday service is nearly cost free if we don't calculate in the 100+ years it took to build the church...
There's no scenario where AI goes away completely.
I don't think the "major AI services go away completely" scenario is realistic at all when you look at those companies' revenue and customer demand, but that's a different debate I guess.
How much would it cost you to deploy a model that you and maybe a few coworkers could effectively use? $400k probably to buy all the hardware required to host a top-tier model that could do a few hundred tokens per second for 10 concurrent users? That's $40k per person. Ammortize the hardware over 5 years, thats $8k per person per year (roughly), with no training costs (that's just you buying hardware and running it yourself). So that means, you need ~$800 per user monthly just to cover hardware to run the model (this is with no staffing costs, internet, taxes, electricity, hosting, housing, etc).
So just food for thought, but $200 claude code is probably still losing money even just on inference.
Since they are in the software realm, they are probably shooting for a 90% profit margin. Using the above example, it would be ($800 + R&D + opex) x 10. My guess is assuming no more training (which probably can never be profitable at current rates), they need $20k per month per user, which is why that number was floated by OpenAI previously.
Earlier quoted context omitted.
> we'd probably happily pay >$500 a month for the current frontier models Try $5,000. OpenAI loses hundreds of billions a year, they need a 100x, not 2x.
$60k/yr still seems like a good deal for the productivity multiplier you get on an experienced engineer costing several times that. Actually, I'm fairly certain that some optimizations I had codex do this week would already pay for that from being able to scale down pod resource requirements, and that's just from me telling it to profile our code and find high ROI things to fix, taking only part of my focus away from…
Maybe, but that's a hard sell to all the workplaces who won't even spring for >1080p monitors for their experienced engineers.
Earlier quoted context omitted.
$60k/yr still seems like a good deal for the productivity multiplier you get on an experienced engineer costing several times that. Actually, I'm fairly certain that some optimizations I had codex do this week would already pay for that from being able to scale down pod resource requirements, and that's just from me telling it to profile our code and find high ROI things to fix, taking only part of my focus away from…
Wildly different experience of frontier models than I have, what's your problem domain? I had both Opus and Gemini Pro outright fail at implementing a dead simple floating point image transformation the other day because neither could keep track of when things were floats and when they were uint8.
I'm surprised it can't keep track of float vs uint8. Mine knew to look at things like struct alignment or places where we had slices (Go) on structures that could be arrays (so unnecessary boxing), in addition to things like timer reuse, object pooling/reuse, places where local variables were escaping to heap (and I never even gave it the compiler escape analysis!), etc. After letting it have a go with the profiler for a couple rounds, it eventually concluded that we were dominated by syscalls and crypto related operations, so not much more could be microoptimized.
I've only been using this thing since right before Christmas, and I feel like I'm still at a fraction of what it can do once you start teaching it about the specifics of your workplace's setup. Even that I've started to kind-of automate by just cloning all of our infra teams' repos too. Stuff I have no idea about it can understand just fine. Any time there's something that requires more than a super pedestrian application programmer's knowledge of k8s, I just say "I don't really understand k8s. Go look at our deployment and go look at these guys' terraform repo to see all of what we're doing" and it tells me what I'm trying to figure out.
I know this is probably an annoying question, but… has the author actually tried self-hosting an AI with one's own hardware? I have; ollama (and various frontends thereof) makes it straightforward, and it's absolutely not cost-prohibitive — I've ran my share of LLMs even on laptops without dedicated GPUs at all, and while the experience wasn't great compared to the commercial options, it wasn't outright unusable, either. Locally-hosted LLMs are already finding their way into various applications; that's only going to get more viable over time, not less (unless the computing hardware industry takes a catastrophic nosedive, in which case AI affordability is arguably the least of our worries).
I'm sure the author understands this and is just being hyperbolic in the article's title, but the AI bubble bursting ≠ AI disappearing, for the same reason the dotcom bubble bursting ≠ the World Wide Web disappearing. The bubble will burst when AI shifts from being novel to being mundane, just as with any other technology-related bubble — and that entails a degree of affordability and ubiquity that's mutually exclusive with any notion of AI “disappearing”. Hopefully it'll mean companies being less motivated to shove AI “features” down everyone's throats, but the virtually-intelligent cat is already out of Pandora's box: the technology's here to stay, and I think it's presumptuous to think the race to the bottom w.r.t. cost is anywhere near the finish line.