Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Outsourcing plus local AI will soon become more economical vs. frontier labs

signalbloom.ai

Post reply on HN
Print view View original

321–330 of 408 posts

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#321

Post by zozbot234 »

jurgenburgen wrote: post #318
adrian_b wrote: post #254

Earlier quoted context omitted.

There is no wear on the SSDs, because the weights are just read, they are not written during inference. For model training, the requirements are very different, and the training of a big LLM cannot be done with home equipment. On the other hand, inference can be done on almost any PC, even for LLMs with thousands of billions of parameters, just very slowly. The only problem is that the inference becomes limited by th…

Does more RAM increase performance? This approach sounds like it could eventually be fast enough for local use as hardware and models improve.

Faster SSD access improves performance more than RAM does, at least until all of the model is being cached in RAM. So older and cheaper HEDT platforms with lots of PCIe lanes to attach storage to are best for this approach.

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#322

Post by i2km »

dminik wrote: post #169
anthonypasq wrote: post #114

Earlier quoted context omitted.

https://www.wsj.com/tech/ai/mind-blowing-growth-is-about-to-...

Does Anthropic really expect to double their income without also doubling their expenses?

This is one of the things people miss. If they double their customers, of course they double their expenses. Unlike SW, the marginal cost here is still high

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#323

Post by [deleted] »

stavros wrote: post #249
alexriddle wrote: post #91

Earlier quoted context omitted.

Anything over 150 seats means you need to pay at token rates plus the $20/user. My day job is operational (no coding at all) and I'm spending ~$300 a month on a few chats with Claude/Cowork a day over the course of a month.

We deployed OpenWebUI with the Claude API the other day for employees. Someone sent ten messages (which appeared to just be reasonable day-to-day work), and we paid $200 for it. There were 44M input tokens, 100k output tokens, no cache hits at all. OpenWebUI reports 3M tokens used, Claude reports 44M, and I have no idea where the rest of the tokens went. This was all on a brand new API key, installed directly to the…

[deleted]

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#324

Post by dns_snek »

adrian_b wrote: post #259
stingraycharles wrote: post #255

Earlier quoted context omitted.

> The difference between datacenter hardware and cheap personal hardware is not in what can be run and what cannot be run. You do realize that a model like Opus is (estimated to be) around 5T parameters, and uses around 5TB of GPU memory? These kind of things are just impossible to run locally.

This kind of things can certainly be run locally, even on a small mini-PC, like a NUC, or even on a laptop, with the weights stored on SSDs. Like I have said, the problem is not that they cannot be run, but that they may run more slowly than it is acceptable for a given application. Depending on the model, the speeds reported for inference with weights stored on SSDs vary from one token every few seconds to at most a…

When people say that you "can't do" something what they actually mean is that it's completely impractical (if not impossible).

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#325

Post by wongarsu »

dminik wrote: post #169
anthonypasq wrote: post #114

Earlier quoted context omitted.

https://www.wsj.com/tech/ai/mind-blowing-growth-is-about-to-...

Does Anthropic really expect to double their income without also doubling their expenses?

There we go back to the original question: are subscriptions profitable, API pricing wildly profitable, and they just lose all that money on fixed costs like model training; or do they actually barely make money on inference?

That's why talking about the profitability of inference without accounting for model training is interesting, because that is the deciding factor in whether more customers would help getting them in the green

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#326

Post by ajb »

stingraycharles wrote: post #77
Chyzwar wrote: post #48

When discussing LLM pricing, people are missing the plot. The subscription token price is 10x-40x cheaper than API pricing. Your 90$ Claude subscriptions give you close to $1000 to $4000 in equivalent API token pricing. The second issue is that the quality of the model “operator” makes a massive difference in the outcomes. Highly skilled senior devs who know how to prompt and have high agency will outperform team peo…

Also, your local hardware is in no way capable of running the types of models that the cloud providers do, it’s just not economically feasible, and it never will be.

SanDisk has designed a flash equivalent to HBM, which has 1.6TB/s of bandwidth. I expect that it will be available initially to server manufacturers only, but once supply ramps up will be built into individual machines. At that point it will be practical to run local inference on much larger models. Of course, maybe the SOTA providers will find some way to use even larger ones, but it seems like the returns to scale aren't as much as they were.

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#327

Post by myzek »

koonsolo wrote: post #179
ecshafer wrote: post #67

I have really been trying to get local models to work. I have tried different harnesses, tooling, skills, prompts, etc. But when I compare claude code with anthropic models or codex with gpt 5.5, vs qwen, glm or gemma and the same harnesses, the frontier models come out massively ahead. I am at the point where I just don't see the point of the non-frontier models, they waste more time than they save.

I came to the same conclusion. For the amount that a query costs, using Opus all the time is the cheapest option.

For now. The prices for using those will increase massively at some point

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#328

Post by [deleted] »

jyounker wrote: post #182

The problem with outsourcing, as opposed to remote developers, is that it takes a really good manager and tech lead to make it work. My experience is that you have to write extremely detailed design documents and work specifications in order to get effective results. These generally have to be as detailed as most effective prompts. Once you've written specs that detailed, why do you need outsourced developers and fro…

[deleted]

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#329

Post by orphea »

gpugreg wrote: post #102
stymaar wrote: post #93

Earlier quoted context omitted.

> Lastly, there is a massive difference in capabilities, determinism, and error handling between 5T SOTA models like Opus What's your source for Opus being a 5T model? > and tiny distillations from DeepSeek that perform well only in benchmarks. I don't think you know what you're talking about. Local models aren't “distillations from Deepseek”. And they don't perform well “only in benchmarks”, Qwen 3.6 is a very decen…

> What's your source for Opus being a 5T model? Elon Musk tweeted that Grok is 0.5T or 1/10th the size of Opus. https://xcancel.com/elonmusk/status/2042123561666855235#m While this source's reliability is certainly debatable, the size matches the results of this paper, in which researchers estimated the parameter count from model knowledge. https://01.me/research/ikp/

  Elon Musk tweeted
Come on. The Onion would be a more credible source.

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#330

Post by redsocksfan45 »

latexr wrote: post #161
danielmarkbruce wrote: post #124

Earlier quoted context omitted.

It's 100% this. Many people suck at prompting. It's likely that habits from search are ingrained. But in general some people are just so bad at it .

According to Google, “there’s no wrong way to prompt”. https://www.youtube.com/watch?v=9bBfYX8X5aU&t=48s

[dead]
Post reply on HN