Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Outsourcing plus local AI will soon become more economical vs. frontier labs

signalbloom.ai

Post reply on HN
Print view View original

351–360 of 408 posts

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#351

Post by kakacik »

Npovview wrote: post #295
0xbadcafebee wrote: post #286

Earlier quoted context omitted.

Most software development teams are pushing back on the deluge of bad changes from AI tools and are moving slower again to regain trust and stability. It is likely that future software development will not actually be higher velocity.

Bad changes will be eliminated because better people are using the AI tools. They will reduce cost as well as slop.

Yes and we will all hold hands together singing and bring world peace from now on forever. Back in reality, I can see some obstacles without even trying

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#352

Post by addedGone »

wahnfrieden wrote: post #285
lunar_mycroft wrote: post #269

Earlier quoted context omitted.

Right, they can do it to sell enterprise plans, but they can't offer said plans to those enterprise customers indefinitely. So if your employer wants to spend $200/month on tokens, you're going to get however many tokens $200 buys you each month, not the order of magnitude more you can get with a consumer subscription.

That’s what I’m saying. Enterprise customers don’t use the subscription plan

Except that they do, we do.

A lot of startups pile up enormous amount of accounts, companies don't need the Enterprise Anthropic solution, they can just subscribe to many accounts and have their own staff KYC for each (1 codex, 1 claude, 1 google and so-on).

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#354

Post by regexorcist »

koyote wrote: post #281
regexorcist wrote: post #237

Earlier quoted context omitted.

Sure, I have a 64G MBP with an M1 Ultra. The best model for me by far has been the 35B A3B, in particular the 8Q_KL unslouth variant. The dense model works but it's much slower, and I don't really see a difference in quality with a good harness.

What do you use as a harness?

I use oh-my-openagent. It does an incredible job at planning and executing by orchestrating subagents with different roles.

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#355

Post by rldjbpin »

context: i work closely with the "proposed".

people dispute the 1k per seat number. i have seen cases where that is the amount billed, while the actual engineer might be paid even less so.

the point we seem to forget is the local compute available to the offshore team. it might be highly optimistic to assume a capable machine for each dev which might cost orders of magnitude their monthly take home. i have seen way too many guys have simple office laptops from dell/lenovo/hp/etc. with dated ultrabook specs and at most 16gb system memory in Windows. just infeasible to assume that they could get away with local ai.

combined with network complications with some (besides corporate vpn, some may be working at times using a mobile hotspot), combined with corporate policy, it becomes a logistics nightmare to make this happen.

while equally infeasible long-term, the "per-seat" coding assistant subscription like github copilot is the most compatible to this scenario. it is an add-on to the existing software licensing anyways, and might be the most ergonomic in that sense. however, recent times have shown that it's been too cheap and subsidised to work anymore.

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#356

Post by dminik »

wongarsu wrote: post #325
dminik wrote: post #169

Earlier quoted context omitted.

Does Anthropic really expect to double their income without also doubling their expenses?

There we go back to the original question: are subscriptions profitable, API pricing wildly profitable, and they just lose all that money on fixed costs like model training; or do they actually barely make money on inference? That's why talking about the profitability of inference without accounting for model training is interesting, because that is the deciding factor in whether more customers would help getting the…

Without actual data I don't know. My gut feeling is that they overall lose money on subscriptions (and especially the free tier that accounts for 95% of all users). And make thin profit (~5%) on API pricing.

But it's just that. A gut feeling.

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#357

Post by dminik »

i2km wrote: post #322
dminik wrote: post #169

Earlier quoted context omitted.

Does Anthropic really expect to double their income without also doubling their expenses?

This is one of the things people miss. If they double their customers, of course they double their expenses. Unlike SW, the marginal cost here is still high

I mean, it's possible that with the new datacenter from SpaceX, they could onboard more users than it costs them to rent. That's fair. But I kind of doubt that.

One thing that really stinks to me is that various AI boosters have been claiming insane profit margins (40%, 50%, ...), yet apparently Anthropic stands to (possibly) make $500M profit on $11B in expenses, that's clearly nowhere near 50%. Not to mention that they're not making profit on inference now.

So where do people get this confidence to pull random numbers from?

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#358

Post by stavros »

SyneRyder wrote: post #339
stavros wrote: post #249

Earlier quoted context omitted.

We deployed OpenWebUI with the Claude API the other day for employees. Someone sent ten messages (which appeared to just be reasonable day-to-day work), and we paid $200 for it. There were 44M input tokens, 100k output tokens, no cache hits at all. OpenWebUI reports 3M tokens used, Claude reports 44M, and I have no idea where the rest of the tokens went. This was all on a brand new API key, installed directly to the…

No cache hits seems ominous, could this be an OpenWebUI issue? It also seems ominous that Anthropic models are basically nowhere on the OpenWebUI leaderboards. I'm only doing a cursory search, but it seems OpenWebUI doesn't support Anthropic caching, and they don't intend to? Other providers handle caching automatically (apparently?) but caching has to be specifically managed by the client with Anthropic. If that's c…

Really? Huh, I've never heard of Anthropic caching needing to be specifically enabled. I'll look into that, thank you! Sounds like the culprit.

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#359

Post by Npovview »

curt15 wrote: post #274
Npovview wrote: post #119

Earlier quoted context omitted.

Even with increased prices, AI enables velocity both in development and bugs fixing. Would companies want that? If prices are biting the company, I think companies will route all development and bugs fixing requests through few superperfomer developers with complete knowledge of the different components within the company (they will be the Queen Bees holding the company on their head). The rest of the company will be…

> Even with increased prices, AI enables velocity both in development and bugs fixing. What about human understanding of the codebase that's essential to any project's long term health? Even "superperformer developers" eventually leave the company.

Ask multiple AIs (if you can't trust one) to explain the project.

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#360

Post by Npovview »

runtime_terror wrote: post #194
Npovview wrote: post #119

Earlier quoted context omitted.

Even with increased prices, AI enables velocity both in development and bugs fixing. Would companies want that? If prices are biting the company, I think companies will route all development and bugs fixing requests through few superperfomer developers with complete knowledge of the different components within the company (they will be the Queen Bees holding the company on their head). The rest of the company will be…

So kinda like how stuff is now at a lot of big companies? I've worked at many different companies and almost always there are a few out-performers and a lot of people just found enough not to get fired (no hate, power to them lol). We're already seeing slash their AI budgets. I expect that will increase till we hit more of an equilibrium.

I think people will start measuring (features * time taken to implement) to tokens consumed ratio and then redistribute token budgets to developers. This will measure how effective/efficient people are LLMs.
Post reply on HN