Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Outsourcing plus local AI will soon become more economical vs. frontier labs

signalbloom.ai

Post reply on HN
Print view View original

161–170 of 408 posts

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#161

Post by latexr »

danielmarkbruce wrote: post #124
kelseyfrog wrote: post #87

Earlier quoted context omitted.

> The quality of the model “operator” makes a massive difference in the outcomes. My hunch is that this is the source of much of the variability in outcomes upstream of HN commenters claiming extremes of, "This model changes everything!" to "This[same] model is crap." We haven't operationalized what it means to "be good at prompting," nor developed proxies/heuristics/shibboleths for accessing prompting skill. There's…

It's 100% this. Many people suck at prompting. It's likely that habits from search are ingrained. But in general some people are just so bad at it .

According to Google, “there’s no wrong way to prompt”.

https://www.youtube.com/watch?v=9bBfYX8X5aU&t=48s

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#162

Post by adrian_b »

colonCapitalDee wrote: post #101
zozbot234 wrote: post #99

Earlier quoted context omitted.

It can run open-weight models that are roughly as capable. It's going to be slow unless you're using actual datacenter hardware, but they'll run.

"roughly" is doing a lot of heavy lifting there

The difference between datacenter hardware and cheap personal hardware is not in what can be run and what cannot be run.

Anything can also be run on a cheap computer.

The difference is in speed. A cheap computer may run a big model up to a few orders of magnitude slower than datacenter hardware, depending on whether the LLM is small enough to fit in GPU memory, or it is small enough to fit in CPU memory or it is so big that it must spill on SSDs.

Depending on the application, the tradeoff between run time and run cost may happen to favor using local hardware, despite a much slower speed.

There are plenty of applications where doing them for negligible cost during an overnight job can be preferable to obtaining faster results at a very high price, for instance scanning for bugs in a mature code base using a great number of different open-weights LLMs, which can achieve similar bug coverage like using a single, but overpriced and unavailable SOTA LLM, e.g. Mythos.

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#163

Post by pyreko »

simonw wrote: post #79
Chyzwar wrote: post #48

When discussing LLM pricing, people are missing the plot. The subscription token price is 10x-40x cheaper than API pricing. Your 90$ Claude subscriptions give you close to $1000 to $4000 in equivalent API token pricing. The second issue is that the quality of the model “operator” makes a massive difference in the outcomes. Highly skilled senior devs who know how to prompt and have high agency will outperform team peo…

I learned today that the Anthropic "Enterprise" plan - the one big companies use because they need governance features and audit logs and all of that jazz - is billed at API token rates (plus $20/seat/month). So large companies are getting billed a lot more than those discount subscription plans.

Yep, where I work I know people easily spending over a few thousand dollars a month.

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#164

Post by overfeed »

ramesh31 wrote: post #153
stymaar wrote: post #106

Earlier quoted context omitted.

[flagged]

People can simultaeneously be reprehensible idiots while being a reliable expert on something they have personally invested billions of dollars into and operate at scale.

> ...while being a reliable expert on something they have personally invested billions of dollars into and operate at scale.

Like "Full Self-Driving" from coast-to-coast by 2016?

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#165

Post by spprashant »

runtime_terror wrote: post #115
spprashant wrote: post #55

Earlier quoted context omitted.

"offshore Indian devs" are no slouches. They have access to the same GPT models and likely cost a tenth of the median US salary. Businesses are always looking to lower marginal cost. They will hire 1 software architect in US to write specs and 10 software developers in India to babysit 100 agents.

Obviously this is just anecdotal but over my 20+ year career I've worked with a lot of outsourced teams in India and my experience has nearly always been that they require a frustratingly specific degree of direction to product anything of quality. Just recently I asked a dev there for a POC of a feature with decent specificity and ended up with about 8k LOC of spaghetti. I re-wrote it later in a few hundred. This is…

Companies are also pivoting away from mere outsourcing to setting up entire GCCs in there.

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#166

Post by DougN7 »

Npovview wrote: post #119
runtime_terror wrote: post #108

Earlier quoted context omitted.

> The subscription token price is 10x-40x cheaper than API pricing This is a temporary phenomenon. Expect either drastic price increases or draconian throttling or both in the coming months. These companies are operating at huge loses and have hundreds of billions in liabilities and commitments. They need to turn on the money faucet sooner than later.

Even with increased prices, AI enables velocity both in development and bugs fixing. Would companies want that? If prices are biting the company, I think companies will route all development and bugs fixing requests through few superperfomer developers with complete knowledge of the different components within the company (they will be the Queen Bees holding the company on their head). The rest of the company will be…

From what I understand, that is sort of how IBM Bob works - multiple models behind the scenes and they route the request to the model that will handle it best at the lowest price.

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#167

Post by lunar_mycroft »

m_kos wrote: post #157
alexriddle wrote: post #91

Earlier quoted context omitted.

Anything over 150 seats means you need to pay at token rates plus the $20/user. My day job is operational (no coding at all) and I'm spending ~$300 a month on a few chats with Claude/Cowork a day over the course of a month.

$300 is my employer's monthly cap on Claude Enterprise. It lasts me at most a week of moderate use. I would much rather get Codex Pro and Claude Pro or Max, which would cost ≤ $200. For $300, one could also add Gemini Ultra to the mix so I could have all three review each other's code, etc. Claude can be very good but enterprise pricing doesn't make sense to me.

The $200 plan you're talking about is subsidized by Anthropic. They cannot afford to keep offering that to everyone indefinitely. Absolute best case scenario for current users is that they can continue to subsidize it as way to sell enterprise plans, but there's no way that they can keep offering it to everyone at those prices.

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#168

Post by overfeed »

alfiedotwtf wrote: post #118
runtime_terror wrote: post #108

Earlier quoted context omitted.

> The subscription token price is 10x-40x cheaper than API pricing This is a temporary phenomenon. Expect either drastic price increases or draconian throttling or both in the coming months. These companies are operating at huge loses and have hundreds of billions in liabilities and commitments. They need to turn on the money faucet sooner than later.

Incentives matter… If prices keep going up, watch for companies to exit frontier models and go to local llama.cpp instances for 6-month-ago SOTA, with the flex of being housed within the office - no more privacy leakage, no more price gouging. To be honest, I’m not sure why a Y-Combinator backed company hasn’t come out yet flooding the market with highly capable OPAI (pronounced “Oh-pah” as in what Greeks shout as th…

> I’m not sure why a Y-Combinator backed company hasn’t come out yet flooding the market with highly capable OPAI

If we momentarily disregard the fact that YC itself owns billions of dollars worth of OpenAI shares[1], YC would plan to find demo-day investors willing to drive down the value of frontier labs. The coöpetition among VCs and the existing web of AI investments will mean no VC will be interested in investing in local AI...until after the frontier labs IPO.

1. Thanks to the self-dea^w foresight of former YC president Sam Altman

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#169

Post by dminik »

anthonypasq wrote: post #114
Forgeties79 wrote: post #100

Earlier quoted context omitted.

Considering not one company is in the black yet I don’t really know how we can say anyone is making bank, unless we want to count absurd levels of VC funding (now slowing down) I guess.

https://www.wsj.com/tech/ai/mind-blowing-growth-is-about-to-...

Does Anthropic really expect to double their income without also doubling their expenses?

Re: Outsourcing plus local AI will soon become more economical vs. frontier labs

#170

Post by causal »

freediddy wrote: post #59

My friend is an exec at a US software company and they are preparing to lay off a few teams of programmers in their Eastern European locations and replacing them with a small number of US programmers + AI. He said they are much more productive and produce new features much faster.

This makes more sense to me. The bottleneck for me is less becoming "understanding code" and more "understanding users". Validating the latter is a task non-programmers can do.
Post reply on HN