Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are OpenAI and Anthropic losing money on inference?

martinalderson.com

Post reply on HN
Print view View original

371–380 of 495 posts

Re: Are OpenAI and Anthropic losing money on inference?

#371

Post by lumost »

_sword wrote: post #276

I've done the modeling on this a few times and I always get to a place where inference can run at 50%+ gross margins, depending mostly on GPU depreciation and how good the host is at optimizing utilization. The challenge for the margins is whether or not you consider model training costs as part of the calculation. If model training isn't capitalized + amortized, margins are great. If they are amortized and need to b…

I wonder how much capex risk there is in this model, depreciating the GPUs over 5 years is fine if you can guarantee utilization. Losing market share might be a death sentence for some of these firms as utilization falls.

Re: Are OpenAI and Anthropic losing money on inference?

#372

Post by Tossrock »

achenet wrote: post #326
simianwords wrote: post #200

Earlier quoted context omitted.

What do you think his strategy is? He has to make money at some point. I don’t buy the logic that he will “scam” his investors and run away at some point.

He makes money by convincing people to buy OpenAI stock. If OpenAI goes down tomorrow, he will be just fine. His incentive is to sell the stock, not actually build and run a profitable business. Look at Adam Neumann as an example of how to lose billions of investor dollars and still walk out of the ensuing crash with over a billion. https://en.wikipedia.org/wiki/Adam_Neumann His strategy is to sell OpenAI stock like…

Altman doesn't have any stock. He's playing a game at a level people caught up on "capitalism bad" can't even conceptualize.

Re: Are OpenAI and Anthropic losing money on inference?

#373

Post by yalogin »

Will these companies ever stop training new models? What does it mean if we get there. Feels like they will have to constantly train and improve the models, not sure what that means either. What ncremental improvements can these models show?

Another question is - will it ever become less costly to train?

Let to see opinions from someone in the know

Re: Are OpenAI and Anthropic losing money on inference?

#374

Post by runako »

renjimen wrote: post #289
meshugaas wrote: post #262

Earlier quoted context omitted.

The "model as company" metaphor makes no sense. It should actually be models are products, like a shoe. Nike spends money developing a shoe, then building it, then they sell it, and ideally those R&D costs are made up in shoe sales. But you still have to run the whole company outside of that. Also, in Nike's case, as they grow they get better at making more shoes for cheaper. LLM model providers tell us that every ne…

But new models to date have cost more than the previous ones to create, often by an order of magnitude, so the shoe metaphor falls apart. A better metaphor would be oil and gas production, where existing oil and gas fields are either already finished (i.e. model is no longer SOTA -- no longer making a return on investment) or currently producing (SOTA inference -- making a return on investment). The key similarity wi…

> new models to date have cost more than the previous ones to create

This largely was the case in software in the '80s-'10s (when versions largely disappeared) and still is the case in hardware. iPhone 17 will certainly cost far more to develop than did iPhone 10 or 5. iPhone 5 cost far more than 3G, etc.

Re: Are OpenAI and Anthropic losing money on inference?

#375

Post by runako »

oblio wrote: post #330
DenisM wrote: post #309

Earlier quoted context omitted.

Fantastic perspective. Basically each new company puts competitive pressure on the previous company, and together they compress margins. They are racing themselves to the bottom. I imagine they know this and bet on AGI primacy.

> I imagine they know this and bet on AGI primacy. Just like Uber and Tesla are betting on self driving cars. I think it's been 10 years now ("any minute now").

Notably, Uber switched horses and now runs Waymos with no human drivers.

Re: Are OpenAI and Anthropic losing money on inference?

#376

Post by xdennis »

efficax wrote: post #95
EcommerceFlow wrote: post #63

I wouldn't be surprised if their profit/query is at a negative for all major Ai companies, but guess what? They have a service which understands a users question/needs 100x better than a traditional Google search does. Once they tap into that for PPC/paid ads, their profit/query should jump into the green. In fact, there's a decent chance a lot of these models will go 100% free once that PPC pipeline is implemented a…

> Once they tap into that for PPC/paid ads, If they start showing ads based on your prompts, and your history of "chats", it will erode the already shaky trust that users have in the bots. "Hallucinations" are one thing, but now you'll be asking yourself all the time: is that the best answer the llm can give me, or has it been trained to respond in ways favourable to its advertisers?

This is the exact same issues Facebook/YouTube/etc had with ads. In the end, ads won.

Google used to segregate ads very clearly in the beginning. Now they look almost the same as results. I've switched to DDG since then, but have the majority of users? Nope. Even if they're not using ad blockers, most people seem to not mind the ads.

With LLMs, the ads will be even more harder to tell apart from non-ads.

Re: Are OpenAI and Anthropic losing money on inference?

#377

Post by Romario77 »

yalogin wrote: post #373

Will these companies ever stop training new models? What does it mean if we get there. Feels like they will have to constantly train and improve the models, not sure what that means either. What ncremental improvements can these models show? Another question is - will it ever become less costly to train? Let to see opinions from someone in the know

current way the models works is that they don't have memory, it's included in training (or has to be provided as context).

So to keep up with times the models have to be constantly trained.

One thing though is that right now it's not just incremental training, the whole thing gets updated - multiple parameters and how the model is trained is different.

This might not be the case in the future where the training could become more efficient and switch to incremental updates where you don't have to re-feed all the training data but only the new things.

I am simplifying here for brevity, but I think the gist is still there.

Re: Are OpenAI and Anthropic losing money on inference?

#378

Post by senko »

Romario77 wrote: post #377
yalogin wrote: post #373

Will these companies ever stop training new models? What does it mean if we get there. Feels like they will have to constantly train and improve the models, not sure what that means either. What ncremental improvements can these models show? Another question is - will it ever become less costly to train? Let to see opinions from someone in the know

current way the models works is that they don't have memory, it's included in training (or has to be provided as context). So to keep up with times the models have to be constantly trained. One thing though is that right now it's not just incremental training, the whole thing gets updated - multiple parameters and how the model is trained is different. This might not be the case in the future where the training could…

Updating the internal knowledge is not the primary motivator here, as you can easily, and more reliably (less hallucination), get that information at inference stage (through web search tool).

They're training new models because the (software) technology keeps improving, (proprietary) data sets keep improving (through a lot of manual labelling but also synthetic data generation), and in general researchers have better understanding of what's important when it comes to LLMs.

Re: Are OpenAI and Anthropic losing money on inference?

#379

Post by supersour »

reliabilityguy wrote: post #269
whatshisface wrote: post #251

Earlier quoted context omitted.

Why is that? It seems like multiple cores requesting the same address would be easier for the MMU to fetch for, not harder.

It’s not that the fetching is the problem, but serving the data to many cores at the same time from a single source.

I'm not familiar with GPU architecture, is there not a shared L2/L3 data cache from which this data would be shared?

Re: Are OpenAI and Anthropic losing money on inference?

#380

Post by black_knight »

vonneumannstan wrote: post #321
BlindEyeHalo wrote: post #303

Earlier quoted context omitted.

Why wouldn't you factor in training? It is not like you can train once and then have the model run for years. You need to constantly improve to keep up with the competition. The lifespan of a model is just a few months at this point.

I suspect we've already reached the point with models at the GPT5 tier where the average person will no longer recognize improvements and this model can be slightly improved at slow intervals and indeed run for years. Meanwhile research grade models will still need to be trained at massive cost to improve performance on relatively short time scales.

Strangely, I feel GPT-5 as the opposite of an improvement over the previous models, and consider just using Claude for actual work. Also the voice mode went from really useful to useless “Absolutely, I will keep it brief and give it to you directly. …some wrong annswer… And there you have it! As simple as that!”
Post reply on HN