Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Are OpenAI and Anthropic losing money on inference?

martinalderson.com

Post reply on HN
Print view View original

91–100 of 495 posts

Re: Are OpenAI and Anthropic losing money on inference?

#91

Post by SubiculumCode »

ninetyninenine wrote: post #81
Aurornis wrote: post #37

Earlier quoted context omitted.

> The models as is are still hugely useful, even if no further training was done. Exactly. The parent comment has an incorrect understanding of what unit economics means. The cost of training is not a factor in the marginal cost of each inference or each new customer. It’s unfortunate this comment thread is the highest upvoted right now when it’s based on a basic misunderstanding of unit economics.

I upvoted it because it aligns most closely with my own perspective. I have a strong dislike for AI and everything associated with it, so my judgment is shaped by that bias. If a post sounds realistic or complex, I have no interest in examining its nuance. I am not concerned with practical reality and prefer to accept it without thinking, so I support ideas that match my personal viewpoint. I don’t understand why peo…

Is this written by a sarcastic AI?

Re: Are OpenAI and Anthropic losing money on inference?

#92

Post by topaz0 »

lkjdsklf wrote: post #48
JCM9 wrote: post #4

These articles (of which there are many) all make the same basic accounting mistakes. You have to include all the costs associated with the model, not just inference compute. This article is like saying an apartment complex isn’t “losing money” because the monthly rents cover operating costs but ignoring the cost of the building. Most real estate developments go bust because the developers can’t pay the mortgage paym…

It’s fun to work backwards, but i was listening to a podcast where the journalists were talking about a dinner that Sam Altman had. This question came up and Sam said they were profitable if you exclude training and the COO corrected him So at least for OpenAI, the answer is “no” They did say it was close And that’s if you exclude training costs which is kind of absurd because it’s not like you can stop training

Worth noting that the post only claims they should be profitable for the inference of their paying customers on a guesstimated typical workload. Free users and users with atypical usage patterns will obviously skew the whole picture. So the argument in the post is at least compatible with them still losing money on inference overall.

Re: Are OpenAI and Anthropic losing money on inference?

#94

Post by EcommerceFlow »

techpineapple wrote: post #84
EcommerceFlow wrote: post #63

I wouldn't be surprised if their profit/query is at a negative for all major Ai companies, but guess what? They have a service which understands a users question/needs 100x better than a traditional Google search does. Once they tap into that for PPC/paid ads, their profit/query should jump into the green. In fact, there's a decent chance a lot of these models will go 100% free once that PPC pipeline is implemented a…

> They have a service which understands a users question/needs 100x better than a traditional Google search does. Source?

A lifetime of using Google and 4 years of using LLMs.

Re: Are OpenAI and Anthropic losing money on inference?

#95

Post by efficax »

EcommerceFlow wrote: post #63

I wouldn't be surprised if their profit/query is at a negative for all major Ai companies, but guess what? They have a service which understands a users question/needs 100x better than a traditional Google search does. Once they tap into that for PPC/paid ads, their profit/query should jump into the green. In fact, there's a decent chance a lot of these models will go 100% free once that PPC pipeline is implemented a…

> Once they tap into that for PPC/paid ads,

If they start showing ads based on your prompts, and your history of "chats", it will erode the already shaky trust that users have in the bots. "Hallucinations" are one thing, but now you'll be asking yourself all the time: is that the best answer the llm can give me, or has it been trained to respond in ways favourable to its advertisers?

Re: Are OpenAI and Anthropic losing money on inference?

#96

Post by CharlesW »

conradev wrote: post #50
JCM9 wrote: post #4

These articles (of which there are many) all make the same basic accounting mistakes. You have to include all the costs associated with the model, not just inference compute. This article is like saying an apartment complex isn’t “losing money” because the monthly rents cover operating costs but ignoring the cost of the building. Most real estate developments go bust because the developers can’t pay the mortgage paym…

I found Dario’s explanation pretty compelling: https://x.com/FinHubIQ/status/1960540489876410404 the short of it: if you do the accounting on a per-model basis, it looks much better

That was worth a watch, thank you!

Re: Are OpenAI and Anthropic losing money on inference?

#97

Post by moduspol »

This kind of presumes you're just cranking out inference non-stop 24/7 to get the estimated price, right? Or am I misreading this?

In reality, presumably they have to support fast inference even during peak usage times, but then the hardware is still sitting around off of peak times. I guess they can power them off, but that's a significant difference from paying $2/hr for an all-in IaaS provider.

I'm also not sure we should expect their costs to just be "in-line with, or cheaper than" what various hourly H100 providers charge. Those providers presumably don't have to run entire datacenters filled to the gills with these specialized GPUs. It may be a lot more expensive to do that than to run a handful of them spread among the same datacenter with your other workloads.

Re: Are OpenAI and Anthropic losing money on inference?

#98

Post by ekelsen »

The math on the input tokens is definitely wrong. It claims each instance (8 GPUs) can handle 1.44 million tokens/sec of input. Let's check that out.

1.44e6 tokens/sec * 37e9 bytes/token / 3.3e12 bytes/sec/GPU = ~16,000 GPUs

And that's assuming a more likely 1 byte per parameter.

So the article is only off by a factor of at least 1,000. I didn't check any of the rest of the math, but that probably has some impact on their conclusions...

Re: Are OpenAI and Anthropic losing money on inference?

#99

Post by smjburton »

Good breakdown of the costs involved. Even if they're running at a loss, OpenAI and Anthropic receive considerable value from the free training data users are providing through their conversations. Looking at it another way, these companies are paying for the training data to make their models better for future profitability.
Post reply on HN