121–130 of 495 posts
The math on the input tokens is definitely wrong. It claims each instance (8 GPUs) can handle 1.44 million tokens/sec of input. Let's check that out. 1.44e6 tokens/sec * 37e9 bytes/token / 3.3e12 bytes/sec/GPU = ~16,000 GPUs And that's assuming a more likely 1 byte per parameter. So the article is only off by a factor of at least 1,000. I didn't check any of the rest of the math, but that probably has some impact on…
1. Companies that train models and license them
2. Companies that do inference on models
Earlier quoted context omitted.
There’s no mention of that in this article about it: https://archive.is/wZslL They quote him as saying inference is profitable and end it at that. Are you saying that the COO corrected him at the dinner, or on the podcast? Which podcast was it?
From a journalist at the dinner: “I think that tends to end poorly because as demand for your service grows, you lose more and more money. Sam Altman actually addressed this at dinner. He was asked basically, are you guys losing money every time someone uses ChatGPT? And it was funny. At first, he answered, no, we would be profitable if not for training new models. Essentially, if you take away all the stuff, all the…
Earlier quoted context omitted.
Exactly. All of the claims that OpenAI is losing money on every request are wrong. OpenAI hasn’t even unlocked all of their possible revenue opportunities from the free tier such as ads (like Google search), affiliate links, and other services. There’s also a lot of comments in this thread who want LLM companies to fail for different reasons, so they’re projecting that wish on to imagined unit economics. I’m having f…
As someone who has been taking the largest part of Google and facebooks ad wallet share away, Let me tell you something. Advertising is now a very very locked in market and will take over a decade to shift even a significant minority it into OpenAIs hands. This is not likely the first or even second monetization strategy imo. But I’m happy to be wrong.
Can you elaborate? You’ve sparked my curiosity.
https://www.wheresyoured.at/deep-impact/
Basically, DeepSeek is _very_ efficient at inference, and that was the whole reason why it shook the industry when it was released.
This kind of presumes you're just cranking out inference non-stop 24/7 to get the estimated price, right? Or am I misreading this? In reality, presumably they have to support fast inference even during peak usage times, but then the hardware is still sitting around off of peak times. I guess they can power them off, but that's a significant difference from paying $2/hr for an all-in IaaS provider. I'm also not sure w…
1. Idle instances don't turn electricity to heat so that reduces their operating cost.
2. Idle instances can be borrowed for training which means flexible training amortizes peak inference capacity.
Earlier quoted context omitted.
The difference is the money printer right now only prints for ~6 months before it needs to be replaced with an even more expensive printer.
And if you ever stop/step off the treadmill and jack up prices to reach profitability, a new upstart without your sunk costs will immediately create a 99% solution and start competing with you. Or more like hundreds of competitors. Like we've seen with Karpathy & Murati, any engineer with pedigree working on the frontline models can easily raise billions to compete with them. Expect the trend to pick up as the pool o…
If OpenAI didn't come along with ChatGPT, we would probably just now be getting Google Bard 1.0 with an ability level of GPT-3.5 and censorship so heavy it would make it useless for anything beyond "Tell me who the first president was".
I guess my use is absolutely nothing compare to someone with a couple of agents running continuously.
The math on the input tokens is definitely wrong. It claims each instance (8 GPUs) can handle 1.44 million tokens/sec of input. Let's check that out. 1.44e6 tokens/sec * 37e9 bytes/token / 3.3e12 bytes/sec/GPU = ~16,000 GPUs And that's assuming a more likely 1 byte per parameter. So the article is only off by a factor of at least 1,000. I didn't check any of the rest of the math, but that probably has some impact on…
37 billion bytes per token? Edit: Oh assuming this is an estimate based on the model weights moving fromm HBM to SRAM, that's not how transformers are applied to input tokens. You only have to do move the weights for every token during generation, not during "prefill". (And actually during generation you can use speculative decoding to do better than this roofline anyways).
And more importantly batches, so taking the example from the blog post, it would be 32 tokens per each forward pass in the decoding phase.
Ok, one issue I have with this analysis is the breakdown between input and output tokens. I'm the kind of person who spend most of my chat asking questions, so I might only use 20ish input tokens per prompt, where Gemini is having to put out several hundred, which would seem to affect the economics quite a bit