21–30 of 63 posts
Earlier quoted context omitted.
A $7 per query estimate is beyond absurd.
Which of my estimates do you find absurd? How would you calculate cost?
If they charged 1 cent per 100B flops that pricing would be $35.000 / 1K tokens. They may be taking a loss, but I doubt it would be at a 1 to 17,500 ratio.
Even GPT-4 with 32k token window is only $0.12 / 1K tokens
Earlier quoted context omitted.
> wager this will materially affect Microsoft's next earnings report How?
They're footing the bill for this whole operation. They provide the compute and storage and they're also so heavily invested in OpenAI after their recent $10B round that they can't offload the costs. See my other comment on this post for a conservative cost estimate for each query.
GPT3 has 175 billion parameters and Altman said 4 would use way more compute than 3. Let's just look at GPT 3. Each forward pass requires 2N=350B flops per token. The computational overhead of the attention mechanism is negligible but the memory overhead of the attention for all users is not, but we can ignore that for now. Let's assume each query involves ~200 tokens of combined input and output. That's 70T flops pe…
I've heard from insiders that the cost is roughly a couple cents per ChatGPT message but that was back with the legacy GPT-3.5 model, not the Turbo model or GPT-4
Earlier quoted context omitted.
Which of my estimates do you find absurd? How would you calculate cost?
Most obviously this part must be off by 3-4 orders of magnitude: > Let's say the cost of compute is 1 cent per 100B flops (probably a lowball) That's like 0.5ms worth of compute on A100. VMs with an A100 cost a few dollars per hour at cloud providers, i.e. about a cent per second or 0.001 cents per ms. OpenAI would obviously get a far better deal than somebody renting a single GPU for an hour. Look, we know what Open…
I sourced my cost/flop estimate from here: https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-fl...
I'll acknowledge that a fleet entirely made up of A100 GPUs running at full capacity 100% of the time with Azure's 3-year commitment pricing (likely very low margin) would make this 5 orders of magnitude cheaper. But the sequential nature of transformers makes it difficult to achieve perfect utilization, and there aren't enough A100 GPUs in the Azure fleet to serve ChatGPT's billion users, so they're certainly running on mostly lesser hardware. Also, my estimate was for GPT-3 not 4, and neglected to address any memory or bandwidth costs, not to mention the costs to train and iterate on the models in the first place, including legions of data curators and many researchers commanding million dollar salaries.
If ChatGPT plus is $20/mo and each user uses it ~10 times per day, then MS/OpenAI would break even if my estimate were 2 orders of magnitude too high. Within the year, we'll have a much better idea of the real cost of these models.
Have to assume they literally can't get enough GPUs to respond to demand. Wonder if $20/mo is even profitable for them?
From my experience with their Plus plan GPT 4 usually just times out no matter what the prompt, forcing me to revert to 3.5. I'm not sure why I haven't asked for a refund yet.
Plus, you can always use Bing Chat. It’s more locked down, but depending on what you’re doing, it’s pretty good and backed by GPT-4.
Earlier quoted context omitted.
Which of my estimates do you find absurd? How would you calculate cost?
Most obviously this part must be off by 3-4 orders of magnitude: > Let's say the cost of compute is 1 cent per 100B flops (probably a lowball) That's like 0.5ms worth of compute on A100. VMs with an A100 cost a few dollars per hour at cloud providers, i.e. about a cent per second or 0.001 cents per ms. OpenAI would obviously get a far better deal than somebody renting a single GPU for an hour. Look, we know what Open…
Earlier quoted context omitted.
Most obviously this part must be off by 3-4 orders of magnitude: > Let's say the cost of compute is 1 cent per 100B flops (probably a lowball) That's like 0.5ms worth of compute on A100. VMs with an A100 cost a few dollars per hour at cloud providers, i.e. about a cent per second or 0.001 cents per ms. OpenAI would obviously get a far better deal than somebody renting a single GPU for an hour. Look, we know what Open…
Basic common sense would tell you that starting with an assumption and working backwards to justify that assumption is called succumbing to confirmation bias. I sourced my cost/flop estimate from here: https://aiimpacts.org/2019-recent-trends-in-gpu-price-per-fl... I'll acknowledge that a fleet entirely made up of A100 GPUs running at full capacity 100% of the time with Azure's 3-year commitment pricing (likely very…
The other thing to note is that OpenAI limits paying users to X queries per Y hours in GPT-4, something like 3 per 5. Which means the GPT-4 fleet is quite limited.
Have to assume they literally can't get enough GPUs to respond to demand. Wonder if $20/mo is even profitable for them?
I'm sure serious competitors are just around the corner, at least Bard will catch up, No doubt Apple has something in the works, there's little to zero doubt about that.
So do they make a huge capital investment hoping that growth will continue at this trajectory, or do they back off the gas pedal ?
I bet there's also some contentious ideas going on internally too. If they go on a huge hiring spree, does that kind of invalidate the product which is supposed to replace everyone? How do the optics look there? Do they hire and fire once GPT-5 becomes a sentient AGI?
Google already has a huge engineering team, operates are unprecedented scale, and IMO has a much more diverse set of products to integrate with, they also solve a lot of novel solutions which they require specialized engineers that can't be replaced easily by "AI".
Be interesting to see how it plays out. Maybe a victim of their own success ?