GPT-4o mini: advancing cost-efficient intelligence • Hacker News Board

Re: GPT-4o mini: advancing cost-efficient intelligence

#21

Post by tedsanders » Thu, Jul 18, 2024, 6:16 PM UTC

minimaxir wrote: post #3

GPT-4o mini is $0.15/1M input tokens, $0.60/1M output tokens. In comparison, Claude Haiku is $0.25/1M input tokens, $1.25/1M output tokens. There's no way this price-race-to-the-bottom is sustainable.

Yeah, to put these prices in perspective: when tokens get this cheap, $1M buys you more than a trillion output tokens.

To earn appreciable revenue at this price, an LLM company needs to be regularly generating multiple internets worth of text.

On the one hand, generating multiple internets of text seems outlandish.

But on the other hand, we're now approaching the point where you can start building LLMs into software without fretting about cost. Now that you can buy ~30 pages for a penny (instead of a dollar) you can really start to throw it into websites, games, search bars, natural language interfaces etc. without every user costing you much.

But small models are not the endgame for these AI companies, as truly general intelligence is a market worth trillions.

What this ~98% cost drop over 2 years hints at is that when AGI does arrive, it might not be horribly expensive.

Re: GPT-4o mini: advancing cost-efficient intelligence

#22

Post by skybrian » Thu, Jul 18, 2024, 6:18 PM UTC

razodactyl wrote: post #12

Here's something interesting to think about: In ML we do a lot of bootstrapping. If a model is 51% wrong on a binary problem you flip the answer and train a 51% correct model then work your way up from there. Small models are trained from synthetic and live data curated and generated by the more advanced models. If I end up with an advanced smaller model capable of running alongside a greater number of similar models…

I’m a little skeptical of processes that seem to create more information than you had to start with. For a game like chess or Go, it makes sense, because winning strategies are implicit in the rules of the game, but it takes a lot of computation to discover the consequences. Similarly for math where theorems are non-obvious consequences of axioms. And computer code can be similar to math.

But how does that work for an LLM in general? They’re trained on everybody’s opinions all at once, both right and wrong answers. They’re trained to generate text supporting all sides of every argument. What does more training on derived text actually do?

Re: GPT-4o mini: advancing cost-efficient intelligence

#23

Post by OutOfHere » Thu, Jul 18, 2024, 6:23 PM UTC

Sohcahtoa82 wrote: post #14

minimaxir wrote: post #3

GPT-4o mini is $0.15/1M input tokens, $0.60/1M output tokens. In comparison, Claude Haiku is $0.25/1M input tokens, $1.25/1M output tokens. There's no way this price-race-to-the-bottom is sustainable.

Take a loss on every sale and make up for it with volume!

> Take a loss on every sale and make up for it with volume!

If you take a loss on every sale, it is impossible to make up for it with volume. The result will be a loss magnified by the volume.

Re: GPT-4o mini: advancing cost-efficient intelligence

#24

Post by alach11 » Thu, Jul 18, 2024, 6:30 PM UTC

petercooper wrote: post #15

k2xl wrote: post #8

This is great - Though I am confused on two things: 1. How is it possible that GPT-4o mini outperforms 3.5 turbo but 3.5 turbo is more expensive? Like why would someone use a worse model and pay more? 2. Why is the GPT4o vision and GPT4o-mini vision cost the same?

I might be wrong, but I've inferred from OpenAI's pricing behavior that they use it to encourage people to migrate to more efficient models. The 3.5 Turbo pricing is maintained to encourage you to stop using it. Look at davinci-002's pricing, for example - it's very high for something that's relatively ancient.

It's also very likely that 3.5-turbo is more expensive for them to run than gpt-4o-mini. Models are getting smaller and more efficient. They just keep 3.5-turbo around for legacy support.

Re: GPT-4o mini: advancing cost-efficient intelligence

#25

Post by skybrian » Thu, Jul 18, 2024, 6:31 PM UTC

razodactyl wrote: post #11

minimaxir wrote: post #3

GPT-4o mini is $0.15/1M input tokens, $0.60/1M output tokens. In comparison, Claude Haiku is $0.25/1M input tokens, $1.25/1M output tokens. There's no way this price-race-to-the-bottom is sustainable.

At scale you should realise that this is still A LOT of money and the models are considerably reduced in cost so the margin probably works out even better. OpenAI are successful, it's a fact, which means they know what they're doing business wise. (Not bootlicking, just trying to be logical). Think about it this way: Imagine if every email you sent or every online forum post you commented on provided incentive for th…

I’m not sure what you mean and I don’t see how profitability follows from that?

Venture-backed companies can lose money for years. Sometimes it pays off in the end, but making predictions about profitability seems hard inside a bubble.

Also, some industries like manufacturing solar panels have high market growth but they’re unprofitable for most manufacturers.

So I think it remains to be seen if OpenAI knows what they’re doing. It doesn’t seem like the sort of thing armchair arguments are good at predicting.

Re: GPT-4o mini: advancing cost-efficient intelligence

#26

Post by thatsnotmepls » Thu, Jul 18, 2024, 6:31 PM UTC

OutOfHere wrote: post #23

Sohcahtoa82 wrote: post #14

Earlier quoted context omitted.

Take a loss on every sale and make up for it with volume!

> Take a loss on every sale and make up for it with volume! If you take a loss on every sale, it is impossible to make up for it with volume. The result will be a loss magnified by the volume.

Guess you missed the sarcasm.

Re: GPT-4o mini: advancing cost-efficient intelligence

#27

Post by laborcontract » Thu, Jul 18, 2024, 6:32 PM UTC

razodactyl wrote: post #12

Here's something interesting to think about: In ML we do a lot of bootstrapping. If a model is 51% wrong on a binary problem you flip the answer and train a 51% correct model then work your way up from there. Small models are trained from synthetic and live data curated and generated by the more advanced models. If I end up with an advanced smaller model capable of running alongside a greater number of similar models…

Sounds like you're describing mixture of experts, the architecture being used in openai's gpt-4 and mistral's mixtral series of models.

Re: GPT-4o mini: advancing cost-efficient intelligence

#28

Post by Sohcahtoa82 » Thu, Jul 18, 2024, 6:32 PM UTC

OutOfHere wrote: post #23

Sohcahtoa82 wrote: post #14

Earlier quoted context omitted.

Take a loss on every sale and make up for it with volume!

> Take a loss on every sale and make up for it with volume! If you take a loss on every sale, it is impossible to make up for it with volume. The result will be a loss magnified by the volume.

It's a joke. Sadly, the origin is unknown, but it's a joke that's well over 10 years old.

Re: GPT-4o mini: advancing cost-efficient intelligence

#29

Post by bryanh » Thu, Jul 18, 2024, 6:32 PM UTC

minimaxir wrote: post #10

mucle6 wrote: post #6

It looks like the vision costs the same for GPT-4o vs mini. Both start with 150x150px and if you click the (i) it says mini uses way more base tokens and way more tile tokens, it still costs the same...

Good catch: the calculators here are bizarre . For GPT-4o, a 512x512 image uses 170 tile tokens. For GPT-4o mini, a 512x512 image uses 5,667 tile tokens. How does that even work in the context of a ViT? The patches and its image encoder should be the same size/output. Since the base token counts increase proportionally (which makes even less sense) I have a hunch there's a JavaScript bug instead.

Confirmed that mini uses ~30x more tokens than base gpt-4o using same image/same prompt: { completionTokens: 46, promptTokens: 14207, totalTokens: 14253 } vs. { completionTokens: 82, promptTokens: 465, totalTokens: 547 }.

Re: GPT-4o mini: advancing cost-efficient intelligence

#30

Post by minimaxir » Thu, Jul 18, 2024, 6:37 PM UTC

bryanh wrote: post #29

minimaxir wrote: post #10

Earlier quoted context omitted.

Good catch: the calculators here are bizarre . For GPT-4o, a 512x512 image uses 170 tile tokens. For GPT-4o mini, a 512x512 image uses 5,667 tile tokens. How does that even work in the context of a ViT? The patches and its image encoder should be the same size/output. Since the base token counts increase proportionally (which makes even less sense) I have a hunch there's a JavaScript bug instead.

Confirmed that mini uses ~30x more tokens than base gpt-4o using same image/same prompt: { completionTokens: 46, promptTokens: 14207, totalTokens: 14253 } vs. { completionTokens: 82, promptTokens: 465, totalTokens: 547 }.

Huh. I am so confused.