GPT-4o mini: advancing cost-efficient intelligence • Hacker News Board

Re: GPT-4o mini: advancing cost-efficient intelligence

#61

Post by tedsanders » Thu, Jul 18, 2024, 9:38 PM UTC

lsaferite wrote: post #59

pants2 wrote: post #44

Earlier quoted context omitted.

I don't expect organizations to need to generate 1T output tokens, but 1T input tokens is common. Consider developers at a large company running queries with their entire codebase as context. Or lawyers plugging in the entire tax code to ask questions about. Each of them running dozens of queries per day on multi-millions of context input, it's going to add up quick.

Wouldn't a lawyer wanting to run queries against the entire tax code have a model that was fine-tuned on all of that data though? I mean, vs. doing RAG by sending the entire tax code on each request.

Unclear, but fine-tuning has many problems not faced by RAG:

- More prone to hallucinations

- Worse at citing sources for people to double check outputs

- Can't be updated without retraining

- Can't impose knowledge access controls for different users

Re: GPT-4o mini: advancing cost-efficient intelligence

#62

Post by ff7250 » Thu, Jul 18, 2024, 9:52 PM UTC

minimaxir wrote: post #3

GPT-4o mini is $0.15/1M input tokens, $0.60/1M output tokens. In comparison, Claude Haiku is $0.25/1M input tokens, $1.25/1M output tokens. There's no way this price-race-to-the-bottom is sustainable.

what if they can make money? then the problem is on claude/gemini...

Re: GPT-4o mini: advancing cost-efficient intelligence

#63

Post by bronco21016 » Thu, Jul 18, 2024, 10:16 PM UTC

wrs wrote: post #20

The big news for me here is the 16k output token limit. The models keep increasing the input limit to outrageous amounts, but output has been stuck at 4k. I did a project to summarize complex PDF invoices (not “unstructured” data, but “idiosyncratically structured” data, as each vendor has a completely different format). GPT-4o did an amazing job at the extraction of line items, but I had to do a heuristic layer on t…

Have you written about this anywhere? Would love to know more about the process you're using!

Re: GPT-4o mini: advancing cost-efficient intelligence

#64

Post by jtonz » Thu, Jul 18, 2024, 10:47 PM UTC

razodactyl wrote: post #12

Here's something interesting to think about: In ML we do a lot of bootstrapping. If a model is 51% wrong on a binary problem you flip the answer and train a 51% correct model then work your way up from there. Small models are trained from synthetic and live data curated and generated by the more advanced models. If I end up with an advanced smaller model capable of running alongside a greater number of similar models…

I have posited a similar idea with some of the people I work with. The issue of having complex, multi-step tasks be completed successfully has already been solved. You don't heavily invest in having one single expert for your business to solve all your problems. You build a team. Multiple specialized experts working in unison to achieve a shared outcome. Some people work on the task simultaneously, others sequentially. All with a specific purpose associated with the goal.

These assets are horizontally and vertically scalable based off skills, quality, or performance required. An efficiently designed AI architecture I believe could do the same. Its not mixture-of-experts as you aren't necessarily asking each model simultaneously but designing and/or having the system intelligently decide when it has completed its task and where the output should travel next.

Think of a platform where you had 'visual design' models, 'coding' models, 'requirements' models, 'testing' models, all wired together. The coding models you incorporate are trained specifically for the languages you use, testing the same. All interchangeable / modularized as your business evolves.

You feed in your required outcome at the front of your 'team' and it funnels through each 'member' before being spit out the other end.

I have yet to see anyone openly discussing this architecture pattern so if anyone could point me in that direction I would thoroughly appreciate it.

Re: GPT-4o mini: advancing cost-efficient intelligence

#65

Post by kristianp » Thu, Jul 18, 2024, 11:10 PM UTC

@dang: This post isn't on the 1st or 2nd page of hacker news. Did it trip some automated controversy detection code for too many comments in the first hour?

Edit: it says 181 points, 6 hours ago, and eyeballing the 1st page it should be in the top 5 right now.

Re: GPT-4o mini: advancing cost-efficient intelligence

#66

Post by BaculumMeumEst » Thu, Jul 18, 2024, 11:35 PM UTC

One of the great things about open source small models such as llama3 is that you can fine-tune them with your own data and run them on your own hardware. I am so excited to see these models continue to improve and am uninterested in this new model from "Open"AI, which is presumably increasingly feeling the heat of competition from all sides.

Re: GPT-4o mini: advancing cost-efficient intelligence

#67

Post by saiansh2525 » Fri, Jul 19, 2024, 12:53 AM UTC

yawnxyz wrote: post #4

minimaxir wrote: post #3

GPT-4o mini is $0.15/1M input tokens, $0.60/1M output tokens. In comparison, Claude Haiku is $0.25/1M input tokens, $1.25/1M output tokens. There's no way this price-race-to-the-bottom is sustainable.

I think it's heavily quantized, so it doesn't cost them (too much). But I think it's still at cost...

Judging from the perplexity scores, the model doesn't seem to be quantized, it seems to simply be a scaled down version of the original GPT-4O or something similar.

Re: GPT-4o mini: advancing cost-efficient intelligence

#68

Post by dragonwriter » Fri, Jul 19, 2024, 12:54 AM UTC

Sohcahtoa82 wrote: post #14

minimaxir wrote: post #3

GPT-4o mini is $0.15/1M input tokens, $0.60/1M output tokens. In comparison, Claude Haiku is $0.25/1M input tokens, $1.25/1M output tokens. There's no way this price-race-to-the-bottom is sustainable.

Take a loss on every sale and make up for it with volume!

Take a loss on every sale to drive less-well-funded competitors out of the market, and then reap monopoly rents.

Re: GPT-4o mini: advancing cost-efficient intelligence

#69

Post by OutOfHere » Fri, Jul 19, 2024, 2:03 AM UTC

thatsnotmepls wrote: post #26

OutOfHere wrote: post #23

Earlier quoted context omitted.

> Take a loss on every sale and make up for it with volume! If you take a loss on every sale, it is impossible to make up for it with volume. The result will be a loss magnified by the volume.

Guess you missed the sarcasm.

Sarcasm is generally expected to be suffixed with /s. In this case, significant historical context is required to detect it.

Re: GPT-4o mini: advancing cost-efficient intelligence

#70

Post by maeil » Fri, Jul 19, 2024, 4:50 AM UTC

So far ever since the initial release of gpt 3.5 turbo every ""upgrade"" has mostly been an actual downgrade. I have a battery of tasks that the initial 3.5 turbo (Nov 2022) was able to perform but the newer ones very consistently fail at, regardless of prompting.

I've been moving tasks from 3.5-turbo to Llama3-70b for this reason.

Very curious to see whether this time it'll be an actual upgrade instead of a downgrade.