Earlier quoted context omitted.
sama said they improved it
We can't just take his word for it. This needs experimental verification. It's likely not even close to solved.
people now have access and can do any verification they want
New models and developer products
371–380 of 561 posts
Re: New models and developer products
#372For the Assistants API with unlimited context length it’s not clear to me how the pricing works. Do you pay only for the incremental tokens per message for follow up messages, or does each new message cost the full prior context amount + the cost of the message itself?
Re: New models and developer products
#373And here I was in bliss with the 32k context increase 3 days ago. 128k context? Absolutely insane. It feels like now the bottle neck in GPT workflows is no longer GPT, but instead its the wallet! Such an amazing time to be alive.
> 128k context? Absolutely insane 128k context is great and all, but how effective are the middle 100,000 tokens? LLMs are known to struggle with remembering stuff that isn't at the start or end of the input. Known as the Lost Middle https://arxiv.org/abs/2307.03172
In fairness, humans have much the same problem.
Re: New models and developer products
#374Earlier quoted context omitted.
While I agree with you, as a happy GPT4 plus customer, I'm worried about the inevitable enshittification downhill roll that will eventually ensue. Once marketing gets in charge of product, it's doomed. And I can't think of a product startup that it hasn't happened to. Particularly with this type of growth, at some point, the suits start to out number the techies 10:1. This is why openeness and healthy competition is…
It's not marketing, it's economics. If you set money on fire -- eventually there's a time when you need to stop doing that.
I think the parent is more talking about the other common situation where organizations start focusing on maximizing profits, rather than just working towards a basic profitability. Eg, Google Maps API pricing comes to mind.
Yes, OpenAI might be (we don't know how much) burning through their $5B capital/Azure credits now, but I think the `turbo` models are starting to addressing this as well. And $20/month from a large user base can also add up pretty quick.
Re: New models and developer products
#375Earlier quoted context omitted.
Everybody's got their own calculus about how competitive their space is and what this tech can do for them, but some might be best off dancing around lock-in by being careful about what they use from OpenAI and how tightly they integrate with it. This is very early in the maturity cycle for this tech. The options that will be available for private inference and fine tuning, for cloud-gpu/timeshare inference and fine…
Do you build on AWS AI services then? Or any other cloud provider? The outcome is the same, right? Technical lock in, cost risks, integration maintenance, etc.
The key line in my comment, for emphasis:
> This is very early in the maturity cycle for this tech.
Think about what value you get out of the services and what migration might look like. If you are making simple completion or chat calls with a clever prompt, then migration will probably be trivial when the time comes. Those features are the commodity that everyone will be offering and you'll be able to shop around for the ideal solution as alternatives become competitive.
Alternately, if you're handing OpenAI a ton of data for them to opaquely digest for fine tuning with no egress tools, or having them accumulate lots of other critical data with no egress, you're obviously getting yourself locked in.
The more features you use, the more idiosyncratic those features are, the more non-transferable those features are, and the more deeply you integrate those features, the more risk you're taking. So you want to consider whether the reward is worth that risk.
Different projects will legitimately have different answers for that.
Re: New models and developer products
#376Earlier quoted context omitted.
It's a good strategy. For me, avoiding the moat means either a big drop in quality and just ending up in somebody elses moat, or a big drop in quality and a lot more money spent. I've looked into it and maybe the most practical end-to-end system for owning my own LLM is to run a couple of 3090s on a consumer motherboard at substantial running cost to keep them up 24/7 and that's not powerful enough to cut it and rath…
Everybody's got their own calculus about how competitive their space is and what this tech can do for them, but some might be best off dancing around lock-in by being careful about what they use from OpenAI and how tightly they integrate with it. This is very early in the maturity cycle for this tech. The options that will be available for private inference and fine tuning, for cloud-gpu/timeshare inference and fine…
While everything you say mighy be true, it's also irrelevant.
Time to market is more important, you build users you get an edge, you can swap models later on (as long as you own the data).
Re: New models and developer products
#377Every day this video ages more and more poorly [1]. categories of startups that will be affected by these launches: - vectorDB startups -> don't need embeddings anymore - file processing startups -> don't need to process files anymore - fine tuning startups -> can fine tune directly from the platform now, with GPT4 fine tuning coming - cost reduction startups -> they literally lowered prices and increased rate limits…
[deleted]
Re: New models and developer products
#378If I had no contact with society from the 29th of November 2022 (the day before ChatGPT was released according to Wikipedia) and came back today to see the OpenAI keynote I would have lost my mind.
The progress and usefulness of these products is absolutely incredible.
Re: New models and developer products
#379Earlier quoted context omitted.
depends on how much developers are willing to embrace the risk of building everything on OpenAI and getting locked onto their platform. What's stopping OpenAI from cranking up the inference pricing once they choke out the competition? That combined with the expanded context length makes it seem like they are trying to lead developers towards just throwing everything into context without much thought, which could be p…
> depends on how much developers are willing to […] getting locked onto their platform. I mean.. the lock in risks have been known with every new technology since forever now, and not just the risk but the actual costs are very real. People still buy HP printers with InkDRM and companies willingly write petabytes of data into AWS that they can’t even afford to egress at current prices. To be clear, I despise this bus…
Combination of 1) short-term business thinking (save $1 today = $1 more of EPS) and 2) fear of competition building AI products and taking share. thus rush to use first usable platform (e.g. openAI).
Psychology and FOMO plays interesting role in walking directly into a snake pit.
Re: New models and developer products
#380If I had no contact with society from the 29th of November 2022 (the day before ChatGPT was released according to Wikipedia) and came back today to see the OpenAI keynote I would have lost my mind. The progress and usefulness of these products is absolutely incredible.
I was in prison when ChatGPT came out. All I knew of it was a headline that flashed past really fast on CNN and I called my buddy and said "What the hell is Chat OPT?"
I'd just finished reading The Singularity is Near for the second time too...