21–30 of 78 posts
>By embracing batch processing and leveraging the power of cost-effective open-source models, you can sidestep the price floor and continue to scale your AI initiatives in ways that are no longer feasible with traditional APIs. Context size is the real killer when you look at running open source alternatives on your own hardware. Has anything even come close to the 100k+ range yet?
> If you’re building batch tasks with LLMs and are looking to navigate this new cost landscape, feel free to reach out to see how Sutro can help. I don't have any reason to doubt the reasoning this article is doing or the conclusions it reaches, but it's important to recognize that this article is part of a sales pitch.
Stopped reading here, if you're positioning yourself as if you have some kind of unique insight when there is none in order to boost youe credentials and sell your product there's little chance you have anything actually insightful to offer. Might sound like an overreaction/nitpicking but it's entirely needless LinkedIn style "thought leader" nonsense.
In reality it was immediately noticed by anyone using these models, have a look at the HN threads at the time, or even on Reddit, let alone the actual spaces dedicated to AI builders.
Is there math backing up the “quadratic” statement with LLM input size? At least in the traffic analogy, I imagine it’s exponential, but for small amounts exceeding some critical threshold, a quadratic term is sufficient
I'm not sure where you're getting an exponential from.
If these corporations had to build a car they would make the largest possible engine, because "MORE ENGINE MORE SPEED", just like they think that bigger models means bigger intelligence, but forget to add steering, or even a chassi.
Earlier quoted context omitted.
Aren't all LLMs loosing money at this point?
I don't believe that's true on inference - I think most if not all of the major providers are selling inference at a (likely very small) margin over what it costs to serve them (hardware + energy). They likely lose money when you take into account the capital cost of training the model itself, but that cost is at least fixed: once it's trained you can serve traffic from it for as long as you chose to keep the model r…
What is holding back AI is this business necessity that models must perform everything. Nobody can push for a smaller model that learns a few simple tasks and then build upon that, similar to the best known intelligent machine: the human. If these corporations had to build a car they would make the largest possible engine, because "MORE ENGINE MORE SPEED", just like they think that bigger models means bigger intellig…
I'll take a model specialized in web scraping. Give me one trained on generating report and documentation templates (I'd commit felonies for one which could spit out a near-conplete report for SSRS).
Models trained for specific helpdesk tasks ("install a printer", "grant this user access to these services with this permission level").
A model for analyzing network traffic and identifying specific patterns.
None of these things should require titanic models nearing trillions of parameters.
What is holding back AI is this business necessity that models must perform everything. Nobody can push for a smaller model that learns a few simple tasks and then build upon that, similar to the best known intelligent machine: the human. If these corporations had to build a car they would make the largest possible engine, because "MORE ENGINE MORE SPEED", just like they think that bigger models means bigger intellig…
What is holding back AI is this business necessity that models must perform everything. Nobody can push for a smaller model that learns a few simple tasks and then build upon that, similar to the best known intelligent machine: the human. If these corporations had to build a car they would make the largest possible engine, because "MORE ENGINE MORE SPEED", just like they think that bigger models means bigger intellig…
I suspect a large part of the reason we've had many decades of exponential improvements in compute is the general purpose nature of computers. It's a narrow set of technologies that are universally applicable and each time they get better/cheaper they find more demand, so we've put an exponentially increasing amount of economical force behind it to match. There needed to be "plenty of room at the bottom" in terms of physics and plenty of room at the top in terms of software eating the world, but if we'd built special purpose hardware for each application I don't think we'd have seen such incredible sustained growth.
I see neural networks and even LLMs as being potentially similar. They're general purpose, a small set of technologies that are broadly applicable and, as long as we can keep making them better/faster/cheaper, they will find more demand, and so benefit from concentrated economic investment.