This only decreases memory cost of input context window, not the memory cost to load and run the models.
"Only"
New LLM optimization technique slashes memory costs
11–20 of 227 posts
Re: New LLM optimization technique slashes memory costs
#12[flagged]
Google Trends make it seem like we're out of the exponential growth phase for LLMs-- search interest is possibly plateauing.
A decline in search interest outside of academia makes sense. The groups who can get by on APIs don't care so much how the sausage is made and just want to see prices come down. Interested parties have likely already found tools that work for them.
There's definitely some academic interest outside of CS in producing tools using LLMs. I know plenty of astro folks working to build domain specific tools with open models as their backbone. They're typically not interested in more operational work, I guess because they operate under the assumption that relevant optimizations will eventually make their way into public inference engines.
And CS interest in these models will probably sustain for at least 5-10 more years, even if performance plateaus, as work continues into how LLMs function.
All that to say, maybe we're just seeing the trend die for laypeople?
Re: New LLM optimization technique slashes memory costs
#13[flagged]
True. Microsoft's all in, Apple's all in, Nvidia is selling shovels, insurance companies are all in, police & military are all in, education is all in, office management is all in. Who is left to pump line up?
Re: New LLM optimization technique slashes memory costs
#14This only decreases memory cost of input context window, not the memory cost to load and run the models.
I agree with you though, the title is misleading.
Re: New LLM optimization technique slashes memory costs
#15This only decreases memory cost of input context window, not the memory cost to load and run the models.
And that’s what matters the most! To me, at small model sizes (1-8B), anyway. A few thousans tokens already bog my RAM down quite a lot and I’d love to have more - I’d go as far as saying that context greatly determines LLM capability at this point.
Re: New LLM optimization technique slashes memory costs
#16[flagged]
True. Microsoft's all in, Apple's all in, Nvidia is selling shovels, insurance companies are all in, police & military are all in, education is all in, office management is all in. Who is left to pump line up?
As far as I know, finance is not all in. I see Goldman Sachs doing experiments, for example, but it doesn't feel like they're convinced yet.
Re: New LLM optimization technique slashes memory costs
#17So it's like a garbage collector for prompts?
Re: New LLM optimization technique slashes memory costs
#18[flagged]
Google Trends make it seem like we're out of the exponential growth phase for LLMs-- search interest is possibly plateauing. A decline in search interest outside of academia makes sense. The groups who can get by on APIs don't care so much how the sausage is made and just want to see prices come down. Interested parties have likely already found tools that work for them. There's definitely some academic interest outs…
well gary marcus a non lay person is helping spread word that ai winter is again upon us.
but maybe statistical learning from pretraining is near its limit. not enough data or not enough juice to squeeze more performance out of averages.
though with all the narrow ais it does seem plausible you might be able to cram all what these narrow ais can do in on big goliath model. wonder if reinforcement learning and reasoning can manage to keep the exponential curve of ai going if there are still hiccups in the short term.
the difficulty in just shoehorning llms as they are in any and every day task without a hitch might be behind the temporary hype-dying down trend.
Re: New LLM optimization technique slashes memory costs
#19This is for inference right? Not training?
doesn't training require inference? so i guess it would help there too?
Re: New LLM optimization technique slashes memory costs
#20[flagged]
For the general public, not the HN audience.