This only decreases memory cost of input context window, not the memory cost to load and run the models.
[dead]
New LLM optimization technique slashes memory costs
21–30 of 227 posts
Re: New LLM optimization technique slashes memory costs
#22Does this mean us plebs can run LLMs on gimped VRAM Nvidia lower end cards?
Re: New LLM optimization technique slashes memory costs
#23Re: New LLM optimization technique slashes memory costs
#24Does this mean us plebs can run LLMs on gimped VRAM Nvidia lower end cards?
I don't think so. It seems to just lower the ram needed for the context window. Not for loading the model on the vram.
Re: New LLM optimization technique slashes memory costs
#25Earlier quoted context omitted.
True. Microsoft's all in, Apple's all in, Nvidia is selling shovels, insurance companies are all in, police & military are all in, education is all in, office management is all in. Who is left to pump line up?
As far as I know, finance is not all in. I see Goldman Sachs doing experiments, for example, but it doesn't feel like they're convinced yet.
Finance is basically all of the reasons not to use (generative, LLM based) AI , all in one vertical. The poster child of determinism.
Re: New LLM optimization technique slashes memory costs
#26Earlier quoted context omitted.
doesn't training require inference? so i guess it would help there too?
Training doesn't require inference. It uses back-propagation, a different algorithm.
Backpropagation happens after some number of inferences. You need to infer to calculate a loss function to then backprop from.
Re: New LLM optimization technique slashes memory costs
#27[flagged]
[deleted]
Re: New LLM optimization technique slashes memory costs
#28[flagged]
True. Microsoft's all in, Apple's all in, Nvidia is selling shovels, insurance companies are all in, police & military are all in, education is all in, office management is all in. Who is left to pump line up?
no one is successfully using LLMs for anything other than customer service related things and text generation(coding, writing)
Re: New LLM optimization technique slashes memory costs
#29[flagged]
Google Trends make it seem like we're out of the exponential growth phase for LLMs-- search interest is possibly plateauing. A decline in search interest outside of academia makes sense. The groups who can get by on APIs don't care so much how the sausage is made and just want to see prices come down. Interested parties have likely already found tools that work for them. There's definitely some academic interest outs…
Or maybe it’s Google Search usage that’s plateauing, as LLM interest is answered elsewhere?
I am only half kidding.
Re: New LLM optimization technique slashes memory costs
#30[flagged]