101–110 of 214 posts
Earlier quoted context omitted.
Low precision: agreed Attention: Isn't it quadratic in context length? I dunno, this feels like the crude first iteration of something that will get inevitably passed by something that scales better.
In theory one could use the nysromformer. Don't know if anyone does in practice
I expect we will see the biggest jump in performance if (when) consumer-grade coprocessors like mobile GPUs start incorporating attention layers as a primitive building block at the hardware level, e.g., with instructions and memory layouts engineered specifically to make ultra-low-precision (say, 4-bit) transformer layers as compute- and memory-efficient as possible on consumer devices. That seems almost inevitable…
Low precision: agreed Attention: Isn't it quadratic in context length? I dunno, this feels like the crude first iteration of something that will get inevitably passed by something that scales better.
Alternative models like S4 have been able to get transformer level performance with O(N) sequence length scaling.
Earlier quoted context omitted.
If they are amazing, then ChatGPT must be God-like in your view. I've been underwhelmed by Alpaca and Alpaca-LORA and LLaMA all at 13B but I have not tried higher params.
In my opinion the problem with these is engineering a good prompt. I read of lots of people only getting nonsense or repetitions, and learned a bit from what they shared. These models are not chat bots. Vicuna is more friendly in that regard. But I’m well aware of their limitations also, and I can see how one can be underwhelmed. They are not jacks of all trades
Earlier quoted context omitted.
It’s in the near past. https://github.com/ggerganov/llama.cpp
Also worth checking out https://github.com/saharNooby/rwkv.cpp which is based on Georgi's library and offers support for the RWKV family of models which are Apache-2.0 licensed.
I’m also in the middle of making it user friendly to run these models on all platforms (built with Flutter). First MacOS release will be out before this weekend: https://github.com/BrutalCoding/shady.ai
Earlier quoted context omitted.
From my experience with llama.cpp and oobaboogas webui I can say they are amazing, at least on my gaming pc. I’m absolutely astonished at the speed and quality of llama, alpaca, galactica and vicuna (the >10B parameters ones). Make no mistake, it’s for tinkerers that do not expect each prompt to be answered human like. I see them as creativity and thought testing tools, also knowledge exploratory.
is there a difference in the quality of llm one would be able to train or run on a gpu with 8, 12, 16, all the way up to 24gb? I'm trying to decide whether it's worth while to splurge on a more expensive 4090 vs a 4070 or whatever.
In order to run large language models, we should all be buying a fully loaded Mac Studio (128GB of ram, 20 CPU cores, a lot of GPU and Neural cores.) and putting Linux on it to remove the artificial restrictions. Yes, we will be running them soon in low end hardware, but we need to get at least to GPT-3.5-turbo level of inference speed and quality before we try to make it small. I already started. $ neofetch -` x@dec…
Which LLM can run on apple's neural cores / GPU cores? I can only run on plain ol' CPU cores (llama), and it runs fine on my Ryzen CPU for less than half the price of that system. That being said I'm switching from Ubuntu to Arch cause I'm sick of all my packages being way out of date!
https://github.com/geohot/tinygrad/tree/master/accel/ane
But I have not tested it on Linux since Asahi has not yet added support.
Same machine but OSX, llama.cpp runs at 18ms per token (7B) and 200ms per token (65B) on CPU using float16.
While many NLP related Apple ML job listings have been added since this article was written, there were several recent listings at the time of its writing. While I feel that Apple does not focus well on intangible technologies, products that can't be readily carried, worn and given their boutique product development fetish focus, I have some hope that they can overcome this bias somewhat, and see how behind they are.
I disagree with the assessment that Apple is behind. Apple is known for executing well and putting their weight behind the things they launch. Like Jack Dorsey would often say "it's not important to be first to market, you can just be best to market". And the world got CashApp. I'm sure however Apple enters the space, it will be fleshed out (vs Bard).
They are absolutely behind in the space and anyone who works in the industry will tell you that. Only feasible way IMO would have to be a very big budget acquisition of one of the major LLM startups, but most of those already have big tech backers.
I don't understand why people are so excited to build this big thing on top of Llama, which is closed source, severely license restricted and we now know for a fact that Meta is going after users with the legal hammer. I'm sure if we'd pool resources together we could build a truly open alternative worthy of building on top of.
This would require like ConstitutionDAO level of resource pooling without direct monetary payoff.
I mean, good luck.
I expect we will see the biggest jump in performance if (when) consumer-grade coprocessors like mobile GPUs start incorporating attention layers as a primitive building block at the hardware level, e.g., with instructions and memory layouts engineered specifically to make ultra-low-precision (say, 4-bit) transformer layers as compute- and memory-efficient as possible on consumer devices. That seems almost inevitable…
There are those Google accelerators that plug into an M.2 slot. You could plausibly do this today, although I am not sure what sort of memory constraints those accelerators have.
It’s used for boosting interference (offline) on Linux, Mac and Windows.
Haven’t bought or used them but I’ve had my eyes on these for a little while!