51–60 of 64 posts
Earlier quoted context omitted.
you can't really compare the utility of an LLM against libreoffice or similar. There is no comparison. Libreoffice is something that you would definitely use unlike an LLM.
I use LLMs to double check/write python pandas code so I don't need to use Libreoffice/spreadsheet software.
Earlier quoted context omitted.
I have. On a Mac Studio with NixOS based Asahi Linux and 128Gb of RAM, mixtral 8x7b uses 49GB of RAM. At the same time I load airflow tasks that deal with world wide datasets (using ~60GB on 16 parallel streams with the performance cores) format is parquet and also mmaped. Computer still has 8 efficiency cores and the whole GPU for visualizing the maps using lonboard / browsing / etc. The computer uses 8-10W when idl…
Interesting, thanks for sharing that! For curiosity, what kind of performance you get with that setup + mixtral 8x7b in terms of tokens/second?
./mixtral-8x7b-instruct-v0.1.Q8_0.llamafile --cli -t 16 -n 200 -p "In terms of Lasso"
I got 15 tokens per second for prompt evaluation and 8 tokens per second for regular eval.
The same hardware can run things much faster on OSX, or if you use more quantization but I prefer to run things at Q8 or f16 even if they are slow. In the future I how to use GPU, ANE and the crazy 1.58 or 0.68 bit quantization but for now this does the trick handsomely.
Earlier quoted context omitted.
>scans my files and can summarize folders for me Can you tell me what exactly you want it to do? You have a bunch of files in folders and you want the AI to tell you what exactly?
"You have a bunch of files in folders and you want the AI to tell you what exactly?" For example: "show me the folder(s), where my old University projects are stored" "Sort this folder into programming projects, general notes, pictures, music, videos and install files" "Find me the folder where I made notes about a novel sorting algorithm"
Like, take this query for example: show me the folder(s), where my old University projects are stored. How would an AI, however powerful, know what are "university projects" if they aren't tagged as such? And if they were, why is the AI necessary?
One approach I've tried before is: if you have a folder /projects/ with so many project folders in it that you don't even know anymore what is what anymore, you just create a text file called /projects/index.txt and write the name of each folder in there and what it's for, so you don't forget later.
NixOS has packages for llama-cpp and ollama: https://search.nixos.org/packages?channel=23.11&from=0&size=... Edit: And Arch packages ollama officially - https://archlinux.org/packages/?sort=&q=llama&maintainer=&fl... - and a few things in the AUR - https://aur.archlinux.org/packages?O=0&K=llama
One thing about LLMs is that they are 6GB+ (and much larger for "smart" ones) just sitting in the background. They suck power and produce heat like nothing else, and they are finicky, especially at smaller sizes. Running one as a background desktop assistant is whole different animal than calling a Microsoft API.
At least with a GPU that can do power save that's not the case. I have a box with some 3090's in it, each card will idle <50W when it's not doing inference with the weights loaded into VRAM. Only when I ask it to do inference it will spin up and start consuming 300-400W.
Earlier quoted context omitted.
You don't need the larger/smarter models for most "assistant" use cases. Small language models like Phi-2 can be enough.
I'd love this to be true, and it might be for some specific well tested situations with a narrow set of data that you can be confident about. But that's a bit wishful, isn't it?
NixOS has packages for llama-cpp and ollama: https://search.nixos.org/packages?channel=23.11&from=0&size=... Edit: And Arch packages ollama officially - https://archlinux.org/packages/?sort=&q=llama&maintainer=&fl... - and a few things in the AUR - https://aur.archlinux.org/packages?O=0&K=llama
NixOS just got tabbyml[1] which is built on llama-cpp. Working on systemsd services the weekend and updating latest tabbyml release which supports rocm in addition to cuda [1] https://github.com/TabbyML/tabby [2] https://github.com/NixOS/nixpkgs/pull/291744