71–80 of 88 posts
Earlier quoted context omitted.
That is a thoroughly bankrupt ethos that should be denounced every time it pops up. It is literally condoning criminality.
No crimes in this case, just license breaches. After a few training iterations, it’ll be very muddled anyways.
Earlier quoted context omitted.
Using nix and then complaining about having to set up your compilation environment libs/etc is kind of like sticking a rod in your bike's wheel spokes and complaining about crashing. Don't give up on the idea of system libraries (ie, use nix) and this doesn't happen. Also, hi? I don't recall you by that nick but the internet is a small place sometimes.
oh, I'm very aware that I've brought this upon myself, but I'm sticking out for the greater good (and stubbornness.) specifically, I'm trying to benchmark a bunch of different GPU configurations on different workloads on vast.ai, which uses Docker containers. I abhor Dockerfiles and my experience building containers with nix has been pleasant, so that's what I'm doing and why. fortunately I think I'm getting past the…
(The irony of holding this opinion while dealing with pre-trained AI models is not lost)
Earlier quoted context omitted.
I've been testing quite a few of these models lately. For me, the absolute best is still the 65B 4-bit quantized llama model with the correct prompt and parameters, both for programming, language and general questions. I am actually getting about 2 tokens/second with the latest llama.cpp using 16 threads on a 5950x with 64 gb of ram. 16 threads seems to be the sweet spot, any higher and it slows down, any lower and i…
Sounds like you've had some more success w/ w/ raw LLaMA - would def be interested in how you're prompting it. BTW, for those interested (looks like the markdown rendering is a bit messed up) but here are some notes I'm taking for some of the nuts and bolts for the local models I'm running: https://mostlyobvious.org/?link=%2FReference%2FSoftware%2FGe...
But here is the bash script[0] I launch my "go to" AI, it's called Omnius :)
As written in the previous comment, it is a modified version of the examples/chat-13b.sh that is included in the llama.cpp github.
Earlier quoted context omitted.
I've been testing quite a few of these models lately. For me, the absolute best is still the 65B 4-bit quantized llama model with the correct prompt and parameters, both for programming, language and general questions. I am actually getting about 2 tokens/second with the latest llama.cpp using 16 threads on a 5950x with 64 gb of ram. 16 threads seems to be the sweet spot, any higher and it slows down, any lower and i…
How do you talk to llama? It doesn't respond to instructions, so it's a bit complicated to have it extract keywords and/or summarize texts. Can you please share examples of llama prompts?
Edit: I have not tried to use it for extracting keywords / summarizing text. Perhaps I shall experiment a bit with it.
Earlier quoted context omitted.
This is incredible to me (not your comment per se, but what you're referencing). I really don't understand how brittle and fragile Python is with all its dependencies. It's crazy to me that a simple bump from 3.10 to 3.11 can break Pytorch . This is like bumping your Ruby version up one level and suddenly Rails doesn't work. Why on earth is Python like this? It's so frustrating coming from other languages where the d…
I have despised Python ever since the 2=>3 transition for the reasons you say. Tools like pyenv help, but it's still a mess. It makes me sad that all the popular ML tooling ends up built in Python.
Earlier quoted context omitted.
I've been testing quite a few of these models lately. For me, the absolute best is still the 65B 4-bit quantized llama model with the correct prompt and parameters, both for programming, language and general questions. I am actually getting about 2 tokens/second with the latest llama.cpp using 16 threads on a 5950x with 64 gb of ram. 16 threads seems to be the sweet spot, any higher and it slows down, any lower and i…
I really want to try out the 65B model, I hear great things! Sadly none of my computers can handle it, slightly tempted to get more RAM but I’m on a 8 core i7.
My biggest hope in the end is that we will get a library that can utilize the unified memory model of the AMD platform and run these things with a combination of system ram and GPU. I think intel also has something similar in their platforms.
Not sure how well llama.cpp runs with 8 cores and so big weights though, I am really pushing how usable it is due to speed with my 5950x already.
Perhaps we'll even get dedicated AI boards in the end, much like GPUs today.
Earlier quoted context omitted.
How do you talk to llama? It doesn't respond to instructions, so it's a bit complicated to have it extract keywords and/or summarize texts. Can you please share examples of llama prompts?
See my sibling comment for a bash script to launch it :) Edit: I have not tried to use it for extracting keywords / summarizing text. Perhaps I shall experiment a bit with it.
No llama.cpp nor any compilation complexity. Run with two Python commands!
Did they release the merged weights, yet? I'd love to try this model. Afaict from the docs, you still need to request the original Llama weights from Meta (or get ahold of them another way), then apply the diff-weights requiring 60GB RAM?
Seems like llama derived model are flourishing. However with llama is licensed as academic only and noncommercial model, what is the path for bringing this to production of for profit purpose? I certainly interested doing so.
The methodology for alpaca has proven powerful and it's being applied to model with better licensing. It's hard to track lineage, but I think openassistant models are the most permissive at the moment, they use a openly sourced set of data to build an instruct model on top of phiia, which itself is a gptneox trained on a duplicated version of the famous the pile dataset. The problem is verifying the licensing claims…
Earlier quoted context omitted.
That is a thoroughly bankrupt ethos that should be denounced every time it pops up. It is literally condoning criminality.
No crimes in this case, just license breaches. After a few training iterations, it’ll be very muddled anyways.