101–110 of 246 posts
Might be a silly question but is GGML a similar/competing library to George Hotz's tinygrad [0]? [0] https://github.com/geohot/tinygrad
Earlier quoted context omitted.
> The problem is, this financial backing and support is via VCs, who will steer the project to close it all up again. How exactly could they meaningfully do that? Genuine question. The issue with the OpenAI business model is that the collaboration within academia and open source circles is creating innovations that are on track to out-pace the closed source approach. Does OpenAI have the pockets to buy the open sourc…
I agree with the spirit but saying that open source is on track to outpace OpenAI in innovation is just not true. Open source models are being compared to GPT3.5, none yet even get close to GPT4 quality and they finished that last year.
Earlier quoted context omitted.
>> run LLaMA 30B at almost 3 tokens/s Please tell me your config! I have an i9-10900 with 32GB of ram that only gets .7 tokens/s on a 30B model
> Please tell me your config! I have an i9-10900 with 32GB of ram that only gets .7 tokens/s on a 30B model Have you quantized it?
I'm running in Windows using koboldcpp, maybe it's faster in Linux?
I tested it out both locally (6c/12t CPU) and on a Hetzner CPX41 instance (8 AMD cores, 16 GB of RAM, no GPU), the latter of which costs about 25 EUR per month and still can generate decent responses in less than half a minute, my local machine needing approx. double that time. While not quite as good as one might expect (decent response times mean maxing out CPU for the single request, if you don't have a compatible GPU with enough VRAM), the technology is definitely at a point where it's possible for it to make people's lives easier in select use cases with some supervision (e.g. customer support).
What an interesting time to be alive, I wonder where we'll be in a decade.
> Nat Friedman and Daniel Gross provided the pre-seed funding. Why? Why should VCs get involved again? They are just going to look for an exit and end up getting acquired by Apple Inc. Not again.
+1. VC involvement in projects like these always pivot the team away from the core competency of what you'd expect them to deliver - into some commercialization aspect that convert only a tiny fraction of the community yet take up 60%+ of the core developer team's time. I don't know why project founders head this way...as the track records of leaders who do this end up disappointing the involved community at some poi…
git clone https://github.com/jart/cosmopolitan
cd cosmopolitan
# cross-compile on x86-64-linux for x86-64 linux+windows+macos+freebsd+openbsd+netbsd
make -j8 o//third_party/ggml/llama.com
o//third_party/ggml/llama.com --help
# cross-compile on x86-64-linux for aarch64-linux
make -j8 m=aarch64 o/aarch64/third_party/ggml/llama.com
# note: creates .elf file that runs on RasPi, etc.
# compile loader shim to run on arm64 macos
cc -o ape ape/ape-m1.c # use xcode
./ape ./llama.com --help # use elf aarch64 binary above
Earlier quoted context omitted.
I agree with the spirit but saying that open source is on track to outpace OpenAI in innovation is just not true. Open source models are being compared to GPT3.5, none yet even get close to GPT4 quality and they finished that last year.
We're basically surviving off the scraps companies like Facebook have been tossing off the table, like LLaMA. The fact that we're even allowed and able to use these things ourselves, at all, is a tremendous victory.
ggml and llama.cpp are such a good platform for local LLMs, having some financial backing to support development is brilliant. We should be concentrating as much as possible to do local inference (and training) based on privet data. I want a local ChatGPT fine tuned on my personal data running on my own device, not in the cloud. Ideally open source too, llama.cpp is looking like the best bet to achieve that!
Can LLaMA be used for commerical purposes though (might limit external contributors)? I believe, FOSS alternatives like DataBricks Dolly / Together RedPajama / Eluether GPT NeoX (et al) is where the most progress is likely to be at.
I believe ggml is the basis of llama.cpp (the OP says it's "used by llama.cpp")? I don't know much about either, but when I read the llama.cpp code to see how it was created so quickly, I got the sense that the original project was ggml, given the amount of pasted code I saw. It seemed like quite an impressive library.
I happen to have RPi 4B with HomeAssistant. Is this something I could set up on it and integrate with HA to control it with speech, or is it overkill?
I doubt it. I'm running 4-bit 30B and 65B models with 64GB ram, a 4080 and a 7900x. The 7B models are less demanding, but even so, You'll need more than an rpi. Even then, it would be a project to get these to control something. This is more 'first baby steps' toward the edge.
Would love voice assistant running locally but probably there are solutions out there - didn't get to do the research yet