371–380 of 436 posts
Earlier quoted context omitted.
Is llama open source? I heard it was pirated from Facebook
I did not claim llama was open source, but I see the url I posted insinuates that (probably for a contorted meaning of open source, as in source available for approved academics). Anyway, T5 being available for download from Huggingface only makes my question more pertinent...
Earlier quoted context omitted.
OpenAI specifically cited scaling costs as a reason for why they switched their org structure from non-profit to "capped profit"[0]. You could potentially crowdfund this, though I should point out that this was already tried and Kickstarter shut it down. The effort in question, "Unstable Diffusion", was kinda sketchy, promising a model specifically tuned for NSFW work. What you'd want is an organization that's respon…
Maybe a lot of people/companies also don't want to give their data and knowledge to OpenAI, so that they can sell it off to the competition.
This isn't really helpful for people who want open AI though, because if your strategy is to deny OpenAI data and knowledge then you aren't going to release any models either.
Has anyone done any comprehensive analysis on exactly how much quantization affects the quality of model output? I haven't seen any more than people running it and being impressed (or not) by a few sample outputs. I would be very curious about some contrastive benchmarks between a quantized and non-quantized version of the same model.
I've done some experiments here with Llama 13B, in my subjective experience the original fp16 model is significantly better (particularly on coding tasks). There are a bunch of synthetic benchmarks such a wikitext2 PPL and all the whiz bang quantization schemes seem to score well but subjectively something is missing. I've been able to compare 4 bit GPTQ, naive int8, LLM.int8, fp16, and fp32. LLM.int8 does impressive…
> Llama.cpp 30B
> LLaMA-65B
the "number B" stands for "number of billions" of parameters... trained on?
like you take 65 billion words (from paragraphs / sentences from like, Wikipedia pages or whatever) and "train" the LLM. is that the metric?
why aren't "more parameters" (higher B) always better? aka return better results
how many "B" parameters is ChatGPT on GPT3.5 vs GPT4?
GPT3: 175b
GPT3.5: ?
GPT4: ?
https://blog.accubits.com/gpt-3-vs-gpt-3-5-whats-new-in-open...
how is Llama with 13B parameters able to compete with GPT3 with 175B parameters? It's 10x+ less? How much RAM goes it take to run "a single node" of GPT3 / GPT3.5 / GPT4?
Earlier quoted context omitted.
llama can run on an m1. T5 still needs a specialized gpu
What is the reason T5 needs a specialized GPU and Llama doesn't? In the end they are mathematical models, so what would prevent someone from loading T5 into a machine with plenty of RAM (like a server)? Would the codebase truly require that much refactoring? How difficult would it be to rewrite the model arhitecture as a set of mathematical equations (Einstein summation) and reimplement inference for CPU?
Most people don't have the hardware or budget to access these specialized high vram GPUs.
Earlier quoted context omitted.
> The laws protecting IP merely gives a moat to incumbents. Yes. These laws are bad. We could fix this with a 2 line change: Section 1. Article I, Section 8, Clause 8 of this Constitution is hereby repealed. Section 2. Congress shall make no law abridging the right of the people to publish information.
Abolishing the copyright clause would not solve this problem because OpenAI is not leveraging copyright or patents. They're just not releasing anything. To fix this, you'd need to ban trade secrecy entirely. As in, if you have some kind of invention or creative work you must publish sufficient information to replicate it "in a timely manner". This would be one of those absolutely insane schemes that only a villain in…
The problem is how in the world is ChatGPT so good compared to the average human being? The answer is that human beings (except for the 1%), have their left hands tied behind their back because of copyright law.
It's possible to run llama.cpp on windows, e.g. see this tutorial: https://www.youtube.com/watch?v=coIj2CU5LMU Would this version (ggerganov) work with one of those methods?
I did the following:
1. Create a new working directory.
2. git clone https://github.com/ggerganov/llama.cpp
3. Download the latest release from https://github.com/ggerganov/llama.cpp/releases (note the CPU requirements in the filename) and unzip directly into the working directory's llama.cpp/ - you'll have the .exe files and .py scripts in the same directory.
4. Open PowerShell, cd to the working directory/llama.cpp, and create a new Python virtual environment: python3 -m venv env and activate the environment: .\env\Scripts\Activate.ps1
5. Obtain the LLaMA model(s) via the magnet torrent link and place them in the models directory. I used 30B and it is slow, but usable, on my system. Not even ChatGPT 3 level especially for programming questions, but impressive.
6. python3 -m pip install torch numpy sentencepiece
7. python3 convert-pth-to-ggml.py models/30B/ 1 (you may delete the original .pth model files after this step to save disk space)
8. .\quantize.exe ./models/30B/ggml-model-f16.bin ./models/30B/ggml-model-q4_0.bin 2
9. I copied the examples/chat-13B.bat to a new chat-30B.bat file, updated the model directory, and changed the last line of the script to: .\main.exe
10. Run using: .\examples\chat-30B.bat
https://github.com/ggerganov/llama.cpp#usage has details, although it assumes 7B and skips a few of the above steps.
Earlier quoted context omitted.
99% of LLM evaluation with PyTorch was already done in C++. These .cpp projects don't improve anything for performance. They just drop dependencies necessary for training and experimentation.
Optimization isn't just about speed. As you said, dropping dependencies makes it portable, embeddable, more versatile
Earlier quoted context omitted.
Maybe using Nix it's a better experience for creating such an environment where you depending also on system utilities.
Everyone is using llama.cpp because we reject the idea of giving up on system libraries like nix does. That kind of tomfoolery (at least in the desktop context) is only required when you use software projects that use libraries/languages which break forwards compatibility every 3 years. If you just write straight c++ (without c++xx, or anything like it) you can compile the code on machines from decades ago if you wan…
Earlier quoted context omitted.
I did not claim llama was open source, but I see the url I posted insinuates that (probably for a contorted meaning of open source, as in source available for approved academics). Anyway, T5 being available for download from Huggingface only makes my question more pertinent...
I made an app for running t5 locally - compiled version allows you to run without installing anything. https://capsizegames.itch.io/chat-ai https://github.com/Capsize-Games/chatai
does it happen to run on CPU on a server with 96GB RAM?
Earlier quoted context omitted.
Any JVM language or .NET language will take more to interface with native libraries, it’s not the same. Ocaml is very niche, I feel it’s an hard sell for a general purpose language. Haskell, 3x that. JS and TS, could be. But are they so much better than Python, if better at all?
Native library interfacing isn't really Python's strong suit, interpreter plugins are quite painful to write. .NET has P/Invoke which is much nicer. JVM is getting Panama+jextract, which is the nicest yet. You can go straight from header files to pure Java bindings which don't need any extra native code at all. But it's not shipped yet :(