191–200 of 238 posts
Earlier quoted context omitted.
SBC
Wtf does SBC mean? God enough with the acronyms people.
I would elaborate and say, anywhere that your computer is resource constrained ( ram, processing power ) but you still want to make up articles for your Amazon Affiliate blog
Earlier quoted context omitted.
By most measures you could think of for intelligence languages models are improving, so I don’t see why you think this wouldn’t lead to something at least almost human-level if you scaled it up enough Of course there could be some wall somewhere but I don’t see why there would be
Because you need more training data for better results and they are running out of new training data.
Earlier quoted context omitted.
(Stupid question) are models still non-deterministic if you set the temperature to zero? Would setting the temperature to zero degrade the quality of response?
Even at T=0 and run deterministically, the answers still have "randomness" with respect to the exact prompt used. Change wording slightly and you've introduced randomness again even if the meaning doesn't change. It would be the same for a person. For an llm, a trivial change in wording could produce a big change in answer, same as running it again with a new random seed. "Prompt engineering" is basically overfitting…
I’ve been working through that repo and managed the 13B dataset on a single Pi4 8gig I’ve also replicated the work in OpenMPI ( from a thread on the llama.cpp GitHub repo ) and today I managed to get the 65B dataset operational on three pi4 nodes. I’m not saying this as any achievement of mine, but as a comment on the current reality of reproducible LLM At home on anything you’ve got. It really feels like this techni…
> I’ve also replicated the work in OpenMPI... Oh cool! How did it perform? I wonder if this would be an exciting test for Amazon's SRD protocol which appears to be built for HPC. I'm looking for an excuse to play with it...
I don't know about SRD, but llama.cpp has MPI configurations built-in. I didn't have to engineer anything or rewrite anything ( I made an optimization patch, but I didn't even make that one up myself ) I just compiled it with flags set.
As far as performance on 65B, I'm still waiting for it to finish to get the timings :)
Earlier quoted context omitted.
SBC
Wtf does SBC mean? God enough with the acronyms people.
The interesting implication about running an LLM on a single board computer is that if it's a proof of concept for an LLM on a smartphone. If you have a model that can produce useful results on a Ras Pi, you have something that could potentially run on hundreds of millions of smartphones. I'm not sure what the use case is for running an LLM on your phone instead of the cloud, but it opens some interesting possibilities. It depends just how useful such a small LLM could be.
Earlier quoted context omitted.
Even at T=0 and run deterministically, the answers still have "randomness" with respect to the exact prompt used. Change wording slightly and you've introduced randomness again even if the meaning doesn't change. It would be the same for a person. For an llm, a trivial change in wording could produce a big change in answer, same as running it again with a new random seed. "Prompt engineering" is basically overfitting…
But if T=0 and you use the exact same input (not a single word or position changes) do you get the same output? Reading your response it implies that the randomness is related to even slight changes.
Look at this minimal implementation (Karpathy's) of LLaMA, the only randomness is in the "sample" function that comes in at non-zero temperature, otherwise its easy to see everything is deterministic: https://github.com/karpathy/llama2.c/blob/master/run.c
Otoh, with MoE like GPT-4 has, it can still vary at zero temperature.
In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…
I was hoping that link would answer the question that's been bugging me for months: what are the penalties that you pay for using a quantized model? Sadly it didn't. It talked about "perplexities" and showed some floating point numbers. I want to see examples like "here's a prompt against a model and the same prompt against a quantized version of that model, see how they differ."
e.g. take a look at these two rows:
llama-65b.ggmlv3.q4_K_M.bin 4.90639 llama.cpp
llama-65b.ggmlv3.q3_K_M.bin 5.01299 llama.cpp
If you want more fine-grained explanations of how generation of certain types of texts get clobbered, you would probably need to prepare datasets comprised of that type of string, and measure the perplexity delta on that subset. i.e.
dperplexity/dquantization(typed_inputs).
dtype/dquantization(all_outputs).Does anyone know what the next breakthroughs will be and their rough timelines regarding locally run models? For instance will anything like chat gpt 4 be runnable on an M1 Mac within the next year?
I don't think there's any likelihood of replicating GPT-4 on your M1 in the next twelve months, especially not if you're expecting responsive performance. What we could see are a plethora of models dedicated to doing particular tasks. Say, specialists in aspects of programming or accounting or law or medicine. Or general knowledge models with access to a local cache of Wikipedia. Individually, none of these models would have to come close to GPT-4's overall level of power and flexibility. But collectively, they could reach that level of utility.
Earlier quoted context omitted.
How do humans write if not by intuiting what word comes after another? Intelligence is the ability of that next word decision procedure to determine a next word that is aligned with our human intuition and model of truth. I believe what you’re getting at is modality , that GPT-4 only provides responses in text. You can’t ask it to drive a car, or paint like Dall-e. And that’s a fair criticism, but it’s mostly just be…
> How do humans write if not by intuiting what word comes after another? We don't know. It may turn out that we use mechanisms similar to LLMs, or it might be something entirely different. As for the rest: nobody knows how to make ChatGPT butter a piece of toast, let alone drive a car. ChatGPT does not reason about text, either.
This leaves a ton of stuff out. - Token generation is serial and bandwidth bound, but prompt ingestion is not and runs in batches of 512+. Short tests are fast on pure CPU llama.cpp, but long prompting (such as with ongoing conversation) is extremely slow compared to other backends. - Llama.cpp now has very good ~4 bit quantization that doesn't affect perplexity much. Q6_K almost has the same perplexity as FP16, but…
What's prompt ingestion?
Speed is particularly important. The response can be streamed word by word, so it doesn't have to be particularly fast, but slow input processing leads to very noticable latency.