151–160 of 238 posts
This leaves a ton of stuff out. - Token generation is serial and bandwidth bound, but prompt ingestion is not and runs in batches of 512+. Short tests are fast on pure CPU llama.cpp, but long prompting (such as with ongoing conversation) is extremely slow compared to other backends. - Llama.cpp now has very good ~4 bit quantization that doesn't affect perplexity much. Q6_K almost has the same perplexity as FP16, but…
In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…
In case anyone is wondering, yes, there is a cost when a model is quantized. https://oobabooga.github.io/blog/posts/perplexities/ Essentially, you lose some accuracy and there might be some weird answers and probably more likely to go off the rail and hallucinate. But the quality loss is lower the more parameters you have. So for very large model sizes the differences might be negligible. Also, this is the cost of in…
This is not generally true, sometimes quantisation can improve accuracy. I haven't seen that with LLMs yet though.
Is it: the weights all happen to be where float is sparse, so quantization ends up increasing fidelity? Or is it more of a “worse is better” dropout-type situation?
This leaves a ton of stuff out. - Token generation is serial and bandwidth bound, but prompt ingestion is not and runs in batches of 512+. Short tests are fast on pure CPU llama.cpp, but long prompting (such as with ongoing conversation) is extremely slow compared to other backends. - Llama.cpp now has very good ~4 bit quantization that doesn't affect perplexity much. Q6_K almost has the same perplexity as FP16, but…
Memory bound token generation is a limitation of transformer decoders. In the past, hardware has motivation algorithm innovations. I’m curious how long it will take until we see more hardware friendly models.
Earlier quoted context omitted.
AGI should have morals, opinions, self-reflection, learn continuously from sensor data, reason, realize when they’re proven wrong and update their model of the world, and be creative. So far LLMs exhibit none of those. But LLMs exhibit a digestible distillation of a very large body of data which may be a component of an AGI. But you can have an AGI that doesn’t have encyclopedic knowledge but it’s still highly intell…
That is not what AGI means. AGI = Artificial General Intelligence. 1. Artificial = we made it 2. General = it can solve problems in any field 3. Intelligence = the ability to solve problems A chess engine is a very strong Artificial Intelligence . But it’s not very General , it can only evaluate chess positions. GPT-4 is very General , you can ask it about any question and get a somewhat reasonable answer. But it’s n…
That's news to me. AGI (or strong AI) is typically defined as "human-level intelligence", or "perform any task that a human or animal can." Humans and animals often perform tasks that are critically reliant on being conscious, emoting, reading body language, reasoning, etc.
Not only that but prominent thinkers who have carved out the notion of AGI (or Strong AI) tend to have consciousness, mental states, and emotions at the core of it.
I think what you're talking about is a multi-task AI, not an AGI.
This leaves a ton of stuff out. - Token generation is serial and bandwidth bound, but prompt ingestion is not and runs in batches of 512+. Short tests are fast on pure CPU llama.cpp, but long prompting (such as with ongoing conversation) is extremely slow compared to other backends. - Llama.cpp now has very good ~4 bit quantization that doesn't affect perplexity much. Q6_K almost has the same perplexity as FP16, but…
Earlier quoted context omitted.
This is not generally true, sometimes quantisation can improve accuracy. I haven't seen that with LLMs yet though.
Interesting, how would that work? Are there any well-known examples? Is it: the weights all happen to be where float is sparse, so quantization ends up increasing fidelity? Or is it more of a “worse is better” dropout-type situation?
Earlier quoted context omitted.
This is not generally true, sometimes quantisation can improve accuracy. I haven't seen that with LLMs yet though.
Interesting, how would that work? Are there any well-known examples? Is it: the weights all happen to be where float is sparse, so quantization ends up increasing fidelity? Or is it more of a “worse is better” dropout-type situation?