81–90 of 238 posts
Is there a Llama2 65b quantized version for Mac M2?
Update, this might work: https://huggingface.co/TheBloke/Llama-2-70B-GGML
Example command mentions the 4bit variant:
./main -m llama-2-70b.ggmlv3.q4_0.bin -gqa 8 -t 13 -p "Llamas are"
Earlier quoted context omitted.
> the answers are amazingly good, highly unreliable and parrot meaning at best. There is no inductance, and no inteospection and no understanding of the deep semantic meaning of the language presented. There's no intelligence. Can say the same about a half of population, tbh
So what you’re saying is.... we’re going to have human-level AI, and it’s going to be incredibly stupid
Earlier quoted context omitted.
Most CPU RAM is much slower than GPU RAM. GPUs typically pack RAM 2 generations ahead with a wider bus than anything you'd find on a consumer motherboard.
For reference, DDR4-3200 in quad channel is ~100 GB/s while a 3090's VRAM is 960 GB/s. Of course, most consumers only have dual channel. M1 Pro is 200 and M1 Max is 400. Which is slow for GPU memory, but incredible for main memory -- although I'm not sure how much of that a single core can actually pull.
Earlier quoted context omitted.
I was hoping that link would answer the question that's been bugging me for months: what are the penalties that you pay for using a quantized model? Sadly it didn't. It talked about "perplexities" and showed some floating point numbers. I want to see examples like "here's a prompt against a model and the same prompt against a quantized version of that model, see how they differ."
I want to see examples like "here's a prompt against a model and the same prompt against a quantized version of that model, see how they differ." We suck at evaluating and comparing models imo. There are metrics and evaluation task, but it's still very subjective. The closer we get to assessing human like performance, the tougher it is, because it becomes more subjective and less deterministic by the nature of the ta…
I think this right here is the answer to measuring and comparing model performance.
Instead of trying to compare models holistically, we should be comparing them for specific problem sets and use cases... the same as we compare humans against one another.
Using people as an example, a hiring manager doesn't compare 2 people holistically, they compare 2 people based on how well they're expected to perform a certain task or set of tasks.
We should be measuring and comparing models discriminately rather than holistically.
Is there a Llama2 65b quantized version for Mac M2?
>> Memory bandwidth is the limiting factor in almost everything to do with sampling from transformers. So how about using an APU - a CPU with GPU built in. The GPU shares the CPU memory, so if you want you can have 128GB RAM and allocate 100GB to the GPU. Sure the GPU i not fast, but if memory is important.....
Most CPU RAM is much slower than GPU RAM. GPUs typically pack RAM 2 generations ahead with a wider bus than anything you'd find on a consumer motherboard.
>> Memory bandwidth is the limiting factor in almost everything to do with sampling from transformers. So how about using an APU - a CPU with GPU built in. The GPU shares the CPU memory, so if you want you can have 128GB RAM and allocate 100GB to the GPU. Sure the GPU i not fast, but if memory is important.....
This project's been a blast to work with. While it's written in C++, it provides a C interface to compile against which makes it especially easy to extend with Go, Python and other runtimes. A few folks and I have been building a tool with it in Go for pulling & running multiple models, and serving them on a REST API: https://github.com/jmorganca/ollama In similar light, you haven't checked it out, llama.cpp also has…
Earlier quoted context omitted.
No, a human isn't born with a set of knowledge like a freshly trained LLM, keeping the model fixed and responding to input. The analog to the model changes based on the human's experience. Just making bigger and bigger LLMs won't give you this.
So a human is different because it keeps training its neural network?
could be a difference that makes no difference, or ...
Earlier quoted context omitted.
The irony in your statement is immense. Yes, Kurzeweil has been saying this for decades. No it doesn't mean AGI is close. These llms do nothing to advance AGI. There is no theoretical basis to the belief in emergent intelligence from statistical language models and the answers are amazingly good, highly unreliable and parrot meaning at best. There is no inductance, and no inteospection and no understanding of the dee…
His prediction was that one human brain's worth of computing power could be acquired for $1000 by 2029. That still seems reasonable. That's not the same as AGI or the singularity.
But I am human, I am conscious and no visible VLSI work or algorithmic model will lead to AGI or a human equivalent computing power by 2029. Let alone for $1000.