Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

github.com

Post reply on HN
Print view View original

41–50 of 298 posts

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#41

Post by lxe »

Szpadel wrote: post #4

I tried llama 14b model by using one of online tools (mentioned in one of other hn comments, it claimed to use that model, but cannot be 100% sure) and I was very disappointed with results. I tried to ask it fairly simple question: to write regex validating email address, tried this 2 times and got responses: "what is your question" and "please wait..." so it just doged response. in contrast chatgpt was able to provi…

I've gotten great q&a / conversational results from llama 7b and 13b using https://github.com/oobabooga/text-generation-webui.

    ],
    [
      "How does the Schrodinger's cat thought experiment relate to the principles of quantum mechanics?",
      "In this famous paradox, a cat inside a box is simultaneously both alive and dead due to the nature of subatomic particles. This demonstrates that at the smallest scales of reality, we cannot predict whether something will happen until we observe it happening."
    ],
    [
      "What are some of the potential consequences of climate change on global food security?",
      "According to researchers at Stanford University, rising temperatures could lead to crop failures and reduced harvests for staple crops such as rice, wheat, corn, and soybeans by midcentury."
    ],
    [
      "Can you provide a brief overview of the history of political philosophy, from Ancient Greece to the present day?",
      "Absolutely! Political philosophy is a branch of philosophy concerning the role of government in society. In ancient times, Plato and Aristotle were among the first thinkers to consider the subject in depth. Since then, other influential philosophers such as Thomas Hobbes and John Locke have expanded upon these ideas. Today, contemporary philosophers such as John Rawls continue to debate the merits of different political systems."
    ]

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#42

Post by 0xbadc0de5 »

2bitencryption wrote: post #25

Could someone with experience explain: what's the theoretical minimum hardware requirement for llama 7B, 15B, etc, that still provides output on the order of It seems like we can pull some tricks, like using F16, and some kind of quantization, etc. At the end of the day, how much overhead is left that can be reduced? What can I expect to have running on 16gb ram with a 3080 and a midrange AMD processor?

The 4-bit GPTQ LLaMA models are the current top-performers. This site has done a lot of the heavy lifting: https://github.com/qwopqwop200/GPTQ-for-LLaMa

With 30b-4bit on a RTX 4090, I'm seeing numbers like:

Output generated in 4.17 seconds (4.03 tokens/s, 21 tokens)

Output generated in 4.38 seconds (4.25 tokens/s, 23 tokens)

Output generated in 4.57 seconds (4.25 tokens/s, 24 tokens)

Output generated in 3.86 seconds (3.40 tokens/s, 17 tokens)

The lower size (7b, 13b) are even faster with lower memory use. A 16GB 3080 should be able to run the 13b at 4-bit just fine with reasonable (>1 token/s) latency.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#43

Post by brookst »

0xbadc0de5 wrote: post #24

That's all fine and good. But to do anything useful, you're going to want a powerful GPU (RTX 3090, RTX 4090 or A6000) with as much VRAM as possible. Unlike the diffusion models, LLM's are very memory-intensive, even at 4-bit GPTQ. The larger models like llama-13b and llama-30b run quite well at 4-bit on a 24GB GPU. The llama-65b-4bit should run on a dual 3090/4090 rig. Coupled with the leaked Bing prompt and text-ge…

Macs have UMA so an off the shelf Mac can use up to about 120GB of vram. Far more than any consumer card, more than the biggest a100.

GPU power is lower, of course, but pure vram is not a problem.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#44

Post by Szpadel »

dmm wrote: post #19
Szpadel wrote: post #4

I tried llama 14b model by using one of online tools (mentioned in one of other hn comments, it claimed to use that model, but cannot be 100% sure) and I was very disappointed with results. I tried to ask it fairly simple question: to write regex validating email address, tried this 2 times and got responses: "what is your question" and "please wait..." so it just doged response. in contrast chatgpt was able to provi…

sillysaurusx made some posts about tweaks that greatly improved output for him. Maybe they would help you: """ Fixed the llama sampler. After turning off top_p, adding top_k 40, setting temp to 0.7, and adding a repetition penalty of 1/0.85, llama 7B is looking nice. """ https://twitter.com/theshawwn/status/1632569215348531201?cxt... https://twitter.com/theshawwn/status/1633456289639542789?cxt...

for reference I tried or here: https://lite.koboldai.net/

with chat mode, and parameters: temp 0.62, top_p 0.9, repetition penalty 1.08

is free to try, but then you have to wait in queue, therefore I had to wait 10min per response and I didn't had patience to test it further

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#45

Post by hot_gril »

potatoman22 wrote: post #20
jcq3 wrote: post #12

Don't know anything about ML can someone can explain me what is this hype about?

This is an advanced language model that can now run quickly on consumer grade hardware. You used to need thousands of dollars of GPUs to run a model as sophisticated as this - now it can be done on a laptop,

Run meaning run inference, not train, right?

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#46

Post by lxe »

lxe wrote: post #41
Szpadel wrote: post #4

I tried llama 14b model by using one of online tools (mentioned in one of other hn comments, it claimed to use that model, but cannot be 100% sure) and I was very disappointed with results. I tried to ask it fairly simple question: to write regex validating email address, tried this 2 times and got responses: "what is your question" and "please wait..." so it just doged response. in contrast chatgpt was able to provi…

I've gotten great q&a / conversational results from llama 7b and 13b using https://github.com/oobabooga/text-generation-webui . ], [ "How does the Schrodinger's cat thought experiment relate to the principles of quantum mechanics?", "In this famous paradox, a cat inside a box is simultaneously both alive and dead due to the nature of subatomic particles. This demonstrates that at the smallest scales of reality, we ca…

And 30b:

    Give me a basic recipe for easy spaghetti carbonara

    Here is a basic recipe for spaghetti carbonara:

    Ingredients: 1 lb spaghetti, 3 tbsp butter, 2 tbsp olive oil, 1/4 cup grated Parmesan cheese, 1 egg, 1/2 teaspoon salt, 1/4 teaspoon pepper, 1/2 cup chopped parsley, 1/2 pound bacon, 1 clove garlic, 1/4 cup heavy cream.

    Directions: Cook spaghetti according to package directions; drain. Heat butter and oil in large skillet on medium heat. Add bacon; cook 4 minutes or until crisp. Remove from skillet. Add garlic to drippings in pan; sauté 1 minute. Stir in egg, salt, pepper, and parsley; stirring constantly. Add bacon and spaghetti; toss well. Sprinkle with cheese. Serves 4.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#47

Post by jonas21 »

stagger87 wrote: post #17
yieldcrv wrote: post #14

Earlier quoted context omitted.

why does it say video unavailable?

I'm pretty sure he only makes his past twitch streams available to subscribers. It's weird they are available on Youtube, maybe he doesn't know that's something he can change.

I don't think he runs the YouTube channel - it's managed by some of his fans. But like what's he gonna do about it? Send them a takedown notice?

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#48

Post by TOMDM »

hot_gril wrote: post #45
potatoman22 wrote: post #20

Earlier quoted context omitted.

This is an advanced language model that can now run quickly on consumer grade hardware. You used to need thousands of dollars of GPUs to run a model as sophisticated as this - now it can be done on a laptop,

Run meaning run inference, not train, right?

Yes

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#50

Post by stu2b50 »

0xbadc0de5 wrote: post #24

That's all fine and good. But to do anything useful, you're going to want a powerful GPU (RTX 3090, RTX 4090 or A6000) with as much VRAM as possible. Unlike the diffusion models, LLM's are very memory-intensive, even at 4-bit GPTQ. The larger models like llama-13b and llama-30b run quite well at 4-bit on a 24GB GPU. The llama-65b-4bit should run on a dual 3090/4090 rig. Coupled with the leaked Bing prompt and text-ge…

VRAM is the thing that Apple Silicon is going to have in excess compared to anything even close in price. MacBook Airs can have 14-15GB of VRAM if necessary.
Post reply on HN