Live data from Hacker News

Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

github.com

291–298 of 298 posts

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#291

Earlier quoted context omitted.

> they could snipe ML away from nvidia. If Nvidia announced tomorrow that they were cancelling every datacenter deal they had, open-sourcing CUDA and publishing their entire patent library to the creative commons, I would still not believe you. This is a fun project for people with Apple Silicon machines who want to participate in the AI happenings, but I don't think you can warp it into a call for Nvidia's head. Let…

Whoa, who’s calling for nvidia’s head? Not me. My point was that the PC architecture of separate system and GPU memory is hitting a wall that means inefficiency and higher prices. I have little doubt that Nvidia’s attempted acquisition of ARM was in part because nvidia recognized this. I expect they are exploring other UMA approaches. But it will be hard in the fragmented, not-vertically-integrated model. Apple’s adv…

Nvidia doesn't need to acquire ARM to sell systems with unified memory. The Tegra boards are all mixed-address-space systems, and CUDA lets you manipulate memory over PCI. They see the writing on the wall, which is why they've been building systems for the past decade that reflect this philosophy.

If you think it's hard to imagine Nvidia hardware running on a developer desktop, wait until you hear about what happened when Macs tried entering the server market.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#293
post #265
post #165

Earlier quoted context omitted.

Siri doesn’t seem as fast or responsive compared to Google assistant at times.

At this point in time, Siri as a voice-driven assistant has become so totally and utterly useless, its not even worth comparing it to anything else. I wonder how a company can work at a feature like that for 10 years, and manage to make it worse with every release they put out. At this point in time, Apple should be so embarrased of Siri that I really think scratching the whole thing would have a net benefit. Scratch…

The logistics aren't that easy, Apple's entire product line runs Siri.

Re: Llama.cpp: Port of Facebook's LLaMA model in C/C++, with Apple Silicon support

#294

Relevant: Since LLaMA leaked on torrent, it has been converted to Huggingface weights and it has been quantisized to 8bit for less vram requirements. A few days ago it has also been quantisized to 4bit and 3bit is coming. The quantization method they use is from the GPTQ paper ( https://arxiv.org/abs/2210.17323 ) which leads to almost no quality degradation compared to the 16bit weights. 4 bit weights: Model, weight…

[dead]
Post reply on HN