Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

github.com

Post reply on HN
Print view View original

121–130 of 241 posts

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#121

Post by throwaway2027 »

ushakov wrote: post #95
simonw wrote: post #84

It's hard to overstate the impact Georgi Gerganov and llama.cpp have had on the local model space. He pretty much kicked off the revolution in March 2023, making LLaMA work on consumer laptops. Here's that README from March 10th 2023 https://github.com/ggml-org/llama.cpp/blob/775328064e69db1eb... > The main goal is to run the model using 4-bit quantization on a MacBook. [...] This was hacked in an evening - I have no…

i am curious, why are your comments always pinned to the top?

Time flies and simonw his AI feedback isn't always received favorably, sometimes he pushes it too much.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#122

Post by logicallee »

liuliu wrote: post #80
echoangle wrote: post #55

Earlier quoted context omitted.

Maybe stupid question but why not just put it in a torrent?

It is very simple. Storage / bandwidth is not expensive. Residential bandwidth is. If you can convince people to install a bandwidth-related software on their residential homes, you can then charge other people $5 to $10 per 1GiB bandwidth (useful for botnet mostly, get around DDOS protections and other reCAPTCHA tasks).

Thank you for your suggestion. Below is only our plans/intentions, we welcome feedback about it:

We are not going to do what you suggest. Instead, our approach is to use the RAM people aren't using at the moment for a fast edge cache close to their area.

We've tried this architecture and get very low latency and high bandwidth. People would not be contributing their resources to anything they don't know about.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#123

Post by fc417fc802 »

vardalab wrote: post #89
Tepix wrote: post #50

Earlier quoted context omitted.

It's insane how much traffic HF must be pushing out of the door. I routinely download models that are hundreds of gigabytes in size from them. A fantastic service to the sovererign AI community.

Yup, I have downloaded probably a terabyte in the last week, especially with the Step 3.5 model being released and Minimax quants. I wonder what my ISP thinks. I hope they don't cut me off. They gave me a fast lane, they better let me use it, lol

Even fairly restrictive data caps are in the range of 6 Tb per month. P2P at a mere 100 Mb works out to 1 TiB per 24 hours.

Hypothetically my ISP will sell me unmetered 10 Gb service but I wonder if they would actually make good on their word ...

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#124

Post by fc417fc802 »

HPsquared wrote: post #31
zozbot234 wrote: post #21

Earlier quoted context omitted.

> We still need good value hardware to run Kimi/GLM in-house If you stream weights in from SSD storage and freely use swap to extend your KV cache it will be really slow (multiple seconds per token!) but run on basically anything. And that's still really good for stuff that can be computed overnight, perhaps even by batching many requests simultaneously. It gets progressively better as you add more compute, of course…

At a certain point the energy starts to cost more than renting some GPUs.

Aren't decent GPU boxes in excess of $5 per hour? At $0.20 per kWhr (which is on the high side in the US) running a 1 kW workstation 24/7 would work out to the same price as 1 hour of GPU time.

The issue you'll actually run into is that most residential housing isn't wired for more than ~2kW per room.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#125

Post by fc417fc802 »

embedding-shape wrote: post #63
sowbug wrote: post #58

Earlier quoted context omitted.

I thought of the tracking and gate questions, too, when I vibed up an HF torrent service a few nights ago. (Super annoying BTW to have to download the files just to hash the parts, especially when webseeds exist.) Model owners could disable or gate torrents the same way they gate the models, and HF could still measure traffic by .torrent downloads and magnet clicks. It's a bit like any legalization question -- the bl…

> Model owners could disable or gate torrents the same way they gate the models, and HF could still measure traffic by .torrent downloads and magnet clicks. But that'll only stop a small part, anyone could share the infohash and if you're using the dht/magnet without .torrent files or clicks on a website, no one can count those downloads unless they too scrape the dht for peers who are reporting they've completed the…

> unless they too scrape the dht for peers who are reporting they've completed the download.

Which can be falsified. Head over to your favorite tracker and sort by completed downloads to see what I mean.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#127

Post by ushakov »

seanhunter wrote: post #120
ushakov wrote: post #103

Earlier quoted context omitted.

of course your comment attracts more upvotes - it's at the top.

It’s at the top because of upvotes. They don’t have an “if simonw: boost” branch in the code.

the code is not public, so we can't know. i think it's much more nuanced and certain users' comments might get a preferential treatment, based on factors other than the upvote count - which itself is hidden from us.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#128

Post by ComplexSystems »

ushakov wrote: post #127
seanhunter wrote: post #120

Earlier quoted context omitted.

It’s at the top because of upvotes. They don’t have an “if simonw: boost” branch in the code.

the code is not public, so we can't know. i think it's much more nuanced and certain users' comments might get a preferential treatment, based on factors other than the upvote count - which itself is hidden from us.

> the code is not public, so we can't know.

I feel like you're making this statement in bad faith, rather than honestly believing the developers of the forum software here have built in a clause to pin simonw's comments to the top.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#129

Post by karmasimida »

Does local AI have a future? The models are getting ridiculously big and any storage hardware is hoarded by few companies for next 2 years and nvidia has stopped making consumer GPU for this year.

It seems to me there is no chance local ML is going to be anywhere out of the toy status comparing to closed source ones in short term

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#130

Post by francispauli »

ushakov wrote: post #95
simonw wrote: post #84

It's hard to overstate the impact Georgi Gerganov and llama.cpp have had on the local model space. He pretty much kicked off the revolution in March 2023, making LLaMA work on consumer laptops. Here's that README from March 10th 2023 https://github.com/ggml-org/llama.cpp/blob/775328064e69db1eb... > The main goal is to run the model using 4-bit quantization on a MacBook. [...] This was hacked in an evening - I have no…

i am curious, why are your comments always pinned to the top?

thanks for reminding me i need to follow his blog weekly again
Post reply on HN