Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

github.com

Post reply on HN
Print view View original

41–50 of 241 posts

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#41

Post by ryandrake »

mft_ wrote: post #38
tkp-415 wrote: post #25

Can anyone point me in the direction of getting a model to run locally and efficiently inside something like a Docker container on a system with not so strong computing power (aka a Macbook M1 with 8gb of memory)? Is my only option to invest in a system with more computing power? These local models look great, especially something like https://huggingface.co/AlicanKiraz0/Cybersecurity-BaronLLM_O... for assisting in p…

There’s no way around needing a powerful-enough system to run the model. So you either choose a model that can fit on what you have —i.e. via a small model, or a quantised slightly larger model— or you access more powerful hardware, either by buying it or renting it. (IME you don’t need Docker. For an easy start just install LM Studio and have a play.) I picked up a second-hand 64GB M1 Max MacBook Pro a while back fo…

I wasn't able to have very satisfying success until I bit the bullet and threw a GPU at the problem. Found an actually reasonably priced A4000 Ada generation 20GB GPU on eBay and never looked back. I still can't run the insanely large models, but 20GB should hold me over for a while, and I didn't have to upgrade my 10 year old Ivy Bridge vintage homelab.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#42

Post by dragonwriter »

option wrote: post #29

Isn't HF banned in China? Also, how are many Chinese labs on Twitter all the time? In either case - huge thanks to them for keeping AI open!

> Isn't HF banned in China?

I think, for some definition of “banned”, that’s the case. It doesn’t stop the Chinese labs from having organization accounts on HF and distributing models there. ModelScope is apparently the HF-equivalent for reaching Chinese users.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#45

Post by superkuh »

I'm glad the llama.cpp and the ggml backing are getting consistent reliable economic support. I'm glad that ggerganov is getting rewarded for making such excellent tools.

I am somewhat anxious about "integration with the Hugging Face transformers library" and possible python ecosystem entanglements that might cause. I know llama.cpp and ggml already have plenty of python tooling but it's not strictly required unless you're quantizing models yourself or other such things.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#46

Post by microsoftedging »

HanClinto wrote: post #7

I'm regularly amazed that HuggingFace is able to make money. It does so much good for the world. How solid is its business model? Is it long-term viable? Will they ever "sell out"?

FT had a solid piece a few weeks back: "Why AI start-up Hugging Face turned down a $500mn Nvidia deal"

https://giftarticle.ft.com/giftarticle/actions/redeem/9b4eca...

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#47

Post by embedding-shape »

sowbug wrote: post #39
mythz wrote: post #18

I consider HuggingFace more "Open AI" than OpenAI - one of the few quiet heroes (along with Chinese OSS) helping bring on-premise AI to the masses. I'm old enough to remember when traffic was expensive, so I've no idea how they've managed to offer free hosting for so many models. Hopefully it's backed by a sustainable business model, as the ecosystem would be meaningfully worse without them. We still need good value…

Why doesn't HF support BitTorrent? I know about hf-torrent and hf_transfer, but those aren't nearly as accessible as a link in the web UI.

> Why doesn't HF support BitTorrent?

Harder to track downloads then. Only when clients hit the tracker would they be able to get download states, and forget about private repositories or the "gated" ones that Meta/Facebook does for their "open" models.

Still, if vanity metrics wasn't so important, it'd be a great option. I've even thought of creating my own torrent mirror of HF to provide as a public service, as eventually access to models will be restricted, and it would be nice to be prepared for that moment a bit better.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#48

Post by Fin_Code »

mythz wrote: post #18

I consider HuggingFace more "Open AI" than OpenAI - one of the few quiet heroes (along with Chinese OSS) helping bring on-premise AI to the masses. I'm old enough to remember when traffic was expensive, so I've no idea how they've managed to offer free hosting for so many models. Hopefully it's backed by a sustainable business model, as the ecosystem would be meaningfully worse without them. We still need good value…

I still don't know why they are not running on torrent. Its the perfect use case.

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#49

Post by liuliu »

logicallee wrote: post #30
dmezzetti wrote: post #14

This is really great news. I've been one of the strongest supporters of local AI dedicating thousands of hours towards building a framework to enable it. I'm looking forward to seeing what comes of it!

>I've been one of the strongest supporters of local AI, dedicating thousands of hours towards building a framework to enable it. Sounds like you're very serious about supporting local AI. I have a query for you (and anyone else who feels like donating) about whether you'd be willing to donate some memory/bandwidth resources p2p to hosting an offline model: We have a local model we would like to distribute but don't h…

> We have a local model we would like to distribute but don't have a good CDN.

That is not true. I am serving models off Cloudflare R2. It is 1 petabyte per month in egress use and I basically pay peanuts (~$200 everything included).

Re: Ggml.ai joins Hugging Face to ensure the long-term progress of Local AI

#50

Post by Tepix »

mythz wrote: post #18

I consider HuggingFace more "Open AI" than OpenAI - one of the few quiet heroes (along with Chinese OSS) helping bring on-premise AI to the masses. I'm old enough to remember when traffic was expensive, so I've no idea how they've managed to offer free hosting for so many models. Hopefully it's backed by a sustainable business model, as the ecosystem would be meaningfully worse without them. We still need good value…

It's insane how much traffic HF must be pushing out of the door. I routinely download models that are hundreds of gigabytes in size from them. A fantastic service to the sovererign AI community.
Post reply on HN