181–190 of 241 posts
Earlier quoted context omitted.
I have a 1.2TB cap before you start getting charged extra, so you might need to recalibrate your restrictive level.
Is that with a WISP by chance? Or in a developing country? Or are there really wired providers with such low caps in the western world in this day and age?
Earlier quoted context omitted.
ModelScope is the Chinese equivalent of Hugging Face and a good back up. All the open models are Chinese anyways
Not true! Mistral is really really good, but I agree that there isn't a single decent open model from the USA.
Can anyone point me in the direction of getting a model to run locally and efficiently inside something like a Docker container on a system with not so strong computing power (aka a Macbook M1 with 8gb of memory)? Is my only option to invest in a system with more computing power? These local models look great, especially something like https://huggingface.co/AlicanKiraz0/Cybersecurity-BaronLLM_O... for assisting in p…
The general rule of thumb is that you should feel free to quantize even as low as 2 bits average if this helps you run a model with more active parameters. Quantized models are not perfect at all, but they're preferable to the models with fewer, bigger parameters. With 8GB usable, you could run models with up to 32B active at heavy quantization.
Earlier quoted context omitted.
I have a 1.2TB cap before you start getting charged extra, so you might need to recalibrate your restrictive level.
Is that with a WISP by chance? Or in a developing country? Or are there really wired providers with such low caps in the western world in this day and age?
Earlier quoted context omitted.
Haha for now our primary goal is to expand the market for local AI and educate people on how to do RL, fine-tuning and running quants :)
Amazing work and people should really appreciate that the opportunity costs of your work are immense (given the hype). On another note: I'm a bit paranoid about quantization. I know people are not good at discerning model quality at these levels of "intelligence" anymore, I don't think a vibe check really catches the nuances. How hard would it be to systematically evaluate the different quantizations? E.g. on the Aid…
Since I don't see it mentioned here, LlamaBarn is an awesome little—but mighty—MacOS menubar program, making access to llama.cpp's great web UI and downloading of tastefully curated models easy as pie. It automatically determines the available model- and context-sizes based on available RAM.
https://github.com/ggml-org/LlamaBarn
Downloaded models live in:
~/.llamabarn
# bind to all interfaces (0.0.0.0)
defaults write app.llamabarn.LlamaBarn exposeToNetwork -bool YES
# or bind to a specific IP (e.g., for Tailscale)
defaults write app.llamabarn.LlamaBarn exposeToNetwork -string "100.x.x.x"
# disable (default)
defaults delete app.llamabarn.LlamaBarn exposeToNetworkEarlier quoted context omitted.
Is that with a WISP by chance? Or in a developing country? Or are there really wired providers with such low caps in the western world in this day and age?
ATT once told me if I don't pay for their TV service then my home gigabit fiber would have a 1TB cap. They had an agreement with the apartment building so I had no other choice of provider.
Earlier quoted context omitted.
Mistral is cool and I wish them success but it consistently ranks extremely low on benchmarks while still being expensive. Chinese models like DeepSeek might rank almost as low as Mistral but they are significantly cheaper. And Kimi is the best of both worlds with incredible benchmark results while still being incredibly cheap I know things change rapidly so I'm not counting them out quite yet but I don't see them as…
Why are you talking price when we are talking local AI? That doesn't make any sense to me. Am I missing something?
Earlier quoted context omitted.
Mistral is cool and I wish them success but it consistently ranks extremely low on benchmarks while still being expensive. Chinese models like DeepSeek might rank almost as low as Mistral but they are significantly cheaper. And Kimi is the best of both worlds with incredible benchmark results while still being incredibly cheap I know things change rapidly so I'm not counting them out quite yet but I don't see them as…
Why are you talking price when we are talking local AI? That doesn't make any sense to me. Am I missing something?
Earlier quoted context omitted.
> We have a local model we would like to distribute but don't have a good CDN. That is not true. I am serving models off Cloudflare R2. It is 1 petabyte per month in egress use and I basically pay peanuts (~$200 everything included).
1 petabyte per month is 1 million downloads of a 1 GB file. We intend to scale to more than 1 million downloads per month. We have a specific scaling architecture in mind. We're qualified to say this because we've ported a billion parameter model to run in your browser - fast - on either webgpu or wasm. (You can see us doing it live at the youtube link in my comment above.) There is a lot of demand for that.