Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi-K3 on HuggingFace

huggingface.co

Post reply on HN
Print view View original

151–160 of 588 posts

Re: Kimi-K3 on HuggingFace

#151

Post by solarengineer »

light_hue_1 wrote: post #124
walrus01 wrote: post #64

Earlier quoted context omitted.

There are a number of use cases where sending the contents of your context and prompts (and the resulting output) to a 3rd party service is off the table as an option, and people will compromise speed for data sovereignty. And not everyone's electricity is equally expensive, I pay about $0.075 USD per kWh. It would for example cost me about $48 a month of electricity (not counting cost of cooling) to run a quad socke…

As someone who has worked in two industries that are at the maximal end of data sensitivity and privacy this comes across as a tinfoil hat issue not a real business requirement. In such cases we've always found ways to trade dollars for the privacy we need without having to run our own inference at excruciating slow speeds.

There are regulated sectors in countries where data sovereignty is important enough that the sector sticks to air-gapped on-prem hardware and does not use cloud services at all. They have the dollars to pay for more than what it would cost to run on the Cloud.

Re: Kimi-K3 on HuggingFace

#153

Post by vb-8448 »

energy123 wrote: post #111
vb-8448 wrote: post #48

Earlier quoted context omitted.

> Then we'll be able to guesstimate if "labs are subsidising tokens on API pricing". No, you don't. Without training cost you can infer only the marginal cost of serving this kind of models. Moreover, you don't know the actual size of closed models (what if Fable is a 10T model? What if it's 1T?)

Gross margins are insanely important, possibly the most important single metric if for some reason you were forced to choose one.

Only if you assume that at some point, for any reason, there will be "the model" that doesn't need costly retraining.

I guess this is one of the reason Anthropic i so "active" for asking for a development break.

Re: Kimi-K3 on HuggingFace

#154

Post by walrus01 »

johndough wrote: post #132
walrus01 wrote: post #59

Earlier quoted context omitted.

I have found that the "mostly didn't lose anything" Q8 large models that I want to run are all too large to run on the "only $3995!" 128GB max RAM systems that some people are buying, and definitely won't fit with any usable amount of context. Things like Qwen 3.5 122B Q8 or deepseek v4 flash Q8, or Laguna S 2.1 Q8 need 170-190GB of RAM including full context, which fits on a 256GB RAM dual socket workstation or rack…

DeepSeek-V4 should use only 5GB for context due to CSA and HCA, see figure here: https://huggingface.co/deepseek-ai/DeepSeek-V4-Pro But not every framework implements it properly yet.

Yeah, or close enough to 5GB for estimation purposes, for example a just spawned qwen 3.5 122B llama-server instance reports as:

0.07.015.888 I common_memory_breakdown_print: | - Host | 170038 = 162913 + 6740 + 384 |

The 6740 is the cache size.

Re: Kimi-K3 on HuggingFace

#157

Post by zargon »

Iolaum wrote: post #116
walrus01 wrote: post #59

Earlier quoted context omitted.

I have found that the "mostly didn't lose anything" Q8 large models that I want to run are all too large to run on the "only $3995!" 128GB max RAM systems that some people are buying, and definitely won't fit with any usable amount of context. Things like Qwen 3.5 122B Q8 or deepseek v4 flash Q8, or Laguna S 2.1 Q8 need 170-190GB of RAM including full context, which fits on a 256GB RAM dual socket workstation or rack…

There's an emerging practice of using Q4 quants and Q8 KV cache for local inference. At that point you can run both Qwen3.5-122B-A10B (my personal choice on Framework Desktop 128gb) and Laguna-S-2.1. Now whether that's good enough for one's use-case remains to be determined. You can also get more out of those (local models and quantizations) if you further tweak the harness you use them with, but tbh this is where it…

> emerging practice of using Q4 quants and Q8 KV cache for local inference

That's not an emerging practice, it's a tested strategy that is these days only used as a last resort by those desperate to fit a model in memory. Some models do better than others, but generally the model quality suffers greatly under those conditions.

Re: Kimi-K3 on HuggingFace

#159

Post by esperent »

maelito wrote: post #12

Did someone run censorship and political bias tests on this ? Must be interesting.

Outside of asking it to talk about Tiananmen Square, are there any standard tests for "bias"? And if so, who created them and what are their biases?

Re: Kimi-K3 on HuggingFace

#160

Post by walrus01 »

zargon wrote: post #157
Iolaum wrote: post #116

Earlier quoted context omitted.

There's an emerging practice of using Q4 quants and Q8 KV cache for local inference. At that point you can run both Qwen3.5-122B-A10B (my personal choice on Framework Desktop 128gb) and Laguna-S-2.1. Now whether that's good enough for one's use-case remains to be determined. You can also get more out of those (local models and quantizations) if you further tweak the harness you use them with, but tbh this is where it…

> emerging practice of using Q4 quants and Q8 KV cache for local inference That's not an emerging practice, it's a tested strategy that is these days only used as a last resort by those desperate to fit a model in memory. Some models do better than others, but generally the model quality suffers greatly under those conditions.

I have never seen anyone report "this produced really great results" from intentionally quantizing their context vs. leaving it at full precision which is the ordinary default.
Post reply on HN