Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Kimi-K3 on HuggingFace

huggingface.co

Post reply on HN
Print view View original

491–500 of 588 posts

Re: Kimi-K3 on HuggingFace

#491

Post by hnfong »

cosmojg wrote: post #478
mock-possum wrote: post #392

Earlier quoted context omitted.

Is that even enforceable?

IANAL, but probably not, at least not in the United States. Under U.S. copyright law, the weights of machine learning models are excluded from copyright as they are the product of an automated optimization process (e.g., stochastic gradient descent, expectation maximization, genetic algorithms) rather than human authorship. Granted, this has yet to be fully tested in court and going to court is expensive, so it's lik…

If your argument is true, what would make LLM weights not copyrightable, but compiled code copyrightable?

Re: Kimi-K3 on HuggingFace

#492

Post by georgeven »

embedding-shape wrote: post #43
davidkunz wrote: post #8

This is historic. For the first time, an open-weights LLM is right at the top. We won't be able to run this ourselves, but many providers can.

> For the first time, an open-weights LLM is right at the top. Hmm, not quite true, I think that honor, for better or worse, goes to OpenAI. When they released GPT2 (or GPT1 for that matter) is was quite literally the SOTA in the ecosystem when it was released.

No! because they did not release GPT-2 XL until analogs appeared a year later. So the SOTA model during GPT-2 days was closed weights.

Re: Kimi-K3 on HuggingFace

#493

Post by [deleted] »

NitpickLawyer wrote: post #3

This will be interesting for a few reasons. First, depending on where the median pricing settles w/ 3rd party providers will tell us what it costs to serve a 3T model. Since it's going to be mxfp4 native, it'll take ~1.5TB of VRAM to host this, which is juuust at the limit of 8xb200s (but realistically you'll need 16x for context / throughput optimisation). Won't be cheap to host, but at least we should get some rang…

[deleted]

Re: Kimi-K3 on HuggingFace

#494

Post by acchow »

ThunderSizzle wrote: post #162
acchow wrote: post #106

Earlier quoted context omitted.

> like 180W-250W TDP > running GLM 5.2 would be cool at like ~100 tokens per second for a single session Your power consumption estimates are off for this generation of GPUs. A 27B dense model gets 50-80 tps on an RTX 6000 using 600 watts.

An AMD R9700 gets 20-50 TPS at ~300 watts on 27B. 100 TPS for the 35B MOE model. And there might be some more optimizations to that as AMD software support gets better with ROCm's latest versions.

That's about right. Half the TPS with half the power.

GLM-5.2 will be much more demanding tho

Re: Kimi-K3 on HuggingFace

#495

Post by whs »

skinfaxi wrote: post #386
ljlolel wrote: post #345

Earlier quoted context omitted.

can send safely context if there’s confidential computing ala my site https://trustedrouter.com/

How do you prove you are running exclusively on Nitro enclave instances or GCP confidential spaces?

Seems clear from their website?

1. Their API server provide an attestation JWT. This JWT is signed by Google's private key. 2. The attestation has details on the running container. I suppose the container host is a Google-provided distro and Google's signer will verify that the OS is theirs and up-to-date. 3. They could've proxy the attestation. To prove this is not the case, the field eat_nonce include the TLS certificate fingerprint, which should match the API server you're connecting to. I suppose you will need to pull their container and verify from the source that the container itself generate the private key, it never leaves the container, and the container has no way to run arbitrary code such as SSH or vulnerabilities.

Re: Kimi-K3 on HuggingFace

#496

Post by 93po »

nancyminusone wrote: post #456
fnordpiglet wrote: post #451

I suggest downloading these frontier models just to have a copy; even though it’s 1.5TB, it’s worth sticking in a cheap disk and putting aside. Seeding torrents would be even more useful. The man is coming to lock these down, like they tried to do with encryption algorithms. The only way open software survives regulation is through distribution. Over time the enormous investment in techniques and hardware manufacturi…

they will just restrict you from buying the hardware these run on

Or more likely, put regulations in place so the hardware can only run allowed models/allows surveillance of what's run. They're already doing stuff like this for 3D printers.

Re: Kimi-K3 on HuggingFace

#497

Post by vunderba »

lmf4lol wrote: post #475
MeProtozoan wrote: post #361

It is online on https://app.fireworks.ai/models/fireworks/kimi-k3 (Uncached Input $3.00/M Cached Input $0.30/M Output $15.00/M)

I love fireworks.ai! They launched it couple of hours ago and we have it now already live on our platform for our users. Just a shame they deprecated the on-demand flux models :( Where do I get my fix for image gen now?

Fal.AI is what I use for running proprietary image models through their paces as part of my GenAI benchmark site. Highly recommended.

https://fal.ai/explore

Re: Kimi-K3 on HuggingFace

#498

Post by ElectricalUnion »

magicalhippo wrote: post #80
walrus01 wrote: post #50

Earlier quoted context omitted.

It will be very interesting to see what kind of 'slow' performance people get from running it on a no GPU, but tons of RAM server (like a dual or quad socket xeon with 1.5 to 3TB of RAM). For the purpose of giving it longer duration tasks to generate a piece of something and come back and check on what it has done in 4 or 6 hours. Even if the output is like 5-6 tok/s, that might be usable for some purposes. Huge pric…

> running it on a no GPU, but tons of RAM server Or from SSD using something like Colibri[1]. Not going to be quick, but at least runable. [1]: https://github.com/JustVugg/colibri

Won't the answer (even for a pretty basic message like "hi") at SSD speeds take like a _entire week_ to _start showing useful output?_ (attempting to do 22k average claude code system prompt + 32k thinking tokens thru 0.1t/s throughput)

Re: Kimi-K3 on HuggingFace

#499

Post by jacobpfeifer »

soerxpso wrote: post #486
schmorptron wrote: post #183

Earlier quoted context omitted.

LLM inference unfortunately also seems to be a task that's poorly formed for moderate consumer hardware,as a single user. For a single user use case, the load is bursty but requires the weights to be in memory already. So a multi user server that keeps the model weights in parts of its memory and then spends some more per user kv cache is wildly more efficient and the wildly expensive gpu cores aren't just sitting id…

A decentralized inference network would be cool. Something that's set up so that I can run a model for personal use on beefy hardware, but also farm out the unused GPU time to the network, probably at much lower prices than normal providers since it would be slower and would lack data security guarantees.

Not quite what you are asking for. Instead of selling the unused GPU, you're donating it to the common good.

https://cocore.dev/

Re: Kimi-K3 on HuggingFace

#500

Post by api »

mwcampbell wrote: post #446
api wrote: post #375

Earlier quoted context omitted.

Sometimes I’m not sure who is more unhinged: the total AI kool aid drinkers who think this will make us all into immortal demigods (or take over the world as it goes “foom”), or the AI doomers and haters who exaggerate everything potentially negative about it and react to it the way a 1980s Christian fundamentalist reacted to rock music. It’s a new fundamental innovation in math and CS that allows large scale lossy c…

To steelman the haters, I think their view is that the industry is so uniquely shitty that it's unconscionable to help the industry at all by using the tech, which is a product of that industry.

It’s far less shitty than the social media industry (except where it overlaps) but that’s IMO.

Still kind of shitty. But if you really hate it use open models hosted commodity.

Post reply on HN