171–180 of 251 posts
Earlier quoted context omitted.
Hear, hear. Even if the model fits, a few tokens per second make no sense. Time is money too.
Maybe for a coding agent, but a daily/weekly report on sensitive info? If it were 2016 and this technology existed but only in 1 t/s, every company would find a way to extract the most leverage out of it.
Huggingface Link: https://huggingface.co/moonshotai/Kimi-K2.5 1T parameters, 32b active parameters. License: MIT with the following modification: Our only modification part is that, if the Software (or any derivative works thereof) is used for any of your commercial products or services that have more than 100 million monthly active users, or more than 20 million US dollars (or equivalent in other currencies) in mont…
Earlier quoted context omitted.
The model absolutely can be run at home. There even is a big community around running large models locally: https://www.reddit.com/r/LocalLLaMA/ The cheapest way is to stream it from a fast SSD, but it will be quite slow (one token every few seconds). The next step up is an old server with lots of RAM and many memory channels with maybe a GPU thrown in for faster prompt processing (low two digits tokens/second). At t…
Its often pointed out in the first sentence of a comment how a model can be run at home, then (maybe) towards the end of the comment it’s mentioned how it’s quantized. Back when 4k movies needed expensive hardware, no one was saying they could play 4k on a home system, then later mentioning they actually scaled down the resolution to make it possible. The degree of quality loss is not often characterized. Which makes…
Earlier quoted context omitted.
Hear, hear. Even if the model fits, a few tokens per second make no sense. Time is money too.
Maybe for a coding agent, but a daily/weekly report on sensitive info? If it were 2016 and this technology existed but only in 1 t/s, every company would find a way to extract the most leverage out of it.
Congratulations, great work Kimi team. Why is that Claude still at the top in coding, are they heavily focused on training for coding or is it their general training is so good that it performs well in coding? Someone please beat the Opus 4.5 in coding, I want to replace it.
Gemini 3 pro is way better than Opus especially for large codebases.
Earlier quoted context omitted.
You might want to clarify that this is more of a "Look it technically works" Not a "I actually use this" The difference between waiting 20 minutes to answer the prompt '1+1=' and actually using it for something useful is massive here. I wonder where this idea of running AI on CPU comes from. Was it Apple astroturfing? Was it Apple fanboys? I don't see people wasting time on non-Apple CPUs. (Although, I did do this fo…
The reason Macs get recommended is the unified memory, which is usable as VRAM for the GPU. People are similarly using the AMD Strix Halo for AI which also has a similar memory architecture. Time to first token for something like '1+1=' would be seconds, and then you'd be getting ~20 tokens per second, which is absolutely plenty fast for regular use. Token/s slows down at the higher end of context, but it's absolutel…
Anyway, we were trying to run a 70B model on a macbook(can't remember which M model) at a fortune 20 company, it never became practical. We were trying to compare strings of character length ~200. It was like 400-ish characters plus a pre-prompt.
I can't imagine this being reasonable on a 1T model, let alone the 400B models of deepseek and LLAMA.
Earlier quoted context omitted.
You might want to clarify that this is more of a "Look it technically works" Not a "I actually use this" The difference between waiting 20 minutes to answer the prompt '1+1=' and actually using it for something useful is massive here. I wonder where this idea of running AI on CPU comes from. Was it Apple astroturfing? Was it Apple fanboys? I don't see people wasting time on non-Apple CPUs. (Although, I did do this fo…
Mac studio way is not "AI on CPU," as M2/M4 are complex SoC, that includes a GPU with unified memory access.
Language is full of issues of particulars vs universals, and you could debate if its just an integrated GPU with different marketing.
Whatever the case, we couldn't use it in production, and NVIDIAs stock price reflects the reality on the ground.
Earlier quoted context omitted.
Models of this size can usually be run using MLX on a pair of 512GB Mac Studio M3 Ultras, which are about $10,000 each so $20,000 for the pair.
You might want to clarify that this is more of a "Look it technically works" Not a "I actually use this" The difference between waiting 20 minutes to answer the prompt '1+1=' and actually using it for something useful is massive here. I wonder where this idea of running AI on CPU comes from. Was it Apple astroturfing? Was it Apple fanboys? I don't see people wasting time on non-Apple CPUs. (Although, I did do this fo…
That said, I wouldn't necessarily recommend spending $20,000 on a pair of Mac Studios to run models like this. The performance won't be nearly as good as the server-class GPU hardware that hosted models run on.
Earlier quoted context omitted.
The reason Macs get recommended is the unified memory, which is usable as VRAM for the GPU. People are similarly using the AMD Strix Halo for AI which also has a similar memory architecture. Time to first token for something like '1+1=' would be seconds, and then you'd be getting ~20 tokens per second, which is absolutely plenty fast for regular use. Token/s slows down at the higher end of context, but it's absolutel…
We are getting into a debate between particulars and universals. To call the 'unified memory' VRAM is quite a generalization. Whatever the case, we can tell from stock prices that whatever this VRAM is, its nothing compared to NVIDIA. Anyway, we were trying to run a 70B model on a macbook(can't remember which M model) at a fortune 20 company, it never became practical. We were trying to compare strings of character l…
Have you all noted that the latest releases (Qwen3 max thinking, now Kimi k2.5) from Chinese companies are benching against Claude opus now and not Sonnet? They are truly catching up, almost at the same pace?
They distill the major western models, so anytime a new SOTA model drops, you can expect the Chinese labs to update their models within a few months.