Earlier quoted context omitted.
This is incredible! Are there other big players in this space (freezing models to silicon)?
take a look at Cerebras, who are doing wafer-scale compute
I imagine Astra is/will soon will be on Cerebras?
Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
121–130 of 154 posts
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#122Earlier quoted context omitted.
I think the point is that it’s running at all…
It actully does the job. Example: every morning it takes 30-40 minutes to generate reports automatically and these reports are being sent as a pdf to read to Telegram.
Do those reports require Kimi K3 though? Qwen3.6+ could probably do the same in a few seconds with similar quality.
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#123Earlier quoted context omitted.
I would agree, but I want to add that I have real issues with combination of opencode plus slow inference (4-5tok/s). I get weird interruptions. I can only guess its related to some kind of timeouts in the harness or something. Its not a problem of the model of course, but it seems impractical atm. I wonder if anyone else had this kind of thing happening.
I think pi handles it better
I just tried pi, absolute junk compared to hermes
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#124There's a tendency to cite only decode speeds, but in practice, an LLM generates far fewer tokens than it has to read (unless you ask general knowledge questions). So the effective performance is much slower than the decode rate suggests, because a 512-token prompt already takes 6 minutes to load
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#125There's a tendency to cite only decode speeds, but in practice, an LLM generates far fewer tokens than it has to read (unless you ask general knowledge questions). So the effective performance is much slower than the decode rate suggests, because a 512-token prompt already takes 6 minutes to load
Generally infill is significantly faster than inference due to batching. Is that not the case here for some reason?
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#126There's a tendency to cite only decode speeds, but in practice, an LLM generates far fewer tokens than it has to read (unless you ask general knowledge questions). So the effective performance is much slower than the decode rate suggests, because a 512-token prompt already takes 6 minutes to load
Generally infill is significantly faster than inference due to batching. Is that not the case here for some reason?
They mention it here: https://github.com/argonautlabsai/deltafin/blob/main/k3-publ...
>device bytes read during the prefill window, all four drives (arm csv) 8,977 GB at 24.1 GB/s aggregate
I.e. low memory bandwidth.
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#127Earlier quoted context omitted.
To the extent that cryptocurrency moved off ASICs, it was because of interest shifting to different cryptocurrencies that were specifically designed to be harder to mine on an ASIC than Bitcoin's compute-heavy, memory-light hashing. I'm not sure there's any reason to expect a similar shift from LLMs. The hardware used for training doesn't dictate what hardware needs to be used for inference, and nobody's going to des…
Yet it doesn't seem that ASICs will have any particular advantage over consumer hardware since AI is very memory heavy, which is (right now) expensive no matter how you package it. And the compute is just simple matrix multiplication, which is almost entirely what GPUs were meant to do anyway.
ASIC vs GPU doesn't make a ton of difference when both are relying on commodity DRAM; in that sense, LLMs are more like the anti-ASIC cryptocurrencies. But the actually interesting ASICs are the ones that ditch the commodity discrete DRAM chips. They lose out on the memory density and thus struggle to scale up to the largest models, but for what does fit onto a Cerebras wafer or a Taalas chip, the speed is phenomenal. They have a real shot at securing the "smart enough, and really fast" segment of the market.
And it seems more plausible to me that an ASIC architecture rather than GPUs would be able to best make use of something like wafer-bonded custom memory to approach the density of discrete DRAM while retaining the extremely high bandwidth that comes with arbitrarily wide interfaces and minimal PHYs.
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#128Earlier quoted context omitted.
To be fair, nobody has upgradable memory in any system that has enough memory bandwidth and compute power to run LLMs with decent performance. It might be interesting to compare against some decade-old x86 server or workstation stuffed full of LRDIMMs to reach 1.5–2TB of RAM, but the bandwidth would be only slightly faster than a desktop today with high-end DDR5: nowhere close to GPU bandwidth. So performance would s…
Take a look at AMD’s 12-channel memory servers. The newer Epycs are up to 16-channels now, 1.6TB/s. Pretty great for inference.
Sure, if you want to make a comparison where the price tags aren't the same order of magnitude, then a recent server is obviously going to be powerful. But since the baseline of this comparison is a laptop and several Thunderbolt SSDs, the kind of servers or workstations with 1.5–2TB of RAM that you can reasonably compare against would have to be the really old ones, barely new enough to support that much total RAM.
And despite the theoretically high memory bandwidth of recent EPYC CPUs, approximately nobody who can afford one is doing LLM inference on them.
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#129But why though? Cannot possibly be useful at such slow speeds, and costs a ton to perform that badly
I've never understood why "Hacker" News so frequently gets "But why though?" comments at the top. The entire history of innovation is filled with people doing something just to see they can get it to work, even if badly, and then people continue to iterate on that until it works better, then works well, and then is so obvious people would never even question it. But it all starts with someone doing it to scratch an i…
Some people play video games as a hobbies, others watch Netflix, some decide to run frontier models on hardware that's not designed for it.
"We choose to go to the moon in this decade and do the other things, not because they are easy, but because they are hard".
Re: Kimi K3 (2.8T) at 1 token/s on a MacBook Pro, streamed from four SSDs
#130[dead]