Standard llama.cpp can mmap the gguf, so it'll stay on disk if it doesn't fit on memory, and the kernel page cache will ensure the hot parts ("resident trunk") stay resident.
What's the benefit of a custom implementation at all?
Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
81–90 of 181 posts
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#82[flagged]
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#83Earlier quoted context omitted.
This is not how thinking works. Claude uses tens of thousands of thinking tokens to get to 100 words. Kimi is no different.
Additionally, GP can use the word ‘concise’ (or similar) in their prompt if they want more concise output from a model.
That doesn’t mean “use fewer thinking tokens”. It might use more as it mulls over how to make its response concise.
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#84Standard llama.cpp can mmap the gguf, so it'll stay on disk if it doesn't fit on memory, and the kernel page cache will ensure the hot parts ("resident trunk") stay resident. What's the benefit of a custom implementation at all?
Letting the kernel use SSD based swap space for something this big would be a good way to destroy its cumulative write endurance over a period of just a couple months. I would be very interested in seeing SMART self reported drive cumulative write and wear out stats if this was done for more than a short test.
In my experience llama-server is better run with --no-mmap on things that will fit entirely into RAM. Though obviously you need a 2TB server for full Kimi k3 and 1M context.
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#85Be advised that the firm behind it ("sqliteai") had a nasty history of using non-open source licenses, e.g. Elastic License. I advise against using anything by them for this reason even if this project currently has an open license.
[dead]
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#86That README hits all my “this is authored by an LLM” instincts. I presume the codebase is also written by an LLM?
i thought, here on HN, we were past the "oooh it's written by a LLM it's bad!". I care about the craft, well designed systems, good clean architecture and code, etc... But i also care about reaching goals. Whether i do it working on my own, or with human coworkers or with AI coworkers doesn't matter that much to me. Yes, the result is sometimes the most important thing.
> I care about the craft
No you don't. Buying a table and sanding the edges off doesn't mean you're a carpenter.
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#87Earlier quoted context omitted.
Additionally, GP can use the word ‘concise’ (or similar) in their prompt if they want more concise output from a model.
That doesn’t mean “use fewer thinking tokens”. It might use more as it mulls over how to make its response concise.
effort level is a separate parameter, and is probably what they want
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#88Earlier quoted context omitted.
>Contributors >... >claude You don't need to presume. If someone is so lazy that they tell claude to commit their code (ie. they're too lazy to run git commit themselves), the chances they reviewed the code is slim.
That’s a needlessly antagonistic and insulting thing to say. This person that you’ve never met and probably never will doesn’t owe any of us anything. They’re out there doing what they want to do how they want to do it and if you don’t like it the correct response isn’t to insult them in front of a bunch of strangers on the internet for clout or whatever. I doubt that you’d ever call them lazy to their face — why do…
> I doubt that you’d ever call them lazy to their face — why do it here?
What makes you so sure about that?
Posting LLM generated obviously un-reviewed slop and wasting people's time deserves scorn.
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#89Earlier quoted context omitted.
>Contributors >... >claude You don't need to presume. If someone is so lazy that they tell claude to commit their code (ie. they're too lazy to run git commit themselves), the chances they reviewed the code is slim.
Honestly, Claude writes better commit messages than most people. Personally I've mostly given in to letting it commit for me now, though I do occasionally take over and hand-write the messages if it's a particularly important concept and Claude's is too verbose. Codex/GPT-x defaults to one-line commit messages, which are too short. Claude likes to write several paragraphs, which is usually too long. If you tell it ho…
The agents write much better commit messages than me, at least.
They also write better PR descriptions. At work we have a PR authoring skill and we’ve included an instruction for it to write a reviewer guide that tells the most logical way to review the code and I’ve found that really helps, so much so that I’m creating a tool to have “literate” PR reviews, where it constructs a narrative interwoven with diffs.
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#90Standard llama.cpp can mmap the gguf, so it'll stay on disk if it doesn't fit on memory, and the kernel page cache will ensure the hot parts ("resident trunk") stay resident. What's the benefit of a custom implementation at all?
Letting the kernel use SSD based swap space for something this big would be a good way to destroy its cumulative write endurance over a period of just a couple months. I would be very interested in seeing SMART self reported drive cumulative write and wear out stats if this was done for more than a short test. In my experience llama-server is better run with --no-mmap on things that will fit entirely into RAM. Though…
The really big thing is the model weights, which are a read workload not a write one, it won't affect an SSD's write endurance.
The write workloads are just the context and any K/V cache - llama.cpp does not mmap those to disk, so they would remain in memory or VRAM as space affords.