It's too bad Optane PMem is dead
Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
41–50 of 181 posts
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#42That README hits all my “this is authored by an LLM” instincts. I presume the codebase is also written by an LLM?
Yup, I hate to engage in anything that looks like a "shallow dismissal" but the project documentation seems to outright contradict itself wrt. whether it's running the model at genuinely native precision (though the claimed 3-bit quant is potentially interesting) and the headline claim of achieving 2 secs/token in a mere 29GB RAM footprint looks outright nonsensical given what we know about K3 itself (~115GB in dense parameters alone at native precision, plus ~25GB active sparse experts per token and some comparatively minor footprint for the KV cache). This is just not very helpful.
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#43That README hits all my “this is authored by an LLM” instincts. I presume the codebase is also written by an LLM?
Does it matter?
yes. its a quality test. pre LLMs you could easily judge if a project was a labor of love by attention to details, like docs and README. Nowadays if even readme is sloped, what else was slop vibecoded? everything? how much effort was put in there besides 3 prompts? 5? you can never tell
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#44Neat! But, what do you do with a 0.5tk/s LLM? Have you tried running it via llamacpp or other software that supports naive SSD offloading to compare speeds?
Have it summarise the week overnight for the meeting in the morning. Then have it summarise the meeting transcription overnight for the report tomorrow. Then someone else will have it summarise the report overnight to read on a 6" handheld screen in the small office the next morning after breakfast.
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#45Earlier quoted context omitted.
I saw someone’s excellent idea that if you have a slow system like this, you should communicate by email. It is no longer meant for realtime iteration, but more pointed questions for which there is more effort and time expected on both parties.
0.5t/s is still too slow even for email. For a moderately large inquiry (1MTok output, let's ignore the 4k context window limitation for now) it'll take the model around 23 days or uninterrupted execution to answer a single email. Real world inquiries are gonna be much slower of course, but this setup is still too slow to do anything meaningfully useful I think.
I’m sure it’s possible, but I really struggle to think of an example that would result in a 1m token output.
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#46That README hits all my “this is authored by an LLM” instincts. I presume the codebase is also written by an LLM?
I wrote tons of software, even a programming language by hand https://github.com/marcobambini/gravity . I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster. As developers, we can choose to adapt to new technologies or become extinct.
[dead]
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#47Earlier quoted context omitted.
Yeah I'm begging these authors to at least *read* the LLM generated README's. They're so, so incomprehensible because the LLM has a super limited theory of mind for readers. They always assume that external readers have access to the full context and history of decisions in the project development. These decisions and instructions from the user are extremely important for the model and almost completely irrelevant fo…
>They're so, so incomprehensible because the LLM has a super limited theory of mind for readers. They always assume that external readers have access to the full context and history of decisions in the project development The transformer does not yet understand the non-transformer.[0] This is probably because all the data we trained it on was created by non-transformers, so it thinks it's a non-transformer, but it is…
[dead]
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#48That README hits all my “this is authored by an LLM” instincts. I presume the codebase is also written by an LLM?
>Contributors >... >claude You don't need to presume. If someone is so lazy that they tell claude to commit their code (ie. they're too lazy to run git commit themselves), the chances they reviewed the code is slim.
If my harness wrote the code, I have it write the commit. I will hand author something at a higher level that expresses my intent for how I promoted the harness.
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#49That README hits all my “this is authored by an LLM” instincts. I presume the codebase is also written by an LLM?
I wrote tons of software, even a programming language by hand https://github.com/marcobambini/gravity . I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster. As developers, we can choose to adapt to new technologies or become extinct.
Please consider writing your Readmes by hand even if the code is computer-generated. I want to read what human authors think about their projects. I virtually never want to read what a computer thought.
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#50Earlier quoted context omitted.
I wrote tons of software, even a programming language by hand https://github.com/marcobambini/gravity . I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster. As developers, we can choose to adapt to new technologies or become extinct.
Please consider writing your Readmes by hand even if the code is computer-generated. I want to read what human authors think about their projects. I virtually never want to read what a computer thought.
Also irrespective of the merits of LLMs, it's simply unpleasant to read LLM generated prose. It can't write well. The annoyance is compounded when you read the same poor writing everywhere.
I don't know how people that shovel AI prose don't realise this. Are they not also reading other people's shitty AI text?
I did see one sloperator who told his agent to copy his writing style. I have no idea if that works but it's got to be better than yet more "Here's the kicker" LinkedIn drivel.