Ok, this one will take just 30h (compared to that other project that would take 6.25 days) to start writing output tokens after you say hi in Claude Code.
Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
51–60 of 181 posts
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#52Neat! But, what do you do with a 0.5tk/s LLM? Have you tried running it via llamacpp or other software that supports naive SSD offloading to compare speeds?
You could use it for long run tasks while you don’t use the laptop.
By the time the tokens start coming out 30h later you might need to use your laptop again...
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#53Fun. I see the novelty.
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#54Neat! But, what do you do with a 0.5tk/s LLM? Have you tried running it via llamacpp or other software that supports naive SSD offloading to compare speeds?
Have it summarise the week overnight for the meeting in the morning. Then have it summarise the meeting transcription overnight for the report tomorrow. Then someone else will have it summarise the report overnight to read on a 6" handheld screen in the small office the next morning after breakfast.
Then have it summarise the meeting transcription over the entire week for the report next week.
ftfy.
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#55Earlier quoted context omitted.
0.5t/s is still too slow even for email. For a moderately large inquiry (1MTok output, let's ignore the 4k context window limitation for now) it'll take the model around 23 days or uninterrupted execution to answer a single email. Real world inquiries are gonna be much slower of course, but this setup is still too slow to do anything meaningfully useful I think.
I’m sure it’s possible, but I really struggle to think of an example that would result in a 1m token output.
it would be 1M tokens worth of work, with some small report for the end
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#56Dear creator: you didn't ship the first draft of your code - why did you ship the first draft of your README??
because claude ships these verbose READMEs with its 'honest' takes and justifications for the naming. Its goal is to prime the next Agent that reads it, not you, human
we need a new HUMANS.md to replace the readme
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#57That README hits all my “this is authored by an LLM” instincts. I presume the codebase is also written by an LLM?
Yes, but very slowly.
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#58looks helpful
your readme is overly verbose
agents don't need that and its extremely low signal for humans too
tell your language model to get it to the point
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#59Earlier quoted context omitted.
Please consider writing your Readmes by hand even if the code is computer-generated. I want to read what human authors think about their projects. I virtually never want to read what a computer thought.
Also irrespective of the merits of LLMs, it's simply unpleasant to read LLM generated prose. It can't write well. The annoyance is compounded when you read the same poor writing everywhere . I don't know how people that shovel AI prose don't realise this. Are they not also reading other people's shitty AI text? I did see one sloperator who told his agent to copy his writing style. I have no idea if that works but it'…
Even worse than that, it's not the LLM, it's the operator. Mimicking your own style given a corpus is the lowest possible bar - you can also select among the great writers (technical or not) from history and have it give you eerily correct stylistic cues. Try Hemingway or Mark Twain next time you feel like generating a doc. Benjamin Franklin if you're feeling particularly Poor Richard.
There's no reason any more to read bad documentation. You can feed any docs in and get them in the precise style you want, so don't impose bad prose on others in your repos. Pick a solid technical writing base, jargon free, and go from there. I sometimes generate architecture proposals from the llm in RFC format and it's word perfect.
Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s
#60Earlier quoted context omitted.
I saw someone’s excellent idea that if you have a slow system like this, you should communicate by email. It is no longer meant for realtime iteration, but more pointed questions for which there is more effort and time expected on both parties.
0.5t/s is still too slow even for email. For a moderately large inquiry (1MTok output, let's ignore the 4k context window limitation for now) it'll take the model around 23 days or uninterrupted execution to answer a single email. Real world inquiries are gonna be much slower of course, but this setup is still too slow to do anything meaningfully useful I think.
Sure, you cannot do 1M, but there are plenty of useful questions you could ask that are far more modest. Simple Q+A, look at this function, how would you design X? All of those could have few paragraphs of outputs that would finish within a day.