Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

github.com

Post reply on HN
Print view View original

61–70 of 181 posts

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#61

Post by jv-k »

Wowfunhappy wrote: post #49
marcobambini wrote: post #13

Earlier quoted context omitted.

I wrote tons of software, even a programming language by hand https://github.com/marcobambini/gravity . I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster. As developers, we can choose to adapt to new technologies or become extinct.

Please consider writing your Readmes by hand even if the code is computer-generated. I want to read what human authors think about their projects. I virtually never want to read what a computer thought.

This. Code written by LLMs using the new agentic SDLC is fine, can even be good code. But we shouldn't expect users to read sloppy slop, it's just plain lazy.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#62

Post by jv-k »

pja wrote: post #4

That README hits all my “this is authored by an LLM” instincts. I presume the codebase is also written by an LLM?

Absolutely. Anyone can remove flag and remove typical LLM slop at least. I wrote a simple tool for it as part of my CI: https://github.com/jv-k/deslopper

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#65

Post by airstrafer »

OutOfHere wrote: post #63

Be advised that the firm behind it ("sqliteai") had a nasty history of using non-open source licenses, e.g. Elastic License. I advise against using anything by them for this reason even if this project currently has an open license.

Are these the sqlite developers..?

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#66

Post by AnotherGoodName »

bgirard wrote: post #29

Approximate calculation is putting the cost at ~$5 per million tokens (assuming 42W sustained, 20¢/kWh), and that's excluding hardware and other costs.

I got to the same conclusion another way. There's ~2.6million seconds a month and this is getting 0.5tok/s which is 1.3million tokens a month. Give some room for overhead and a reasonable rule of thumb;

The cost to run the machine per month is the cost per million tokens.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#67

Post by zeryx »

IshKebab wrote: post #50
Wowfunhappy wrote: post #49

Earlier quoted context omitted.

Please consider writing your Readmes by hand even if the code is computer-generated. I want to read what human authors think about their projects. I virtually never want to read what a computer thought.

Also irrespective of the merits of LLMs, it's simply unpleasant to read LLM generated prose. It can't write well. The annoyance is compounded when you read the same poor writing everywhere . I don't know how people that shovel AI prose don't realise this. Are they not also reading other people's shitty AI text? I did see one sloperator who told his agent to copy his writing style. I have no idea if that works but it'…

Rewrite the readme in my voice, use all of my comments and responses in current context as source. Don't use em dashes or other LLM things.

Done, solved. Never had a problem with a readme or email since.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#68

Post by Dylan16807 »

andai wrote: post #28
bglazer wrote: post #19

Earlier quoted context omitted.

Yeah I'm begging these authors to at least *read* the LLM generated README's. They're so, so incomprehensible because the LLM has a super limited theory of mind for readers. They always assume that external readers have access to the full context and history of decisions in the project development. These decisions and instructions from the user are extremely important for the model and almost completely irrelevant fo…

>They're so, so incomprehensible because the LLM has a super limited theory of mind for readers. They always assume that external readers have access to the full context and history of decisions in the project development The transformer does not yet understand the non-transformer.[0] This is probably because all the data we trained it on was created by non-transformers, so it thinks it's a non-transformer, but it is…

I think the models would need far far more introspection for the problem to be it understanding how it thinks but not how others think. I really doubt it understands how it thinks.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#70

Post by Dylan16807 »

ElectricalUnion wrote: post #54
dotancohen wrote: post #44

Earlier quoted context omitted.

Have it summarise the week overnight for the meeting in the morning. Then have it summarise the meeting transcription overnight for the report tomorrow. Then someone else will have it summarise the report overnight to read on a 6" handheld screen in the small office the next morning after breakfast.

Then have it summarise the meeting transcription over the entire week for the report next week . ftfy.

If we estimate a meeting with pauses between speakers as 2.25 words per second, and .75 words per token, then a meeting generates 3 tokens per second. This says prefill and decode are both .5 tokens per second? Then each hour of meeting turns into 6 hours to read and 1 hour to output a summary. You could summarize two hours of meeting overnight, not too bad.
Post reply on HN