Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

github.com

Post reply on HN
Print view View original

111–120 of 181 posts

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#111

Post by matheusmoreira »

zeryx wrote: post #67
IshKebab wrote: post #50

Earlier quoted context omitted.

Also irrespective of the merits of LLMs, it's simply unpleasant to read LLM generated prose. It can't write well. The annoyance is compounded when you read the same poor writing everywhere . I don't know how people that shovel AI prose don't realise this. Are they not also reading other people's shitty AI text? I did see one sloperator who told his agent to copy his writing style. I have no idea if that works but it'…

Rewrite the readme in my voice, use all of my comments and responses in current context as source. Don't use em dashes or other LLM things. Done, solved. Never had a problem with a readme or email since.

No idea why you're getting downvoted. Asking the AI to just copy my writing patterns dramatically reduced the amount of code comment and commit message rewriting I had to do. Ten years worth of HN comments finally became useful for something!

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#112

Post by matheusmoreira »

bglazer wrote: post #19
pja wrote: post #4

That README hits all my “this is authored by an LLM” instincts. I presume the codebase is also written by an LLM?

Yeah I'm begging these authors to at least *read* the LLM generated README's. They're so, so incomprehensible because the LLM has a super limited theory of mind for readers. They always assume that external readers have access to the full context and history of decisions in the project development. These decisions and instructions from the user are extremely important for the model and almost completely irrelevant fo…

Yeah, I agree with you. The one time I tried to generate technical documentation for my project, I ended up rewriting almost all of the LLM output. They're extremely verbose, and needlessly so. Fable takes it up to eleven by having an obtuse sentence structure.

The thing about documentation though is humans won't actually read any of it. Maybe tailoring the documentation to the needs of LLMs isn't so bad since they're the ones who will actually consume all of those documents.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#114

Post by darkwater »

IshKebab wrote: post #110
darkwater wrote: post #104

Earlier quoted context omitted.

And what if I have PV?

In the UK you can sell electricity domestically for 20c/kWh so at least here that's the cost if you have PV. If you don't, the cost is higher.

It's not a cost if you self-consume your energy, even if you get paid for what you inject. You are just amortizing differently the cost of the PV installation. But what you get for the energy you sell is not real money, it's just money you MIGHT consume as energy in other periods.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#115

Post by dannyw »

darkwater wrote: post #114
IshKebab wrote: post #110

Earlier quoted context omitted.

In the UK you can sell electricity domestically for 20c/kWh so at least here that's the cost if you have PV. If you don't, the cost is higher.

It's not a cost if you self-consume your energy, even if you get paid for what you inject. You are just amortizing differently the cost of the PV installation. But what you get for the energy you sell is not real money, it's just money you MIGHT consume as energy in other periods.

Actually, saving $1 on something you would pay anyway is more valuable than earning $1 of taxable income.

I think that's just semantics -- unless you genuinely keep racking up non-redeemable energy bill credits for feed-in.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#117

Post by tim-projects »

Once the tech catches up to the point that we can accurately select the right model for the task, then this ends up becoming a valuabke thing to have. You would spin it up sparingly as part of an automated discovery process maybe for 30 mins a day. And the rest of the time is spent using tiny models.

I could see a future like that.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#118

Post by freedomben »

forty wrote: post #96
freedomben wrote: post #36

Earlier quoted context omitted.

I often let Claude write my commit messages even when I'm the one who wrote the code. Claude is often damn good at writing commit messages, and they frequently end up much better than if I wrote it all by hand. I nearly always edit them somewhat, but it's like starting from 80% instead of 0%. Some might call it laziness, but I call it working smarter rather than harder.

A good commit message needs to explain why this change was needed/done rather than what is the content of the commit. Unless Claude also has access to the context via for example the associated issue, it simply cannot write a good commit message since it cannot generally guess why something was done from just the change. This is also the case for humans and the reason why you need to include this in the commit messag…

I fully agree, but you'd be amazed at how good Claude is at figuring out the why based on the what. I would never have believed it if I hadn't tried it myself. If it doesn't get the why, then I will typically add it

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#120

Post by adrian_b »

walrus01 wrote: post #84
mappu wrote: post #81

Standard llama.cpp can mmap the gguf, so it'll stay on disk if it doesn't fit on memory, and the kernel page cache will ensure the hot parts ("resident trunk") stay resident. What's the benefit of a custom implementation at all?

Letting the kernel use SSD based swap space for something this big would be a good way to destroy its cumulative write endurance over a period of just a couple months. I would be very interested in seeing SMART self reported drive cumulative write and wear out stats if this was done for more than a short test. In my experience llama-server is better run with --no-mmap on things that will fit entirely into RAM. Though…

It is much better to not use any swap in Linux, so you will never have this probem, or any other problem caused by swap.

I stopped using swap on Linux about a quarter of century ago, when it was a great improvement, and since then I have never seen a case when swap would have been useful, and I use Linux on a variety of laptops, desktops and servers.

Even if you do not use swap, you can have memory-mapped files that are much bigger than your physical memory. The LLM weights files must be mapped as read-only. In this case, the pages that have not been used recently will be freed when memory is needed to load other pages from the files.

The weights files must be mapped using huge pages, otherwise an excessive amount of physical memory would be wasted and reading new pages would be very slow.

It is likely that it is not possible to reach a good enough performance with a memory-mapped file without using carefully "madvise", with which it is possible to force the reading of the pages that you know that they will be needed in the future and also the freeing of the pages that you know that they will not be needed soon.

On Linux, it is possible to execute "madvise" asynchronously (with liburing). An alternative to liburing is to execute "madvise" from a concurrent thread, which synchronizes with requests to do "madvise" from the orchestrating thread.

Post reply on HN