Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

github.com

Post reply on HN
Print view View original

161–170 of 181 posts

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#161

Post by dwedge »

marcobambini wrote: post #13
pja wrote: post #4

That README hits all my “this is authored by an LLM” instincts. I presume the codebase is also written by an LLM?

I wrote tons of software, even a programming language by hand https://github.com/marcobambini/gravity . I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster. As developers, we can choose to adapt to new technologies or become extinct.

Look I'll be honest in that I didn't read your code (or even open the link) so this totally might not apply to you, but:

> I'm using my skills to orchestrate LLMs and agents, and I can write better code much faster.

Everyone says that. And most of them are wrong about the better part. Referring to your previous ability to write code is almost an appeal to authority at this point because it's coming with the assumption that you review the code with he same vigor, and that you are just as good as you were before you offloaded these tasks regularly to LLMs. Maybe you do and maybe you are, but I will say that a lot of the people I've worked with who I used to respect and whose code was good now produce absolute crap. The models are getting better so their code is getting a bit better, but I no longer believe they had anything to do with the quality, they no longer seem to care.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#162

Post by matheusmoreira »

IshKebab wrote: post #131
matheusmoreira wrote: post #111

Earlier quoted context omitted.

No idea why you're getting downvoted. Asking the AI to just copy my writing patterns dramatically reduced the amount of code comment and commit message rewriting I had to do. Ten years worth of HN comments finally became useful for something!

> Ten years worth of HN comments finally became useful for something! This is a great idea! How do you do it practically? I assume you don't feed the whole corpus in every prompt - do you condense it somehow?

I just created a ~/social directory, launched Claude Code in it and told Claude I'd like to start tracking and studying my own social media presence. Before long there were scripts to dump all of my public posts into a SQLite database. I pulled HN, reddit, lobsters, stackoverflow and its sister sites, even mailing lists. Given a corpus inside a database, any sort of analysis can be done.

I started with linguistic analysis to reduce commit message rewriting. That one also used years worth of commit messages as input.

I immediately started doing more though. Sentiment analysis, interlocutors, negativity, positivity, recurring arguments, best and worst threads by voting patterns.

Discovered a lot of fun stuff. For example, it was hilarious and a little terrifying to realize I was randomly interacting with some industry heavy hitters without even realizing who they were. Claude on the other hand seems to know quite a bit about many people on HN.

Mapped out all of my free and open source software contributions. Counted my successes and failures, did post-mortems, saw what I did right and where I could improve. Got some very useful results out of that. For example, I will never attempt to contribute to GNU ever again.

Ask it to steelman every single argument I get into and every single worldview I have. Turned up some excellent literature too. This is great for sharpening my mind, and sometimes Claude even changes my mind.

The database also acts as a preservation archive in case any of these sites get wiped out.

It's also the best search engine I've ever used. Saying something like "I remember posting something along the X and Y lines" is enough to find pretty much anything.

Gets downright nostalgic at times. Claude found my GitHub projects dating to my teenage years.

> I assume you don't feed the whole corpus in every prompt

Claude has the local comment database and runs SQL against it to pull whatever it wants.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#163

Post by IshKebab »

knollimar wrote: post #150
IshKebab wrote: post #146

Earlier quoted context omitted.

But unless you're off-grid you can always use extra PV energy by selling it to the grid.

The grid doesn't give you money in these locations. They give you credits for future consumption. You can only spend this on energy. If you always have a surplus, you will just rack up large amounts of credits.

What locations? In the UK they give you actual money. You can withdraw it.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#164

Post by make3 »

fennecbutt wrote: post #121

Idk about you guys but I'd find 0.5t/s useless. Even for long tasks. I'd rather just shell out the money to offload as much as possible to say 2x 4060ti 16gb with tensor parallelisation. Anything but that low token rate. This is the sort of thing I'd expect in 20 years for some cyberpunk esque "turtlebot" that thinks at 0.5t/s, is solar powered and performs some menial civic maintenance background task like cutting g…

paying for the hardware is also too expensive for no reason, you're not querying the llm 24/7. Better use a service that hosts a lot of popular open weights LLMs, unless you have a very good reason not to.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#165

Post by make3 »

Garlef wrote: post #126
ikurei wrote: post #107

They say it's a waste that you pay for the tokens and then the inference provider pays for the electricity. Isn't that how everything works? I pay cucumbers and the farmers have to pay for the water and the fertilizer... I hope that reasoning is an after-the-fact justification by the LLM that wrote this. It's a ver interesting idea and I wouldn't mind trying it out, but with a smaller model. At 0.5t/s and reading man…

If you grow your own tomatoes, you'll have free tomatoes! (Doesn't really get you a BLT but hey... at least you'll have saved the world a bit because they're not from the supermarket) /s

here at 0.5 tok/s, it's like getting 1/100 of a tomato for your lunch per hour, and you can't just let it grow, you have to look at it growing just as you're about to eat, and it doesn't accumulate between meals

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#167

Post by Lerc »

fennecbutt wrote: post #121

Idk about you guys but I'd find 0.5t/s useless. Even for long tasks. I'd rather just shell out the money to offload as much as possible to say 2x 4060ti 16gb with tensor parallelisation. Anything but that low token rate. This is the sort of thing I'd expect in 20 years for some cyberpunk esque "turtlebot" that thinks at 0.5t/s, is solar powered and performs some menial civic maintenance background task like cutting g…

It's about a million tokens a week. So you can do the usual calculations of rent Vs buy.

There are definitely some tasks, that if the system can run unsupervised (a largeish if), it doesn't matter as long as the result happens before a deadline.

In that respect it is easy to tell if this works for you or not. As a stepping stone to more efficiency in the future it has more value.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#168

Post by rcxdude »

ikurei wrote: post #107

They say it's a waste that you pay for the tokens and then the inference provider pays for the electricity. Isn't that how everything works? I pay cucumbers and the farmers have to pay for the water and the fertilizer... I hope that reasoning is an after-the-fact justification by the LLM that wrote this. It's a ver interesting idea and I wouldn't mind trying it out, but with a smaller model. At 0.5t/s and reading man…

It is worth thinking about, when you're buying something, how much of what you're paying is the supplier's margin. And whether they have efficiency advantages over you doing it yourself. But for large LLMs it does seem like there's a pretty big efficiency advantage to the rack-scale hardware in datacenters compared to hacks like this.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#169

Post by danw1979 »

Lerc wrote: post #167
fennecbutt wrote: post #121

Idk about you guys but I'd find 0.5t/s useless. Even for long tasks. I'd rather just shell out the money to offload as much as possible to say 2x 4060ti 16gb with tensor parallelisation. Anything but that low token rate. This is the sort of thing I'd expect in 20 years for some cyberpunk esque "turtlebot" that thinks at 0.5t/s, is solar powered and performs some menial civic maintenance background task like cutting g…

It's about a million tokens a week. So you can do the usual calculations of rent Vs buy. There are definitely some tasks, that if the system can run unsupervised (a largeish if), it doesn't matter as long as the result happens before a deadline. In that respect it is easy to tell if this works for you or not. As a stepping stone to more efficiency in the future it has more value.

That’s a good way to think about it - do you want your expensive laptop grinding away constantly 24/7 consuming $5 of power to generate what amounts to $15 worth of tokens per week ?

You can get a frontier model subscription for about the same cost as the electricity (heavily subsidised by someone else’s money !) with instant results.

I know there are applications for this and it’s cool people are pushing the boundaries with local models. I run oMLX and Qwen myself, but it’s a toy really. It’s not anywhere near replacing Claude for my purposes.

Re: Run Kimi K3 using 29 GB of RAM at 0.50 tok/s

#170

Post by root_axis »

preommr wrote: post #147
fennecbutt wrote: post #121

Idk about you guys but I'd find 0.5t/s useless. Even for long tasks. I'd rather just shell out the money to offload as much as possible to say 2x 4060ti 16gb with tensor parallelisation. Anything but that low token rate. This is the sort of thing I'd expect in 20 years for some cyberpunk esque "turtlebot" that thinks at 0.5t/s, is solar powered and performs some menial civic maintenance background task like cutting g…

I pay $20 for codex, use it daily for coding, and still haven't dipped below 50% for weekly usage. I wouldn't even be able to afford buying 29 gigs of ram, or a new video card with hardware prices the way they are now. Maybe if it was 2016-2018 prices, I'd think about it.

Well, if you only ever needed a $20 subscription's worth of tokens it would never make any sense to buy hardware since even at 2018 prices you could buy 20 years of subscriptions.
Post reply on HN