Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: sllm – Split a GPU node with other developers, unlimited tokens

sllm.cloud

Post reply on HN
Print view View original

71–80 of 117 posts

Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens

#71

Post by tcdent »

jrandolf wrote: post #43
QuantumNomad_ wrote: post #25

> How does billing work? > When you join a cohort, your card is saved but not charged until the cohort fills. Stripe holds your card information — we never store it. Once the cohort fills, you are charged and receive an API key for the duration of the cohort. Have any cohorts filled yet? I’m interested in joining one, but only if it’s reasonable to assume that the cohort will be full within the next 7 days or so. (Es…

No cohorts have been filled yet. We're still early. We are seeing reservations pick up quickly, but I'd be able to give you a more concrete estimate of fill velocity after about a week. That said, we're planning to add a 7-day window: if a cohort doesn't fill within 7 days of your reservation, it cancels automatically and your card is released. We don't want anyone's payment method sitting in limbo indefinitely.

This is a fantastic idea.

On a nonzero number of occasions I have priced the cost of running an inference server with a model that is actually usable and the annual cost is astronomical.

Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens

#72

Post by mememememememo »

pokstad wrote: post #31
freedomben wrote: post #13

This is an excellent idea, but I worry about fairness during resource contention. I don't often need queries, but when I do it's often big and long. I wouldn't want to eat up the whole system when other users need it, but I also would want to have the cluster when I need it. How do you address a case like this?

This problem sounds like an excellent opportunity. We need a race to the bottom for hosting LLMs to democratize the tech and lower costs. I cheer on anyone who figures this out.

This is classic queuing theory, rate limits etc. I don't have an answer but I would look there.

Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens

#73

Post by mememememememo »

varunr89 wrote: post #15

$40/mo for deepseek r1 seems steep compared to a pro sub on open ai /claude unless you run 24x7. im not sure how sharing is making this affirdable.

Yes you don't choose this for the price. But because you want to control yout dependencies.

Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens

#74

Post by ycui1986 »

artificialprint wrote: post #48
MuffinFlavored wrote: post #46

> Running DeepSeek V3 (685B) requires 8×H100 GPUs which is about $14k/month. Most developers only need 15-25 tok/s. > deepseek-v3.2-685b, $40/mo/slot for ~20 tok/s, 465 slots total > 465 users × 20 tok/s = 9,300 tok/s needed > The node peaks at ~3,000 tok/s total. So at full capacity they can really only serve: > 3,000 ÷ 20 = 150 concurrent users at 20 tok/s > That's only 32% of the cohort being active simultaneously…

People work 8 hours a day presumably, I guess they are banking on this idea

only works if the users are evenly distributed around the globe (which is likely more of less the case). if the user concentrates in on century, the token rate will be terrible.

Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens

#75

Post by gravypod »

jrandolf wrote: post #62
johndough wrote: post #59

Earlier quoted context omitted.

Isn't this a bad deal? Or is there an error in my math? For $40, I'd get 20 tok/s * 2.6M seconds per month = 52M tokens of DeepSeek v3.2 per month if I run it 24/7, which is not realistic for most workloads. On OpenRouter [1], $40 buys 105M tokens from the same model, which is more than 52M tokens, and I can freely choose when to use them. [1]: https://openrouter.ai/deepseek/deepseek-v3.2

20 tok/s is an average. It can be more, it can be less. If you are running off-peak I'm sure you'd get some crazy number.

Why wouldn't developers just do llm arbitrage against openrouter if it is a better deal?

Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens

#76

Post by jrandolf »

gravypod wrote: post #75
jrandolf wrote: post #62

Earlier quoted context omitted.

20 tok/s is an average. It can be more, it can be less. If you are running off-peak I'm sure you'd get some crazy number.

Why wouldn't developers just do llm arbitrage against openrouter if it is a better deal?

The problem is different. OpenRouter is a router to LLMs. It doesn't solve GPU underutilization.

Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens

#78

Post by gravypod »

jrandolf wrote: post #76
gravypod wrote: post #75

Earlier quoted context omitted.

Why wouldn't developers just do llm arbitrage against openrouter if it is a better deal?

The problem is different. OpenRouter is a router to LLMs. It doesn't solve GPU underutilization.

What I am saying is if your system lets me pay $x/token and open router lets me pay $y/token if x<y then someone could make money just by providing those tokens through the open router API. That would either drive up demand for your systems increasing costs or drive up supply on open router decreasing costs. Eventually the costs would converge, no?

Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens

#79

Post by lelanthran »

wavemode wrote: post #77

> nobody is charged until the cohort fills So then what happens if some people's payment method fails once you do charge?

> So then what happens if some people's payment method fails once you do charge?

I expect its a pre-auth, like car rental companies do; a pre-auth gives you a code from the card issuer and an expiry. The issuer will reserve the amount on the cardholders account, and only perform the transaction to the merchant once the merchant sends a second message with the pre-auth code.

Post reply on HN