This is great, thanks! I personally would like something like this but with "regular" GPU access. Some people still use them for something other than LLMs ^^.
There is vast.ai!
Show HN: sllm – Split a GPU node with other developers, unlimited tokens
51–60 of 117 posts
Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens
#52This is an excellent idea, but I worry about fairness during resource contention. I don't often need queries, but when I do it's often big and long. I wouldn't want to eat up the whole system when other users need it, but I also would want to have the cluster when I need it. How do you address a case like this?
Also, cache ejection during contention qill degrade everyones service.
I question whether they actually understand LLMs at scale.
Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens
#53I read the FAQ, and I can't imagine this is going to work the way you want it to. It fundamentally doesn't make sense as a business model. I can sign up for a cohort today, but there's not even a hint of how long it will take the cohort to fill up. The most subscribed cohort is only at 42% (and dropping), so maybe days to weeks? That's a long time to wait if you have a use case to satisfy. And then the cohort expires…
Going on ChatGPT.com and using their AI for 24 hours doesn't mean you are actually using their LLM for 24 hours. It's only live for as long as the output is being generated. You reading, waiting for tool calls, etc. don't count toward concurrency. Factor in time-zones, lunch times, etc...it's more likely that we'd have an underutilization problem.
For filling up the cohorts, I agree and we're launching for a week to gather feedback.
Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens
#54This is an excellent idea, but I worry about fairness during resource contention. I don't often need queries, but when I do it's often big and long. I wouldn't want to eat up the whole system when other users need it, but I also would want to have the cluster when I need it. How do you address a case like this?
Also, cache ejection during contention qill degrade everyones service. I question whether they actually understand LLMs at scale.
I suppose it's meant to be a "minimum viable" third-party inference platform, where you're literally selling subscription-based access (i.e. fixed price, not PAYGO by token) to a single GPU cluster, and then only once enough users subscribe to make it viable (which is very nice from them, it works like a Kickstarter/group coupon model and creates a guaranteed win-win for the users). But they could easily expand to more than just the minimum cluster size, which would somewhat improve efficiency. (Deepseek themselves scale out their model over huge amounts of GPUs, which is how they manage to price their tokens quite cheap.)
Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens
#55[flagged]
Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens
#56This is an excellent idea, but I worry about fairness during resource contention. I don't often need queries, but when I do it's often big and long. I wouldn't want to eat up the whole system when other users need it, but I also would want to have the cluster when I need it. How do you address a case like this?
Ultimately the most sensible way of handling this is you end up with "surge pricing" for the highest-priority tokens whenever the inference platform is congested, over and above the base subscription (but perhaps ultimately making the subscription a bit cheaper).
Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens
#57[dead]
Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens
#58[flagged]
There's a big difference between non-compliant, illegal, and criminal.
Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens
#59Can you explain the benefits over something like openrouter?
24/7 LLM for $10/month.
Isn't this a bad deal? Or is there an error in my math?
For $40, I'd get 20 tok/s * 2.6M seconds per month = 52M tokens of DeepSeek v3.2 per month if I run it 24/7, which is not realistic for most workloads.
On OpenRouter [1], $40 buys 105M tokens from the same model, which is more than 52M tokens, and I can freely choose when to use them.
Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens
#60Earlier quoted context omitted.
24/7 LLM for $10/month.
Isn't this a bad deal? Or is there an error in my math? For $40, I'd get 20 tok/s * 2.6M seconds per month = 52M tokens of DeepSeek v3.2 per month if I run it 24/7, which is not realistic for most workloads. On OpenRouter [1], $40 buys 105M tokens from the same model, which is more than 52M tokens, and I can freely choose when to use them. [1]: https://openrouter.ai/deepseek/deepseek-v3.2
[deleted]