Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: sllm – Split a GPU node with other developers, unlimited tokens

sllm.cloud

Post reply on HN
Print view View original

61–70 of 117 posts

Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens

#61

Post by lelanthran »

singpolyma3 wrote: post #29
lelanthran wrote: post #19

Earlier quoted context omitted.

> 25 t/s is barely usable. Maybe for a background runner That's over a 1000 words/s if you were typing. If 1000 words/s is too slow for your use-case, then perhaps $5/m is just not for you. I kinda like the idea of paying $5/m for unlimited usage at the specified speed. It beats a 10x higher speed that hits daily restrictions in about 2 hours, and weekly restrictions in 3 days.

Sure if it was just a matter of typing. But in practise it means sitting and staring for minutes at nothing happening with a "thinking" until something finally happens. I mean my local 122b is only 20t/s so for background stuff it can be used for that. But not for anything interactive IME.

> I mean my local 122b is only 20t/s so for background stuff it can be used for that. But not for anything interactive IME.

What are you running that local 122b on? I mean, this looks attractive to me for $5/m running unlimited at 20t/s-25t/s, but if I could buy hardware to get that running locally, I don't mind doing so.

Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens

#62

Post by jrandolf »

johndough wrote: post #59
jrandolf wrote: post #41

Earlier quoted context omitted.

24/7 LLM for $10/month.

Isn't this a bad deal? Or is there an error in my math? For $40, I'd get 20 tok/s * 2.6M seconds per month = 52M tokens of DeepSeek v3.2 per month if I run it 24/7, which is not realistic for most workloads. On OpenRouter [1], $40 buys 105M tokens from the same model, which is more than 52M tokens, and I can freely choose when to use them. [1]: https://openrouter.ai/deepseek/deepseek-v3.2

20 tok/s is an average. It can be more, it can be less. If you are running off-peak I'm sure you'd get some crazy number.

Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens

#63

Post by moralestapia »

jrandolf wrote: post #51
moralestapia wrote: post #45

This is great, thanks! I personally would like something like this but with "regular" GPU access. Some people still use them for something other than LLMs ^^.

There is vast.ai!

Wow!

I recall hearing about them years ago.

Good to see they're thriving!

Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens

#66

Post by delusional »

mogili1 wrote: post #23
jrandolf wrote: post #20

Earlier quoted context omitted.

We implement rate-limiting and queuing to ensure fairness, but if there are a massive amount of people with huge and long queries, then there will be waits. The question is whether people will do this and more often than not users will be idle.

Rate limit essentially is a token limit

Time x capacity is also a limit. There's always a limit.

Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens

#67

Post by spencer9714 »

Interesting concept. One thing I’m curious about if I’m in a cohort for something like DeepSeek V3 and another user spins up a heavy 24/7 job, how do you keep TTFT from degrading? vLLM’s continuous batching helps, but there’s still a physical limit with shared VRAM/compute. I’ve been grappling with this exact 'noisy neighbor' issue while building Runfra. We actually ended up moving toward a credit per task model on idle GPUs specifically to avoid that resource contention entirely.

Curious how you’re thinking about isolation here. Is there any hard guarantee on a 'slice' of the GPU, or is it mostly just handled by the vLLM scheduler?

Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens

#69

Post by singpolyma3 »

lelanthran wrote: post #61
singpolyma3 wrote: post #29

Earlier quoted context omitted.

Sure if it was just a matter of typing. But in practise it means sitting and staring for minutes at nothing happening with a "thinking" until something finally happens. I mean my local 122b is only 20t/s so for background stuff it can be used for that. But not for anything interactive IME.

> I mean my local 122b is only 20t/s so for background stuff it can be used for that. But not for anything interactive IME. What are you running that local 122b on? I mean, this looks attractive to me for $5/m running unlimited at 20t/s-25t/s, but if I could buy hardware to get that running locally, I don't mind doing so.

Framework desktop

Re: Show HN: sllm – Split a GPU node with other developers, unlimited tokens

#70

Post by vineyardmike »

ssl-3 wrote: post #40
freedomben wrote: post #22

Earlier quoted context omitted.

Is there any way to buy into a pool of people with similar usage patterns? Maybe I'm overthinking it, but just wondering

I think it'd be best to pool with people with different patterns, not the same patterns. Perhaps it would be best to pool with people in different timezones, and/or with different work/sleep schedules. If everyone in a pool uses it during the ~same periods and sleeps during the ~same periods, then the node would oscillate between contention and idle -- every day. This seems largely avoidable. (Or, darker: Maybe the c…

This is basically why the big companies can sell subscriptions for cheaper than API costs. First priority can go to API users, lower priority subscription users get slotted in as space/SLO allows, and then sell the remaining idle GPU to batch users and spare training. Oh and geography shift as necessary for different nations working hours.
Post reply on HN