Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

github.com

Post reply on HN
Print view View original

111–120 of 126 posts

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#111

Post by baristaGeek »

I just used it, went through the whole installation (took like 1 hour approx). Long but straightforward. If I put my computer to sleep will it continue?

I started the server (very curiously I was running oLlama in the same prot slotserve uses by default, instead of switching it which I know you can do, I just ditched oLlama, perhaps an insight for you) and built a small html hello world served via Python. The thing pointed me to the localhost link, nice!

As an early user, my advice is to focus on efficiency. The efficiency of the installation but more importantly, the efficiency of running the thing. 8.1GB per slotserve process is a lot! Is that in your control?

Also, I've seen an interest of certain kinds of programmers for open-weight models. "We all know agree that LLMs for coding are very useful but we're giving money to a small set of big, evil corporations. They're Trump donors. I heard it's bad for the environment because it uses water". If it's local and open-weight, this could be marketed this way I think.

Finally, what's the actual, real use case for slotserve?

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#114

Post by carloslfu »

baristaGeek wrote: post #111

I just used it, went through the whole installation (took like 1 hour approx). Long but straightforward. If I put my computer to sleep will it continue? I started the server (very curiously I was running oLlama in the same prot slotserve uses by default, instead of switching it which I know you can do, I just ditched oLlama, perhaps an insight for you) and built a small html hello world served via Python. The thing p…

thanks!

> ditched oLlama"

yeah! this is interesting.

> 8.1GB per slotserve process is a lot! Is that in your control?

yes, it is hard, but I agree the smaller the better. I'll work on that

> If it's local and open-weight, this could be marketed this way I think.

I like this!

> what's the actual, real use case for slotserve?

I'm working rn on an app on top of it that closes the loop and is a fully local AI app, an experiment. I'll publish it as soon as it is usable!

> built a small html hello world served via Python

What did you use as a harness here?

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#115

Post by baristaGeek »

carloslfu wrote: post #114
baristaGeek wrote: post #111

I just used it, went through the whole installation (took like 1 hour approx). Long but straightforward. If I put my computer to sleep will it continue? I started the server (very curiously I was running oLlama in the same prot slotserve uses by default, instead of switching it which I know you can do, I just ditched oLlama, perhaps an insight for you) and built a small html hello world served via Python. The thing p…

thanks! > ditched oLlama" yeah! this is interesting. > 8.1GB per slotserve process is a lot! Is that in your control? yes, it is hard, but I agree the smaller the better. I'll work on that > If it's local and open-weight, this could be marketed this way I think. I like this! > what's the actual, real use case for slotserve? I'm working rn on an app on top of it that closes the loop and is a fully local AI app, an exp…

For the harness... just the shell. No client library. Does that answer your question?

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#119

Post by carloslfu »

nerdsnipe wrote: post #117

Brilliant! Have you had any success integrating it into a MacOS Swift app. I'd love to see it in action before I consider adding it to a future build. My biggest issue is getting these opensource models to use tools well enough for production.

This is next! in the works rn.

Re: Show HN: Running 104GB Qwen3.8-Flash-Next on 48GB Mac with at ~12 tok/s

#120

Post by carloslfu »

baristaGeek wrote: post #115
carloslfu wrote: post #114

Earlier quoted context omitted.

thanks! > ditched oLlama" yeah! this is interesting. > 8.1GB per slotserve process is a lot! Is that in your control? yes, it is hard, but I agree the smaller the better. I'll work on that > If it's local and open-weight, this could be marketed this way I think. I like this! > what's the actual, real use case for slotserve? I'm working rn on an app on top of it that closes the loop and is a fully local AI app, an exp…

For the harness... just the shell. No client library. Does that answer your question?

thanks! in part, I was wondering how you got the code into files. I guess you copy pasted it inside a file, am I right?
Post reply on HN