Nvidia on NixOS WSL – Ollama up 24/7 on your gaming PC
yomaq.github.io
Nvidia on NixOS WSL – Ollama up 24/7 on your gaming PC
1–10 of 51 posts
Re: Nvidia on NixOS WSL – Ollama up 24/7 on your gaming PC
#2Remove windows. And this is amazing.
Re: Nvidia on NixOS WSL – Ollama up 24/7 on your gaming PC
#3Given that Ollama runs quite fine on Windows if you have NVIDIA, why such complicated setup?
It would make more sense for AMD I suppose where Ollama's Windows support is lacking compared to Linux.
That said, neat tricks useful for other stuff as well.
Re: Nvidia on NixOS WSL – Ollama up 24/7 on your gaming PC
#4This is great work, it solves an exact problem I too am having. Now I just need to upgrade my 12 year old GPUs to something that can run an LLM.
Re: Nvidia on NixOS WSL – Ollama up 24/7 on your gaming PC
#5it's unclear to me what you gain running ollama in wsl like this compared to switching to a productive native operating system [like nixos] or just installing the windows release of ollama and quietly forgetting about it.
i use nixos.wsl at work to have the same emacs configuration as on my laptop, and that's fine except the windows filesystem performance makes me want to throw the whole system in a dumpster. but on my home gaming machine i have some games that only run on windows so i just installed ollama's windows installer which works with my GPU and installs an autostart entry.
these days the windows box sits in a dark corner on my network with tailscale (again just the windows install), running sunshine too to start steam games on my laptop.
Re: Nvidia on NixOS WSL – Ollama up 24/7 on your gaming PC
#6Given that Ollama runs quite fine on Windows if you have NVIDIA, why such complicated setup? It would make more sense for AMD I suppose where Ollama's Windows support is lacking compared to Linux. That said, neat tricks useful for other stuff as well.
Yeah… was about to ask that. I just run Ollama on windows. What am I actually solving by running it virtualized in WSL?
Re: Nvidia on NixOS WSL – Ollama up 24/7 on your gaming PC
#7Just to awnser the question why not using windows. Here are some ideas why the author might have used nix instead:
- reproducible (with minor adjustments even on non-WSL systems)
- if you are used to nix, there is not much which beats it in terms of stability, maintainability, upgradability, and fun (?)
- additional services are typically easier to set up like tailscale-acl, used by the author, which uses pulumi under the hood
- despite some downsides (disc speed was an issue when I used it), WSL is surprisingly capable
Re: Nvidia on NixOS WSL – Ollama up 24/7 on your gaming PC
#8Given that Ollama runs quite fine on Windows if you have NVIDIA, why such complicated setup? It would make more sense for AMD I suppose where Ollama's Windows support is lacking compared to Linux. That said, neat tricks useful for other stuff as well.
Yeah… was about to ask that. I just run Ollama on windows. What am I actually solving by running it virtualized in WSL?
It's probably marginally faster for inference as long as you're willing to pay the extremely annoying cost of all the stupid filesystem issues. Given that ollama specifically has very good cross platform support it seems like a fool's errand.
Re: Nvidia on NixOS WSL – Ollama up 24/7 on your gaming PC
#9I agree with the other commenters, this post does not explain why you would not just run Ollama or Koboldcpp on Windows. What exactly makes running Ollama within virtualized NixOS in WSL in some way better than running natively?
If it's just the novelty aspect of it or some idealogical reason, that's fine, but it should be explained in the blog post before someone thinks this is a sane and logical way to run Ollama on a gaming PC.
Re: Nvidia on NixOS WSL – Ollama up 24/7 on your gaming PC
#10As others have already pointed out if you're going to run Ollama in Windows anyway, why not use the native build? And if you want to use WSL, then I'd suggest using something like LocalAI which gives you a lot more control and support for additional formats (GGML, GGUF, GPTQ, ONNX, etc).