Viewing profile — alexellisuk

Additionally: https://www.openwall.com/lists/oss-security/2026/07/06/7

Funnily enough - I built this (delegation) over the weekend with Fable for a local voice chat running 100% on local LLMs, Parakeet and Kokoro. I say "...ask the thinking model..." …

Yeah, I'm surprised Justin posted this like it was new(s). Wasn't it doing the rounds on the 22nd when it launched?

For self-hosting, have a look at what we're building with SlicerVM.com (disclosure: I'm the founder). Also runs just as well on Apple Silicon. We run quite a few Slicer instances o…

This is clever work, especially given that Proxmox is already a very viable VMware replacement and wasn’t originally designed around microVMs as the primary abstraction. I’m glad t…

Thanks for the comment ZDR is mentioned in the post - in particular many the coding plans that are not from the two major leaders have questionable IP/ownership claims on inputs/ou…

1. On the technical: The cache only makes generation fast, it doesn't influence what gets chosen next. The loops that hurt the most (point 2 below) are when the model re-decides to…

vLLM is great at continuous batching and model serving in production, but it's a very different beast and much less versatile for the prosumer category (where we sit for our usage)…

We did run vLLM on the 3090s — measured ~3 tok/s slower on generation for our single-to-few-user pattern, plus less flexibility on quant and slower startup (actual minutes vs singl…

Fair enough, that sentence was fairly compressed. I’ve reworded it - the meaning remains the same. The post is not AI generated, I use AI for code generation and write my own artic…

I think that's quite telling Gorgi replied that he uses Qwen with 131k context. https://x.com/ggerganov/status/2067539416436867230?s=20 We also use it with 200-256k (native) contex…

Ha, you underestimate how dogged you need to be to get this stuff working well. The RTX 3090 in question was used from eBay, no way to return it. The RTX 6000 Pro is the "new card"…

The important thing about MoEs which I mention in the conclusion is that they carry fewer (way fewer) active tokens during inference/generation. 35B-A3B is what we started out with…

One of the things I mentioned in the post: > Local models can quickly read and explain codebases, even if they can't write them - this is a superpower Might have been buried lower …

Author here. Thanks for the question. I'll answer assuming this is a question you have for me. As explained in the post - the 3090s were what were the test bed that proved the inve…

[dead]

Hi - the author of the post here. I wanted to write up something that was a bit more than "Qwen is the goat" or "Cancelled Claude, run everything local now" or even "The model orga…

What quant?

I was thinking about the RPi 6 yesterday whilst realising I couldn't set up my RPi Zero 2W anymore - the OS has become burdensome - tied strictly to an imager, that gives me an all…

Not a surprise at all. If you look at https://slicervm.com you'll see he's copied our terminal animation from the top of the website. Took out a monthly subscription for 1x month, …