461–470 of 653 posts
Earlier quoted context omitted.
What on earth hardwares do you guys have to be able to run 100gb models locally?! That's crazy! I'm here struggling to even get 27b models to run in somewhat usable way
MoE models can use system memory along with a GPU.
Earlier quoted context omitted.
A local model needs 0 investment and 0 commitment, takes literal minutes to get started (especially if you have someone who is into that stuff showing you the ropes) and if you end up disliking the experience of using AI you can just `rm -fr` it and forget the whole thing existed.
Needs 0 investment and 0 committment? - You at least need a capable machine, so that's not 0 monetary investment. - You need to spend at least an hour decicding between ollama, llamacp, mlx, etc. - You need to find the correct quantized version of the model that works for you based on the architecture. - You need to figure out the correct context window size to get reasonable performance. - You need to setup a harnes…
For me at least the local AI stuff, powered with solar has been pretty great. Would that scale to a large business? Goodness no, but for my tinkering and learning, it works great.
Earlier quoted context omitted.
> All requests to an LLM are idempotent, for every API call you need to send it the entire conversation history A more appropriate term is “stateless”. LLM responses are certainly not idempotent, as they are not even deterministic.
>LLM responses are certainly not idempotent, as they are not even deterministic. Isn't that more due to an optimization and not how the LLM itself runs? Like a MoE LLM run on a single input should give the same output each time. But this is inefficient, as any given token is hitting 1 (or maybe 2 or 3) experts at a time, meaning all the other experts are doing absolutely nothing. So you upgrade it to take in multiple…
In the simplest possible case, a distributed addition of floating point numbers is not deterministic if you don't specify the order of the addition operations. If floats are added in a first-come first-serve fashion (the simplest implementation of a reduction operation) you already lose determinism. These could be activations from multiple experts, but it could also be adding chunks of a matrix operation that uses multiple CUDA cores.
If you manage to achieve determinism in temperature zero, it is possible to extend it to determinism at higher temperatures, because at that point you only have to keep the pseudo-random-number generator state in sync across parallel instances and this problem has been solved.
Earlier quoted context omitted.
> You at least need a capable machine, so that's not 0 monetary investment It is 0 monetary investment if I already have said machine lying around doing nothing. Which is exactly the story OP talked about.
But most people don't have an RTX 5090 lying around, so the story doesn't apply to them, right?
Earlier quoted context omitted.
This is the diametric opposite of the rent-vs-buy scenario that this entails. Local: You need to invest $thousands into GPU and/or very-high-end CPU+Memory hardware. Vendor: You can use any existing device, even a phone or tablet. A very low-end laptop is fine. > takes literal minutes to get started Local: Typical scenario is hours just to download the software, the model weights, and then faffing around with CUDA an…
It took me about three hours total to set up a local model. I already have a GPU and I have fiber for the download. llama.cpp is not difficult to compile and has many backends. It can run parts of the model on different backends, like in the common case that the GPU doesn't have enough VRAM for everything. There are many step-by-step guides available.
Earlier quoted context omitted.
Qwen3.6-35B is my daily driver for AI, and what convinced me to cancel my Claude subscription back in April. The Qwen3.6 line is easily the best local model I've tried, and I've tried a lot. I've got it diligently grinding away on my laptop right now, reviewing and fixing some bugs in my F# code.
What do you use to pair it with web search?
Earlier quoted context omitted.
You can always wait until the page loads before posting your thoughts on the Internet
Or let them just ask and if someone asking a question on the internet bothers you, just move along and ignore it.
Earlier quoted context omitted.
What on earth hardwares do you guys have to be able to run 100gb models locally?! That's crazy! I'm here struggling to even get 27b models to run in somewhat usable way
Strix Halo, 128GB RAM. I got a refurbished Corsair AI Workstation for a smoking price ($2100) about two months ago. Lucky timing that it was in stock.
Earlier quoted context omitted.
Qwen3.6-35B is my daily driver for AI, and what convinced me to cancel my Claude subscription back in April. The Qwen3.6 line is easily the best local model I've tried, and I've tried a lot. I've got it diligently grinding away on my laptop right now, reviewing and fixing some bugs in my F# code.
What kind of machine do you have running that? My attempts at local have always resulted in a very hot lap