Viewing profile — roadside_picnic

> and I would not call it cheap It is compared to the cost of making an authentic ramen broth at home at the scale of feeding 2-4 people. Pho has a similar property. I've made home…

> We do this and then no one reads it. This to me is an anti-pattern that I see everywhere because we're still in the early stages of understanding how to adopt these things and cr…

MoEs don't route like most people imagine. They aren't learning topic based experts despite the name The original Mixtral paper [0] (in the "Routing analysis" section) found: "surp…

> from 40 to 50 tok/s generation That's actually much better than I would have thought! Thanks for the answer and it does make this approach make more sense as a budget solution to…

What's the tokens/sec you're getting on that setup (genuinely curious because it's a setup I haven't actually run myself)?

> no other widely accessible path for young people to build a future This is exactly right. Widespread gambling has a fundamental nihilism baked into it. If you believe hard work a…

If you're using a MoE model, then why do you care about the larger RAM offered by these devices? That's the main problem with low bandwidth devices: they limit the effective ram yo…

> Yet those 2-3M channels get the lions share of the views. ...because they're directly tied to YouTube own revenue. As someone who creates a fair bit of content, and am fortunate …

In general if you're setting up a local LLM you should assume it's going to be primarily working as a server and talking to various clients. I use my MBP, but that's because I don'…

It depends on your use case. There's a lot of hype around machines like the DGX spark (I'm assuming this is the type of device you're referring to) because they look awesome, and a…

My experience working in the open model space pretty deeply (both LLMs and diffusion models) for years now is that it is not quite as simple as that. In the open model space an ins…

In addition to models getting better, the quantization methods have also got much better. If you already have an RTX 3080 it's absolutely worth the time to just mess around and see…

M3-Max laptop: ~55 token/sec RTX 4090: ~190 token/sec I don't have the number around but there is a notable latency for pre-fill on the M3, but once it's running the delay is negli…

There are a couple of things, but basically it boils down to the same reason people prefer Linux to Windows/MacOs: customization, control and privacy (arguably all of these are rea…

Just running it through `llama-cli` so that there's absolutely no persistent state related to the chat (and least I believe this to be the case).

See my comment to parent. I've been using local LLMs for practical, personal tasks for a few months now very successfuly. You can run fantastic local models if you have either: - M…

I have a home server that runs Qwen3.6-35B-A3B through llama.cpp with Open WebUI for the user facing interface. My teen isn't super interested in AI, but whenever they do feel curi…

> It's like the infinite monkeys on typewrighters that will type whatever you are looking for, given infinite time. In the monkey example the infinite time is doing a lot of work t…

I used to think comparisons of AI with the web where ridiculous, but increasingly it looks like they're not that dissimilar as far as how they change how we work. But as someone wh…

It's more insidious than that. These IPOs aren't being rushed, they were waiting for all the pieces to be in place to force 401ks and other retirement plans to buy these IPOs. The …

As you likely know, rules have recently been changed that basically force many 401k funds to invest in these IPOs while simultaneously having a relatively small number of the initi…

Have you personally used any of the latest batch of even smaller local models? They certainly don't beat SotA models at coding... but with a good harness they are able to achieve t…

It also underplays what I've personally witnessed that I would consider true AI psychosis. I worked with someone who sincerely believed he was spiritually co-evolving with his army…

There's a difference between "red flags" and "imperfections". Every team has faults, which if you're experienced at interviewing/working many places, are usually pretty easy to fig…