91–100 of 108 posts
This logic seems mad. If people only need SLMs then hyperscalers can also centrally host higher-efficiency models, and still gain efficiencies of scale and convenience over hosting locally.
The privacy cost of sending everything to a third party is huge. Running locally fully resolves that, so it is the obvious choice if only cost can be managed.
A remaining advantage of large language models is that as they get larger, they tend to hallucinate less, simply because the odds of the training set containing a desired answer improve with size. If a solid "I don't know" detector is developed for inference, then you can try a small language model first. An implication is that successful research in "I don't know" detection could destroy hundreds of billions in shar…
>> A remaining advantage of large language models is that as they get larger, they tend to hallucinate less First time I hear that...not really true. "Understanding Why Language Models Hallucinate: Testing Reasoning Against Priors" - https://arxiv.org/abs/2607.00447 "Calibrated Language Models Must Hallucinate" - https://arxiv.org/abs/2311.14648 "TruthfulQA: Measuring How Models Mimic Human Falsehoods" - https://arxi…
From the "TruthfulQA" paper: "Models generated many false answers that mimic popular misconceptions and have the potential to deceive humans. The largest models were generally the least truthful. This contrasts with other NLP tasks, where performance improves with model size. However, this result is expected if false answers are learned from the training distribution." That's more of a garbage-in, garbage out problem. If the large model is trained by shoveling in random web content, that's going to happen. Not a hallucination problem. The LLM just fed back what it had been told.
Earlier quoted context omitted.
> Exactly the type of thing I’d pay someone not to deal with. Very much a personality and/or lifestyle thing, but I've also been building infra for decades. I mean it was two container startups and a phone app download (10min). Not exactly difficult. If I didn't like it I certainly wouldn't be in tech! > pay someone I'd do that with something like 3 story roof work or foundation work with a contractor, but tech? If I…
To be clear I don’t mean pay someone to set it up for me, but rather pay for a saas platform that’s already doing it. It’s not that I couldn’t run something locally, I just don’t want to deal with the hassle. I was into home automation and self hosting media and some other stuff for a while. It’s the kind of thing I’d only do again if i was equally or more interested in the process than the outcome.
We're looking for different outcomes.
I am also looking to actively cut my SaaS / services cost to $0 or near $0. I don't want another company touching my fucking data or prompts.
Earlier quoted context omitted.
> And they will use it! Just out of curiosity: Use it for what?
Whatever it is suited for that they can get a decent margin on. Or, you know, hobbies. The demand is there and the market will find pricing that works.
From what’s presented this seems to be the lower end of Q&A and reasoning tasks and not long horizon agentic work. I agree that the search engine replacement AI usage is something that can run anywhere (though it’s still better run in the cloud for speed, context length, sandboxing and convenience) but this isn’t the engine of AI growth. Also, the average consumer is not going to be running a local model until they a…
I want somebody to make a local box:
- router
- wifi
- thread/matter support
- 1-3 nvme in RAID
- GPU, running GPT-OSS OOTB with other models downloadable
- running containers
- allows people to share and store files with friends and family
- runs a very simplified homekit
- has phone app
But with the prices of hardware right now... Probably out of reach for a general computing product.
Earlier quoted context omitted.
> I tried running a smaller model locally, and it's not usable for me. If you have the hardware, a MacBook Pro for Qwen 3.6 35B A3B and Gemma 4 26B A4B for example, they are absolutely usable, both in terms of speed and quality. Anecdotally, I can use Qwen for day-to-day coding tasks in TS and Go, without hickups.
You let a hiccup slip through in your comment though.
"The future could well be specialized edge models that know only ONE thing - and know it well." That design paradigm sounds familiar Everyday I use smalll applications that do only one thing, some written in the 1970's This submission got [flagged]. Later the "[flagged]" label was removed
In fact sounds like an expert system from decades ago.
I think one of the watershed developments will be fast models. Imagine current frontier models at 20k tokens/second.
Maybe this is interesting for an agent?
Earlier quoted context omitted.
The valuations of the hyperscalars won't sustain just being more efficient than something you can run locally. There's a market there, but it's for margin on a commodity. They're priced for oligopoly on unique, premium products.
I very much don’t want to run it locally. I want the same one running somewhere else that I can interact with from all my devices. Look at something like Grok Bot. Nobody is going to run this locally. You can already self host almost anything, yet most people and businesses don’t.