Earlier quoted context omitted.
It is, I think, a common feeling among Echo/Alexa users. Now that people are getting used to the amazing understanding capabilities of ChatGPT and the likes, it probably increases the frustration level because you get a hint of how good it could be. I believe it boils down to two main issues: - The narrow AI systems used for intent inference have not scaled with the product features. - Amazon is stuck and can't signi…
I think the economics here are wrong by orders of magnitude. It doesn't make sense to deploy to the home an expensive GPU that will sit idle 99% of the time, unless running an LLM gets much cheaper, computationally. It's much cheaper to run it on-premise and charge a subscription, otherwise nobody would pay for ChatGPT and would have an LLM rig at home instead.
Here, home assistant is telling you: you can use your own infra (most people won't) or you can use our cloud.
It works because most likely the user base will be rather small and home assistant can get cloud resources as if it was infinite on that scale.
If their product was amazing, and suddenly millions of people wanted to buy the cloud version, they would have a big problem: cloud infrastructure is never infinite at scale. They would be limited by how much compute their cloud provider is able/willing to sell them, rather than how much of that small boxes they could sell, possibly loosing the opportunity to corner the market with a great product.
If you package everything, you don't have that problem (you only have the one to be able to make the product, which I agree is also not small). But in term of energy efficiency, it also does not have to be that bad: the apple silicon line has shown that you can have very efficient hardware with significant AI capabilities, if you design a SOC for that purpose, it can be energy efficient.
Maybe I'm wrong that the approach will get common, but the fact that scaling AI services to millions of users is hard stand.