Earlier quoted context omitted.
https://news.ycombinator.com/item?id=49728887 How much ram has a fair amount of flexibility since MoE can be kept on flash and swapped in at a performance cost, and depending on how small a context you can use.
Well the linked thread mention 14700KF which is a 400$ processor and RTX 2070. Beside its mentioned 5 or 6 gb of ram. People run Firefox on pentium with less than 8gb of ram, just 4 or 5 years ago 4gb computer were still extremely common. Even Microsoft sold its surface devices with 4gb of ram, I was dumb enough to buy one. To give a counterpoint, you could disable it on those low end devices. But then people would c…
Though low memory devices like 4-8gb probably does need specialized inference to give fair numbers.
> you could disable it on those low end devices
it's not like it does anything if you don't ask for it-- and good thing, because the privacy invasion would be all the worse if it did!
> mistral was the safest
Sending the user's confidential information to third parties while falsely suggesting that it is private can cause them irrecoverable harm.
An alternative is that a feature is slow for some users on slow hardware, and there is a setting to make it faster at the expense of privacy and security that they can switch on. The large body of users that find the default config fast enough have no reason to flip the switch.
> But then people would complain about the size, like what happened with Google Chrome embedded llm.
People or astroturf accounts? :P but having read some of the commentary on it there was some pretty weird takes, like generalized complaints about AI and people thinking google was using their computer to serve other people. Google's business model generally prevents correctly marketing the functionality in any case: it's not like they're going to make a proper pitch for how important it is for your privacy when the rest of their business is centered on hoovering everything up. Mozilla is not so constrained :P