You're quite right there. I've wondered those myself. Here are some theories.
> What's hard to to understand is how we allow them.
I think that this has to do with two factors - motivation and organization. Ordinary people are not motivated enough to seek long term affordability and market health. They are easily tempted by anything convenient and cheap (in the short term). That's how big chains and big online shops were able to out-compete brick and mortar and mom and pop shops. Once the competition is gone, the big shops show their true colors and hold the market hostage to extract as much revenue as possible. BigCos can do this because they're motivated strongly by the promise of great profits to seek long term strategies like this (as delayed gratification). Meanwhile the consumers make short term gains and accumulate massive long term losses!
The second factor is organization. Consumers hardly ever organize to make a concerted effort to force the hand of big companies. For example, let's assume that someone is marketing a simple, good quality and long-lasting washing machine. The only catch is that it's pricier than the other 'smart washing machines' because of the lower scale of production. Assume that the people organized together and decided to buy only that washing machine or any other that competes with it on merits (but not the price). The new washing machine will eventually become cheaper and better because they can now increase the production capacity as profits roll in. Meanwhile, the other companies will be forced to make their offerings cheaper and remove any offending 'features' if they want to sell any of it. And when they do, it will restore the market competition and drive the market further in favor of the consumers. This is what we want.
However, what happens is exactly the opposite of the above. Someone introduces a smart washing machine into a market full of regular dumb washing machines. They make it cheaper by collaborating with other companies - like recording and selling user data to third parties. Since the consumers are not organized, a sizable portion of the population will start choosing it. That population doesn't have to be the majority. It needs to be just big enough for other manufacturers to notice the slump in their sales. Even if malpractice in the product is obvious, they'll choose to sacrifice their privacy for the short term savings with some justification like 'I don't have anything to hide'. What happens next is well known. Other companies notice the loss of sales and are forced to follow suite. At some point, even the consumers who were never willing to compromise will be left without any choice. This is a repeating story with a lot of products. But one where this is very egregiously obvious is the smart TV market.
Meanwhile, the big companies actually organize to drive the market in that direction - again motivated by profits. One well known example of this is the 'Geneva Cartel' where manufactures banded together to mandate planned obsolescence of electric bulbs. Another example is the US telecom industry. Those companies would have preferred to create a monopoly first and then do this. However due to the anti-monopoly laws, they're motivated to organize together as co-monopolies instead. To this end, even the billionaires that own these rival multinational giants maintain an exclusive and secretive social club where they conduct all these scheming. They behave exactly like those old royal families and modern crime syndicate families to protect their privileged position in society.
> How the market hasn't seen an opening; why someone else hasn't started making machines to fill these niches.
> There are so many pro-consumer ideas (I've only listed a few) that a company could seize upon to market themselves. And unlike pocket consumer-electronics, there's little barrier to entry. You don't need to reinvent anything (quite the opposite).
I think the above explanation answers these questions too. The final point is that the larger consumer community lacks the long-term strategic planning that involves sacrificing short term savings, and the ability to unite for a common cause. The much smaller business community uses these same skills effectively against the consumers to consolidate wealth as much as possible. This weakness of the consumer community is clearly evident even on HN. Despite being a technical community, a few here would rather argue that privacy is not important to them, than unite with everyone and use their weight to push the market in the opposite direction.
The above situation is not a lost cause though. People have united together to achieve much harder goals. What's needed is a solid motivation. And in this situation, that motivation can come from the awareness of the class war and the exploitation they're subjected to under it. That needs a lot of public campaigns. It needn't start big outright. It can start with token signs of protest and gradually build up mass and momentum from there as the people take notice. We already have such a campaign in progress right now - the clippy propic campaign that Louis Rossmann kicked off. We need more people to take similar initiatives.