> A healthy regulated will encourage maximizing profit for power and bring in competition which drives the cost down until energy is a commodity and the cost of electricity is actually based on the price of production and a small profit based on the cost of capital.
That is not how the electricity market works, in Australia anyway, and somewhat fundamentally everywhere. The network needs to maintain a frequency and voltage for it to be reliable. These change as load and production change. So consumers don't get a choice of which electrons power their house, only who they pay. They pay a 'retailer' who usually has nothing to do with production for a known cost per kwh + fees ahead of time. The market then operates where agreements between parties including retailers and produces (traders and others as well) has a 'market rate' that essentially arbitrage between longer term fixed rates and market rates.
The fact that the stability is tied to frequency and voltage (and infrastructure) means there is a limit to the rate of production and consumption, not to mention electricity is a necessity in the modern age.
In Australia at least we are finding out the hard way about what happens when you privatize a necessity. People will pay whatever it costs, and since the market needs a high level of regulation just to function, a market IMO is just a bad fit for trying to bring costs down rather than just rent seeking.
> The fact that you can add to the grid by installing solar and battery and connect to the grid in a single afternoon
That has become harder in recent time due to areas being over saturated by solar. Cities in Australia can deny you connecting to the grid if there is too much, as well as we have high network voltage detection on inverters which now kick in on many sunny days due to again, too much solar. Electricity network operators pay a large amount of money to services to predict/model how much of the power in their network is coming from solar and where because they commonly don't know, so it becomes a difficult balance of how much solar you are allowed to connect.
> but with the right equipment you can have community sized islands that can be much more immune to instability.
Agreed, but due to the required _shared_ infrastructure for this to work will need public land to connect these islands or even within an 'island', as well as the now private vested interests in rent seeking, this will be a fringe solution only available to those with larger amounts of land like communes or other rural setups. Again, speaking to Australia.