It's hard to compete with grocery stores: 1. They're local. Most people have a grocery store within 5 minutes of their house. 2. They're well-stocked. You have ten thousand products to choose from. Who could ask for more? 3. They're timely. If you go to a grocery store, it will take you a certain maximum amount of time - the amount of time it takes to go down each aisle once. 4. They're accurate. You picked every ite…
To your specific points:
1. Locality is a huge competitive disadvantage. Real-estate, staffing, infrastructure, etc.
2. More-efficiently managing a wide selection and optimizing for product that frequently turns over is a key advantage of warehouse/delivery operations.
3. It makes no sense to compare an interruption of your day with an asynchronous order/delivery method. No-one uses Netflix the way they used Blockbuster. No-one misses Blockbuster.
4. Personally picking fruit directly off plants was more accurate than personally picking fruit in a grocery store. It's also something most people don't miss as a regular chore. Warehouse/delivery has a massive advantage here as well, as they'll be able to better manage inventory flow and don't have to suffer the same exposure to elements/contaminants as a grocery store.
They can even offer "ugly produce" at a discount, bringing the same implicit assumption of responsibility for odds and ends and generating sufficient extra revenue from the particular buyers, to effectively serve them and any returns.
There's no "niche" for this. The niche will be serving the people who enjoy farmer's markets and those who dropped a dozen eggs and will pay a premium for their replacement.