Okay, I'll try.
Given the unmeasurable social costs of people starving and dying when there isn't enough food (you can say similarily for people not having houses and freezing to death when there's a blizzard, but also read the sidenote at the end), the capitalists involved in selling and distributing food should not be compensated equally in monetary terms (by selling at higher prices) when there is a shortage of food, since they are the ones at fault of not having reserve resources and prepare their production line from various disasters such as storms and droughts. Even people from the ancient times built and used granaries to store food for years and opened it up to their fellow villagers when storms/droughts happen (even without any profit motives, there wasn't even a notion of "profit" at all!) Capitalists in the current era should prepare far better than them, as our knowledge of biology, climate, and logistics are much more sophisticated than before; a shortage of food should be outright unimaginable given the advanced technology we have. Once there are empty food shelves, the disaster's already happened (at least in the short term you can't materialize food anymore by doing smart "economics"), and really at that dire point capitalists don't deserve the right to receive profit in the same way as normal times.
If the moral argument for this doesn't convince you - then there is a much more convincable political argument (of "dialectical materialism"), which basically goes as: once basic things like food and shelter aren't a given, and then capitalists gouge prices up creating an even bigger moral catastrophe among the poor, enough people would be fucking mad at them and band together wreck shit up, and some people would eventually have their heads cut off (as happened in many points in society, for example the French revolution). It's only that the current structures of power (the state & capital) are much more powerful than the past that this is more unlikely to happen, but if I were really in power I wouldn't dare try to steer closer to this event, as bloodshed doesn't really feel that good to participate. And that's the most uncontroversial center-wing take I can create.
Sidenote: I think your analogy of bread isn't really comparable to the housing problem though, since housing acts less of an ideal commodity than food in economic terms. Housing is far more troublesome if you approach it in a free-market lense, since it's far easier to create a natural monopoly on land than anything else, and thus it is hard for free markets to create a "fair" price that matches with the actual use-value. That's why even some libertarians in support of free markets also think economic rent from land is unfair business practice and should be controlled by heavy taxes (such as the economic ideology of Georgism: https://en.wikipedia.org/wiki/Georgism)