I've downsized a few times over the years. 1080 > 920 > 720 > 660. How much have I missed having a bigger space? None. I could probably drop down to about 550 tomorrow and still have tons of room.
The first thing is "usable square footage". In the larger units (> 720) I had things like hallways. Closets for a water heater or furnace. Etc. When I dropped down to 720, I actually had more usable square footage than when I had 1080 square feet. This is because the smaller spaces were better designed for their intended activities, and other things were inlined or made more efficient (in-line electric water heaters, in-wall thermal pump) as a result of the space constraints.
Additionally space isn't to store stuff, it's the support the activities within the space. When I had 1080 square feet, the kitchen was larger than the bedroom I grew up in. It meant a lot of unnecessary walking around to get anything done. When the work triangle shrunk to about 20 square feet, everything was in reach, and I still had tons of room for all the prep work, and an excess of space to store everything. A 160 square foot kitchen was excessively large when 50 works just as well. Just like a 50 square foot "laundry room" is worthless when there wasn't any room to put an ironing board -- now I have a laundry closet (9 square feet?) with some stacked Bosch units that allow me to get just as much done (with a fold-out ironing board on the door). The bedroom went from having a 90-square foot walk in closet, to 14 square feet of reach-ins -- of which I use one of them. I guess if I had a live-in girlfriend she'd use the other. Bathroom? Also shrunk. However the bedroom got larger, as did the main living area. Big wins.
As for stuff… I've gotten rid of tons of it. Every year, clean things out. Every year, be baffled at home much gets tossed out. I've zero clutter now, yet I still have everything I care about. I still have a home-office built into a 15square foot reach-in closet with two 27" displays and a laser printer that's very comfortable to work in. I still have some collectibles stashed in a storage bench at the bottom of the bedroom closet. However if it doesn't have sentimental, monetary, or immediate value, it's gotten tossed at one point. It forces you to think hard about what you value, and stick to it if you don't want to live in clutter or with a giant stack of boxes somewhere.
Coworkers always seem to "feel sorry for me". "How do you live without being able to stock up on toilet paper at Costco?" I guess I don't need nearly as much TP in the bathroom as you do. "How do you live with such a small car?" Yeah, a 2-seat roadster is really roughing it, but y'know, I soldier on. I have everything I want, nothing that wastes my time or attention, less to clean, less space to heat/cool, and a car I drive very sideways. 220 square feet I probably couldn't immediately shift to, but I'm pretty certain I could go to 350 very hastily if I had to. 220 would just require a lot of thought, and giving up activities such being able to host thanksgiving, etc.