I run my company in a similar way. It's myself, three employees and a bunch of contractors. (
https://asylumsquare.com/team )
We have an office, but everyone can work from where-ever he wants to (actually the freelancers are from all around the world). Since Covid hit we're operating 100% remote before most of the time we worked with 2 people in the office the rest of us remotely.
Everyone works as much or less as he wants, can take days off whenever he feels like.
We do have voice calls from time to time when it's appropriate, but in 99.9% only two people are involved and it usually happens when we have the feeling it would clarify faster as by asynchronous chat (for example when screen share is involved to demonstrate something). Actually I wouldn't even call it meetings because we mostly don't even setup a schedule, it just naturally happens (for example chatting about a specific topic, then deciding a call would be easier / faster).
It works well for us, but I think it would be a lot harder if we were more people as we don't have anyone who organizes or manages the project and the people.