I (in the CEST time zone) am currently collaborating with a colleague who lives in Australia, because academics often don't get to choose where they end up.
Its a fucking mess, most of the time its ok, we can meet in the early morning here which is late at night there and do most work offline, but when deadlines are looming or issues crop and we have to talk back and forth over stuff urgently it basically cuts the useful time in the day down to like 3 hours at most, and even that results in at least one of us working very unsociable hours.
Remote work I like, I'm very happy working from home and can do so fairly productively, but timezones are a big issue. My example is with two people working in two timezones, if you've got a bunch of people working in different timezones (the person above mentioned 5!) it can quickly become very messy.