Purchasing/licensing software is always a hassle. Even aside from the monetary cost, you need to decide which to buy, go through the purchase process, and sometimes you also have some activation hassle.
With open source software, you just download it and use it. If you decide to change to another version of the software or different software entirely, you just do that.
Startups are hard, removing anything that takes time or effort is a good thing.
But even with this, cost is still an issue. Yes, salaries are the biggest cost, but in a startup, you might be working for yourself, so no salaries are being paid. Having to shell out any money at all becomes something you avoid. (Yes, there are Microsoft programs to give you Microsoft software at very low prices. But the procedure for those programs is always more complicated than "download and run".)