One thing no one has mentioned is the tax implications of the different choices for startups - something which can be as important as the technical side of things. The OECD model tax convention, which has been adopted by many pairs of countries for their double taxation avoidance agreements (DTAAs) uses the term "permanent establishment" - "a fixed place of business through which the business of the an enterprise is…
Have you seen this be an issue in practice? Our place of business is our registered office.
The definition is somewhat ambiguous, but it is likely that if there is an actual physical place (even if it is just a server) used to offer goods or services for sale to the public, there is a good chance that those sales are attributable to that server, and will be taxed in the country that server is located in.
This might not be a problem for an established corporation with the sales volume to justify the tax accountant expenditure and payment overhead needed to comply with tax law in multiple countries, but for a bootstrapped startup it can be an important consideration.