On a longer timeframe, if you don't grow, you die.
Scenario time...
Let's say I have a company that is growing pretty well. After some time, I feel the easy growth stops. Let's say this is the "right size" point you are alluding to, and I was able to recognize it, spot on. I stop hiring and just work with the existing resources that I have. In theory everything should be good for a long time.
Unfortunately, I do not exist in a vacuum. I have competition, and this competition continues to expand. Eventually, if they continue to invest and grow, even past their "right size", they will take away my market share. As they grow stronger, I will grow weaker. This may cause my company to lose enough sales that I will have to fire people.
The fallacy is that there is a "right size" that will continue to be the "right size" through time. A market is a dynamic system that does not stand still. What was the right size at one point, may be too large or too small in the future, due to things beyond my direct control.
As a smart businessman, I have to understand this. I have to expand when there is room, and try to outperform the competition when the market is crowded. I need to downsize when the time is right to protect the rest of the business.
What you are saying is easier done on a smaller scale. There are many so-called "lifestyle" businesses which live by a philosophy that is closer to yours and farther from Twitter's. On a global scale, it much more difficult to follow the "lifestyle" business philosophy and succeed.