Here's why many "rock star" designers don't like (generally) working for a single company or brand like a startup. First you have to understand not all designers are created equal. You tend to have "Rock Stars" (or in the agency speak a Creative Director) and a spectrum down to "production" designers. "Creative Directors" tend to be good at a lot of the different design disciplines, they understand business objectives, they can help with product and feature vision, they understand usability and they can make things pretty. Its the entire package.
If they go to work for a brand new startup they start with a blank slate, all the needs and challenges are large and the designer is fully engaged because they have to draw on all their different skill sets to solve problems.
But as time progresses the needs of a startup tends to shift from where there is less need to do full UX the product is now defined and its about making small incremental changes. This is where they go from being a "Creative Director" to a production designer. As a CD they get to design the web site with a brand new look and feel, as a production designer they're doing a new module on the home page which matches the existing design standards.
Basically the challenges keep getting smaller, and the designer is less engaged. There's a similar arc in development, however while at first you need to do "Big Things" (get the site up) and some of the changes become smaller, at least those small things can be as challenging (or more so) then the "Big Things" (optimize the load time).
Finally as a designer you live and die by your portfolio, and having a one site portfolio gets you nowhere. Unless you're looking for a big exit you're much better off getting more work, and more projects it both grows your portfolio and expands your experience and abilities.