I agree. I think a lot of the advice that I see above is: "come as you are, and if they won't take you, you didn't want to work there anyway."
This is wrong. I work for a startup (my third one). I wore a suit to the interview. My now-boss said to me, "We should have told you that you didn't have to wear a tie to the interview. Sorry!". I would have worn the tie anyway.
I have personally hired 16 people since starting work at my current startup. They all dressed professionally for the interview, and subsequently changed into shorts and t-shirts.
The issue is: while the startup is casual dress, I might need to take you with me to pitch Fidelity Investments for the B-round of funding. If I'm taking you with me to my investment bank, you better wear a suit and be comfortable in it.