* Hardware : Amazon EC2. We also use cloudfront (CDN) and a couple of other services. It's worked out amazingly well really.
* Accounting: My wife (PhD in Accounting). She uses Quickbooks I think. We use ADP for our payroll for quite awhile and that was nightmare. They were late with W2's and failed to pay the proper taxes they said they were paying. Just awful. My wife does payroll now.
* Email: We use authsmtp as our gateway. They've been very solid. Very few spam box issues. We also use mailchimp to manage our actual contact lists.
* Analytics: we use a combination of homegrown tools, google analytics, UserFly and CrazyEgg for usability and conversion analysis.
* Email: we use Google Apps. That's also worked out extremely well for us. No real issues and we rely on Google Docs quite a bit.
* Other services: We use Twilio to handle phone-tracking (it's an integral part of our product). We used UserVoice at one point and that was excellent.
I'll update with other stuff as I think about it.