Whoa, I love the sound of this as far as development process... But what really blew me away is: 3D Chat makes $1M/month? Really? Or did I find the wrong IMVU?
Continuous Deployment at IMVU: Doing the impossible fifty times a day.
61–70 of 76 posts
Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.
#62Earlier quoted context omitted.
what do you want to know?
Everything :) No, seriously, I'd be much obliged if you could tell what tools go into your setup, how much of it is created in house - and thus unavailable - and how much of it is off the shelf, preferably open source. I'd very much like to spend time on recreating what you've done there.
I've written in light detail about this in a few places; I'd be glad to share more. Here's an assortment off the top of my head. Feel free to ask anything else you'd like to know.
http://startuplessonslearned.blogspot.com/2009/02/continuous...
http://startuplessonslearned.blogspot.com/2008/11/five-whys....
http://startuplessonslearned.blogspot.com/2008/09/new-versio...
http://startuplessonslearned.blogspot.com/2008/12/continuous...
Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.
#63Earlier quoted context omitted.
Everything :) No, seriously, I'd be much obliged if you could tell what tools go into your setup, how much of it is created in house - and thus unavailable - and how much of it is off the shelf, preferably open source. I'd very much like to spend time on recreating what you've done there.
I've written in light detail about this in a few places; I'd be glad to share more. Here's an assortment off the top of my head. Feel free to ask anything else you'd like to know. http://startuplessonslearned.blogspot.com/2009/02/continuous... http://startuplessonslearned.blogspot.com/2008/11/five-whys.... http://startuplessonslearned.blogspot.com/2008/09/new-versio... http://startuplessonslearned.blogspot.com/2008/1…
Thank you, I'll be reading all of that later today.
Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.
#64Earlier quoted context omitted.
While the "one in a million" better is a cool blurb, what does it really mean? Let's say your team makes 25 commits per day. 25 * ~300 working days = 7,500 commits per year That would take 133+ years to reach 1 in a million. The more interesting metric to me is how often the build gets broken.
The part you're missing is the 15000 tests, multiplied by a new commit every 9 minutes, which in 8 working hours, is roughly 50 commit-test cycles, so 750,000 tests run in a day's timespan... Edit: of course that assumes a peak commit rate matching or exceeding the commit-test cycle period. The point being that even a considerably low rate of failure in the testing mechanism could manifest itself as a blocked commit-…
We empirically have on average 70 builds a day. The number is higher than your calculation because we don't all work 9-5, we're commiting frequently from around 8am to 9pm. We also run builds repeatedly overnight to flush out any intermittently failing tests we may have recently introduced. We'll run the builds as fast as they can go from 2am-4am.
Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.
#65Whoa, I love the sound of this as far as development process... But what really blew me away is: 3D Chat makes $1M/month? Really? Or did I find the wrong IMVU?
Yep, that IMVU. Here are a few more staggering statistics that we've published: http://www.vator.tv/news/show/2009-01-22-recession-not-affec...
Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.
#66I was thinking of doing a basic Django + Selenium + Hudson continuous integration how-to blog post, but this blows me out of the water :)
I'd still like to see that post, this blog post gives the overview, yours should give the details.
Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.
#67From what I have read Facebook use a similar method: commit and deploy often and rollback if something messes up. We also use this method on Plurk.com and have done so for about a year. Thought, IMVU's case is pretty extreme :) The major problem is rolling back client side changes (that are located in scripts or CSS). This is pretty costly to rollback, because of browser cache - we solve this by having real versionin…
deploy/rollback is probably ok for a consumer site. But not everything is a public website (no really...) - if you're deploying a service with an SLA with dollar penalties for downtime you might want to stick to a more traditional release cycle. I sure hope the phone network, the stock exchange and my bank aren't using deploy/rollback and releasing 50 times a day!
Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.
#68I tell people that we should aim for this sort of automation and they pat me on the head and say, "No, no, that will never do." I think there's an idea that if something goes wrong because you let an automated system do it, it's somehow much worse than if something goes wrong because there was human error. I don't really understand the reasoning.
There's a difference between automation and how often your customers see something going wrong. I'm all for the automation. But lets say the error rate is so low that just 0.1% of automated releases go wrong. Rolling out 50 times a day means you'll expose an error every 20 days. Compare that to a monthly, weekly or even daily cycle and you can see you're exposing yourself and your customers to problems without much corresponding gain.
Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.
#69Earlier quoted context omitted.
"rock solid" is "very reliable" "one-in-a-million-or-better tests" is "tests which fail less than one in a million times". Our Internet-Explorer-based tests are very reliable; they fail less than once per million executions.
While the "one in a million" better is a cool blurb, what does it really mean? Let's say your team makes 25 commits per day. 25 * ~300 working days = 7,500 commits per year That would take 133+ years to reach 1 in a million. The more interesting metric to me is how often the build gets broken.
As I read it, the issue he's talking about is that it's easy to accidentally write a test that passes the first 100 000 times you write it, but then fails the next time because of a timeout that was set too low or something like that. A test like that can waste a lot of your time tracking down a nonexistent bug.
It's true that any particular test that spuriously fails one in a million times may never fail. But if you have tens of thousands of tests, and you do tens of test runs per day, you'll have a test spuriously fail once a day or so.
Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.
#70Why do all changes have to end up in production immediately?
When something breaks in production, it's easier to figure out what it was and fix it if you only changed a few things since the last time you updated production.