Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Continuous Deployment at IMVU: Doing the impossible fifty times a day.

timothyfitz.wordpress.com

Post reply on HN
Print view View original

11–20 of 76 posts

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#11

Post by jeremyw »

mechanical_fish wrote: post #3

Don't be too disappointed if a single submission gets a lukewarm or confused response on HN. The upmods and comments on here are a lot less consistent than what you're used to. ;) Just keep writing. It's really valuable. Also, it's clear to me why your daily routine might sound like science fiction to the median HN reader: A lot of programmers have never seen a system like this. As those of us who were online during…

Hmm, on your Google point, we know that they use partial-cluster deployments extensively, and several presentations point to sophisticated testing of these momentary guinea pig users. I wouldn't hold a one-time lack of a sanity check against their total uptime history. Tests ain't perfect.

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#12

Post by lgriffith »

Interesting idea but....

Looks like meeting that goal would constrain you to write code to be used by a robot and not by a human. There may be many cases where this is both doable and acceptable to the end user. So no problem with that.

I am greatly challenged to see how this could be done for a highly interactive, visually oriented, subtle pattern generating response to user input, type application. Computers are still not as bright as earth worms when it comes to generalized pattern recognition. Which means we programmers are about as bright as earth worms when it comes to writing such code.

How then could computers automatically test all the software reactions to the wonderful and totally unpredictable behavior of mere humans as they interact with your software? The test cases would expand to consume all the resources available for development. All you would get done is writing all but impossible test cases. At least you wouldn't ship bugs.

This does not consider the explosion of combination and permutations of inputs that prohibits exhaustive testing that no matter how many systems you run tests on.

It would be much easier and cheaper to go out of business. Your certainty of being free of shipped bugs would be much better than one in a million.

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#13

Post by amix »

From what I have read Facebook use a similar method: commit and deploy often and rollback if something messes up. We also use this method on Plurk.com and have done so for about a year. Thought, IMVU's case is pretty extreme :)

The major problem is rolling back client side changes (that are located in scripts or CSS). This is pretty costly to rollback, because of browser cache - we solve this by having real versioning of the static files so we can force a refresh of browser cache (real versioning = script_{timestamp}.js and not script.js?v={timestamp}).

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#14

Post by delano »

amix wrote: post #13

From what I have read Facebook use a similar method: commit and deploy often and rollback if something messes up. We also use this method on Plurk.com and have done so for about a year. Thought, IMVU's case is pretty extreme :) The major problem is rolling back client side changes (that are located in scripts or CSS). This is pretty costly to rollback, because of browser cache - we solve this by having real versionin…

What do you mean by real versioning?

The major advantage to using "script.js?v={timestamp}" is that it maintains a consistent URI for the resource. Whereas with "script_{timestamp}.js", everything that points to it needs to be updated every time it changes.

You could create a symbolic link or rewrite rule that directs requests for "script.js" to the latest "script_{timestamp}.js" but it's more convenient to use a URI parameter.

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#15

Post by ivey »

plinkplonk wrote: post #7
patio11 wrote: post #2

One of the greatest lines I have ever read on a blog: It may be hard to imagine writing rock solid one-in-a-million-or-better tests that drive Internet Explorer to click ajax frontend buttons executing backend apache, php, memcache, mysql, java and solr. I am writing this blog post to tell you that not only is it possible, it’s just one part of my day job.

"writing rock solid one-in-a-million-or-better tests that drive Internet Explorer" I find this unparseable. (English is not my native language). As far as I know "one in a million" means something like "very rare". Help?

When I say reliable, I don’t mean “they can fail once in a thousand test runs.” I mean “they must not fail more often than once in a million test runs.” We have around 15k test cases, and they’re run around 70 times a day. That’s a million test cases a day. Even with a literally one in a million chance of an intermittent failure per test case we would still expect to see an intermittent test failure every day.

He means that test failures are not acceptable in their culture, and they have a LOT of tests.

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#16

Post by willwagner »

amix wrote: post #13

From what I have read Facebook use a similar method: commit and deploy often and rollback if something messes up. We also use this method on Plurk.com and have done so for about a year. Thought, IMVU's case is pretty extreme :) The major problem is rolling back client side changes (that are located in scripts or CSS). This is pretty costly to rollback, because of browser cache - we solve this by having real versionin…

I've worked at places that have done this sort of thing too. We basically dropped css and js files in new directory with the svn rev number which made it very easy to deploy and break through the client side cache.

The urls would be something like:

/static/r12345/foo.js and /static/r12345/foo.css

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#17

Post by abstractbill »

amix wrote: post #13

From what I have read Facebook use a similar method: commit and deploy often and rollback if something messes up. We also use this method on Plurk.com and have done so for about a year. Thought, IMVU's case is pretty extreme :) The major problem is rolling back client side changes (that are located in scripts or CSS). This is pretty costly to rollback, because of browser cache - we solve this by having real versionin…

commit and deploy often and rollback if something messes up

This describes pretty well what we do at Justin.TV too. These days I push new code about 5 times a day.

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#18

Post by briansmith »

plinkplonk wrote: post #7
patio11 wrote: post #2

One of the greatest lines I have ever read on a blog: It may be hard to imagine writing rock solid one-in-a-million-or-better tests that drive Internet Explorer to click ajax frontend buttons executing backend apache, php, memcache, mysql, java and solr. I am writing this blog post to tell you that not only is it possible, it’s just one part of my day job.

"writing rock solid one-in-a-million-or-better tests that drive Internet Explorer" I find this unparseable. (English is not my native language). As far as I know "one in a million" means something like "very rare". Help?

"rock solid" is "very reliable" "one-in-a-million-or-better tests" is "tests which fail less than one in a million times".

Our Internet-Explorer-based tests are very reliable; they fail less than once per million executions.

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#19

Post by amix »

delano wrote: post #14
amix wrote: post #13

From what I have read Facebook use a similar method: commit and deploy often and rollback if something messes up. We also use this method on Plurk.com and have done so for about a year. Thought, IMVU's case is pretty extreme :) The major problem is rolling back client side changes (that are located in scripts or CSS). This is pretty costly to rollback, because of browser cache - we solve this by having real versionin…

What do you mean by real versioning? The major advantage to using "script.js?v={timestamp}" is that it maintains a consistent URI for the resource. Whereas with "script_{timestamp}.js", everything that points to it needs to be updated every time it changes. You could create a symbolic link or rewrite rule that directs requests for "script.js" to the latest "script_{timestamp}.js" but it's more convenient to use a URI…

The problem with script.js?v={timestamp} is that it's ignored by some browsers while script_{timestamp}.js isn't. And with script.js?v={timestamp} you can't set good cache headers.

Also, if you ever move to a CDN, then you are forced to use real versioning (at least with Amazon Cloudfront).

The versioning scheme we use is `md5 hash of name + file contents + file extension` (and not timestamp).

Post reply on HN