Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Continuous Deployment at IMVU: Doing the impossible fifty times a day.

timothyfitz.wordpress.com

Post reply on HN
Print view View original

51–60 of 76 posts

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#51

Post by TimothyFitz »

abstractbill wrote: post #17
amix wrote: post #13

From what I have read Facebook use a similar method: commit and deploy often and rollback if something messes up. We also use this method on Plurk.com and have done so for about a year. Thought, IMVU's case is pretty extreme :) The major problem is rolling back client side changes (that are located in scripts or CSS). This is pretty costly to rollback, because of browser cache - we solve this by having real versionin…

commit and deploy often and rollback if something messes up This describes pretty well what we do at Justin.TV too. These days I push new code about 5 times a day.

I've read your post on unit tests, and I didn't understand what you were trying to say.

Were you saying don't write automated tests that test your code, instead focus on monitoring the actual production invironment?

Or were you saying that specifically the "unit test" class of automated tests are not worth their time?

I can imagine a system that monitors the business metrics well enough to prevent defects from slipping into production (it's a stretch, metrics are soft and squishy moving targets), but I can't imagine using only those metrics to find every bug you ever slip into production. Metrics are so distant from the bug that caused their downturn; you'd waste so many cycles debugging. The gap between writing the code and finding the problem would be much larger than if unit tests found them; that has to slow things down as well.

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#52

Post by JabavuAdams »

mechanical_fish wrote: post #3

Don't be too disappointed if a single submission gets a lukewarm or confused response on HN. The upmods and comments on here are a lot less consistent than what you're used to. ;) Just keep writing. It's really valuable. Also, it's clear to me why your daily routine might sound like science fiction to the median HN reader: A lot of programmers have never seen a system like this. As those of us who were online during…

> As those of us who were online during a specific half-hour period a couple weeks ago can attest, even Google doesn't have a system that's remotely as reliable as this: It appears to be possible to break all of Google search, worldwide, in ten minutes by misplacing a single character in a text file.

How can we conclude that the same isn't true of IMVU? The fact that such a rare event hasn't happened to them yet tells us very little.

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#53

Post by [deleted] »

plinkplonk wrote: post #7
patio11 wrote: post #2

One of the greatest lines I have ever read on a blog: It may be hard to imagine writing rock solid one-in-a-million-or-better tests that drive Internet Explorer to click ajax frontend buttons executing backend apache, php, memcache, mysql, java and solr. I am writing this blog post to tell you that not only is it possible, it’s just one part of my day job.

"writing rock solid one-in-a-million-or-better tests that drive Internet Explorer" I find this unparseable. (English is not my native language). As far as I know "one in a million" means something like "very rare". Help?

[deleted]

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#54

Post by lgriffith »

jmathes wrote: post #38
lgriffith wrote: post #34

Earlier quoted context omitted.

Then why do such an exhaustive automated test? Why not have your local tests, automated or not, cover the common cases and error conditions to catch programmer stupidities? Then let the actual humans do the strange corner cases. If your design is even close to correct, testing repeatedly tested code is pointless. If your design is corrupt and your implementation is sloppy, no amount of testing is going to save your a…

I think the source of your confusion is that you're a one man team. You don't have to solve problems that 20-man teams have to face. At least half of all the code I depend on is code I do not understand, so I have to depend on its tests, and I have to make the same promise to consumers of my code. If my change breaks code someone else wrote that I didn't foresee, I am depending on his tests to tell me what I screwed…

Maybe the problem is that you have the 20-man team. There is no coherence in the code. The design is wrong, coupling is too high, and the module cohesion is too low. The large team makes certain that is the case no matter how "tight" (aka heavy) your quality control process.

I have found from working in large teams, there is a core four who get things done. The rest are simply dead weight dedicated to shuffling paper and attending meetings. At best, they do nothing. At worst they create more work than they do.

Use the right four and dump the other sixteen. You will get at least ten times more productivity and ten times higher quality without even breaking a sweat. If you don't have the right four, you are hosed from the start.

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#55

Post by sbraford »

briansmith wrote: post #18
plinkplonk wrote: post #7

Earlier quoted context omitted.

"writing rock solid one-in-a-million-or-better tests that drive Internet Explorer" I find this unparseable. (English is not my native language). As far as I know "one in a million" means something like "very rare". Help?

"rock solid" is "very reliable" "one-in-a-million-or-better tests" is "tests which fail less than one in a million times". Our Internet-Explorer-based tests are very reliable; they fail less than once per million executions.

While the "one in a million" better is a cool blurb, what does it really mean?

Let's say your team makes 25 commits per day.

25 * ~300 working days = 7,500 commits per year

That would take 133+ years to reach 1 in a million.

The more interesting metric to me is how often the build gets broken.

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#56

Post by sbraford »

TimothyFitz wrote: post #42
inerte wrote: post #33

I think the original article misguided some people. It all looked very simple, update the code and put it in production. That _is a horrible idea_, as some have noted. What's not horrible is having thousands of tests, on dozen of machines, 9 minutes to-live, with selective updating of users, and rollbacks, as this article has explained. The original post was too light on details, I guess. Its intention was not to be…

Honestly I think it's a gradient. I'm also one of the developers on a hobby project called http://TIGdb.com (Jeff Lindsay is the other, and has written the majority of the website) We don't have a big Continuous Deploy infrastructure, but we also don't have the users and business requirements of IMVU. We started with the usual, completely manual deploys and hard-to-setup sandboxes, and have been iterating towards a f…

Just curious - who maintains the Selenium tets, and how big is the development / "QA" team?

I've never worked in a team big enough that it could devote resources to maintaining all of the following kinds of tests: * unit * functional * AND acceptance * plus writing the actual code

IMHO, a neutral third-party group like QA should be responsible for writing & maintaining acceptance tests.

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#57

Post by mudetroit »

lgriffith wrote: post #54
jmathes wrote: post #38

Earlier quoted context omitted.

I think the source of your confusion is that you're a one man team. You don't have to solve problems that 20-man teams have to face. At least half of all the code I depend on is code I do not understand, so I have to depend on its tests, and I have to make the same promise to consumers of my code. If my change breaks code someone else wrote that I didn't foresee, I am depending on his tests to tell me what I screwed…

Maybe the problem is that you have the 20-man team. There is no coherence in the code. The design is wrong, coupling is too high, and the module cohesion is too low. The large team makes certain that is the case no matter how "tight" (aka heavy) your quality control process. I have found from working in large teams, there is a core four who get things done. The rest are simply dead weight dedicated to shuffling paper…

This works fine if you are tackling a problem that can be sufficiently addressed by 4 developers. Depending on the size and scale of the problem you are trying to suggest and the time line required for delivery you may need a larger team.

When you begin to take that into account you realize you have to find ways for the larger team to work together and still produce a quality product. Hence the techniques being used by the author and other companies out there trying to address similar problems.

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#58

Post by abstractbill »

TimothyFitz wrote: post #51
abstractbill wrote: post #17

Earlier quoted context omitted.

commit and deploy often and rollback if something messes up This describes pretty well what we do at Justin.TV too. These days I push new code about 5 times a day.

I've read your post on unit tests, and I didn't understand what you were trying to say. Were you saying don't write automated tests that test your code, instead focus on monitoring the actual production invironment? Or were you saying that specifically the "unit test" class of automated tests are not worth their time? I can imagine a system that monitors the business metrics well enough to prevent defects from slippi…

Here's where we are putting our effort:

- Monitoring the production environment, tons of effort. We record and analyze an incredible amount of data about everything that happens on the site, and have more and more automated processes looking for anomalies (though still nowhere near as many as I would like).

- Automated testing not including unit tests, some effort. I wouldn't be opposed to us doing more of this, but it's not incredibly high-priority and there always seems to be something else that's more important.

- Unit testing, yeah, not worth our time as far as I'm concerned.

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#60

Post by nuclear_eclipse »

sbraford wrote: post #55
briansmith wrote: post #18

Earlier quoted context omitted.

"rock solid" is "very reliable" "one-in-a-million-or-better tests" is "tests which fail less than one in a million times". Our Internet-Explorer-based tests are very reliable; they fail less than once per million executions.

While the "one in a million" better is a cool blurb, what does it really mean? Let's say your team makes 25 commits per day. 25 * ~300 working days = 7,500 commits per year That would take 133+ years to reach 1 in a million. The more interesting metric to me is how often the build gets broken.

The part you're missing is the 15000 tests, multiplied by a new commit every 9 minutes, which in 8 working hours, is roughly 50 commit-test cycles, so 750,000 tests run in a day's timespan...

Edit: of course that assumes a peak commit rate matching or exceeding the commit-test cycle period. The point being that even a considerably low rate of failure in the testing mechanism could manifest itself as a blocked commit-test-deploy cycle at least once a day, hence the importance placed on rock-solid testing systems that should only ever fail when the tested code itself fails.

Post reply on HN