Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Continuous Deployment at IMVU: Doing the impossible fifty times a day.

timothyfitz.wordpress.com

Post reply on HN
Print view View original

71–76 of 76 posts

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#71

Post by Retric »

akronim wrote: post #68
DannoHung wrote: post #4

I tell people that we should aim for this sort of automation and they pat me on the head and say, "No, no, that will never do." I think there's an idea that if something goes wrong because you let an automated system do it, it's somehow much worse than if something goes wrong because there was human error. I don't really understand the reasoning.

There's a difference between automation and how often your customers see something going wrong. I'm all for the automation. But lets say the error rate is so low that just 0.1% of automated releases go wrong. Rolling out 50 times a day means you'll expose an error every 20 days. Compare that to a monthly, weekly or even daily cycle and you can see you're exposing yourself and your customers to problems without much c…

If you're doing 10 releases a day and someone notices a bug then:

A) You can quickly fix it reducing the number of people that see that bug.

B) You can probably release the fix without breaking other things.

Is it better for you to release 5 bugs that 5% of your users sees for a few hours or release 1 bug that 100% of your users see for 2 weeks?

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#72

Post by lgriffith »

mudetroit wrote: post #57
lgriffith wrote: post #54

Earlier quoted context omitted.

Maybe the problem is that you have the 20-man team. There is no coherence in the code. The design is wrong, coupling is too high, and the module cohesion is too low. The large team makes certain that is the case no matter how "tight" (aka heavy) your quality control process. I have found from working in large teams, there is a core four who get things done. The rest are simply dead weight dedicated to shuffling paper…

This works fine if you are tackling a problem that can be sufficiently addressed by 4 developers. Depending on the size and scale of the problem you are trying to suggest and the time line required for delivery you may need a larger team. When you begin to take that into account you realize you have to find ways for the larger team to work together and still produce a quality product. Hence the techniques being used…

"...you have to find ways for the larger team to work together and still produce a quality product. "

I am not sure its possible. The communication overhead of so many linkages forces incoherence. The resultant incoherence forces still more additions to process and body count. That adds still more communication overhead. The result is still more incoherence - not less. If something is "finished", its simply because time, money, resources, and toleration ran out. The end result was simply called "done".

Maybe that is the best we can do but I am hard pressed to call products produced that way quality products. See Vista et.al. for instructive detail.

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#73

Post by eries »

sbraford wrote: post #56
TimothyFitz wrote: post #42

Earlier quoted context omitted.

Honestly I think it's a gradient. I'm also one of the developers on a hobby project called http://TIGdb.com (Jeff Lindsay is the other, and has written the majority of the website) We don't have a big Continuous Deploy infrastructure, but we also don't have the users and business requirements of IMVU. We started with the usual, completely manual deploys and hard-to-setup sandboxes, and have been iterating towards a f…

Just curious - who maintains the Selenium tets, and how big is the development / "QA" team? I've never worked in a team big enough that it could devote resources to maintaining all of the following kinds of tests: * unit * functional * AND acceptance * plus writing the actual code IMHO, a neutral third-party group like QA should be responsible for writing & maintaining acceptance tests.

Just a thought: perhaps these two facts are related?

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#75

Post by sbraford »

TimothyFitz wrote: post #64
nuclear_eclipse wrote: post #60

Earlier quoted context omitted.

The part you're missing is the 15000 tests, multiplied by a new commit every 9 minutes, which in 8 working hours, is roughly 50 commit-test cycles, so 750,000 tests run in a day's timespan... Edit: of course that assumes a peak commit rate matching or exceeding the commit-test cycle period. The point being that even a considerably low rate of failure in the testing mechanism could manifest itself as a blocked commit-…

We empirically have on average 70 builds a day. The number is higher than your calculation because we don't all work 9-5, we're commiting frequently from around 8am to 9pm. We also run builds repeatedly overnight to flush out any intermittently failing tests we may have recently introduced. We'll run the builds as fast as they can go from 2am-4am.

So how often does a commit get checked in that causes a test (or tests) to fail?

It just seemed to me like you were bragging that tests get run over and over again. They only need to get run if any new code is committed, of course.

And what kind of commit is being checked in every 9 minutes? How big is the dev team? Seems like an awful lot of commits. Is each one a full-fledged feature / bug fix for the site, or are many 1-line changes to the code?

Re: Continuous Deployment at IMVU: Doing the impossible fifty times a day.

#76

Post by jmathes »

shiranaihito wrote: post #45
BeefingJection wrote: post #39

Earlier quoted context omitted.

The obvious solution, of course, is to think things through.

Right, but doing that 50 times per day is more challenging than once, for example.

Whether you commit your code once in one batch at the end of the day or 50 times in 50 smaller chunks, you have the same amount of complexity about which to be careful. In fact it's more complex in the former case, because in the latter, for each push, you know that all the previous pushes are working.
Post reply on HN