Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

As Digg Struggles, VP Of Engineering Is Shown The Door

techcrunch.com

Post reply on HN
Print view View original

111–120 of 124 posts

Re: As Digg Struggles, VP Of Engineering Is Shown The Door

#111

Post by rwmj »

davidw wrote: post #102
MartinCron wrote: post #77

Earlier quoted context omitted.

I would advocate an incremental rewrite of parts of the unmaintainable spaghetti classic asp until there's none of the spaghetti left. It's easier to rewrite part of a system than an entire system. Release to production dozens, if not hundreds of times. Releases are non-events, rollbacks are non-events. A system-wide ground-up rewrite with a big-bang switchover at the end is a classic clusterfuck recipe. It's a shame…

> incremental rewrite of parts of the unmaintainable spaghetti classic Sounds good in theory. In practice? Part of the problem with many big ball of mud systems is that all the parts depend on and talk to all the other parts. Want to fix that horrid DB schema? You'll have to rewrite all the code that talks to it, or rewrite it to talk to an intermediary. Want to rewrite that horrid bit of code that's called foobar_20…

In reality yes, it works, I've done it. We took a horrible, accreted web application, and rewrote it in stages over a period of about 12 months. At the same time we were making regular releases, and needless to say the site stayed up the whole time.

You just have to plan things carefully, work hard, and keep your head screwed on. (Just like with many things in life ...)

Re: As Digg Struggles, VP Of Engineering Is Shown The Door

#112

Post by zeemonkee »

MartinCron wrote: post #77
squidsoup wrote: post #68

Earlier quoted context omitted.

I've heard this time and time again, but what if your application is a genuine ball of mud? Would you really not advocate a rewrite for an unmaintainable spaghetti classic asp app still in production today?

I would advocate an incremental rewrite of parts of the unmaintainable spaghetti classic asp until there's none of the spaghetti left. It's easier to rewrite part of a system than an entire system. Release to production dozens, if not hundreds of times. Releases are non-events, rollbacks are non-events. A system-wide ground-up rewrite with a big-bang switchover at the end is a classic clusterfuck recipe. It's a shame…

Another issue is whether you are just rewriting the code, or fundamentally changing your data store (as Digg have done).

Rewriting the code is fine - if it goes wrong, just back up to the old code, users won't notice the difference. You can do it on a page-by-page basis, just route URLs selectively to the new install.

Migrating data is something that should be done under only the most extreme circumstances - and something will inevitably go horribly wrong, so be prepared to rollback.

Re: As Digg Struggles, VP Of Engineering Is Shown The Door

#114

Post by TheCondor »

sliverstorm wrote: post #61
TheCondor wrote: post #57

Earlier quoted context omitted.

So what's the digg story here? I get that some VP suggested a new buzzwordy technology, they gave him enough rope to hang himself and he did and left the company with a broken pile of crap. It could happen, if you have a healthy company you give trust to people. That it got this far doesn't speak well of the rest of management. It doesn't speak really well of the rest of the team either. Shouldn't there have been som…

Never said I supported it. If my comment came across that way, I apologize.

No, I apologize. I didn't mean to sound like I was attacking you or suggest that you supported it. It just seemed like a good place to interject. It's such a transparent move on Digg's part.

Re: As Digg Struggles, VP Of Engineering Is Shown The Door

#115

Post by [deleted] »

mikeryan wrote: post #42
cookiecaper wrote: post #14

Is there any reason that Cassandra is the focus of this article? It is really silly and irresponsible to peg a nascent project like that without any reasoning or sources. I'm sure something changed besides just a Cassandra rollout, and wasn't Digg using it on v3 too? I think Cassandra is pretty well tested. There have been lots of super-large-scale deployments. It just seems lame to blame it on that, but I guess mayb…

So if you watch the Revision3 video where they talk about the stability issues Kevin throws Cassandra out there nothing really specific but when discussing downtime its the only technical factor he mentions. He calls it "still beta software" and states they were fixing bugs in Cassandra during the days leading up to the release of v4.

[deleted]

Re: As Digg Struggles, VP Of Engineering Is Shown The Door

#116

Post by davidw »

rwmj wrote: post #111
davidw wrote: post #102

Earlier quoted context omitted.

> incremental rewrite of parts of the unmaintainable spaghetti classic Sounds good in theory. In practice? Part of the problem with many big ball of mud systems is that all the parts depend on and talk to all the other parts. Want to fix that horrid DB schema? You'll have to rewrite all the code that talks to it, or rewrite it to talk to an intermediary. Want to rewrite that horrid bit of code that's called foobar_20…

In reality yes, it works, I've done it. We took a horrible, accreted web application, and rewrote it in stages over a period of about 12 months. At the same time we were making regular releases, and needless to say the site stayed up the whole time. You just have to plan things carefully, work hard, and keep your head screwed on. (Just like with many things in life ...)

This would make a great topic for blogs/books/whatever. "How to dig yourself out of a pile of shit."

Re: As Digg Struggles, VP Of Engineering Is Shown The Door

#117

Post by MartinCron »

pilif wrote: post #100
MartinCron wrote: post #78

Earlier quoted context omitted.

In a way, the one-time migration might be harder than near-realtime bidirectional synchronization. That way, you could move portions of users to the new system and back as needed. A sudden leap from one backend to another is like jumping over the grand canyon on a motorcycle. Personally, I would rather build a bridge.

For the concurrent configuration to work consistently, digg would probably have to port v3 to the new backend. Now in the current case, the most issues apparently come from the non-working backend as opposed to the changed featureset. So while they could have run the two versions in parallel, they would not have gained anything. Likely, this was their rationale behind not doing so in the first place.

No, for the current configuration to work, you would have to create a way to move data in both directions from the "old" backend to the "new" backend. Ideally, you would slice up the beast so you didn't have to do this with everything all at once.

It's not a system that would be ideal for something transactional like a bank, but it may have been possible for an organization like Digg.

Re: As Digg Struggles, VP Of Engineering Is Shown The Door

#118

Post by MartinCron »

zeemonkee wrote: post #112
MartinCron wrote: post #77

Earlier quoted context omitted.

I would advocate an incremental rewrite of parts of the unmaintainable spaghetti classic asp until there's none of the spaghetti left. It's easier to rewrite part of a system than an entire system. Release to production dozens, if not hundreds of times. Releases are non-events, rollbacks are non-events. A system-wide ground-up rewrite with a big-bang switchover at the end is a classic clusterfuck recipe. It's a shame…

Another issue is whether you are just rewriting the code, or fundamentally changing your data store (as Digg have done). Rewriting the code is fine - if it goes wrong, just back up to the old code, users won't notice the difference. You can do it on a page-by-page basis, just route URLs selectively to the new install. Migrating data is something that should be done under only the most extreme circumstances - and some…

Because migration is a scenario that may go horribly wrong, you should be prepared to tackle it in small pieces. So often (especially with relational databases, I don't have much experience with the NoSQL realm) everything is in one logical and physical data store, even for non-related features. That makes any migration harder than it needs to be.

Re: As Digg Struggles, VP Of Engineering Is Shown The Door

#119

Post by MartinCron »

davidw wrote: post #102
MartinCron wrote: post #77

Earlier quoted context omitted.

I would advocate an incremental rewrite of parts of the unmaintainable spaghetti classic asp until there's none of the spaghetti left. It's easier to rewrite part of a system than an entire system. Release to production dozens, if not hundreds of times. Releases are non-events, rollbacks are non-events. A system-wide ground-up rewrite with a big-bang switchover at the end is a classic clusterfuck recipe. It's a shame…

> incremental rewrite of parts of the unmaintainable spaghetti classic Sounds good in theory. In practice? Part of the problem with many big ball of mud systems is that all the parts depend on and talk to all the other parts. Want to fix that horrid DB schema? You'll have to rewrite all the code that talks to it, or rewrite it to talk to an intermediary. Want to rewrite that horrid bit of code that's called foobar_20…

In practice? It's not easy, but nothing is. I see three options to the big ball of mud problem.

1. Wallow in it (work with existing structure). Sadly, this is what a lot of people do. I left a job once because that was the only way out of the ball of mud. I was afraid I would turn into a mud-person.

2. Slowly crawl out of it (incremental rewrite). This is hard, but do-able. It involves setting up barriers to mitigate ripple effects, automated testing to be comfortable with frequent releases, and tolerance for temporary imperfection (basically, you need to be willing to frequently release things that are only a tiny bit better than status quo, even if it's not ideal). Not everyone is willing to accept this persistent imperfection and lack of conceptual consistency, especially when option #3 is more exciting and fun.

3. Try to leap out of it and land wherever (total rewrite). This is very, very hard, and prone to total failure or spending valuable money and time to effectively stand still in the market.

I've seen development teams leap directly out of one ball of mud into a different one. One where nobody even knew their way around anymore. How is that anything but a huge waste of time and resources?

Re: As Digg Struggles, VP Of Engineering Is Shown The Door

#120

Post by MartinCron »

davidw wrote: post #116
rwmj wrote: post #111

Earlier quoted context omitted.

In reality yes, it works, I've done it. We took a horrible, accreted web application, and rewrote it in stages over a period of about 12 months. At the same time we were making regular releases, and needless to say the site stayed up the whole time. You just have to plan things carefully, work hard, and keep your head screwed on. (Just like with many things in life ...)

This would make a great topic for blogs/books/whatever. "How to dig yourself out of a pile of shit."

I agree, although it's a somewhat sensitive thing to write about if you're doing it in practice. I sure as hell don't want to be known as the guy who comes in and calls all of the existing code a pile of shit.

I guess I could "change the names to protect the innocent" and tell some stories about digging out of tight places incrementally. If it would convince even one development team that they didn't absolutely have to do a total rewrite, it would be worth it.

Post reply on HN