Live data from Hacker News

"How hard can it be to implement?"

answers.37signals.com

51–60 of 102 posts

Re: "How hard can it be to implement?"

#51

Earlier quoted context omitted.

> (let me guess, early 20s?), No. > but in this case you literally don't know what you're talking about. I don't need to know what exact decisions they made to be able to tell that they were poor decisions. Easy things should be easy, and when they're hard, they're hard because of incompetence somewhere in the process. FWIW, I've worked on a site extremely similar to basecamp (but with far more traffic, at least acco…

Should you implement something with an array or a linked list? That depends. Should it be fast to locate an element in the middle? Or should it be cheap to add another element regardless of the size? Ideally, both, but you weigh the pros and cons and choose one. And even if it is the right choice, you might still run into situations where the other choice would have been better and people will comment that your decis…

This is not necessarily about whether the denormalized database structure is appropriate. It is more about:

Why the h... doesn't their ORM, or even better their database, already take care of cascading updates?

Why does it have to be implemented manually for every code that updates something? (... which is of course expensive and prone to errors)

Re: "How hard can it be to implement?"

#52

Earlier quoted context omitted.

Yes, in your armchair analysis that takes into account none of the issues that they've dealt with in getting where they are today, any past architectural decisions that make this particular feature difficult to implement indicate incompetence. Your confidence probably serves you well (let me guess, early 20s?), but in this case you literally don't know what you're talking about.

> (let me guess, early 20s?), No. > but in this case you literally don't know what you're talking about. I don't need to know what exact decisions they made to be able to tell that they were poor decisions. Easy things should be easy, and when they're hard, they're hard because of incompetence somewhere in the process. FWIW, I've worked on a site extremely similar to basecamp (but with far more traffic, at least acco…

Pick 10,000 random features and aspects of a software program to be implemented over a 10 year period. Some architectural choices will benefit some and harm others. There is no choice that will make "all easy problems easy". You can't just arrive with one specific feature pulled out of thin air 6 years later and claim that if it is not easy than the system architects were incompetent. Do you really not see that?

Re: "How hard can it be to implement?"

#53
post #50

Earlier quoted context omitted.

HP sells servers with up to 64 cores and 2 TB of memory. Considering the car that DHH just bought himself, they should be able to buy/lease several of them.

Are these x86 servers? Or do you mean a single chip with 64 cores?

This thing: http://h10010.www1.hp.com/wwpc/us/en/sm/WF04a/15351-15351-33...

Re: "How hard can it be to implement?"

#54

Earlier quoted context omitted.

> (let me guess, early 20s?), No. > but in this case you literally don't know what you're talking about. I don't need to know what exact decisions they made to be able to tell that they were poor decisions. Easy things should be easy, and when they're hard, they're hard because of incompetence somewhere in the process. FWIW, I've worked on a site extremely similar to basecamp (but with far more traffic, at least acco…

Should you implement something with an array or a linked list? That depends. Should it be fast to locate an element in the middle? Or should it be cheap to add another element regardless of the size? Ideally, both, but you weigh the pros and cons and choose one. And even if it is the right choice, you might still run into situations where the other choice would have been better and people will comment that your decis…

Whatever the case, when I see a programmer doing a linear search through a linked list, I think you must agree that I'm justified in saying that I have no reason, based on that observation, to trust in that programmer's competence. In fact, I think it could easily be argued that in the absence of additional data, it would actually constitute a reason not to trust the programmer's competence.

While I fully agree that trade offs like yours do occur in our profession, their mere existence is no argument against my initial non-trust of 37signals' competence: you need to actually demonstrate that this problem is one of those cases, and I think that would be difficult to do.

The problem of associating entities in a way that their associations can be modified easily and consistently (i.e., normalization) is effectively a solved problem. Just like the programmer complaining that his program is slow because he's doing a linear search through a linked list, a BaseCamp programmer complain that a requested feature is difficult to implement because their data is in an abnormal form does not inspire confidence, and certainly gives no reason to trust his competence.

Re: "How hard can it be to implement?"

#55

Earlier quoted context omitted.

> (let me guess, early 20s?), No. > but in this case you literally don't know what you're talking about. I don't need to know what exact decisions they made to be able to tell that they were poor decisions. Easy things should be easy, and when they're hard, they're hard because of incompetence somewhere in the process. FWIW, I've worked on a site extremely similar to basecamp (but with far more traffic, at least acco…

Pick 10,000 random features and aspects of a software program to be implemented over a 10 year period. Some architectural choices will benefit some and harm others. There is no choice that will make "all easy problems easy". You can't just arrive with one specific feature pulled out of thin air 6 years later and claim that if it is not easy than the system architects were incompetent. Do you really not see that?

> You can't just arrive with one specific feature pulled out of thin air 6 years later and claim that if it is not easy than the system architects were incompetent.

The problem is easy. Seriously, it really is. Most of the programmers here have probably done something almost exactly like it ten or more times in their relatively short careers; I know I have. There is no real intrinsic difficulty to the problem.

I'm not saying that the programmers are 37signals were incompetent. I'm saying that I have do not trust that they are competent. I have no reason, especially in the face of this evidence, to believe that they're competent. They made a successful software product: so have lots of other programmers of questionable competency.

The fact remains that at least in this example, one of their programmers points at an intrinsically easy problem and says, "This is really hard because I'd have to do a lot of stuff that I really shouldn't have to do" except he doesn't seem to realize that he shouldn't have do those things. So tell me, in face of that obvious fact, why should I trust 37signals' engineers' competence?

Re: "How hard can it be to implement?"

#57

Earlier quoted context omitted.

What's the point of having ACID when you don't use its advantages, one might wonder.

So your solution is to reimplement Basecamp on top of another database?

Rails has always been very MySQL-centric, probably because 37S & DHH use it for Basecamp. Perhaps now that PostgreSQL v9.0 has built in master/slave replication they will finally make the switch.

Re: "How hard can it be to implement?"

#58

Earlier quoted context omitted.

Should you implement something with an array or a linked list? That depends. Should it be fast to locate an element in the middle? Or should it be cheap to add another element regardless of the size? Ideally, both, but you weigh the pros and cons and choose one. And even if it is the right choice, you might still run into situations where the other choice would have been better and people will comment that your decis…

Whatever the case, when I see a programmer doing a linear search through a linked list, I think you must agree that I'm justified in saying that I have no reason, based on that observation, to trust in that programmer's competence. In fact, I think it could easily be argued that in the absence of additional data, it would actually constitute a reason not to trust the programmer's competence. While I fully agree that…

> Whatever the case, when I see a programmer doing a linear search through a linked list, I think you must agree that I'm justified in saying that I have no reason, based on that observation, to trust in that programmer's competence. In fact, I think it could easily be argued that in the absence of additional data, it would actually constitute a reason not to trust the programmer's competence

In Erlang, the most common data structure is the linked list. There's no way to search them other than linearly. You are claiming that in general we should suspect that Erlang programmers are incompetent?

Re: "How hard can it be to implement?"

#59

Is there really no simpler way to solve this problem? "Moving a message needs to move all of the message's comments, and all of the comments' files, and all of the comments' files' versions." Why can't you just change some top level reference in the database? I'm imagining a Projects table and a TodoLists table. Each TodoList has something like a projectID foreign key right? Why can't you just change that and automat…

You're falling into the same trap all programmers do when they perpetually underestimate the time to accomplish any task. The main difficulty in estimating time is that you don't know what you don't know, so you have to actually get into the nitty gritty of implementing it, and if you're smart enough you'll hopefully catch all the requirements before you actually launch it. I'm going to take Sam's word on this that t…

> The main difficulty in estimating time is that you don't know what you don't know, so you have to actually get into the nitty gritty of implementing it, and if you're smart enough you'll hopefully catch all the requirements before you actually launch it.

That's certainly true for a low-level, high-precision estimate, which you can only make when you've got detailed requirements. Prior to that you still need to be able to deliver an estimate, and the best approach to take is to provide an estimate with Quantified Uncertainty. The phrase is important, because the estimate is for managers and managers are all about quantifying risks, and uncertainty in a development estimate is definitely a risk.

When you say "I don't know how long this will take, because I don't know how much I don't know about the solution" you're giving an estimate with uncertainty, but you're not quantifying the uncertainty. It's got no bounds, it's limitless. Managers can't make any decisions with that, so it's useless information. But if you say "This will take 4 to 8 effort weeks, and I'll be able to give a more precise estimate after 2 effort weeks of investigation" that gives your manager something to work with. You're still uncertain, but you've quantified the bounds on the uncertainty, and decisions can be made. For example, solving the problem might not be worthwhile unless it happens within the next two weeks (eg: to meet a release deadline) so the manager can just delay the work until later.

The trick with this approach is setting the bounds. The man who taught me this style of estimating called it a Surprise Range. The lower bound should be an estimate such that you'd be surprised if it took less effort than that, and the upper bound should be an estimate such that you'd be surprised if it took more effort than that. Given whatever information you have about the requirements, you just keep pushing your bounds until you can honestly say "I really don't think it would take less/more effort than that."

As you get more experienced, you'll naturally start accounting for the unknown unknowns, because your past experiences with them will be nagging in the back of your mind while you're trying to come up with an estimate for some new task. This is especially true if you're estimating work on a project you've worked on before; you'll have a feel for where the trouble spots are and whether or not the task in question is going to get into those spots.

The other trick with this approach is the "more precise estimate after some investigation" bit of the statement. You need to provide some estimate now, buy some investigation time, and commit to another estimate later. Most of the time the second estimate will fall somewhere within the bounds of the first one, but not always, and your manager should be aware of that possibility. But with experience, you'll usually wind up within the original bounds and narrower than the original bounds too, which is quantifiably less uncertainty than before so you can show progress.

Re: "How hard can it be to implement?"

#60
post #30
post #24

Earlier quoted context omitted.

37signals don't use sharding. They use one database server with enough RAM to load the db into memory

One database server for the whole of basecamp? Edit: I found this: > With that in mind, we went looking for an option to host the Basecamp database, which is becoming a monster. As of this writing, the database is 325GB and handles several thousand queries per second at peak times. 325GB RAM?! But now they have multiple servers. Read more: http://37signals.com/svn/posts/2479-nuts-bolts-database-serv...

You can get servers that can house 512GB or 1TB of RAM from HP/Sun/Dell. Scaling out your app servers and scaling up your DB is common in systems that don't need to be truly "web scale".
Post reply on HN