Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

"How hard can it be to implement?"

answers.37signals.com

Post reply on HN
Print view View original

71–80 of 102 posts

Re: "How hard can it be to implement?"

#71

Post by gaius »

dabeeeenster wrote: post #20
prodigal_erik wrote: post #15

Earlier quoted context omitted.

From a couple of web searches it sounds like they're on MySQL. Targeting a database that has working transactions has to be less work than implementing transactions yourself from scratch, which it seems everyone stuck on MySQL eventually has to do (that's what they're describing, and I know we've done similar things).

Depends what MySQL table types they are using. I think MyISAM has no transactional ability.

I'd argue any database on modern hardware where you can't do a single transaction on a few hundred rows without noticeably affecting performance does not have "working transactions".

Re: "How hard can it be to implement?"

#73

Post by jemfinch »

dasil003 wrote: post #65
jemfinch wrote: post #55

Earlier quoted context omitted.

> You can't just arrive with one specific feature pulled out of thin air 6 years later and claim that if it is not easy than the system architects were incompetent. The problem is easy. Seriously, it really is. Most of the programmers here have probably done something almost exactly like it ten or more times in their relatively short careers; I know I have. There is no real intrinsic difficulty to the problem. I'm no…

The problem is easy. Seriously, it really is. It's easy in the simplified mental model that you have constructed devoid of real world context. Any engineer worth their salt knows the devil's in the details. I'm going to stop arguing now because you're not even responding to my actual argument. It's not a zen thing, you should be able to get it if you actually read my comments, but just in case you need a koan, ponder…

> It's easy in the simplified mental model that you have constructed devoid of real world context.

I constructed my mental model based on the detailed description by a BaseCamp engineer of what he'd have to do in order to solve the problem in his system!

> I'm going to stop arguing now because you're not even responding to my actual argument.

Welcome to my world: you've been arguing this whole time as if I were claiming that BaseCamp engineers were incompetent when I've only been saying that I don't trust that they are competent.

> There exists a potential feature which could be implemented in Basecamp faster than in your product (and vice versa).

There are probably many such features. That's really beside the point, unless you're claiming some additional knowledge here that the decisions made in this particular design contributed to the ease of those potential features. If you're only assuming that because you trust the competence of 37signals, your entire argument is circular and depends on the very claim I'm contesting.

Re: "How hard can it be to implement?"

#74

Post by jemfinch »

amix wrote: post #27
thisisnotmyname wrote: post #2

Is there really no simpler way to solve this problem? "Moving a message needs to move all of the message's comments, and all of the comments' files, and all of the comments' files' versions." Why can't you just change some top level reference in the database? I'm imagining a Projects table and a TodoLists table. Each TodoList has something like a projectID foreign key right? Why can't you just change that and automat…

The strategy you usually take with scaling a relational database is sharding, partitioning and de-normalization. And given that Basecamp has millions of users I don't imagine their database structure to be pretty or normalized. This said, if they have their database sharded by project_id then it would not be that big an issue, but it can be that their database structure is very complex or messed up...

> And given that Basecamp has millions of users I don't imagine their database structure to be pretty or normalized.

Millions of users is not really that many. It's certainly within the realm of what can be reasonably vertically scaled.

Re: "How hard can it be to implement?"

#75

Post by whatusername »

jules wrote: post #50
die_sekte wrote: post #44

Earlier quoted context omitted.

HP sells servers with up to 64 cores and 2 TB of memory. Considering the car that DHH just bought himself, they should be able to buy/lease several of them.

Are these x86 servers? Or do you mean a single chip with 64 cores?

http://www-03.ibm.com/systems/x/hardware/enterprise/x3850x5/...

Re: "How hard can it be to implement?"

#76

Post by vog »

gaius wrote: post #67
vog wrote: post #51

Earlier quoted context omitted.

This is not necessarily about whether the denormalized database structure is appropriate. It is more about: Why the h... doesn't their ORM, or even better their database, already take care of cascading updates? Why does it have to be implemented manually for every code that updates something? (... which is of course expensive and prone to errors)

Exactly. I can imagine that they thought they were being very clever using ORM + MySQL instead of writing SQL to run on a more mature RDBMS. Well, decisions like that often result in you painting yourself into a corner. There's a reason that MySQL is free yet people still pay for Oracle...

I was not thinking of Oracle here. PostgreSQL is free software and offers almost everything you expect from a real database.

So I guess the main reason people pay for Oracle is because they only know about MySQL and not about PostgreSQL.

Re: "How hard can it be to implement?"

#77

Post by anthonyb »

kristiandupont wrote: post #46
jemfinch wrote: post #41

Earlier quoted context omitted.

> (let me guess, early 20s?), No. > but in this case you literally don't know what you're talking about. I don't need to know what exact decisions they made to be able to tell that they were poor decisions. Easy things should be easy, and when they're hard, they're hard because of incompetence somewhere in the process. FWIW, I've worked on a site extremely similar to basecamp (but with far more traffic, at least acco…

Should you implement something with an array or a linked list? That depends. Should it be fast to locate an element in the middle? Or should it be cheap to add another element regardless of the size? Ideally, both, but you weigh the pros and cons and choose one. And even if it is the right choice, you might still run into situations where the other choice would have been better and people will comment that your decis…

Basecamp is written in Ruby, so the choice is really a hash or a list. Either way, it's not likely to be your bottleneck in a web app.

Re: "How hard can it be to implement?"

#78

Post by amix »

jemfinch wrote: post #74
amix wrote: post #27

Earlier quoted context omitted.

The strategy you usually take with scaling a relational database is sharding, partitioning and de-normalization. And given that Basecamp has millions of users I don't imagine their database structure to be pretty or normalized. This said, if they have their database sharded by project_id then it would not be that big an issue, but it can be that their database structure is very complex or messed up...

> And given that Basecamp has millions of users I don't imagine their database structure to be pretty or normalized. Millions of users is not really that many. It's certainly within the realm of what can be reasonably vertically scaled.

You don't want to scale vertically once you hit millions of users and most other web companies like Google, Facebook, Yahoo and Microsoft have proven that horizontally scaling without big iron is the way to go. Currently the only way to scale a relational database such as MySQL or Postgre is by sharding and partitioning - these things ruin most good relational properties that your database structure may have.

Re: "How hard can it be to implement?"

#79

Post by brown9-2 »

gaius wrote: post #66
SMrF wrote: post #3

Earlier quoted context omitted.

From the response; "We can't use database transactions because performing a big move would slow Basecamp down for everyone. So we have to log the process of each step of the move, and make it so any failure in the move can be rolled back gracefully. That means a move is actually a series of copies and deletions instead of just changing a field for each moved item"

How amusing. Absolutely not an issue in any grown-up database that implements row versioning/MVCC a la Postgres or Oracle. The sad thing is he (and most MySQL users) probably believe this is inherent to all RDBMSs.

I mean this as a genuine question since I don't use it, but: MySQL's engine locks the whole table when a transaction touches it?

Re: "How hard can it be to implement?"

#80

Post by gaius »

brown9-2 wrote: post #79
gaius wrote: post #66

Earlier quoted context omitted.

How amusing. Absolutely not an issue in any grown-up database that implements row versioning/MVCC a la Postgres or Oracle. The sad thing is he (and most MySQL users) probably believe this is inherent to all RDBMSs.

I mean this as a genuine question since I don't use it, but: MySQL's engine locks the whole table when a transaction touches it?

The default MyISAM engine does. The behavior of InnoDB is similar to that of SQL Server: a row lock is a data structure held in memory, seperate from the row data. It's computationally expensive to lock this way. However SQL Server escalates row locks into page locks if it thinks it will help (e.g. "lock this row, and the next one, and the next one..." is translated on the fly to "lock this entire page" where a page is an on-disk allocation containing many rows) so now we have a few page locks to manage rather than many row locks. MySQL can't so it struggles when you need to lock many rows at once. Oracle manages this by keeping a row's lock status in-line (e.g. in the block buffer cache) so there is no additional overhead per-lock.

I see this all the time; developers who have "grown up" in an environment where locks and cursors are expensive pick up some odd habits that don't translate well when they code in an environment (such as Oracle) where locks are cheap and cursors are free.

Post reply on HN