Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

"How hard can it be to implement?"

answers.37signals.com

Post reply on HN
Print view View original

81–90 of 102 posts

Re: "How hard can it be to implement?"

#81

Post by gaius »

vog wrote: post #76
gaius wrote: post #67

Earlier quoted context omitted.

Exactly. I can imagine that they thought they were being very clever using ORM + MySQL instead of writing SQL to run on a more mature RDBMS. Well, decisions like that often result in you painting yourself into a corner. There's a reason that MySQL is free yet people still pay for Oracle...

I was not thinking of Oracle here. PostgreSQL is free software and offers almost everything you expect from a real database. So I guess the main reason people pay for Oracle is because they only know about MySQL and not about PostgreSQL.

You're probably right. I've never understood why Postgres has such a small mindshare. Between Postgres and SQLite, there ought to be no ecological niche for MySQL to exist...

Re: "How hard can it be to implement?"

#82

Post by richardhenry »

Groxx wrote: post #6

Sounds like they're in need of some hefty normalization. This is the sort of thing that should be handled by changing a single foreign key on each todo / on a single todo list entry; everything else should be pointing to each individual entity, and need no modification. Unless they're denormalized for optimization purposes? Threaded comments can result in some massively deep queries.

> Unless they're denormalized for optimization purposes?

Bingo?

Re: "How hard can it be to implement?"

#84

Post by JoeAltmaier »

DougWebb wrote: post #59
dasil003 wrote: post #31

Earlier quoted context omitted.

You're falling into the same trap all programmers do when they perpetually underestimate the time to accomplish any task. The main difficulty in estimating time is that you don't know what you don't know, so you have to actually get into the nitty gritty of implementing it, and if you're smart enough you'll hopefully catch all the requirements before you actually launch it. I'm going to take Sam's word on this that t…

> The main difficulty in estimating time is that you don't know what you don't know, so you have to actually get into the nitty gritty of implementing it, and if you're smart enough you'll hopefully catch all the requirements before you actually launch it. That's certainly true for a low-level, high-precision estimate, which you can only make when you've got detailed requirements. Prior to that you still need to be a…

Company wanted to port an OS to 32-bit (yes I know that dates me). I spent a day, reviewed the code (500 modules), wrote a report - two manyears. Project deferred (ashcanned).

A month later my peer, a senior Engineer with the Company, thought "We should run 32-bit! How long can it take?", checked out the source and started changing files. Gradually people noticed what he was doing, resources got added under the table, finally a year later he had something working. Lots of kudos, smiles, what a hero!

I showed him the report I filed a year ago - and he said "yeah, that's about what I had to do. I'm glad I didn't see that to begin with, I would never have started".

Re: "How hard can it be to implement?"

#85

Post by jemfinch »

amix wrote: post #78
jemfinch wrote: post #74

Earlier quoted context omitted.

> And given that Basecamp has millions of users I don't imagine their database structure to be pretty or normalized. Millions of users is not really that many. It's certainly within the realm of what can be reasonably vertically scaled.

You don't want to scale vertically once you hit millions of users and most other web companies like Google, Facebook, Yahoo and Microsoft have proven that horizontally scaling without big iron is the way to go. Currently the only way to scale a relational database such as MySQL or Postgre is by sharding and partitioning - these things ruin most good relational properties that your database structure may have.

> You don't want to scale vertically once you hit millions of users

Hundreds of the top websites in the world (the majority, I would hazard, though without the data to support it) have scaled vertically to millions and tens of millions of entities just fine. Far more than have scaled horizontally. It works. It's been done. And it doesn't give up the sort of transactional niceties that make problems like this easier.

> most other web companies like Google, Facebook, Yahoo and Microsoft

You're confusing "the very biggest web companies" with "most other web companies." Most other web companies continue to use commodity products, and Google/Facebook/Yahoo!/MS certainly would (and do) insofar as it's possible at that scale. Expending resources now to be as horizontally scalable as Google is wasteful premature optimization.

Notably, Yahoo! runs the largest PostgreSQL installation in the world, and Google and Facebook both continue to use MySQL.

> horizontally scaling without big iron is the way to go.

You can get 32-core machines with 128GB of ram from Dell (a mildly tweaked R910) for $30k these days. Is that big iron? How does its price compare with the amount of developer salary and benefits you'll have to spend to grok a non-relational data store, migrate your data to it, and reimplement the ACID features of a relational store in the code for your app? How many developer-days will you spend maintaining that code and how many developer-nights will you spend triaging a crashed site because of the complexity and likely bugginess of that reimplementation? How many users' feature requests will you have to reject as "too difficult to implement" because you feel the need to scale to Google/Facebook levels despite having only a few million users now and predicted growth which shows you'll never in a million years catch up to them?

> Currently the only way to scale a relational database such as MySQL or Postgre is by sharding and partitioning

It will be years before the vast majority of startups exhaust reasonable, cost-effective options for vertical scaling. The recent fervor for non-relational, horizontally scalable data stores is simply the new way of scratching the intellectually masturbatory premature optimization itch that programmers have had since ENIAC.

For what it's worth, I'm not the only crank who thinks this; Dennis Forbes has argued it much more eloquently and compellingly on his blog, e.g. http://blog.yafla.com/Getting_Real_about_NoSQL_and_the_SQL_P... .

Re: "How hard can it be to implement?"

#86

Post by scott_s »

DougWebb wrote: post #59
dasil003 wrote: post #31

Earlier quoted context omitted.

You're falling into the same trap all programmers do when they perpetually underestimate the time to accomplish any task. The main difficulty in estimating time is that you don't know what you don't know, so you have to actually get into the nitty gritty of implementing it, and if you're smart enough you'll hopefully catch all the requirements before you actually launch it. I'm going to take Sam's word on this that t…

> The main difficulty in estimating time is that you don't know what you don't know, so you have to actually get into the nitty gritty of implementing it, and if you're smart enough you'll hopefully catch all the requirements before you actually launch it. That's certainly true for a low-level, high-precision estimate, which you can only make when you've got detailed requirements. Prior to that you still need to be a…

This is a reasonable method, but I think it's worth remembering that you have not actually "quantified the bounds on the uncertainty." What you have done is quantified what you think are the bounds on the uncertainty.

Re: "How hard can it be to implement?"

#87

Post by jemfinch »

gaius wrote: post #81
vog wrote: post #76

Earlier quoted context omitted.

I was not thinking of Oracle here. PostgreSQL is free software and offers almost everything you expect from a real database. So I guess the main reason people pay for Oracle is because they only know about MySQL and not about PostgreSQL.

You're probably right. I've never understood why Postgres has such a small mindshare. Between Postgres and SQLite, there ought to be no ecological niche for MySQL to exist...

MySQL does (did, at least; Postgres has been improving) replication better. Replication is really important for high availability, which almost everyone needs.

The irony, of course, is that a single Postgres server is frequently (IME) more reliable than multiple MySQL servers even when the latter is setup for HA.

Re: "How hard can it be to implement?"

#88

Post by amix »

jemfinch wrote: post #85
amix wrote: post #78

Earlier quoted context omitted.

You don't want to scale vertically once you hit millions of users and most other web companies like Google, Facebook, Yahoo and Microsoft have proven that horizontally scaling without big iron is the way to go. Currently the only way to scale a relational database such as MySQL or Postgre is by sharding and partitioning - these things ruin most good relational properties that your database structure may have.

> You don't want to scale vertically once you hit millions of users Hundreds of the top websites in the world (the majority, I would hazard, though without the data to support it) have scaled vertically to millions and tens of millions of entities just fine. Far more than have scaled horizontally. It works. It's been done. And it doesn't give up the sort of transactional niceties that make problems like this easier.…

Scaling depends a lot of how much data you got, how much data you generate and how much you plan to grow. Given the size of Basecamp and 37 Signal's future projection I doubt it would be wise to hope that they can scale vertically - - because once you hit the limit you are pretty screwed and need to buy _much_ more expensive hardware or rewrite most of your database related code and do lots of migrations. (And rewriting database related code to support sharding is usually error prone since you can't use joins, foreign keys, need to copy data around etc.)

Do note that I am not saying that small websites should shard or scale horizontally - - but big sites with millions of users and tons of data should not scale vertically (it can't payoff and at some point they'll hit the limit).

Re: "How hard can it be to implement?"

#89

Post by dasil003 »

jemfinch wrote: post #73
dasil003 wrote: post #65

Earlier quoted context omitted.

The problem is easy. Seriously, it really is. It's easy in the simplified mental model that you have constructed devoid of real world context. Any engineer worth their salt knows the devil's in the details. I'm going to stop arguing now because you're not even responding to my actual argument. It's not a zen thing, you should be able to get it if you actually read my comments, but just in case you need a koan, ponder…

> It's easy in the simplified mental model that you have constructed devoid of real world context. I constructed my mental model based on the detailed description by a BaseCamp engineer of what he'd have to do in order to solve the problem in his system! > I'm going to stop arguing now because you're not even responding to my actual argument. Welcome to my world: you've been arguing this whole time as if I were claim…

Welcome to my world: you've been arguing this whole time as if I were claiming that BaseCamp engineers were incompetent when I've only been saying that I don't trust that they are competent.

Do you realize how weaselly that is? What does that mean? Connotatively it takes a huge swipe at the 37s team without actually making any commitment. You might as well have said nothing at all if you're not going to take a real stand.

Re: "How hard can it be to implement?"

#90

Post by jemfinch »

amix wrote: post #88
jemfinch wrote: post #85

Earlier quoted context omitted.

> You don't want to scale vertically once you hit millions of users Hundreds of the top websites in the world (the majority, I would hazard, though without the data to support it) have scaled vertically to millions and tens of millions of entities just fine. Far more than have scaled horizontally. It works. It's been done. And it doesn't give up the sort of transactional niceties that make problems like this easier.…

Scaling depends a lot of how much data you got, how much data you generate and how much you plan to grow. Given the size of Basecamp and 37 Signal's future projection I doubt it would be wise to hope that they can scale vertically - - because once you hit the limit you are pretty screwed and need to buy _much_ more expensive hardware or rewrite most of your database related code and do lots of migrations. (And rewrit…

> Scaling depends a lot of how much data you got, how much data you generate and how much you plan to grow.

No doubt, but 37Signals shouldn't have a lot of relational data. The bulk of their per-project bytes, it seems likely, is non-relational stuff like attachments.

> Given the size of Basecamp and 37 Signal's future projection I doubt it would be wise to hope that they can scale vertically

Isn't it less wise to pay a cost you don't yet need and may never need to pay? You pay a significant price in development velocity by forgoing a relational database and using a non-relational data store. Certainly any reasonable organization should be able to project when they will actually need to pay that cost.

> big sites with millions of users

Single digit millions of users isn't that big.

> and tons of data should not scale vertically (it can't payoff and at some point they'll hit the limit).

It can certainly pay off if you never actually need to convert to a non-relational data store. The limit is a lot higher than you seem to think: banks and financial institutions process billions of transactions for hundreds of millions of users daily on the same ACID, relational data stores that you're saying a site like BaseCamp will hit the limit of. I don't buy it.

Post reply on HN