Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The database ruins all good ideas

squarism.com

Post reply on HN
Print view View original

131–140 of 165 posts

Re: The database ruins all good ideas

#131

Post by ahachete »

lmm wrote: post #114
ahachete wrote: post #100

Before thinking of removing the RDBMS; replacing itwith a NewSQL/NoSQL; or trying to horizontally shard, ask yourself: what is the performance I really need? * Read-only queries can easily be scaled out to read replicas. * Write transactions. As an example of numbers publicly available, GitLab.com runs more than 250K read-only txs and more than 60K write txs on a single Postgres cluster, with room for further vertica…

Why have the RDBMS as the default? Why assume you need ACID and transactions when they're overwhelmingly likely to bring you nothing but trouble (the number of web applications that make effective use of database-level transactions is approximately zero). You're far better off starting with a system that does seamless active-active by default, and then figuring out what kind of transactional guarantees you need. An R…

> Why have the RDBMS as the default?

It's possibly the sanest default for a data persistence layer. I'd argue otherwise: use a RDBMS unless you know what you are doing.

> the number of web applications that make effective use of database-level transactions is approximately zero

Almost every driver and layer (ORMs, etc) that connect to databases use transactions. Transparently or not, but it uses them. So actually they are using transactions all the time.

> You're far better off starting with a system that does seamless active-active

Think about what means active-active. Either it means that you serialize everything in every active node (pointless); or you need to deal with concurrency and data synchronization. Which in turn means either supporting distributed transactions (e.g. via Paxos or RAFT protocols) or doing conflict resolution (which essentially is a nice way of saying "data loss"). Both problems are much, much harder than the (usually harmless) consequences of using transactions on a "classical" RDBMS.

The only drawbacks of transactions are that are limited to scale in write on a single node. But if this single node can scale to the numbers I mentioned ---and they do, and more-- most use cases will likely fit within this envelope.

Therefore being RDBMSs like Postgres the best and safest, the default, approach that should be taken for data persistence layers.

Re: The database ruins all good ideas

#132

Post by nesarkvechnep »

cornel_io wrote: post #95
sonthonax wrote: post #93

Earlier quoted context omitted.

Also, developers never design their own database schemas anymore, then when the some crappy ORM (like Django’s) generates a tediously slow schema they blame the database. It’s weird how developers get asked about OO design patterns in interviews, but not once have I ever been asked about database design (beyond some useless stuff about 2nd normal form).

Huh, almost every lead backend job I've interviewed for had a lot of DB design questions, more than normal coding stuff because it's so critical to scaling. It's core to the role even for lower level positions, so it really should be more common.

Clearly you are not a Node.is dev.

Re: The database ruins all good ideas

#133

Post by namibj »

tyingq wrote: post #72
benlivengood wrote: post #30

And this is why Google wrote Spanner. I think cockroachdb tries to solve the same problems. If you need ACID compliance and you need a lot of it, everywhere, all the time, now there are better options than giant Sun/IBM boxes. Databases are not the problem.

It's still a fairly new thing for people outside of Google. It's been a few years, so I imagine it's better now...but when I tried CockroachDB, it was pretty easy to trigger horrible write latency. Bad enough to be a non-starter.

Which version, and what networking did you have between the cluster nodes?

Re: The database ruins all good ideas

#134

Post by jayd16 »

jeltz wrote: post #110
jayd16 wrote: post #56

Earlier quoted context omitted.

It might be a waste. Think RAID 1 vs RAID 5. There might be plenty of ways were leaving performance on the table.

Which one of those leaves performance on the table? One is optimized for performance and the other for disk usage.

Maybe its a bit tortured of a metaphor, but perhaps with continued research we can find a way to use more of the hardware just like how RAID 5 provides more storage over RAID 1 while still keeping parity.

I don't think its foolish to keep asking for more. We still might find it.

Re: The database ruins all good ideas

#135

Post by justinclift »

lmm wrote: post #121
justinclift wrote: post #118

Earlier quoted context omitted.

Why would you be input data in a way that violates referential integrity? That would be pretty bizarre. That being said, if your code doesn't catch and handle errors, then there's a couple of deeper problems already.

> Why would you be input data in a way that violates referential integrity? That would be pretty bizarre. Well if you can't get input that violates that integrity then what are you gaining by enforcing that integrity? > That being said, if your code doesn't catch and handle errors, then there's a couple of deeper problems already. Sure, but how can you do error handling without data storage, given that your applicati…

> Well if you can't get input that violates that integrity then what are you gaining by enforcing that integrity?

Perhaps I'm wording this badly?

Your application should _not_ be generating data which can't go into the database correctly. If it is, that's a different set of problems. ;)

Re: The database ruins all good ideas

#136

Post by simonw »

MakersF wrote: post #99
simonw wrote: post #29

The title of this piece is great: very catchy. But I don't think the content supports the title - by the end of it I wasn't at all clear /why/ the database ruins all good ideas. A few other points. First, horizontally scaling database reads is actually reasonably straight-forward these days: one leader, multiple replicas, load balance reads to the replicas and use a mechanism such that users that have just performed…

> The article talks briefly about mocking your database: definitely never do this. I definitely recommend doing this for same tests: it's the only way to check how your application behaves in case of failures. The best thing would be to have a DB where you can inject failures, but I'm not aware of any. So test - the happy path on the real db - the sad path on the mock/fake (which might be a light wrapper around a rea…

That's true: the one place where it's a good idea to mock the database is when you want to simulate what would happen if a specific database error occurred that can't be triggered another way.

Re: The database ruins all good ideas

#137

Post by simonw »

lmm wrote: post #123
L0stLink wrote: post #119

Earlier quoted context omitted.

RDBMS and Non-RDBMS both have there place, I have used both in the same system several times, all for things that they were good at. Transactions allow you to be confident while making complex changes that in case a failure occurs all partial changes will be rolled back. making use of database level validations and enforcing referential integrity is essential for keeping data consistent over the long term and making…

If scalability is your concern then you can't use any of the supposedly core features of an RDBMS, since fundamentally there is no way to have a transaction across multiple nodes without solving a much bigger problem. Validation is vital but the datastore is not the place to do it, because handling invalid data by dropping it on the floor is almost never the right behaviour. There is no substitute for actually unders…

My experience is the exact opposite: once you understand your data model, 99% of the time you will find that NOT using a RDBMS comes with minimal benefits and significant costs.

Re: The database ruins all good ideas

#138

Post by tyingq »

namibj wrote: post #133
tyingq wrote: post #72

Earlier quoted context omitted.

It's still a fairly new thing for people outside of Google. It's been a few years, so I imagine it's better now...but when I tried CockroachDB, it was pretty easy to trigger horrible write latency. Bad enough to be a non-starter.

Which version, and what networking did you have between the cluster nodes?

I don't recall other than it was before 2.x, but I don't think it's terribly important. The cockroachdb folks were very transparent about it, including a couple of major re-works to address performance. Here's a blog post about one of them: https://www.cockroachlabs.com/blog/2-dot-0-perf-strides/

I was just trying to say that a stable, high performance SQL database with distributed writes is a fairly new thing.

Re: The database ruins all good ideas

#139

Post by heurisko »

noisy_boy wrote: post #128
heurisko wrote: post #120

Earlier quoted context omitted.

If speed is one of the things holding you back from testing with the native database, running the database data on a tmpfs mountpoint is worth looking at. https://vladmihalcea.com/how-to-run-integration-tests-at-war...

For me, it is more of zero setup while getting most of the benefits which is "good enough" (tmpfs works great but every new developer would need to do this + setup/permissions needed for docker + differences between Windows/Linux and/or desktop vs CI host etc).

It looks like tmpfs is Linux-only feature.

Otherwise I have a single docker-compose.yml file that sets up postgres using "Docker for Desktop" on Mac and Windows and works natively on Linux. CI (Linux) uses the same docker-compose.yml.

I haven't used docker in production, but it has proved very useful in test.

Post reply on HN