Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The database ruins all good ideas

squarism.com

Post reply on HN
Print view View original

121–130 of 165 posts

Re: The database ruins all good ideas

#121

Post by lmm »

justinclift wrote: post #118
lmm wrote: post #116

Earlier quoted context omitted.

"Maintain data integrity" sounds good, but what does it actually mean? In practice with an RDBMS it means you drop a write on the floor (or worse, deadlock) if it violates a constraint. And in practice that's almost never good enough, so you end up having to build the same kind of application-level validation logic that you would have written in a non-RDBMS system anyway.

Why would you be input data in a way that violates referential integrity? That would be pretty bizarre. That being said, if your code doesn't catch and handle errors, then there's a couple of deeper problems already.

> Why would you be input data in a way that violates referential integrity? That would be pretty bizarre.

Well if you can't get input that violates that integrity then what are you gaining by enforcing that integrity?

> That being said, if your code doesn't catch and handle errors, then there's a couple of deeper problems already.

Sure, but how can you do error handling without data storage, given that your application itself is stateless? Maybe you retry the failure a few times, but ultimately your only option is to drop the data on the floor, which is almost always not what you want.

Re: The database ruins all good ideas

#122

Post by ekidd »

lmm wrote: post #114
ahachete wrote: post #100

Before thinking of removing the RDBMS; replacing itwith a NewSQL/NoSQL; or trying to horizontally shard, ask yourself: what is the performance I really need? * Read-only queries can easily be scaled out to read replicas. * Write transactions. As an example of numbers publicly available, GitLab.com runs more than 250K read-only txs and more than 60K write txs on a single Postgres cluster, with room for further vertica…

Why have the RDBMS as the default? Why assume you need ACID and transactions when they're overwhelmingly likely to bring you nothing but trouble (the number of web applications that make effective use of database-level transactions is approximately zero). You're far better off starting with a system that does seamless active-active by default, and then figuring out what kind of transactional guarantees you need. An R…

> Why assume you need ACID and transactions when they're overwhelmingly likely to bring you nothing but trouble

Maybe this for consumer web applications, or for CRUD apps.

But my experience with B2B and SaaS applications is that sooner or later, I always need transactions (or locks) for something. Maybe it's for the billing code. Maybe it's for batch jobs or workflow logic. And that's when I'm really happy to have a transactional database.

The alternative to having transactions is setting up a Raft or Paxos server to handle distributed locks, and those require a lot more ops effort than a copy of PostgreSQL.

Re: The database ruins all good ideas

#123

Post by lmm »

L0stLink wrote: post #119
lmm wrote: post #116

Earlier quoted context omitted.

"Maintain data integrity" sounds good, but what does it actually mean? In practice with an RDBMS it means you drop a write on the floor (or worse, deadlock) if it violates a constraint. And in practice that's almost never good enough, so you end up having to build the same kind of application-level validation logic that you would have written in a non-RDBMS system anyway.

RDBMS and Non-RDBMS both have there place, I have used both in the same system several times, all for things that they were good at. Transactions allow you to be confident while making complex changes that in case a failure occurs all partial changes will be rolled back. making use of database level validations and enforcing referential integrity is essential for keeping data consistent over the long term and making…

If scalability is your concern then you can't use any of the supposedly core features of an RDBMS, since fundamentally there is no way to have a transaction across multiple nodes without solving a much bigger problem.

Validation is vital but the datastore is not the place to do it, because handling invalid data by dropping it on the floor is almost never the right behaviour.

There is no substitute for actually understanding your data model, but once you do 99% of the time you'll find using an RDBMS comes with minimal benefits and significant costs.

Re: The database ruins all good ideas

#124

Post by lmm »

ekidd wrote: post #122
lmm wrote: post #114

Earlier quoted context omitted.

Why have the RDBMS as the default? Why assume you need ACID and transactions when they're overwhelmingly likely to bring you nothing but trouble (the number of web applications that make effective use of database-level transactions is approximately zero). You're far better off starting with a system that does seamless active-active by default, and then figuring out what kind of transactional guarantees you need. An R…

> Why assume you need ACID and transactions when they're overwhelmingly likely to bring you nothing but trouble Maybe this for consumer web applications, or for CRUD apps. But my experience with B2B and SaaS applications is that sooner or later, I always need transactions (or locks) for something . Maybe it's for the billing code. Maybe it's for batch jobs or workflow logic. And that's when I'm really happy to have a…

Well if it's not distributed then it's a single point of failure, almost by definition. I hate maintaining a Zookeeper cluster as much as anyone, but anything that runs on a single server is not a real alternative (and if a single-server SPoF is acceptable, then just running the batch job / workflow processor / whatever it is on a single server works just as well - maybe with a pool of workers to scale out the actual processing, but a single coordinator, sort of like how Jenkins or Hadoop do it).

Re: The database ruins all good ideas

#125

Post by kayodelycaon »

lmm wrote: post #121
justinclift wrote: post #118

Earlier quoted context omitted.

Why would you be input data in a way that violates referential integrity? That would be pretty bizarre. That being said, if your code doesn't catch and handle errors, then there's a couple of deeper problems already.

> Why would you be input data in a way that violates referential integrity? That would be pretty bizarre. Well if you can't get input that violates that integrity then what are you gaining by enforcing that integrity? > That being said, if your code doesn't catch and handle errors, then there's a couple of deeper problems already. Sure, but how can you do error handling without data storage, given that your applicati…

If you allow invalid records and multiple versions of a schema in a collection, you will forever being paying for that mistake every time you read from the database.

Re: The database ruins all good ideas

#126

Post by otoolep »

samatman wrote: post #22

SQLite is a remarkably good solution to most of these problems, if deployed correctly. For your main line-of-business database? Of course not. But a deployment of rqlite[0] for your service workers in a read-heavy workload? cuts out a round-trip out of the VM, mocking is trivial, there's a lot to like there. [0]: https://github.com/rqlite/rqlite

rqlite author here. Happy to answer any questions about it.

Re: The database ruins all good ideas

#127

Post by tarsinge »

lmm wrote: post #123
L0stLink wrote: post #119

Earlier quoted context omitted.

RDBMS and Non-RDBMS both have there place, I have used both in the same system several times, all for things that they were good at. Transactions allow you to be confident while making complex changes that in case a failure occurs all partial changes will be rolled back. making use of database level validations and enforcing referential integrity is essential for keeping data consistent over the long term and making…

If scalability is your concern then you can't use any of the supposedly core features of an RDBMS, since fundamentally there is no way to have a transaction across multiple nodes without solving a much bigger problem. Validation is vital but the datastore is not the place to do it, because handling invalid data by dropping it on the floor is almost never the right behaviour. There is no substitute for actually unders…

The point is that you can go pretty far on a single cluster (GitLab example). That 99% figure is trivially wrong in that case.

Re: The database ruins all good ideas

#128

Post by noisy_boy »

heurisko wrote: post #120
noisy_boy wrote: post #97

Earlier quoted context omitted.

That is a good idea. Though H2 supports specifying the database dialect so it is a pretty close and usually quicker approximation.

If speed is one of the things holding you back from testing with the native database, running the database data on a tmpfs mountpoint is worth looking at. https://vladmihalcea.com/how-to-run-integration-tests-at-war...

For me, it is more of zero setup while getting most of the benefits which is "good enough" (tmpfs works great but every new developer would need to do this + setup/permissions needed for docker + differences between Windows/Linux and/or desktop vs CI host etc).

Re: The database ruins all good ideas

#129

Post by ekidd »

lmm wrote: post #124
ekidd wrote: post #122

Earlier quoted context omitted.

> Why assume you need ACID and transactions when they're overwhelmingly likely to bring you nothing but trouble Maybe this for consumer web applications, or for CRUD apps. But my experience with B2B and SaaS applications is that sooner or later, I always need transactions (or locks) for something . Maybe it's for the billing code. Maybe it's for batch jobs or workflow logic. And that's when I'm really happy to have a…

Well if it's not distributed then it's a single point of failure, almost by definition. I hate maintaining a Zookeeper cluster as much as anyone, but anything that runs on a single server is not a real alternative (and if a single-server SPoF is acceptable, then just running the batch job / workflow processor / whatever it is on a single server works just as well - maybe with a pool of workers to scale out the actual…

My most common batch/workflow usecases tend to involve a few hundred workers, and tolerance for brief downtime (especially if jobs don't fail outright).

PostgreSQL works surprisingly well for something like this. It has a rich set of coordination and locking primitives, and it can usually be recovered and restarted in under 10 minutes if the server fails. Yes, connection overhead from 300 workers is a real problem, so it doesn't hurt to put a thin layer in front of it.

One thing that an RDBMS buys me is flexible feature set and a solid ecosystem. Want write-ahead logs and automatic point-in-time recovery? It's out there. Need a specific locking primitive? PostgreSQL probably supports it. Etc.

If I were looking at 2,000 workers, yeah, it would be time to set up a Raft server. But a good RDBMS will do the job for a surprisingly long time.

Re: The database ruins all good ideas

#130

Post by hinkley »

Ozzie_osman wrote: post #23
hughrr wrote: post #4

No it doesn’t. They scale amazingly well if you throw money at the problem. Most people never get there. When you do you will know. I’ve been there. When you’re spending $3 million on hardware and licenses a year you either have a viable business or fucked up badly. That’s the real decider. The answer is to start siloing customers or application concerns out into separate clusters depending on your operating model. I…

> The answer is to start siloing customers or application concerns out into separate clusters depending on your operating model. This is such an underrated solution (siloing or sharding your data in some way). I think people don't do it because: 1. The tooling doesn't make it super-easy (e.g. good luck sharding Postgres unless you're willing to pay for Citus) 2. "Trendy" companies in the past decade have been network…

On paper I’m not dealing with the same problem as the people in this thread, but I can’t help thinking that the root causes are the same.

I have a team that won’t even look at the fact that we measure performance by response time, but we are lumping together write traffic and two classes of read traffic into a single monolithic chunk of code. We are SaaS where our customers write and their customers read. And of course search queries are by far the slowest traffic.

But there’s some sort of mental block about splitting things up that is only slowly changing, and our app is so “flexible” that a load balancer would struggle to tell which urls involve search functionality.

I think people just want to “know where to look” but if your code is on a cluster there already is no “there”. You’re probably looking at some log aggregator anyway, so who gives a tinker’s damn if they come from separate log files on the same box or the same log file?

Post reply on HN