The database ruins all good ideas • Hacker News Board

Re: The database ruins all good ideas

#81

Post by seanhunter » Sun, Jul 18, 2021, 5:32 AM UTC

Ozzie_osman wrote: post #23

hughrr wrote: post #4

No it doesn’t. They scale amazingly well if you throw money at the problem. Most people never get there. When you do you will know. I’ve been there. When you’re spending $3 million on hardware and licenses a year you either have a viable business or fucked up badly. That’s the real decider. The answer is to start siloing customers or application concerns out into separate clusters depending on your operating model. I…

> The answer is to start siloing customers or application concerns out into separate clusters depending on your operating model. This is such an underrated solution (siloing or sharding your data in some way). I think people don't do it because: 1. The tooling doesn't make it super-easy (e.g. good luck sharding Postgres unless you're willing to pay for Citus) 2. "Trendy" companies in the past decade have been network…

Sharding by customer has a lot of benefits for b2b applications also. You can run each customer in a seperate instance in their own vpc if you want to, and guarantee to your customers that their data is never commingled with another customer. This makes a lot of sense if you have relatively low number relatively high value customers.

It radically reduces the infosec/screw up blast radius. For example it means however badly you screw up it is just about impossible to accidentally show data from customer A to a user from customer B.

You can let customers specify region/availability zone and bring their own keys for encryption making lots of enterprise security compliance things easier (for regulated use cases).

There is no one true way of scaling that suits all possible use cases and as engineers, we should think for ourselves and build a solution that works well in our situation, rather than thinking you have to do a certain thing because that's what google/facebook etc do. Their scaling problems are most likely really different to yours.

Re: The database ruins all good ideas

#82

Post by coderintherye » Sun, Jul 18, 2021, 5:42 AM UTC

"I learned a ton about splitbrain mostly through trauma"

The most typical way to learn about splitbrain it seems. I don't think I've experienced more work trauma than split brain trauma.

Re: The database ruins all good ideas

#83

Post by noisy_boy » Sun, Jul 18, 2021, 5:45 AM UTC

simonw wrote: post #29

The title of this piece is great: very catchy. But I don't think the content supports the title - by the end of it I wasn't at all clear /why/ the database ruins all good ideas. A few other points. First, horizontally scaling database reads is actually reasonably straight-forward these days: one leader, multiple replicas, load balance reads to the replicas and use a mechanism such that users that have just performed…

> The article talks briefly about mocking your database: definitely never do this. How your database behaves should be considered part of your application code under test. Running a temporary database for your tests is a solved problem for most development frameworks these days (Django supports this out of the box).

Spring Boot for Java does this too (generally using H2 in memory database). I have found that for CRUD services, doing this vs. pure unit tests that mock the db, yields far better benefit (provided you don't involve heavy volumes in the tests to avoid test data maintenance issues). With this approach, I have found database side edge cases/constraint violations that wouldn't be caught by mocking the database. Also, it is a much more comprehensive test if you take the approach of a) loading an initial state, b) doing business logic mutations and c) compare the final state with the expected state; e.g. you might find side effects that you hadn't considered testing for and so on.

Re: The database ruins all good ideas

#84

Post by gregjor » Sun, Jul 18, 2021, 5:56 AM UTC

Article would benefit from an example or two of ideas ruined by databases. The examples given all have to do with implementation details, not ideas. Databases actually enable a lot of good ideas, and reliable well-understood implementations.

Pointing out that database servers don’t scale horizontally like web servers had the same level of insight as pointing out cars don’t scale like kittens.

Re: The database ruins all good ideas

#85

Post by simonw » Sun, Jul 18, 2021, 6:02 AM UTC

mikeyjk wrote: post #69

simonw wrote: post #29

The title of this piece is great: very catchy. But I don't think the content supports the title - by the end of it I wasn't at all clear /why/ the database ruins all good ideas. A few other points. First, horizontally scaling database reads is actually reasonably straight-forward these days: one leader, multiple replicas, load balance reads to the replicas and use a mechanism such that users that have just performed…

But what if the data setup is convoluted? How does one avoid a horrific domino stack of fixtured data?

I find using libraries like factory_boy and pytest's fixtures mechanism makes that horrific domino stack pretty manageable.

Re: The database ruins all good ideas

#86

Post by zaptheimpaler » Sun, Jul 18, 2021, 6:24 AM UTC

Databases are the reason all that other shit is so easy. Databases are the heart of computing but somehow this clown industry still doesn't get that at a large scale.

I’ll replicate your clown ass todo list app with a few DB tables, a few queries and 1 HTML table dont make me fucking do it :D

Re: The database ruins all good ideas

#87

Post by heurisko » Sun, Jul 18, 2021, 6:37 AM UTC

noisy_boy wrote: post #83

simonw wrote: post #29

The title of this piece is great: very catchy. But I don't think the content supports the title - by the end of it I wasn't at all clear /why/ the database ruins all good ideas. A few other points. First, horizontally scaling database reads is actually reasonably straight-forward these days: one leader, multiple replicas, load balance reads to the replicas and use a mechanism such that users that have just performed…

> The article talks briefly about mocking your database: definitely never do this. How your database behaves should be considered part of your application code under test. Running a temporary database for your tests is a solved problem for most development frameworks these days (Django supports this out of the box). Spring Boot for Java does this too (generally using H2 in memory database). I have found that for CRUD…

I always go further and use the same database for my tests e.g. use postgres rather than H2.

It is easy enough to setup and tear down using docker.

Re: The database ruins all good ideas

#88

Post by _benedict » Sun, Jul 18, 2021, 7:13 AM UTC

hugofirth wrote: post #51

bobthebuilders wrote: post #13

Earlier quoted context omitted.

CockroachDB is a terrible technology that causes almost guaranteed data corruption due to its lack of ACID guarantees and is written in a language with a GC which contirbutes to GC pauses. The dev team refuses to listen to feedback to port their code to C ;(.

This is daft. CockroachDB exhibits extremely strong isolation and consistency levels. I believe it is strictly serializable under most circumstances? Also - anyone who says that x database must be rewritten “because GC” is just making an incredibly un nuanced argument about a nuanced problem. People have built production ready databases in both Java and Go. If you care about low/predictable tail latencies then you ha…

I believe it only guarantees serializable isolation. You may get strict serializable, but it doesn’t appear to be the case that you will know for sure if any transactions were not linearizable.

Re: The database ruins all good ideas

#89

Post by danielheath » Sun, Jul 18, 2021, 7:14 AM UTC

heurisko wrote: post #87

noisy_boy wrote: post #83

Earlier quoted context omitted.

> The article talks briefly about mocking your database: definitely never do this. How your database behaves should be considered part of your application code under test. Running a temporary database for your tests is a solved problem for most development frameworks these days (Django supports this out of the box). Spring Boot for Java does this too (generally using H2 in memory database). I have found that for CRUD…

I always go further and use the same database for my tests e.g. use postgres rather than H2. It is easy enough to setup and tear down using docker.

After a bug slipped through because of it, I switched CI to run the same point release of the database, and validate that the schema is identical to production.

Re: The database ruins all good ideas

#90

Post by danielheath » Sun, Jul 18, 2021, 7:18 AM UTC

l0k3ndr wrote: post #17

I have been working on rewriting a monolith into individual services during past year at work - and what we finally implemented for consistency across different databases of services was that we keep a signal queue - where whenever we encounter an inconsistency, a doc is pushed containing the info and we have a corrective service always reading from that queue and doing the necessary updates to tables. We made a heir…

… is that less operational overhead than running a single process on a server?