91–100 of 165 posts
I'd have liked to learn more about why it's only one db. For instance because they're often insanely efficient and one server can handle many queries.
I was also hoping that it'd suggest specific alternatives.
Earlier quoted context omitted.
But what if the data setup is convoluted? How does one avoid a horrific domino stack of fixtured data?
I find using libraries like factory_boy and pytest's fixtures mechanism makes that horrific domino stack pretty manageable.
Databases are the reason all that other shit is so easy. Databases are the heart of computing but somehow this clown industry still doesn't get that at a large scale. I’ll replicate your clown ass todo list app with a few DB tables, a few queries and 1 HTML table dont make me fucking do it :D
It’s weird how developers get asked about OO design patterns in interviews, but not once have I ever been asked about database design (beyond some useless stuff about 2nd normal form).
The title of this piece is great: very catchy. But I don't think the content supports the title - by the end of it I wasn't at all clear /why/ the database ruins all good ideas. A few other points. First, horizontally scaling database reads is actually reasonably straight-forward these days: one leader, multiple replicas, load balance reads to the replicas and use a mechanism such that users that have just performed…
Agreed re: db mocks, database bugs and quirks should be captured and addressed in tests, not discovered only in production. I don't care how mature the DB you're using is, there's always going to be some edge case behavior that you can't predict until you actually see it happen, even just simple stuff like exactly how an error filters back to the application when you send data that violates a constraint. I've seen way too many tests that make assumptions about these things that turn out to be incorrect, leading to horrible behavior like writes getting through on prod that should have been blocked because of pre-checks that didn't turn out to be enforced the way the test writer assumed they would be.
Databases are the reason all that other shit is so easy. Databases are the heart of computing but somehow this clown industry still doesn't get that at a large scale. I’ll replicate your clown ass todo list app with a few DB tables, a few queries and 1 HTML table dont make me fucking do it :D
Also, developers never design their own database schemas anymore, then when the some crappy ORM (like Django’s) generates a tediously slow schema they blame the database. It’s weird how developers get asked about OO design patterns in interviews, but not once have I ever been asked about database design (beyond some useless stuff about 2nd normal form).
For example, for https://corridorchat.com/, we have a relatively small number of business accounts (tenants), but with a many users per tenant. And new tenants are created relatively infrequently in the scheme of things.
So I have an architecture with a central 'corridorchat central' PostgreSQL database and a scalable number of shard clusters, all managed with Patroni + Consul, and fall-backs that are read-only until they need to be promoted. Consul DNS allows the application to look up either a read-only replica or a write one.
To know what shard a tenant is hosted on, it is necessary to read from the central database. This requires one of the read replicas - and I can create as many of these as I need. Many transactions then require writing to the shard database for that tenant - but since I balance tenants between shards (and have several shard databases per cluster to allow for future scaling if a shard becomes too hot) I can add more shard clusters as needed. Writing to the central database is constrained, but it is a very rare operation, so there is no expected scaling problem there.
Earlier quoted context omitted.
> The article talks briefly about mocking your database: definitely never do this. How your database behaves should be considered part of your application code under test. Running a temporary database for your tests is a solved problem for most development frameworks these days (Django supports this out of the box). Spring Boot for Java does this too (generally using H2 in memory database). I have found that for CRUD…
I always go further and use the same database for my tests e.g. use postgres rather than H2. It is easy enough to setup and tear down using docker.
The title of this piece is great: very catchy. But I don't think the content supports the title - by the end of it I wasn't at all clear /why/ the database ruins all good ideas. A few other points. First, horizontally scaling database reads is actually reasonably straight-forward these days: one leader, multiple replicas, load balance reads to the replicas and use a mechanism such that users that have just performed…
I think I better title would be “reality reveals that ideas were actually bad”. The issues described in this blog are about the challenges of persisting a consistent set of data, when you need to support a lot of read and write access. That is just a fundamentally difficult problem. Databases are the most sophisticated tools we have for doing that, but really they just tend to be a set of (hopefully) well implemented…
If you can't actually retain your business data, it really doesn't matter how many "good ideas" or fancy deployment strategies for the presentation tier you have.
The title of this piece is great: very catchy. But I don't think the content supports the title - by the end of it I wasn't at all clear /why/ the database ruins all good ideas. A few other points. First, horizontally scaling database reads is actually reasonably straight-forward these days: one leader, multiple replicas, load balance reads to the replicas and use a mechanism such that users that have just performed…
I definitely recommend doing this for same tests: it's the only way to check how your application behaves in case of failures.
The best thing would be to have a DB where you can inject failures, but I'm not aware of any. So test - the happy path on the real db - the sad path on the mock/fake (which might be a light wrapper around a real db, but with the ability of injecting failures)
* Read-only queries can easily be scaled out to read replicas.
* Write transactions.
As an example of numbers publicly available, GitLab.com runs more than 250K read-only txs and more than 60K write txs on a single Postgres cluster, with room for further vertical scalability [1].
Do you need millions of transactions per second? Then go ahead. Don't? Then ou can scale very very far with a properly tuned and administered relational database like Postgres.
[1]: https://about.gitlab.com/blog/2020/09/11/gitlab-pg-upgrade/