> since our work touched many parts of the codebase and demanded collaboration with lots of different devs, we now have a strong distributed knowledge base about the existing system
Great to see this cultural side-effect called out.
Squeeze the hell out of the system you have
21–30 of 383 posts
Re: Squeeze the hell out of the system you have
#22The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…
If you don't use joins, how do you associate records from two different tables when displaying the UI? Do you just join in the application? Or something else?
Re: Squeeze the hell out of the system you have
#23> Split up the monolith into multiple interconnected services, each with its own data store that could be scaled on its own terms.
Just to note: you don't have to split out all the possible microservices at this junction. You can ask, "what split would have the most impact?"
In my case, we split out some timeseries data from Mongo into Cassandra. Cass's table structure was a much better fit — that dataset had a well defined schema, so Cass could pack the data much more efficiently; for that subset, we didn't need the flexibility of JSON docs. And it was the bulk of our data, and so Mongo was quite happy after that. Only a single split was required. (And technically, we were a monolith before and after: the same service just ended up writing to two databases.)
Ironically, later, an airchair architect wanted to merge all the data into a JSON document store, which resulted in numerous "we've been down that road, and we know where it goes" type discussions.
Re: Squeeze the hell out of the system you have
#24The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…
I’m wondering if indexes and materialized views can be used to do basically the same thing? That is, assuming they contain all the columns you want.
Re: Squeeze the hell out of the system you have
#25Isn’t Rails wasteful in its database access patterns?
Generally No. But it can be easy to write bad queries using ActiveRecord ORM if you're not aware of N + 1 problems.
Re: Squeeze the hell out of the system you have
#26Earlier quoted context omitted.
Normalization is not only about data storage but most importantly, data integrity.
Yes, but I assert that it's possible to use transactions to update everything consistently. Serializable transactions weren't really common when MySQL/Postgres first came out, but now that they're common in new DBs + ACID, I think it's not possible to do with reasonable difficulty. If you agree with this, than its easy to prove that denormalized tables performance increase is well worth the annoyance of updating ever…
You always need to compare write vs read performance.
Turning a single table update into a 10 table one could tip your lock contention to the point where you are write bound or worse start hitting retries.
Certainly it makes sense to move rarely updated fields to where they are used makes sense.
Similarly "build your table against your queries not your ideal data model" is always sage advice.
Re: Squeeze the hell out of the system you have
#27The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…
>Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins.
Anybody has documentation about this with examples?
Re: Squeeze the hell out of the system you have
#28The bit on the database performance issues leads me to my hottest, flamiest take for new projects: - Design your application's hot path to never use joins. Storage is cheap, denormalize everything and update it all in a transaction. It's truly amazing how much faster everything is when you eliminate joins. For your ad-hoc queries you can replicate to another database for analytical purposes. On this note, I have mixe…
If you don't use joins, how do you associate records from two different tables when displaying the UI? Do you just join in the application? Or something else?
this has opinionated answers.
if you ask Amazon, they might suggest that you design around a single table (https://aws.amazon.com/blogs/compute/creating-a-single-table...).
in my opinion it's easier to use join tables. which are what are sometimes temporarily created when you do a join anyways. in this case, you permanently create table1, table2, and table1_join_table2, and keep all three in sync transactionally. when you need a join you just select on table1_join_table2. you might think this is a waste of space, but I'd argue storage is too cheap for you to be thinking about that.
that being said, you really have to design around your access patterns, don't design your application around your schema. most people do the latter because it seems more natural. what this might mean in practice is that you do mockups of all of the expected pages and what data is necessary on each one. then you design a schema that results in you never having to do joins on the majority, if not all, of them.
Re: Squeeze the hell out of the system you have
#29[The Grug Brained Developer](https://grugbrain.dev/)
Re: Squeeze the hell out of the system you have
#30Earlier quoted context omitted.
Normalization is not only about data storage but most importantly, data integrity.
Yes, but I assert that it's possible to use transactions to update everything consistently. Serializable transactions weren't really common when MySQL/Postgres first came out, but now that they're common in new DBs + ACID, I think it's not possible to do with reasonable difficulty. If you agree with this, than its easy to prove that denormalized tables performance increase is well worth the annoyance of updating ever…
Denormalized transactions are not trivial unless you are using serializable isolation level which will kill performance. If you don't use serializable isolation level, then you risk either running into deadlocks (which will kill performance) or inconsistency.
Decent SQL databases offer materialized views, which probably give you what you want without all the headache of maintaining denormalized tables yourself.