51–60 of 61 posts
Just writing sql and maintaining and testing it like regular code is a perfectly reasonable thing to do.
Not to mention that having sql be callable as an actual function in the programming language that the rest of your application is written in is just so much more comfortable than having small bits of sql scattered about in strings.
Earlier quoted context omitted.
> Where is this meme that RDBMSs do not scale coming from? They don't scale to Google or Facebook operational sizes. Once you get to a billion customers or so the ol' RDBMS tends to struggle. Because everyone wants to be Google they imagine they have Google's problems. I've been in a meeting where the client was talking about their severe scaling issues for their "big data" which could only possibly be resolved by st…
> They don't scale to Google or Facebook operational sizes. You may or may not know this, but the primary datastore used at both Google and FB is MySQL. Sure, they use replication and sharding, but I would strongly argue that MySQL with sharding scales better than some multi master NoSQL thing like Cassandra. Related, you should check out https://github.com/vitessio/vitess if you haven't seen it. It's what Youtube an…
I'm almost certain this isn't true, BigTable and Spanner are much more widely used at Google because.... well, MySql doesn't really scale.
It is a great tragedy of our day that so many software engineers are poor at SQL. Use tools like Jinja and DBT to modularize it, and Airflow to schedule it; but use of SQL should be maximized wherever possible.
Shameless plug: https://github.com/socialpoint-labs/sqlbucket
Just changed job though, and the data engineering team I just joined is literally banning raw SQL and enforcing scala instead... I'm a bit sad.
Today the thinking "keep all domain logic in code" would take you to crazy places given the diversity of systems we need to build applications.
"A foolish consistency is the hobgoblin of little minds" and all that.
If anything it seems more sensible to centralize storage of all business rules in the DB.
Earlier quoted context omitted.
I thought about making a "shadow database" on a server since my work PC is only 8 GB ram but ultimately decided against it. The first reason was GDPR compliance and the second is that we have an already complex architecture as it is (and we're only 2 to manage it) so I don't want to add one more brittle layer that will need debugging at the worst time.
About GDPR: is it possible that holding data in a non-storage/volatile medium could be legally distinct from non-volatile storage, especially if it's essentially syncing with an authoriative data source that's responsible for managing GDPR? Because if not, it seems to me every proxy and persistence layer runs legal risks....
Earlier quoted context omitted.
I thought about making a "shadow database" on a server since my work PC is only 8 GB ram but ultimately decided against it. The first reason was GDPR compliance and the second is that we have an already complex architecture as it is (and we're only 2 to manage it) so I don't want to add one more brittle layer that will need debugging at the worst time.
About GDPR: is it possible that holding data in a non-storage/volatile medium could be legally distinct from non-volatile storage, especially if it's essentially syncing with an authoriative data source that's responsible for managing GDPR? Because if not, it seems to me every proxy and persistence layer runs legal risks....
They do.
One thing this leaves out is the danger of concurrency issues, which can sometimes be worsened by using an ORM. Writing correct, race-free SQL can be very hard (with READ COMMITTED transactions) or require complex retry logic (with SERIALIZABLE). Preventing deadlocks can also be a concern. ORMs tend to hide the underlying SQL operations, making it even harder to verify whether operations are concurrency safe.
> complex retry logic READ COMMITTED and REPEATABLE READ benefit from retry logic as well, not just SERIALIZABLE. For a long time, I sought to write deadlock free code. But that is very hard. For example in PostgreSQL every UPDATE must be ordered, every DELETE must be ordered. [1] Finally, I did myself a favor and create application-level retires. This is 100% cool so long as (1) your deadlocks aren't so frequent so…
One thing this article doesn't mention is that in many cases databases scale poorly, and only vertically. You can throw a bunch of domain logic in complex SQL queries and see significant performance benefits, but at some point you may find that your application has grown and your SQL server is running into IOPS and CPU limits that are difficult to fix. At that point your answer could be, pull logic out of those compl…
Earlier quoted context omitted.
+1 Also, in my experience, SQL query optimizers can also be unpredictable on complex queries (behavior changes with data size and statistics) which can cause them to suddenly change to slow execution plan when data grows or something is added , despite having all indexes for a good plan possible (which for interactive apps is to avoid table scans always -- they might be faster in some bad corner cases, but if data do…
That could be a fault of complex physical data models that need simplification.