141–150 of 165 posts
Earlier quoted context omitted.
Huh, almost every lead backend job I've interviewed for had a lot of DB design questions, more than normal coding stuff because it's so critical to scaling. It's core to the role even for lower level positions, so it really should be more common.
Clearly you are not a Node.is dev.
Earlier quoted context omitted.
RDBMS and Non-RDBMS both have there place, I have used both in the same system several times, all for things that they were good at. Transactions allow you to be confident while making complex changes that in case a failure occurs all partial changes will be rolled back. making use of database level validations and enforcing referential integrity is essential for keeping data consistent over the long term and making…
If scalability is your concern then you can't use any of the supposedly core features of an RDBMS, since fundamentally there is no way to have a transaction across multiple nodes without solving a much bigger problem. Validation is vital but the datastore is not the place to do it, because handling invalid data by dropping it on the floor is almost never the right behaviour. There is no substitute for actually unders…
Earlier quoted context omitted.
Clearly you are not a Node.is dev.
Or a Django dev, or a Rails dev, or any other ‘framework developer’.
But you are correct, interviews for those types of jobs for whatever reason don't tend to focus as much on DB stuff, though they really really should given that a quarter of your job will end up being unfucking the system away from the poor choices that your predecessors made (if you're guaranteed to be serving 1mm DAU and everyone will be generating thousands of records per month, maaaybe don't use normal ints for uids...).
Earlier quoted context omitted.
If scalability is your concern then you can't use any of the supposedly core features of an RDBMS, since fundamentally there is no way to have a transaction across multiple nodes without solving a much bigger problem. Validation is vital but the datastore is not the place to do it, because handling invalid data by dropping it on the floor is almost never the right behaviour. There is no substitute for actually unders…
It is extremely difficult to design a good RDBMS schema without understanding the data model, and once you do, it is there documented in its entirety with best in class tooling for anyone else to come along, pickup and be up to speed with it, additionally you don't have to forgo document storage, most if not all modern RDBMS suppord json(b) types.
Earlier quoted context omitted.
> Why would you be input data in a way that violates referential integrity? That would be pretty bizarre. Well if you can't get input that violates that integrity then what are you gaining by enforcing that integrity? > That being said, if your code doesn't catch and handle errors, then there's a couple of deeper problems already. Sure, but how can you do error handling without data storage, given that your applicati…
> Well if you can't get input that violates that integrity then what are you gaining by enforcing that integrity? Perhaps I'm wording this badly? Your application should _not_ be generating data which can't go into the database correctly. If it is, that's a different set of problems. ;)
Earlier quoted context omitted.
> Why would you be input data in a way that violates referential integrity? That would be pretty bizarre. Well if you can't get input that violates that integrity then what are you gaining by enforcing that integrity? > That being said, if your code doesn't catch and handle errors, then there's a couple of deeper problems already. Sure, but how can you do error handling without data storage, given that your applicati…
If you allow invalid records and multiple versions of a schema in a collection, you will forever being paying for that mistake every time you read from the database.
Fixing up data that was written wrongly takes work, there's no getting away from that. But if you have a record of the original write attempt then at least you can do that work and recover the data (if you decide the cost/benefit is worth it). Whereas if you have the datastore reject it at write time then you're SOL before you've even started.
Earlier quoted context omitted.
Well if it's not distributed then it's a single point of failure, almost by definition. I hate maintaining a Zookeeper cluster as much as anyone, but anything that runs on a single server is not a real alternative (and if a single-server SPoF is acceptable, then just running the batch job / workflow processor / whatever it is on a single server works just as well - maybe with a pool of workers to scale out the actual…
My most common batch/workflow usecases tend to involve a few hundred workers, and tolerance for brief downtime (especially if jobs don't fail outright). PostgreSQL works surprisingly well for something like this. It has a rich set of coordination and locking primitives, and it can usually be recovered and restarted in under 10 minutes if the server fails. Yes, connection overhead from 300 workers is a real problem, s…
I do see where you're coming from, but I find they're more trouble than they're worth in the long term. YMMV I guess.
Earlier quoted context omitted.
I see. I mean, one of my ideas requires many transactions and records that require data consistency. I have no idea how to make sure it can scale to Internet scale. Do you know any good resources? Or is it just "get mysql, pay thousands a month for either a cloud provider or colocated hardware"
I would say that most resources would be a mixed bag in most situations but much of the advice skips over something very basic that experienced people tend to assume you know: you are always going to be limited by the number of times you can actually sync a write to a disk and/or how fast you can read back from the disk and these are the main upper bounds. One mistake I see inexperienced engineers make in modern envi…
Did you read the article a little while back about Discord's challenges with storing tons of messages and how they upgraded their tech stack? Did it make sense to you?
Earlier quoted context omitted.
Why have the RDBMS as the default? Why assume you need ACID and transactions when they're overwhelmingly likely to bring you nothing but trouble (the number of web applications that make effective use of database-level transactions is approximately zero). You're far better off starting with a system that does seamless active-active by default, and then figuring out what kind of transactional guarantees you need. An R…
> Why have the RDBMS as the default? It's possibly the sanest default for a data persistence layer. I'd argue otherwise: use a RDBMS unless you know what you are doing. > the number of web applications that make effective use of database-level transactions is approximately zero Almost every driver and layer (ORMs, etc) that connect to databases use transactions. Transparently or not, but it uses them. So actually the…
They're using them but they're not getting any useful effect out of them, since the database-level transactions don't actually align with business-level transactions. Fundamentally you can't actually use database-level transactions in a useful way in a web application because you can't share a transaction between multiple HTTP requests (without introducing much bigger problems).
> Either it means that you serialize everything in every active node (pointless)
Not pointless - you gain redundancy which is the whole point. Performance will be bad but for a low-traffic system it's often actually fine.
> Which in turn means either supporting distributed transactions (e.g. via Paxos or RAFT protocols) or doing conflict resolution (which essentially is a nice way of saying "data loss"). Both problems are much, much harder than the (usually harmless) consequences of using transactions on a "classical" RDBMS.
The consequence of using transactions on a classical RDBMS at scale without thinking carefully about it is deadlocks, which are just as bad as distributed transaction problems, and you still have to do conflict resolution in the case where a transaction gets aborted, which will happen sooner or later (most RDBMS fans just ignore it and lose data). If you want decent performance then eventually you'll have to model your data properly using CRDTs (this is true even in a classical RDBMS) and at that point you gain nothing from traditional RDBMS transactions.
> Therefore being RDBMSs like Postgres the best and safest, the default, approach that should be taken for data persistence layers.
RDBMSes are not safe. The way they achieve their integrity guarantees is by rejecting, and dropping, unexpected writes. This almost always loses data in practice.