11–20 of 207 posts
This was an interesting read for a database novice. It seems like a lot of the quoted stats are about in memory datasets - is that realistic?
In memory provides real-time transactions you can't guarantee when using disk-based storage.
This was an interesting read for a database novice. It seems like a lot of the quoted stats are about in memory datasets - is that realistic?
I'm speaking out of little experience though. I just think that a lot of us can get away with traditional vertical scaling and not think too much about it.
When it comes to replication & automatic failover in PG what is the 2021 gold standard setup?
In 2016 I was expecting some sane defaults like MySQL would have arrived by 2018 or 2019. Looks like 2021 isn't that much different to 2016. There are work being done, but doesn't seems to be anywhere close to the level of MySQL.
There is almost no consideration for the target being up to date, I have personally experienced missed inserts, replication lags, and the replica being set to read/write.
PGs integrated replication is far superior, it even has support for bootstrapping a new replica node without rsync and a command to `promote` the replica to write master.
I wonder when using a distributed database (like CockroachDB) will be the default for new applications. Right now it seems that they are less feature and harder to set up than traditional RDBMSes but I can only assume that this gap will narrow and at some point in the future things will be "scalable by default". (Of course no DB is going to prevent all ways to shoot yourself in the foot)
CockroachDB understandably wants you to use their Cloud or Enterprise products: the OSS version is quite limited. For example it doesn't support row-level partitioning (https://www.cockroachlabs.com/docs/stable/configure-replicat...). Which means, if I understand correctly, it is not of much help for scaling writes to a single big table.
In the opinion of a last semester CS student who has never written an application from scratch that needed more than a SQLite DB (so take me with a half grain of salt), it seems like premature optimization, while always talked about, is very common. I see people talking about using Kubernetes for internal applications and I just can't figure out why. If it's a hobby project and you want to learn Kubernetes, that's a…
I wonder when using a distributed database (like CockroachDB) will be the default for new applications. Right now it seems that they are less feature and harder to set up than traditional RDBMSes but I can only assume that this gap will narrow and at some point in the future things will be "scalable by default". (Of course no DB is going to prevent all ways to shoot yourself in the foot)
This question is similar to asking on a car forum when using a 40 foot lorry will be the default starter car for everyone. The answer is "probably never" because while it does offer superior cargo transport scalability, the tradeoffs are not worth it for the vast majority of users. The question is posed like distributed databases have no disadvantages over non-distributed databases, but that is simply not the case. C…
https://nickcraver.com/blog/2016/02/17/stack-overflow-the-ar... (2013 post had much less redundancy, but even their 2016 architecture is pretty undistributed in terms of being able to recreate everything from the single source of truth database)
I wonder when using a distributed database (like CockroachDB) will be the default for new applications. Right now it seems that they are less feature and harder to set up than traditional RDBMSes but I can only assume that this gap will narrow and at some point in the future things will be "scalable by default". (Of course no DB is going to prevent all ways to shoot yourself in the foot)
This question is similar to asking on a car forum when using a 40 foot lorry will be the default starter car for everyone. The answer is "probably never" because while it does offer superior cargo transport scalability, the tradeoffs are not worth it for the vast majority of users. The question is posed like distributed databases have no disadvantages over non-distributed databases, but that is simply not the case. C…
- Switching from a car to a van to a lorry is fairly low cost. You don't need to recreate your product (probably).
- You don't need to run distributed databases in a cluster to start.
But I think most importantly the decrease in dev speed and performance is an investment in future scalability. And I only imagine that this different will shrink over time to where for example a 1 node "distributed" DB isn't that different to work with than a 1 node "traditional" DB. And that small difference pays off because adding a second node doesn't have huge tradeoffs.
I agree that right now it doesn't make sense. If I was starting a new product I would still fire up a PostgreSQL instance. However I think that the day will come where the difference is small enough and software support is complete enough that we will start with something distributed, much like people don't often start with SQLite today, even though it is quicker to get going (also not a great comparison because it has less features).
This was an interesting read for a database novice. It seems like a lot of the quoted stats are about in memory datasets - is that realistic?
Yes. Memory can reach 768GB on a single instance today and I imagine that to expand. From there you can scale by sharding. In memory provides real-time transactions you can't guarantee when using disk-based storage.
https://letsencrypt.org/2021/01/21/next-gen-database-servers...
They have 2TB RAM in there, and I suspect that is not the largest possible amount if you're willing to spend more money (though probably the largest possible amount for that particular server).
I played around a bit on the Dell website, and the largest server I could find supported up to 6TB RAM, with a price at ~300k EUR (I assume nobody pays list price for these).