Just goes to show how much a single SQL server can scale before having to worry about sharing and horizontal scaling
This is especially true with your own hardware. Trying this kind of thing in the cloud is usually prohibitively expensive.
The database servers powering Let's Encrypt
21–30 of 236 posts
Re: The database servers powering Let's Encrypt
#22As someone unfamiliar with db management, is it really less operational overhead to have to physically scale your hardware than using a distributed option with more elastic scalability capabilities?
Relational databases enable some very flexible data access patterns. Once you shard, you lose a lot of that flexibility. If you move away from a relational model, you lose even more flexibility and start having to do much more work in your application layer, and usually start having to use more resources and developer time every step of the way. The productivity enabled by having one master RDBMS is a big deal, and i…
If I had a billion dollars, I'd put a research group together to study the prospects of index sharding.
That is, full table replication, but individual servers maintaining differing sets of indexes. OLAP and single request transactions could be routed to specialized replicas based on query planning, sending requests to machines that have appropriate indexes, and preferably ones where those indexes are hot.
Re: The database servers powering Let's Encrypt
#23Just goes to show how much a single SQL server can scale before having to worry about sharing and horizontal scaling
This is especially true with your own hardware. Trying this kind of thing in the cloud is usually prohibitively expensive.
It’s doesn’t have to. Unless you’re conditioned to believe that aws is cheap
Re: The database servers powering Let's Encrypt
#24I'm guessing someone out there's thinking: Why aren't they hosting in the cloud? The cloud being either Amazon or Azure. Surely nothing else exists. Is it really possible to host your own PHYSICAL machine? Does that count as the cloud?!
Yes it's still possible to put your own physical machines in a datacentre. For example : https://www.scaleway.com/en/dedibox/dedirack/ I'm not sure you can say it's the cloud though. They are not hosting their database in the cloud like Amazon or Azure because no cloud provider offers such high performances at a comparable price. Actually I'm not even sure you can get a cloud VM with that many IOs, if you don't mind…
We have a server with the hostname "cloud.example.com".
It can help if someone wants their data "in the cloud".
Re: The database servers powering Let's Encrypt
#25I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me.
64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU.
If you price it out on Dell's site, you get a retail price north of $200k. That is really what made it clear for me. That you could fit $200k+ worth of DIMMS, Drives, CPUS into a 2U spot :)
Re: The database servers powering Let's Encrypt
#26Just goes to show how much a single SQL server can scale before having to worry about sharing and horizontal scaling
Based on their stated 225M sites and a renewal period of 90 days, they're probably averaging around 40 certificates per second. That's only an order of magnitude higher than bitcoin; I wouldn't call it an indication of an ability to scale to a particularly large amount of traffic.
Re: The database servers powering Let's Encrypt
#27Earlier quoted context omitted.
Relational databases enable some very flexible data access patterns. Once you shard, you lose a lot of that flexibility. If you move away from a relational model, you lose even more flexibility and start having to do much more work in your application layer, and usually start having to use more resources and developer time every step of the way. The productivity enabled by having one master RDBMS is a big deal, and i…
If I had a billion dollars, I'd put a research group together to study the prospects of index sharding. That is, full table replication, but individual servers maintaining differing sets of indexes. OLAP and single request transactions could be routed to specialized replicas based on query planning, sending requests to machines that have appropriate indexes, and preferably ones where those indexes are hot.
The problem is the network. You need billion dollars to fix the network so it's as fast as local ram/nvme.
Re: The database servers powering Let's Encrypt
#28What a great read. I think the authors here made great hardware and software decisions. OpenZFS is the way to go, and is so much easier to manage than the legacy RAID controllers imho. Ah, I miss actual hardware.
I enjoyed it as well, i'm also appreciative that they shared their configuration notes here. I've been running multiple data stores on ZFS for years now and it's taken a while to get out of the hardware mindset (albeit you still need a nice beefy controller anyway).
Re: The database servers powering Let's Encrypt
#29As someone unfamiliar with db management, is it really less operational overhead to have to physically scale your hardware than using a distributed option with more elastic scalability capabilities?
Also worth noting that scalability != efficiency. With enough NVMe drives, a single server can do millions of IOPS and scan data at over 100 GB/s. A single PCIe 4.0 x4 SSD on my machine can do large I/Os at 6.8 GB/s rate, so 16 of them (with 4 x quad SSD adapter cards) in a 2-socket EPYC machine can do over 100 GB/s. You may need clusters, duplicated systems, replication, etc for resiliency reasons of course, but a s…
> 16 of them (with 4 x quad SSD adapter cards) in a 2-socket EPYC machine can do over 100 GB/s.
It is more interesting if actual CPU can handle such traffic in context of DB load: encode/decide records, sort, search, merge etc.
Re: The database servers powering Let's Encrypt
#30What are they storing on this server that requires 150Tb of storage and millions of IOPS?
> What exactly are we doing with these servers? Our CA software, Boulder, uses MySQL-style schemas and queries to manage subscriber accounts and the entire certificate issuance process.
There's nothing in that sentence that implies they'd need even 100 IOPS, much less 20 million.