So scaling up instead of scaling out. I’m not sure if it’s a viable strategy long term, at the same time we probably don’t want a single CA to handle too many certificates?
The database servers powering Let's Encrypt
31–40 of 236 posts
Re: The database servers powering Let's Encrypt
#32>> We currently use MariaDB, with the InnoDB database engine. It is kind of funny how long InnoDB was the most reliable storage engine. I am not sure if MyISAM is still trying to catch up, it used to be much worse than InnoDB. With the emergence of RocksDB there are multiple options today.
MyISAM days are gone, no one will seriously consider it as suitable engine in MySQL.
It is quite likely unless you've meticulously avoided it that MySQL is using ISAM on-disk temp tables in the service of your queries.
Re: The database servers powering Let's Encrypt
#33Earlier quoted context omitted.
This is especially true with your own hardware. Trying this kind of thing in the cloud is usually prohibitively expensive.
It’s doesn’t have to. Unless you’re conditioned to believe that aws is cheap
I'm curious what the cost would be to run this type of hardware at any cloud vendor? Does it even exist?
Re: The database servers powering Let's Encrypt
#34I'm curious why they didn't go with the larger 64 core Epyc. I mean it's double the cost, but I suspect that the huge amount of NVMe SSDs is by far the largest part of the cost anyway. And it seems like CPU was the previous bottleneck as it was at 90%.
Re: The database servers powering Let's Encrypt
#35As someone unfamiliar with db management, is it really less operational overhead to have to physically scale your hardware than using a distributed option with more elastic scalability capabilities?
I’m guessing that since it is for registration and all, the usage might be write-driven, or at least equally balance between writes and reads.
In addition, you really care about integrity of your data so you probably want serializability, avoid concurrency and potential write/update conflicts, and to only do the writes on a single server.
For this reason it sounds to me that partitioning/sharding is the only way to really scale this: have different write servers that care about different primary keys.
Re: The database servers powering Let's Encrypt
#36Earlier quoted context omitted.
Also worth noting that scalability != efficiency. With enough NVMe drives, a single server can do millions of IOPS and scan data at over 100 GB/s. A single PCIe 4.0 x4 SSD on my machine can do large I/Os at 6.8 GB/s rate, so 16 of them (with 4 x quad SSD adapter cards) in a 2-socket EPYC machine can do over 100 GB/s. You may need clusters, duplicated systems, replication, etc for resiliency reasons of course, but a s…
> 16 of them (with 4 x quad SSD adapter cards) in a 2-socket EPYC machine can do over 100 GB/s. It is more interesting if actual CPU can handle such traffic in context of DB load: encode/decide records, sort, search, merge etc.
Yes, with modern storage, throughput is a CPU problem.
And CPU problem for OLTP databases is largely a memory access latency problem. For columnar analytics & complex calculations it's more about CPU itself.
When doing 1 MB sized I/Os for scanning, my 16c/32t (AMD Ryzen Threadripper Pro WX) CPUs were just about 10% busy. So, with a 64 core single socket ThreadRipper workstation (or 128-core dual socket EPYC server), there should be plenty of horsepower left.
Re: The database servers powering Let's Encrypt
#37I'm curious why they didn't go with the larger 64 core Epyc. I mean it's double the cost, but I suspect that the huge amount of NVMe SSDs is by far the largest part of the cost anyway. And it seems like CPU was the previous bottleneck as it was at 90%.
We didn't go with the 64-core chips because they have significantly lower clock speeds.
Dual 32-core chips give us plenty of cores while keeping clocks higher for single-threaded performance.
You are correct that the price of the CPUs is almost irrelevant to the overall cost of a system with this much memory and storage. We were picking the ideal CPU, not selecting on CPU price.
Re: The database servers powering Let's Encrypt
#38Earlier quoted context omitted.
It’s doesn’t have to. Unless you’re conditioned to believe that aws is cheap
I'm curious what the cost would be to run this type of hardware at any cloud vendor? Does it even exist?
That's the wrong way to think about the cloud. A better way to think about it would be "how much database traffic (and storage) can I serve from Cloud Whatever for $xxx". Then you need to think about what your realistic effective utilization would be. This server has 153600 GB of raw storage. That kind of storage would cost you $46000 (retail) in Cloud Spanner every month, but I doubt that's the right comparison. The right math would probably be that they have 250 million customers and perhaps 1KB of real information per customer. Now the question becomes why you would ever buy 24×6400 GB of flash memory to store this scale of data.
Re: The database servers powering Let's Encrypt
#39Just goes to show how much a single SQL server can scale before having to worry about sharing and horizontal scaling
Based on their stated 225M sites and a renewal period of 90 days, they're probably averaging around 40 certificates per second. That's only an order of magnitude higher than bitcoin; I wouldn't call it an indication of an ability to scale to a particularly large amount of traffic.
Bitcoin is ECDSA verification, letsencrypt is generating RSA signatures, the two aren't even remotely comparable.
Re: The database servers powering Let's Encrypt
#40Earlier quoted context omitted.
Based on their stated 225M sites and a renewal period of 90 days, they're probably averaging around 40 certificates per second. That's only an order of magnitude higher than bitcoin; I wouldn't call it an indication of an ability to scale to a particularly large amount of traffic.
Bitcoin is ECDSA verification, letsencrypt is generating RSA signatures, the two aren't even remotely comparable.
But the cryptographic bottleneck is in HSMs, not in database servers (database servers don't generate the digital signatures, they just have to store them after they've been generated).