Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The database servers powering Let's Encrypt

letsencrypt.org

Post reply on HN
Print view View original

61–70 of 236 posts

Re: The database servers powering Let's Encrypt

#61

Post by sradman »

e12e wrote: post #46
sradman wrote: post #41

> We can clearly see how our old CPUs were reaching their limit. In the week before we upgraded our primary database server, its CPU usage (from /proc/stat) averaged over 90% This strikes me as odd. In my experience, traditional OLTP row stores are I/O bound due to contention (locking and latching). Does anyone have an explanation for this? > Once you have a server full of NVMe drives, you have to decide how to manag…

> The Let's Encrypt post does not describe how they implement off-machine and off-site backup-and-recovery. I'd like to know if and how they do this. The section: > There wasn’t a lot of information out there about how best to set up and optimize OpenZFS for a pool of NVMe drives and a database workload, so we want to share what we learned. You can find detailed information about our setup in this GitHub repository.…

> Which sounds like traditional master/slave setup, with fail over?

Yes, thank you. I assumed that the emphasis on the speed of NVMe drives meant that master/slave synchronous replication was avoided and asynchronous replication could not keep up. In my mind, this leaves room for interesting future efficiency/performance gains, especially surrounding the "...and is backed up daily" approach mentioned in your quote.

The bottom line is that the old RPO (Recovery Point Objective) and RTO (Recovery Time Objective) are as important as ever.

Re: The database servers powering Let's Encrypt

#64

Post by stefan_ »

dewey wrote: post #7
ryanworl wrote: post #5

What are they storing on this server that requires 150Tb of storage and millions of IOPS?

> What exactly are we doing with these servers? Our CA software, Boulder, uses MySQL-style schemas and queries to manage subscriber accounts and the entire certificate issuance process.

What exactly needs to be stored once the certificate is created and published in the hash tree? It seems like the kind of data that possibly needn't be stored at all or onto something like Glacier for archival.

Re: The database servers powering Let's Encrypt

#65

Post by Conclusionist »

stefan_ wrote: post #64
dewey wrote: post #7

Earlier quoted context omitted.

> What exactly are we doing with these servers? Our CA software, Boulder, uses MySQL-style schemas and queries to manage subscriber accounts and the entire certificate issuance process.

What exactly needs to be stored once the certificate is created and published in the hash tree? It seems like the kind of data that possibly needn't be stored at all or onto something like Glacier for archival.

Going to guess it's for OCSP responses.

Re: The database servers powering Let's Encrypt

#66

Post by zimmerfrei »

sgt wrote: post #3

I'm guessing someone out there's thinking: Why aren't they hosting in the cloud? The cloud being either Amazon or Azure. Surely nothing else exists. Is it really possible to host your own PHYSICAL machine? Does that count as the cloud?!

They are a Public CA and they must undergo a third party compliance audit to operate. The conditions are such you cannot really pass if your infra is in any of those public clouds.

Re: The database servers powering Let's Encrypt

#67

Post by jabberwcky »

cbg0 wrote: post #10

Unless I misunderstood something, it seems they have a single primary that handles read+write and multiple read replicas for it. It shouldn't be too difficult given the current use of MariaDB to start using something like Galera to create a multi-master cluster and improve redundancy of the service, unless there are some non-obvious reasons why they wouldn't be doing this. I think I also see redundant PSUs, would be…

Multi-master hardly comes for free in terms of complexity or performance, you're at the mercy of latency. Either host the second master in the same building, in which case the redundancy is an illusion, or host it somewhere else in which case watch your write rate tank

Asynchronous streaming to a truly redundant second site often makes more sense

Re: The database servers powering Let's Encrypt

#69

Post by tanelpoder »

jdsully wrote: post #56
tanelpoder wrote: post #15

Earlier quoted context omitted.

Also worth noting that scalability != efficiency. With enough NVMe drives, a single server can do millions of IOPS and scan data at over 100 GB/s. A single PCIe 4.0 x4 SSD on my machine can do large I/Os at 6.8 GB/s rate, so 16 of them (with 4 x quad SSD adapter cards) in a 2-socket EPYC machine can do over 100 GB/s. You may need clusters, duplicated systems, replication, etc for resiliency reasons of course, but a s…

It can burst to millions of IOPS, but you get killed on the sustained write workload. Even a high end enterprise NVMe drive will be limited to around 60k IOPs once you exceed its write cache.

Yup indeed it's an issue with NAND SSDs - and it heavily depends on the type (SLC, MLC, TLC, QLC), vendor (controller, memory, "write buffer") sizes etc. I'm doing mostly read tests right now and will move to writes after.

The Samsung PRO 980 I have, are TLC for main storage, but apparently are using some of that TLC storage as a faster write buffer (TurboWrite buffer) - I'm not an expert, but apparently the controller can decide to program the TLC NAND with only 1-bit "depth", they call it "simulated SLC" or something like that. On the 1 TB SSD, the turbowrite buffer can dynamically extend to ~100 GB, if there's unused NAND space on the disk.

Btw, the 3DXpoint storage (Intel Optane SSDs & Micron X1) should be able to sustain crazy write rates too.

Re: The database servers powering Let's Encrypt

#70

Post by cdcarter »

mike_d wrote: post #60
sradman wrote: post #41

> We can clearly see how our old CPUs were reaching their limit. In the week before we upgraded our primary database server, its CPU usage (from /proc/stat) averaged over 90% This strikes me as odd. In my experience, traditional OLTP row stores are I/O bound due to contention (locking and latching). Does anyone have an explanation for this? > Once you have a server full of NVMe drives, you have to decide how to manag…

> traditional OLTP row stores are I/O bound due to contention (locking and latching). Does anyone have an explanation for this? I have seen CPU bound database servers when developers push application logic in to the database. Everything from using server-side functions like MD5() to needless triggers and stored procedures that could have been done application side.

Indeed, at $PLACE_OF_WORK over the last 15 years a lot of logic was built in PL/SQL and dbcpu has become one of our most precious resources. For some applications its perfectly reasonable until you need to horizontally scale.
Post reply on HN