Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The database servers powering Let's Encrypt

letsencrypt.org

Post reply on HN
Print view View original

171–180 of 236 posts

Re: The database servers powering Let's Encrypt

#171

Post by toast0 »

1MachineElf wrote: post #110

I'm thankful for their OpenZFS tuning doc which they developed as part of this server migration: https://github.com/letsencrypt/openzfs-nvme-databases The one thing that I get hung up on when it comes to RAID and SSDs is the wear pattern vs. HDDs. Take for example this quote from the README.md: We use RAID-1+0, in order to achieve the best possible performance without being vulnerable to a single-drive failure. Failu…

> Failure on SSDs is predictable and usually expressed with Terabytes Written (TBW). Failure on spinning disk HDDs is comparatively random.

That wasn't my experience with thousands of SSDs and spinning drives. Spinning drives failed more often, but usually with SMART sector counts increasing before hand. Our SSDs never got close to media wearout, but that didn't stop them from dropping off the bus. Literally working fine, then boom can't detect; all data gone.

Then there's the incidents where the power on hours value rolls over and kills the firmware. I believe these have happened on disks of all types, but here's a recent one on SSDs [1]. Normally when building a big server, all the disks are installed and powered on at the same time, which risks catastrophic failure in case of a firmware bug like this. If you can, try to get drives from different batches, and stagger the power on times.

[1] https://www.zdnet.com/article/hpe-says-firmware-bug-will-bri...

Re: The database servers powering Let's Encrypt

#172

Post by tialaramex »

SulfurHexaFluri wrote: post #162
dylan604 wrote: post #135

Earlier quoted context omitted.

They might average that, but we all know averages only work on paper. For example, AWS has tutorials that provide instructions on how to setup TLS in a LAMP stack running on Linux 2 EC2s. As part of the Let's Encrypt setup, they provide a crontab entry that runs twice a day with a copy option to paste. How many EC2s all hit the Let's Encrypt server at that exact time? Since EC2s default to UTC time, that means that s…

Its worth noting that running certbot / LE twice a day doesn't actually hit the LE server twice a day. It just checks the certificate dates locally and if they have been renewed in the last month it does nothing. I guess you still have a peak time of 00:00 UTC every day though unless people are using servers set to their local time.

I believe modern Certbot also does OCSP. The intent here is, if for any reason your certificate was revoked it makes sense to try to obtain a new certificate even if it hasn't expired. Even if it can't, perhaps the operator will notice a logged reason from Certbot when they did not yet notice their certificate was revoked.

Examples of reasons your certificate might have been revoked:

* You re-used a private key from somewhere, perhaps because you don't understand what "private" means, and other copies of that key leaked

* You didn't re-use the private key but your "clever" backup strategy involves putting the private key file in a public directory named /backup/server/ on your web server and somebody found it

* You use the same certificate for dozens of domain names in your SEO business and yesterday you sold a name for $10k. Hooray. The new owner immediately revoked all certificates for that name which they're entitled to do.

* Your tooling is busted and the "random" numbers it picked aren't very random. (e.g. Debian OpenSSL bug)

* A bug at Let's Encrypt means their Ten Blessed Methods implementation was inadequate for some subset of issuances and rather than cross their fingers the team decided to revoke all certificates issued with the inadequate control.

* Let's Encrypt discovers you're actually a country sanctioned by the US State Department, perhaps in some thin disguise such as an "independent" TV station in a country that doesn't have any independent media whatsoever. It is illegal for them to provide you with services and you were supposed to already know that.

So that is a network connection, but not to the Let's Encrypt servers described in this story.

In practice OCSP is done by a big CA by periodically computing OCSP responses for every single unexpired certificate (either saying it's still valid, or not), and then providing those to a CDN and the CDN acts as "OCSP server" returning the appropriate OCSP response when asked without itself having possession of any cryptographic materials.

Re: The database servers powering Let's Encrypt

#173

Post by killingtime74 »

kilburn wrote: post #170
PeterCorless wrote: post #95

Earlier quoted context omitted.

Yes. They are not doing a very heavy computational workload. Typical heavy-duty servers these days can do 100k's or millions of TPS. 40 TPS is a really, really, really light load. Further, I was looking at those new server specs. There's an error I think? The server config on the Dell site shows 2x 8 GB DRIMMs, for 16 GB RAM per sever, whereas the article says 2 TB! With only 16GB of RAM, but 153.6 TB of NVMe storage…

Why are you assuming that their workload includes just one query per emitted certificate? The reality is that they are storing information during challenges, implementing rate limiting per-account, supporting OCSP validation and a few other things. You can investigate further if you really want to see the queries that they make against the database since their software (Boulder) is open source [1]. Most queries are i…

Kudos for sharing the location of relevant code. This is the kind of referencing we need

Re: The database servers powering Let's Encrypt

#174

Post by spijdar »

segmondy wrote: post #155
tyingq wrote: post #25

I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me. 64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU. If you price it out on Dell's site, you get a r…

I'm surprised that's what did it for me. I admined some sun enterprise servers and was blown away when I bought a pi 3 yrs and realized the pi was probably faster than it. That enterprise server would have 100-200 engineering students logged in and working at once. :-/. We're living in the future.

> the pi was probably faster than it.

If we're talking about 90s or early 2000s Sun, probably. Even then, though, these systems probably had substantially better I/O performance.

I have a SPARC laptop lying around from 1995 that gets a whopping 20 MB/s in disk read/write speeds across its dual half height SCSI drives. That still beats all but the best SD cards.

The full size systems with SAS and other options would get even better disk performance.

Re: The database servers powering Let's Encrypt

#175

Post by tialaramex »

cm2187 wrote: post #118
PeterCorless wrote: post #95

Earlier quoted context omitted.

Yes. They are not doing a very heavy computational workload. Typical heavy-duty servers these days can do 100k's or millions of TPS. 40 TPS is a really, really, really light load. Further, I was looking at those new server specs. There's an error I think? The server config on the Dell site shows 2x 8 GB DRIMMs, for 16 GB RAM per sever, whereas the article says 2 TB! With only 16GB of RAM, but 153.6 TB of NVMe storage…

Don’t these certificates have long RSA keys which are more expensive computationally? Though I guess that doesn’t have to happen on the database server.

The only RSA computations Let's Encrypt need to do are:

* Signature by their RSA Intermediate (currently R3, with R4 on hot standby) - which will be a dedicated piece of hardware - to issue a subscriber's certificate. In practice this happens twice, as a poisoned pre-certificate to obtain proof of logging from public log servers, and then the real certificate with the proofs baked inside it.

* Signatures by their OCSP signer periodically on an OCSP response for each certificate saying it's still trustworthy for a fixed period. Again this will be inside an HSM.

* Signature verification on a subscriber's CSR. To "complete the circuit" it's helpful that Let's Encrypt actually confirms you know the private key corresponding to the public key you wanted a certificate for, the signature on your CSR does this. Some people don't think this is necessary, but I believe Let's Encrypt do it anyway.

You're correct that none of this happens on the database servers. I guess it's possible their servers use TLS to secure the MariaDB connections, in which case a small amount of either ECDSA or RSA computation happens each time such a connection is set up or torn down like at any outfit using TLS, but those database connections are cached in a sane system so that wouldn't be very often.

Re: The database servers powering Let's Encrypt

#176

Post by tialaramex »

stefan_ wrote: post #74
Conclusionist wrote: post #65

Earlier quoted context omitted.

Going to guess it's for OCSP responses.

I'm not sure, e.g. Chrome doesn't do OCSP by default, lots of embedded clients like curl won't either. Unless the protocol is terribly broken, that also seems like the kind of use case where 99% of queries just come out of cache and should never hit a database.

Let's Encrypt still has to publish OCSP responses for every non-expired leaf certificate, at least in time that you can always get a new OCSP response before the previous one expires. In practice they have a tighter schedule so that there's a period between "We are not meeting our self-imposed deadline" and "The Internet broke, oops" in which staff can figure out the problem and fix it.

To do this they automatically generate and sign OCSP responses (the vast majority of which will just say the certificate is still good) on a periodic cycle, and then they deliver them in bulk to a CDN. The CDN is who your client (or server if you do OCSP stapling, which you ideally should) talks to when checking OCSP.

To generate those responses they need a way (hey, a database) to get the set of all certificates which have not yet expired and whether those certificates are revoked or not.

Re: The database servers powering Let's Encrypt

#177

Post by angled »

tyingq wrote: post #166
dan-robertson wrote: post #132

Earlier quoted context omitted.

For a comparison, you could get a 64 processor, 256Gb ram Sun Starfire around 20 years ago[1]. Wikipedia claims these cost well over a million dollars ($1.5 million in 2021 dollars). This machine was enormous (bigger than a rack), would have had more non-uniform memory access to deal with, and the processors were clocked at something like 250-650MHz. [1] https://en.wikipedia.org/wiki/Sun_Enterprise

That's a pretty good comparison. It weighed 2000lbs, and was 38 inches wide, or basically 2 full racks, which I guess you could call an 84U server. It was also 49 inches deep, versus a standard rack which is 36 inches deep.

iirc, anton got a kernel compile down to about a minute with -j32 back in the day while at ozlabs?

Re: The database servers powering Let's Encrypt

#178

Post by Aeolun »

WatchDog wrote: post #92
dastbe wrote: post #76

Earlier quoted context omitted.

(I work at AWS, but this is just for fun) Checking out AWS side, the closest I think you'd get is the x1.32xlarge, which would translate to 128 vCPU (which on intel generally means 64 physical cores) and close to 2TB of RAM. nvme storage is only a paltry 4TB, so you'd have to make up the rest with EBS volumes. You'd also get a lower clock speed than they are getting out of the EPICs

If you eschew RDS, the largest you can go up to seems to be a u-24tb1.metal. 448 vcpu, 24TiB of RAM, $70 an hour. ~$52k per month.

Oh, only $70 per hour. I think I can start that for an hour or so before my wallet is exhausted :P

Re: The database servers powering Let's Encrypt

#179

Post by tialaramex »

cipherboy wrote: post #90
siwyd wrote: post #82

Earlier quoted context omitted.

FYI, the intermediate CA's signed by their new Root X2 certificate won't have OCSP URLs anymore. Source: https://letsencrypt.org/2020/09/17/new-root-and-intermediate...

AFAIK, nobody has suggested removal of OCSP from end-entity certificates. This article you linked (and the comment you wrote) is purely about removal from intermediate CA certificates. The majority of OCSP traffic will probably be for end-entity certificates; most OCSP validation (in browsers and cryptographic libraries) is end-entity validation, not leaf-and-chain. Removal of intermediate CA's OCSP is probably not r…

There's an argument for not doing OCSP on end-entity certificates if you can approach the lifetime for the certificates that you'd realistically need for OCSP responses anyway.

Suppose you promise to issue OCSP revocations within 48 hours if it's urgent, and your OCSP responses are valid for 48 hours. That means after a problem happens OCSP revocation takes up to 96 hours to be effective.

If you only issue certificates with lifetimes of 96 hours then OCSP didn't add anything valuable - the certificates expire before they can effectively be revoked anyway.

Let's Encrypt is much closer to this idea (90 days) than many issuers were when it started (offering typically 1-3 years) but not quite close enough to argue revocation isn't valuable. However, the automation Let's Encrypt strongly encourages makes shortening lifetimes practical. Many of us have Let's Encrypt certs automated enough that if they renewed every 48 hours instead of every 60 days we'd barely care.

The solution to excessive OCSP traffic and privacy risk is supposed to be OCSP stapling instead, but TLS servers that can't get stapling right are still ridiculously popular so that hasn't gone so well.

Re: The database servers powering Let's Encrypt

#180

Post by wahern »

dylan604 wrote: post #135
cperciva wrote: post #26

Earlier quoted context omitted.

Based on their stated 225M sites and a renewal period of 90 days, they're probably averaging around 40 certificates per second. That's only an order of magnitude higher than bitcoin; I wouldn't call it an indication of an ability to scale to a particularly large amount of traffic.

They might average that, but we all know averages only work on paper. For example, AWS has tutorials that provide instructions on how to setup TLS in a LAMP stack running on Linux 2 EC2s. As part of the Let's Encrypt setup, they provide a crontab entry that runs twice a day with a copy option to paste. How many EC2s all hit the Let's Encrypt server at that exact time? Since EC2s default to UTC time, that means that s…

OpenBSD recently added the "~" random range separator to their crontab syntax. The manual for acme-client, the native Let's Encrypt tool, provides

  ~  *  *  *  *  acme-client example.com && rcctl reload httpd
as the example crontab entry. Though, for maximum friendliness to Let's Encrypt that should probably be something like "~ ~ * * ~", which would run the command at a random time once per week. I think you could accomplish something similar using systemd timer units and RandomizedDelaySec, assuming it permits a delay as high as 604800.
Post reply on HN