The database servers powering Let's Encrypt

Post by Denvercoder9 » Fri, Jan 22, 2021, 1:56 AM UTC

dylan604 wrote: post #135

cperciva wrote: post #26

Earlier quoted context omitted.

Based on their stated 225M sites and a renewal period of 90 days, they're probably averaging around 40 certificates per second. That's only an order of magnitude higher than bitcoin; I wouldn't call it an indication of an ability to scale to a particularly large amount of traffic.

They might average that, but we all know averages only work on paper. For example, AWS has tutorials that provide instructions on how to setup TLS in a LAMP stack running on Linux 2 EC2s. As part of the Let's Encrypt setup, they provide a crontab entry that runs twice a day with a copy option to paste. How many EC2s all hit the Let's Encrypt server at that exact time? Since EC2s default to UTC time, that means that s…

Post by tyingq » Fri, Jan 22, 2021, 1:58 AM UTC

walrus01 wrote: post #127

tyingq wrote: post #109

Earlier quoted context omitted.

Yes, that's worth noting. The $200k is the list/retail, which nobody would pay for a purchase of this size.

At this size/scale I wonder what the actual price would be for a comparable Supermicro system. If I had to make a wild guess, well under half the 200k previously quoted.

Post by zerd » Fri, Jan 22, 2021, 2:08 AM UTC

tanelpoder wrote: post #15

multifascia wrote: post #4

As someone unfamiliar with db management, is it really less operational overhead to have to physically scale your hardware than using a distributed option with more elastic scalability capabilities?

Also worth noting that scalability != efficiency. With enough NVMe drives, a single server can do millions of IOPS and scan data at over 100 GB/s. A single PCIe 4.0 x4 SSD on my machine can do large I/Os at 6.8 GB/s rate, so 16 of them (with 4 x quad SSD adapter cards) in a 2-socket EPYC machine can do over 100 GB/s. You may need clusters, duplicated systems, replication, etc for resiliency reasons of course, but a s…

Post by tyingq » Fri, Jan 22, 2021, 2:22 AM UTC

walrus01 wrote: post #144

tyingq wrote: post #143

Earlier quoted context omitted.

The NVME drives are what's making it so expensive. Dell retail is $6044 for each of the Intel P4610 6.4TB drives. That's $145k.

after 15 seconds of googling, the P4610 U.2 format 6.4TB seems something more like a single unit street price of $2400 from non-Dell vendors. I'm mildly surprised it's that low considering that the U.2 format stuff, for serious servers, will always command a premium price. Probably in the range of $2100 to $2200 per unit from a x86-64 component distributor in moderate quantities.

Post by sradman » Fri, Jan 22, 2021, 2:34 AM UTC

jeffbee wrote: post #136

mike_d wrote: post #130

Earlier quoted context omitted.

It sounds like you just hit the threshold where defaults don't cut it anymore. With >100 concurrent clients you need to tune your DB for your workload. innodb_thread_concurrency and innodb_concurrency_tickets would be a good starting point, and optimal values depend on your r/w balance and number of rows touched per type of query.

I'm saying the innodb buffer pool mutex doesn't scale over 100 contenders and you are saying that I can tune mysql so there's never more than that, which it seems to me like we're in agreement.