Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The database servers powering Let's Encrypt

letsencrypt.org

Post reply on HN
Print view View original

131–140 of 236 posts

Re: The database servers powering Let's Encrypt

#131

Post by ggregoire »

tyingq wrote: post #25

I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me. 64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU. If you price it out on Dell's site, you get a r…

We made so much progress in that field over the last 40 years... I sometimes get lost trying to imagine what will be the computing performance available in 100, 1000, 10000 years from now...

Re: The database servers powering Let's Encrypt

#132

Post by dan-robertson »

tyingq wrote: post #25

I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me. 64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU. If you price it out on Dell's site, you get a r…

For a comparison, you could get a 64 processor, 256Gb ram Sun Starfire around 20 years ago[1]. Wikipedia claims these cost well over a million dollars ($1.5 million in 2021 dollars). This machine was enormous (bigger than a rack), would have had more non-uniform memory access to deal with, and the processors were clocked at something like 250-650MHz.

[1] https://en.wikipedia.org/wiki/Sun_Enterprise

Re: The database servers powering Let's Encrypt

#133

Post by sradman »

PeterCorless wrote: post #99
sradman wrote: post #41

> We can clearly see how our old CPUs were reaching their limit. In the week before we upgraded our primary database server, its CPU usage (from /proc/stat) averaged over 90% This strikes me as odd. In my experience, traditional OLTP row stores are I/O bound due to contention (locking and latching). Does anyone have an explanation for this? > Once you have a server full of NVMe drives, you have to decide how to manag…

> This strikes me as odd. In my experience, traditional OLTP row stores are I/O bound due to contention (locking and latching). Does anyone have an explanation for this? Yes. My CTO, Avi Kivity did a great talk about this at Core C++ 2019: https://www.scylladb.com/2020/03/26/avi-kivity-at-core-c-201... Let me boil it down to a few points; some beyond Avi's talk: • Traditional RDBMS with strong consistency and ACID gu…

> But the main point is that you have to really think about how to re-architect your software to take advantage of huge multi-processor machines.

I appreciate the response but it doesn't address my question: given that Let's Encrypt's MySQL-family RDBMS does not implement any of the multi-core/multi-socket/cpu-affinity/lock-free/asyncIO techniques used by databases like ScyllaDB, MemSQL, and VoltDB, why were they seeing 90% CPU utilization on their old Intel servers while the upgraded AMD servers were 25% (the expected range)?

I think mike_d's suggestion is most plausible: they probably included custom functions/procedures that invoke CPU-expensive code. I also thought this was a single-node scale-up architecture but since they are using a three-or-more node master/slave architecture, network I/O could somehow be involved.

Re: The database servers powering Let's Encrypt

#134

Post by gnarbarian »

ggregoire wrote: post #131
tyingq wrote: post #25

I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me. 64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU. If you price it out on Dell's site, you get a r…

We made so much progress in that field over the last 40 years... I sometimes get lost trying to imagine what will be the computing performance available in 100, 1000, 10000 years from now...

Sticks and rocks most likely.

Re: The database servers powering Let's Encrypt

#135

Post by dylan604 »

cperciva wrote: post #26
jarym wrote: post #18

Just goes to show how much a single SQL server can scale before having to worry about sharing and horizontal scaling

Based on their stated 225M sites and a renewal period of 90 days, they're probably averaging around 40 certificates per second. That's only an order of magnitude higher than bitcoin; I wouldn't call it an indication of an ability to scale to a particularly large amount of traffic.

They might average that, but we all know averages only work on paper. For example, AWS has tutorials that provide instructions on how to setup TLS in a LAMP stack running on Linux 2 EC2s. As part of the Let's Encrypt setup, they provide a crontab entry that runs twice a day with a copy option to paste. How many EC2s all hit the Let's Encrypt server at that exact time? Since EC2s default to UTC time, that means that servers are not offsetting those requests by timezones, so that means an even bigger spike.

Re: The database servers powering Let's Encrypt

#136

Post by jeffbee »

mike_d wrote: post #130
jeffbee wrote: post #71

Earlier quoted context omitted.

Any MySQL with more than about 100 concurrent queries of the same InnoDB table is going to be CPU bound on locks. Their whole locking scheme doesn't scale; it's designed to look great in benchmarks with few clients.

It sounds like you just hit the threshold where defaults don't cut it anymore. With >100 concurrent clients you need to tune your DB for your workload. innodb_thread_concurrency and innodb_concurrency_tickets would be a good starting point, and optimal values depend on your r/w balance and number of rows touched per type of query.

I'm saying the innodb buffer pool mutex doesn't scale over 100 contenders and you are saying that I can tune mysql so there's never more than that, which it seems to me like we're in agreement.

Re: The database servers powering Let's Encrypt

#137

Post by zerd »

1MachineElf wrote: post #110

I'm thankful for their OpenZFS tuning doc which they developed as part of this server migration: https://github.com/letsencrypt/openzfs-nvme-databases The one thing that I get hung up on when it comes to RAID and SSDs is the wear pattern vs. HDDs. Take for example this quote from the README.md: We use RAID-1+0, in order to achieve the best possible performance without being vulnerable to a single-drive failure. Failu…

> The reason is that the mirrors are expected to fail after the same amount of TBW

You could fix that by writing a bit more to one of the disks, e.g. run badblocks for different amounts of time before putting them in service.

Re: The database servers powering Let's Encrypt

#138

Post by Rafuino »

rkagerer wrote: post #123

What form factor are those NVMe drives and how are the connected? I see cables, so I'm assuming they're not all plugged straight into their own PCIe slot. Are there a bunch of M.2 headers on the motherboard?

Almost definitely U.2 drives that slot into the front of the server. Here's a cartoony view of what that looks like (on a P5800X Optane SSD): https://www.servethehome.com/new-intel-optane-p5800x-100-dwp...

Re: The database servers powering Let's Encrypt

#139

Post by eppp »

ggregoire wrote: post #131
tyingq wrote: post #25

I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me. 64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU. If you price it out on Dell's site, you get a r…

We made so much progress in that field over the last 40 years... I sometimes get lost trying to imagine what will be the computing performance available in 100, 1000, 10000 years from now...

Single threaded performance isn't keeping pace unfortunately.

Re: The database servers powering Let's Encrypt

#140

Post by tanelpoder »

PeterCorless wrote: post #101
gpsar wrote: post #85

Earlier quoted context omitted.

If the problem involves independent traversals, interleaving with coroutines is a practical way to hide latency https://dl.acm.org/doi/10.1145/3329785.3329917 https://www.linkedin.com/pulse/dont-stall-multitask-georgios...

We've made Scylla as async, shared nothing as possible, and we've also started adding C++20 coroutines (to replace futures/promises). We'll be doing more of that in 2021. https://www.scylladb.com/2021/01/12/making-scylla-a-monstrou...

I plan to test Scylla (and Postgres with TimescaleDB) out soon after done with basic Linux tests :)
Post reply on HN