The database servers powering Let's Encrypt • Hacker News Board

Re: The database servers powering Let's Encrypt

#121

Post by rkagerer » Fri, Jan 22, 2021, 12:42 AM UTC

I'm more interested in how they used ZFS to provide redundancy. I always thought ZFS was optimized for spinning platters with SSD's used for persistent caching. In this scenario they used it to set up all their SSD's in mirrored pairs then stripe across that. No ZIL.

They've tweaked a few other settings as well [1].

I'd be curious to see more benchmarks and latency data (especially as they're utilizing compression, and of course checksums are computed over all data not just metadata like some other filesystems).

[1] https://github.com/letsencrypt/openzfs-nvme-databases

Re: The database servers powering Let's Encrypt

#122

Post by mark242 » Fri, Jan 22, 2021, 12:43 AM UTC

Stopping by to say, 9ms API response time is just ridiculously quick. You're starting to run into the laws of physics and client proximity to the datacenter where those machines live. That's a pretty amazing feat. I would assume the next step for scaling is getting those read replicas deployed across the world in order to cut down on RTT.

Re: The database servers powering Let's Encrypt

#123

Post by rkagerer » Fri, Jan 22, 2021, 12:48 AM UTC

What form factor are those NVMe drives and how are the connected? I see cables, so I'm assuming they're not all plugged straight into their own PCIe slot. Are there a bunch of M.2 headers on the motherboard?

Re: The database servers powering Let's Encrypt

#124

Post by nicoburns » Fri, Jan 22, 2021, 12:49 AM UTC

dingaling wrote: post #80

barkingcat wrote: post #9

Earlier quoted context omitted.

For a service like letsencrypt, the independence factor is also a major reason for self hosting. I can forsee letsencrypt in the future going to building their own cloud (on their own physical infrastructure), but speaking as a letsencrypt user of their free certificate program, I would lose respect and interest in their service if they went with an AWS or GCP or Azure approach. The independence from other major play…

Funny you mention AWS as they're one of the corporate sponsors of LE. So long as they don't have a viable independent revenue stream they're arguably less independent than commercial CAs.

"one of" being the key point here. Let's encrypt has a huge number of sponsors (AWS being only 1 of 9 even if you only count the "platinum level" sponsors), which should allow them to maintain their independence.

https://letsencrypt.org/sponsors/

Re: The database servers powering Let's Encrypt

#125

Post by shitloadofbooks » Fri, Jan 22, 2021, 12:50 AM UTC

mark242 wrote: post #122

Stopping by to say, 9ms API response time is just ridiculously quick. You're starting to run into the laws of physics and client proximity to the datacenter where those machines live. That's a pretty amazing feat. I would assume the next step for scaling is getting those read replicas deployed across the world in order to cut down on RTT.

Why would they need to bother when it's mostly machines talking to machines? Certbot doesn't care that it took 90ms vs 9ms.

Re: The database servers powering Let's Encrypt

#126

Post by dan-robertson » Fri, Jan 22, 2021, 12:52 AM UTC

rkagerer wrote: post #121

I'm more interested in how they used ZFS to provide redundancy. I always thought ZFS was optimized for spinning platters with SSD's used for persistent caching. In this scenario they used it to set up all their SSD's in mirrored pairs then stripe across that. No ZIL. They've tweaked a few other settings as well [1]. I'd be curious to see more benchmarks and latency data (especially as they're utilizing compression, a…

ZFS was started around 2001 when SSDs weren’t really a thing. It’s goals were, amongst other things, to manage multiple volumes (providing redundancy), to be reliable (most filesystems aim for this), and to support cheap snapshotting. The last two we’re supposed to come from being copy-on-write and this model was an advantage when SSDs became popular as it worked a bit better with their semantics.

Re: The database servers powering Let's Encrypt

#127

Post by walrus01 » Fri, Jan 22, 2021, 12:52 AM UTC

tyingq wrote: post #109

jaas wrote: post #104

Earlier quoted context omitted.

Just so everyone here is aware re: pricing... If you buy through a VAR and/or Dell reps you don't pay the price on the website. What you actually pay is typically significantly lower. I don't think anyone actually buys servers like these by just ordering from the website. We (Let's Encrypt) certainly don't. These are expensive servers, crossing into six digits, but not $200k.

Yes, that's worth noting. The $200k is the list/retail, which nobody would pay for a purchase of this size.

At this size/scale I wonder what the actual price would be for a comparable Supermicro system. If I had to make a wild guess, well under half the 200k previously quoted.

Re: The database servers powering Let's Encrypt

#128

Post by walrus01 » Fri, Jan 22, 2021, 12:53 AM UTC

shitloadofbooks wrote: post #125

mark242 wrote: post #122

Stopping by to say, 9ms API response time is just ridiculously quick. You're starting to run into the laws of physics and client proximity to the datacenter where those machines live. That's a pretty amazing feat. I would assume the next step for scaling is getting those read replicas deployed across the world in order to cut down on RTT.

Why would they need to bother when it's mostly machines talking to machines? Certbot doesn't care that it took 90ms vs 9ms.

The 9ms is one indicator that the new hardware platform has ample extra capacity for future growth in load and traffic, it probably won't need to be replaced or upgraded for some years.

Re: The database servers powering Let's Encrypt

#129

Post by walrus01 » Fri, Jan 22, 2021, 12:55 AM UTC

1MachineElf wrote: post #110

I'm thankful for their OpenZFS tuning doc which they developed as part of this server migration: https://github.com/letsencrypt/openzfs-nvme-databases The one thing that I get hung up on when it comes to RAID and SSDs is the wear pattern vs. HDDs. Take for example this quote from the README.md: We use RAID-1+0, in order to achieve the best possible performance without being vulnerable to a single-drive failure. Failu…

> Failure on spinning disk HDDs is comparatively random.

comparatively, yes, but when averaged out over a large number of hard drives it definitely tends to follow a typical bathtub curve failure model seen in any mechanical product with moving parts.

https://www.itl.nist.gov/div898/handbook/apr/section1/gifs/b...

early failures will be HDDs that die within a few months of being put into service

in the middle of the curve, there will be a constant steady rate of random failures

towards the end of the lifespan of the hard drives, as they've been spinning and seeking for many years, failures will increase.

Re: The database servers powering Let's Encrypt

#130

Post by mike_d » Fri, Jan 22, 2021, 12:59 AM UTC

jeffbee wrote: post #71

mike_d wrote: post #60

Earlier quoted context omitted.

> traditional OLTP row stores are I/O bound due to contention (locking and latching). Does anyone have an explanation for this? I have seen CPU bound database servers when developers push application logic in to the database. Everything from using server-side functions like MD5() to needless triggers and stored procedures that could have been done application side.

Any MySQL with more than about 100 concurrent queries of the same InnoDB table is going to be CPU bound on locks. Their whole locking scheme doesn't scale; it's designed to look great in benchmarks with few clients.

It sounds like you just hit the threshold where defaults don't cut it anymore. With >100 concurrent clients you need to tune your DB for your workload.

innodb_thread_concurrency and innodb_concurrency_tickets would be a good starting point, and optimal values depend on your r/w balance and number of rows touched per type of query.