The database servers powering Let's Encrypt • Hacker News Board

Re: The database servers powering Let's Encrypt

#71

Post by jeffbee » Thu, Jan 21, 2021, 9:20 PM UTC

mike_d wrote: post #60

sradman wrote: post #41

> We can clearly see how our old CPUs were reaching their limit. In the week before we upgraded our primary database server, its CPU usage (from /proc/stat) averaged over 90% This strikes me as odd. In my experience, traditional OLTP row stores are I/O bound due to contention (locking and latching). Does anyone have an explanation for this? > Once you have a server full of NVMe drives, you have to decide how to manag…

> traditional OLTP row stores are I/O bound due to contention (locking and latching). Does anyone have an explanation for this? I have seen CPU bound database servers when developers push application logic in to the database. Everything from using server-side functions like MD5() to needless triggers and stored procedures that could have been done application side.

Any MySQL with more than about 100 concurrent queries of the same InnoDB table is going to be CPU bound on locks. Their whole locking scheme doesn't scale; it's designed to look great in benchmarks with few clients.

Re: The database servers powering Let's Encrypt

#72

Post by gautamcgoel » Thu, Jan 21, 2021, 9:24 PM UTC

ed25519FUUU wrote: post #20

What a great read. I think the authors here made great hardware and software decisions. OpenZFS is the way to go, and is so much easier to manage than the legacy RAID controllers imho. Ah, I miss actual hardware.

Can you explain the advantages of OpenZFS over other filesystems? I know FreeBSD uses ZFS, but I never really understood how it stacks up relative to other technologies...

Re: The database servers powering Let's Encrypt

#73

Post by foota » Thu, Jan 21, 2021, 9:25 PM UTC

vmception wrote: post #48

tyingq wrote: post #25

I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me. 64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU. If you price it out on Dell's site, you get a r…

I have a motherboard from 2012 and I just put 2x 8TB NVMe SSDs on it, on a PCIe 2.0 x16 slot Works great. The PCIe card itself has 2 more slots for SSDs The GPU is on the 2.0 x8 slot because they don't really transfer that much data over the lanes. I honestly didn't realize PCIe was up to 4.0 now, and I am pushing up against the limits of PCIe 2.0 but it still works! And I’m “only” at the limits, and its only a limit…

What drives did you get? I think you need PCI 4 to stress most SSDs these days?

Re: The database servers powering Let's Encrypt

#74

Post by stefan_ » Thu, Jan 21, 2021, 9:27 PM UTC

Conclusionist wrote: post #65

stefan_ wrote: post #64

Earlier quoted context omitted.

What exactly needs to be stored once the certificate is created and published in the hash tree? It seems like the kind of data that possibly needn't be stored at all or onto something like Glacier for archival.

Going to guess it's for OCSP responses.

I'm not sure, e.g. Chrome doesn't do OCSP by default, lots of embedded clients like curl won't either. Unless the protocol is terribly broken, that also seems like the kind of use case where 99% of queries just come out of cache and should never hit a database.

Re: The database servers powering Let's Encrypt

#75

Post by iruoy » Thu, Jan 21, 2021, 9:28 PM UTC

Neil44 wrote: post #52

sgt wrote: post #3

I'm guessing someone out there's thinking: Why aren't they hosting in the cloud? The cloud being either Amazon or Azure. Surely nothing else exists. Is it really possible to host your own PHYSICAL machine? Does that count as the cloud?!

Not sure if you're being sarcastic but couple of mil a year on AWS I rekon for anything similar, vs a one off 200k, not a bad saving.

And a sizable internet bill I'd assume. This puppy ain't running on Gigabit.

Re: The database servers powering Let's Encrypt

#76

Post by dastbe » Thu, Jan 21, 2021, 9:30 PM UTC

ed25519FUUU wrote: post #33

jbverschoor wrote: post #23

Earlier quoted context omitted.

It’s doesn’t have to. Unless you’re conditioned to believe that aws is cheap

I'm curious what the cost would be to run this type of hardware at any cloud vendor? Does it even exist?

(I work at AWS, but this is just for fun)

Checking out AWS side, the closest I think you'd get is the x1.32xlarge, which would translate to 128 vCPU (which on intel generally means 64 physical cores) and close to 2TB of RAM. nvme storage is only a paltry 4TB, so you'd have to make up the rest with EBS volumes. You'd also get a lower clock speed than they are getting out of the EPICs

Re: The database servers powering Let's Encrypt

#77

Post by nine_k » Thu, Jan 21, 2021, 9:32 PM UTC

jarym wrote: post #18

Just goes to show how much a single SQL server can scale before having to worry about sharing and horizontal scaling

Read performance is much easier to scale (in one box or several) than write performance. It's usually the writes that make you look at Cassandra and similar, instead of adding more disks and RAM, or spinning another read-only replica.

24 NVMEs should have a lot of write throughput, though.

Re: The database servers powering Let's Encrypt

#78

Post by jasonwatkinspdx » Thu, Jan 21, 2021, 9:39 PM UTC

stu2010 wrote: post #8

multifascia wrote: post #4

As someone unfamiliar with db management, is it really less operational overhead to have to physically scale your hardware than using a distributed option with more elastic scalability capabilities?

Relational databases enable some very flexible data access patterns. Once you shard, you lose a lot of that flexibility. If you move away from a relational model, you lose even more flexibility and start having to do much more work in your application layer, and usually start having to use more resources and developer time every step of the way. The productivity enabled by having one master RDBMS is a big deal, and i…

I agree this is an under appreciated strategy. Someone in my family worked for a hedge fund where one of their simple advantages was they just ran MS SQL on the biggest physical machine available at any given moment. Lots of complexity dodged by just having a lot of brute capacity.

Re: The database servers powering Let's Encrypt

#79

Post by uncledave » Thu, Jan 21, 2021, 9:44 PM UTC

I’d like to understand more about the workload here. Queries per second, average result size, database size, query complexity etc.

Re: The database servers powering Let's Encrypt

#80

Post by dingaling » Thu, Jan 21, 2021, 9:50 PM UTC

barkingcat wrote: post #9

sgt wrote: post #3

I'm guessing someone out there's thinking: Why aren't they hosting in the cloud? The cloud being either Amazon or Azure. Surely nothing else exists. Is it really possible to host your own PHYSICAL machine? Does that count as the cloud?!

For a service like letsencrypt, the independence factor is also a major reason for self hosting. I can forsee letsencrypt in the future going to building their own cloud (on their own physical infrastructure), but speaking as a letsencrypt user of their free certificate program, I would lose respect and interest in their service if they went with an AWS or GCP or Azure approach. The independence from other major play…

Funny you mention AWS as they're one of the corporate sponsors of LE.

So long as they don't have a viable independent revenue stream they're arguably less independent than commercial CAs.