The database servers powering Let's Encrypt • Hacker News Board

Re: The database servers powering Let's Encrypt

#51

Post by tanelpoder » Thu, Jan 21, 2021, 8:01 PM UTC

tanelpoder wrote: post #36

riku_iki wrote: post #29

Earlier quoted context omitted.

> 16 of them (with 4 x quad SSD adapter cards) in a 2-socket EPYC machine can do over 100 GB/s. It is more interesting if actual CPU can handle such traffic in context of DB load: encode/decide records, sort, search, merge etc.

Yes, with modern storage, throughput is a CPU problem. And CPU problem for OLTP databases is largely a memory access latency problem. For columnar analytics & complex calculations it's more about CPU itself. When doing 1 MB sized I/Os for scanning, my 16c/32t (AMD Ryzen Threadripper Pro WX) CPUs were just about 10% busy. So, with a 64 core single socket ThreadRipper workstation (or 128-core dual socket EPYC server),…

As I mentioned memory access latency - I just posted my old article series about measuring RAM access performance (using different database workloads) to HN and looks like it even made it to the front page (nice):

https://news.ycombinator.com/item?id=25863093

Re: The database servers powering Let's Encrypt

#52

Post by Neil44 » Thu, Jan 21, 2021, 8:03 PM UTC

sgt wrote: post #3

I'm guessing someone out there's thinking: Why aren't they hosting in the cloud? The cloud being either Amazon or Azure. Surely nothing else exists. Is it really possible to host your own PHYSICAL machine? Does that count as the cloud?!

Not sure if you're being sarcastic but couple of mil a year on AWS I rekon for anything similar, vs a one off 200k, not a bad saving.

Re: The database servers powering Let's Encrypt

#53

Post by jandrewrogers » Thu, Jan 21, 2021, 8:05 PM UTC

hinkley wrote: post #22

stu2010 wrote: post #8

Earlier quoted context omitted.

Relational databases enable some very flexible data access patterns. Once you shard, you lose a lot of that flexibility. If you move away from a relational model, you lose even more flexibility and start having to do much more work in your application layer, and usually start having to use more resources and developer time every step of the way. The productivity enabled by having one master RDBMS is a big deal, and i…

If I had a billion dollars, I'd put a research group together to study the prospects of index sharding. That is, full table replication, but individual servers maintaining differing sets of indexes. OLAP and single request transactions could be routed to specialized replicas based on query planning, sending requests to machines that have appropriate indexes, and preferably ones where those indexes are hot.

This has been done in popular commercial databases for decades, and is thoroughly researched. As far as I know, these types of architectures are no longer used at this point due to their relatively poor scalability and write performance. I don't think anyone is designing new databases this way anymore, since it only ever made sense in the context of spinning disk.

The trend has been away from complex specialization of data structures, secondary indexing, etc and toward more general and expressive internal structures (but with more difficult theory and implementation) that can efficiently handle a wider range of data models and workloads. Designers started moving on from btrees and hash tables quite a while ago, mostly for the write performance.

Write performance is critical even for read-only analytical systems due to the size of modern data models. The initial data loading can literally take several months with many popular systems, even for data models that are not particularly large. Loading and indexing 100k records per second is a problem if you have 10T records.

Re: The database servers powering Let's Encrypt

#54

Post by _joel » Thu, Jan 21, 2021, 8:09 PM UTC

jaas wrote: post #37

fabian2k wrote: post #34

I'm curious why they didn't go with the larger 64 core Epyc. I mean it's double the cost, but I suspect that the huge amount of NVMe SSDs is by far the largest part of the cost anyway. And it seems like CPU was the previous bottleneck as it was at 90%.

We didn't go with the 64-core chips because they have significantly lower clock speeds. Dual 32-core chips give us plenty of cores while keeping clocks higher for single-threaded performance. You are correct that the price of the CPUs is almost irrelevant to the overall cost of a system with this much memory and storage. We were picking the ideal CPU, not selecting on CPU price.

ooi how is the NUMA on that setup? Does it still use QPI or is there a newer technology now (I've been out of this space for a few years now)

Re: The database servers powering Let's Encrypt

#55

Post by kirugan » Thu, Jan 21, 2021, 8:12 PM UTC

jeffbee wrote: post #32

kirugan wrote: post #6

Earlier quoted context omitted.

MyISAM days are gone, no one will seriously consider it as suitable engine in MySQL.

It is quite likely unless you've meticulously avoided it that MySQL is using ISAM on-disk temp tables in the service of your queries.

I didn't know that, thanks!

Re: The database servers powering Let's Encrypt

#56

Post by jdsully » Thu, Jan 21, 2021, 8:16 PM UTC

tanelpoder wrote: post #15

multifascia wrote: post #4

As someone unfamiliar with db management, is it really less operational overhead to have to physically scale your hardware than using a distributed option with more elastic scalability capabilities?

Also worth noting that scalability != efficiency. With enough NVMe drives, a single server can do millions of IOPS and scan data at over 100 GB/s. A single PCIe 4.0 x4 SSD on my machine can do large I/Os at 6.8 GB/s rate, so 16 of them (with 4 x quad SSD adapter cards) in a 2-socket EPYC machine can do over 100 GB/s. You may need clusters, duplicated systems, replication, etc for resiliency reasons of course, but a s…

It can burst to millions of IOPS, but you get killed on the sustained write workload. Even a high end enterprise NVMe drive will be limited to around 60k IOPs once you exceed its write cache.

Re: The database servers powering Let's Encrypt

#57

Post by gbrown_ » Thu, Jan 21, 2021, 8:17 PM UTC

dewey wrote: post #7

ryanworl wrote: post #5

What are they storing on this server that requires 150Tb of storage and millions of IOPS?

> What exactly are we doing with these servers? Our CA software, Boulder, uses MySQL-style schemas and queries to manage subscriber accounts and the entire certificate issuance process.

The post doesn't specify requirements or application level targets for performance. They show a couple of good latency improvements but don't describe the business or technical impact. The closest we get is this.

> If this database isn’t performing well enough, it can cause API errors and timeouts for our subscribers.

What are the SLO's? How was this being met (or not) before vs after the hardware upgrade? There's a lot of additional context that could have been added in this post. It's not a bad post but instead it simply reduces down to this new hardware is faster than our old hardware.

Re: The database servers powering Let's Encrypt

#58

Post by rantwasp » Thu, Jan 21, 2021, 8:21 PM UTC

gets huge server. does not properly resize the jpeg on the page (it’s 5megs in size and i see it loading). we don’t al have 5TB of ram you know

Re: The database servers powering Let's Encrypt

#59

Post by mike_d » Thu, Jan 21, 2021, 8:29 PM UTC

tyingq wrote: post #50

cperciva wrote: post #26

Earlier quoted context omitted.

Based on their stated 225M sites and a renewal period of 90 days, they're probably averaging around 40 certificates per second. That's only an order of magnitude higher than bitcoin; I wouldn't call it an indication of an ability to scale to a particularly large amount of traffic.

Does "certbot renew" talk to the mothership at all if no certs are ready for renewal? If it does, most setups I've seen run the renewal once or twice a day since it only does the renew when you're down to 30 days left. There may also be some OCSP related traffic.

Certbot will look at the expiration timestamp on your local certs without talking to Lets Encrypt.

Re: The database servers powering Let's Encrypt

#60

Post by mike_d » Thu, Jan 21, 2021, 8:32 PM UTC

sradman wrote: post #41

> We can clearly see how our old CPUs were reaching their limit. In the week before we upgraded our primary database server, its CPU usage (from /proc/stat) averaged over 90% This strikes me as odd. In my experience, traditional OLTP row stores are I/O bound due to contention (locking and latching). Does anyone have an explanation for this? > Once you have a server full of NVMe drives, you have to decide how to manag…

> traditional OLTP row stores are I/O bound due to contention (locking and latching). Does anyone have an explanation for this?

I have seen CPU bound database servers when developers push application logic in to the database. Everything from using server-side functions like MD5() to needless triggers and stored procedures that could have been done application side.