Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The database servers powering Let's Encrypt

letsencrypt.org

Post reply on HN
Print view View original

101–110 of 236 posts

Re: The database servers powering Let's Encrypt

#101

Post by PeterCorless »

gpsar wrote: post #85
tanelpoder wrote: post #51

Earlier quoted context omitted.

As I mentioned memory access latency - I just posted my old article series about measuring RAM access performance (using different database workloads) to HN and looks like it even made it to the front page (nice): https://news.ycombinator.com/item?id=25863093

If the problem involves independent traversals, interleaving with coroutines is a practical way to hide latency https://dl.acm.org/doi/10.1145/3329785.3329917 https://www.linkedin.com/pulse/dont-stall-multitask-georgios...

We've made Scylla as async, shared nothing as possible, and we've also started adding C++20 coroutines (to replace futures/promises). We'll be doing more of that in 2021.

https://www.scylladb.com/2021/01/12/making-scylla-a-monstrou...

Re: The database servers powering Let's Encrypt

#102

Post by amluto »

tyingq wrote: post #25

I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me. 64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU. If you price it out on Dell's site, you get a r…

Dell makes it easier than it should be by having absurd markups on storage.

Re: The database servers powering Let's Encrypt

#103

Post by [deleted] »

jagger27 wrote: post #98
wpietri wrote: post #93

Earlier quoted context omitted.

Totally. 2 TB of RAM! In one box! I think the first servers I had in production had 8 MB RAM. No more, certainly. Soon we'll be at 1000x that. My dad's first "server" was 3 orders of magnitude smaller, with 8 KB of RAM (hand-wound wire core memory). In that time, the US population hasn't even doubled.

More like 1,000,000x

[deleted]

Re: The database servers powering Let's Encrypt

#104

Post by jaas »

tyingq wrote: post #25

I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me. 64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU. If you price it out on Dell's site, you get a r…

Just so everyone here is aware re: pricing...

If you buy through a VAR and/or Dell reps you don't pay the price on the website. What you actually pay is typically significantly lower. I don't think anyone actually buys servers like these by just ordering from the website. We (Let's Encrypt) certainly don't.

These are expensive servers, crossing into six digits, but not $200k.

Re: The database servers powering Let's Encrypt

#105

Post by walrus01 »

tyingq wrote: post #25

I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me. 64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU. If you price it out on Dell's site, you get a r…

One of the interesting things about threadripper and epyc systems with a lot of PCI-E 4.0 lanes on one motherboard, if your goal is not NVME disk I/O, but rather network throughput, it also provides the opportunity for a DIY approach to very high capacity multi-100GbE ethernet interface routers.

Such as a fairly low cost 4U threadripper box, running VyOS (ultimately based on a very normal debian Linux) with multiple Intel 100GbE NICs in PCI-E slots.

The Intel linux kernel (and FreeBSD) driver support for those is generally excellent and robust. Intel has had full time developers doing the Linux drivers for that series of cards since the earliest days of their 64-bit/66MHz PCI-X 10Gbps NICs about 17 years ago.

Also worth mentioning that FRR is now an official Linux Foundation supported project.

https://frrouting.org/

https://www.intel.com/content/www/us/en/products/docs/networ...

Re: The database servers powering Let's Encrypt

#106

Post by flatiron »

mike_d wrote: post #59
tyingq wrote: post #50

Earlier quoted context omitted.

Does "certbot renew" talk to the mothership at all if no certs are ready for renewal? If it does, most setups I've seen run the renewal once or twice a day since it only does the renew when you're down to 30 days left. There may also be some OCSP related traffic.

Certbot will look at the expiration timestamp on your local certs without talking to Lets Encrypt.

There is a —force though. I had to do it once because they thought I got a bad cert (don’t remember the details)

Re: The database servers powering Let's Encrypt

#107

Post by edoceo »

PeterCorless wrote: post #95
cperciva wrote: post #26

Earlier quoted context omitted.

Based on their stated 225M sites and a renewal period of 90 days, they're probably averaging around 40 certificates per second. That's only an order of magnitude higher than bitcoin; I wouldn't call it an indication of an ability to scale to a particularly large amount of traffic.

Yes. They are not doing a very heavy computational workload. Typical heavy-duty servers these days can do 100k's or millions of TPS. 40 TPS is a really, really, really light load. Further, I was looking at those new server specs. There's an error I think? The server config on the Dell site shows 2x 8 GB DRIMMs, for 16 GB RAM per sever, whereas the article says 2 TB! With only 16GB of RAM, but 153.6 TB of NVMe storage…

They don't only GET 40 TPS, that value is an estimate what they serve (above it was site/90 days).

They have capacity for much, much more with that hardware

Re: The database servers powering Let's Encrypt

#108

Post by [deleted] »

foota wrote: post #73
vmception wrote: post #48

Earlier quoted context omitted.

I have a motherboard from 2012 and I just put 2x 8TB NVMe SSDs on it, on a PCIe 2.0 x16 slot Works great. The PCIe card itself has 2 more slots for SSDs The GPU is on the 2.0 x8 slot because they don't really transfer that much data over the lanes. I honestly didn't realize PCIe was up to 4.0 now, and I am pushing up against the limits of PCIe 2.0 but it still works! And I’m “only” at the limits, and its only a limit…

What drives did you get? I think you need PCI 4 to stress most SSDs these days?

[deleted]

Re: The database servers powering Let's Encrypt

#109

Post by tyingq »

jaas wrote: post #104
tyingq wrote: post #25

I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me. 64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU. If you price it out on Dell's site, you get a r…

Just so everyone here is aware re: pricing... If you buy through a VAR and/or Dell reps you don't pay the price on the website. What you actually pay is typically significantly lower. I don't think anyone actually buys servers like these by just ordering from the website. We (Let's Encrypt) certainly don't. These are expensive servers, crossing into six digits, but not $200k.

Yes, that's worth noting. The $200k is the list/retail, which nobody would pay for a purchase of this size.

Re: The database servers powering Let's Encrypt

#110

Post by 1MachineElf »

I'm thankful for their OpenZFS tuning doc which they developed as part of this server migration: https://github.com/letsencrypt/openzfs-nvme-databases

The one thing that I get hung up on when it comes to RAID and SSDs is the wear pattern vs. HDDs. Take for example this quote from the README.md:

We use RAID-1+0, in order to achieve the best possible performance without being vulnerable to a single-drive failure.

Failure on SSDs is predictable and usually expressed with Terabytes Written (TBW). Failure on spinning disk HDDs is comparatively random. In my mind, it makes sense to mirror SSD-based vdevs only for performance reasons and not for data integrity. The reason is that the mirrors are expected to fail after the same amount of TBW, and thus the availability/redundancy guarantee of mirroring is relatively unreliable.

Maybe someone with more experience in this area can change my mind, but if it were up to me, I would have configured the mirror drives as spares, and relied on a local HDD-based zpool for quick backup/restore capability. I imagine that would be a better solution, although it probably wouldn't have fit into tryingq's ideal 2U space.

Post reply on HN