Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The database servers powering Let's Encrypt

letsencrypt.org

Post reply on HN
Print view View original

151–160 of 236 posts

Re: The database servers powering Let's Encrypt

#151

Post by jeffbee »

sradman wrote: post #149
jeffbee wrote: post #136

Earlier quoted context omitted.

I'm saying the innodb buffer pool mutex doesn't scale over 100 contenders and you are saying that I can tune mysql so there's never more than that, which it seems to me like we're in agreement.

I did a quick search and it looks like InnoDB implements a spin-lock. Do you see increased CPU utilization when the buffer pool is overloaded? This could explain the behavior described in the article.

Correct, it uses an aggressive spinlock without fairness features for waiters like you'd see with a proper production-quality mutex. This makes it very efficient in the absence of contention (i.e. in trivial benchmarks) and extremely poorly behaved in times of contention.

Re: The database servers powering Let's Encrypt

#153

Post by paxys »

jaas wrote: post #104
tyingq wrote: post #25

I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me. 64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU. If you price it out on Dell's site, you get a r…

Just so everyone here is aware re: pricing... If you buy through a VAR and/or Dell reps you don't pay the price on the website. What you actually pay is typically significantly lower. I don't think anyone actually buys servers like these by just ordering from the website. We (Let's Encrypt) certainly don't. These are expensive servers, crossing into six digits, but not $200k.

Literally the only reason to display a retail price on the website is so sales reps can say "see we are giving you a massive XX% discount!"

Re: The database servers powering Let's Encrypt

#154

Post by dastbe »

PeterCorless wrote: post #97
dastbe wrote: post #76

Earlier quoted context omitted.

(I work at AWS, but this is just for fun) Checking out AWS side, the closest I think you'd get is the x1.32xlarge, which would translate to 128 vCPU (which on intel generally means 64 physical cores) and close to 2TB of RAM. nvme storage is only a paltry 4TB, so you'd have to make up the rest with EBS volumes. You'd also get a lower clock speed than they are getting out of the EPICs

spittakes reading the suggestion of replacing NVMe with EBS I mean, yeah, I guess you can . But a lot depends on your use case and SLA. If you need to keep ultra-low p99s — single digits — then EBS is not a real option. But if you don't mind latencies, then yeah, fine. Don't get me wrong: EBS is great. But it's not a panacea and strikes me as a mismatch for a high performance monster system. If you need NVMe, you nee…

It wasn't a suggestion of replacement, it was saying that even the closest instance still doesn't get you 24TB of on-host storage, having to use EBS instead.

Re: The database servers powering Let's Encrypt

#155

Post by segmondy »

tyingq wrote: post #25

I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me. 64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU. If you price it out on Dell's site, you get a r…

I'm surprised that's what did it for me. I admined some sun enterprise servers and was blown away when I bought a pi 3 yrs and realized the pi was probably faster than it. That enterprise server would have 100-200 engineering students logged in and working at once. :-/. We're living in the future.

Re: The database servers powering Let's Encrypt

#156

Post by jart »

walrus01 wrote: post #105
tyingq wrote: post #25

I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me. 64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU. If you price it out on Dell's site, you get a r…

One of the interesting things about threadripper and epyc systems with a lot of PCI-E 4.0 lanes on one motherboard, if your goal is not NVME disk I/O, but rather network throughput, it also provides the opportunity for a DIY approach to very high capacity multi-100GbE ethernet interface routers. Such as a fairly low cost 4U threadripper box, running VyOS (ultimately based on a very normal debian Linux) with multiple…

Altavista used to run their entire search engine on a single computer twenty years ago. Imagine what Mike Burrows could accomplish today with one of these 2U EPYC NVME 200gbps PCs. It'd probably trigger the singularity.

Re: The database servers powering Let's Encrypt

#157

Post by hedora »

1MachineElf wrote: post #110

I'm thankful for their OpenZFS tuning doc which they developed as part of this server migration: https://github.com/letsencrypt/openzfs-nvme-databases The one thing that I get hung up on when it comes to RAID and SSDs is the wear pattern vs. HDDs. Take for example this quote from the README.md: We use RAID-1+0, in order to achieve the best possible performance without being vulnerable to a single-drive failure. Failu…

SSD’s still fail, just not often.

State of the art systems keep ~1.2 copies (e.g. 10+2 raid 6) on SSD, and an offsite backup or two. The bandwidth required for timely rebuilds is usually the bottleneck.

These systems can be ridiculously dense; a few petabytes easily fits in 10U. With that many NAND packages, drive failures are common.

Re: The database servers powering Let's Encrypt

#159

Post by jart »

mark242 wrote: post #122

Stopping by to say, 9ms API response time is just ridiculously quick. You're starting to run into the laws of physics and client proximity to the datacenter where those machines live. That's a pretty amazing feat. I would assume the next step for scaling is getting those read replicas deployed across the world in order to cut down on RTT.

9ms 50%ile node latency is good, but that number is normally 1ms for big internet services. See https://dl.acm.org/cms/attachment/3658918e-7081-4676-beec-aa... Mission critical stuff goes even faster like BigTable which has numbers 4x better than the figure.
Post reply on HN