Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The database servers powering Let's Encrypt

letsencrypt.org

Post reply on HN
Print view View original

181–190 of 236 posts

Re: The database servers powering Let's Encrypt

#181

Post by robk »

jaas wrote: post #104
tyingq wrote: post #25

I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me. 64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU. If you price it out on Dell's site, you get a r…

Just so everyone here is aware re: pricing... If you buy through a VAR and/or Dell reps you don't pay the price on the website. What you actually pay is typically significantly lower. I don't think anyone actually buys servers like these by just ordering from the website. We (Let's Encrypt) certainly don't. These are expensive servers, crossing into six digits, but not $200k.

There's a good thread on the sysadmin reddit where folks share the actual prices they're paying

Re: The database servers powering Let's Encrypt

#182

Post by baby »

thamer wrote: post #91
baby wrote: post #31

So scaling up instead of scaling out. I’m not sure if it’s a viable strategy long term, at the same time we probably don’t want a single CA to handle too many certificates?

Scaling up means each query is faster (3x in this particular case). Scaling out means they can support more clients/domains (more DB shards, more web servers, more concurrency, etc). These are two distinct axes that are not incompatible with each other.

Not always true though, scaling out can make your queries faster by alleviating the load

Re: The database servers powering Let's Encrypt

#183

Post by PeterCorless »

sradman wrote: post #133
PeterCorless wrote: post #99

Earlier quoted context omitted.

> This strikes me as odd. In my experience, traditional OLTP row stores are I/O bound due to contention (locking and latching). Does anyone have an explanation for this? Yes. My CTO, Avi Kivity did a great talk about this at Core C++ 2019: https://www.scylladb.com/2020/03/26/avi-kivity-at-core-c-201... Let me boil it down to a few points; some beyond Avi's talk: • Traditional RDBMS with strong consistency and ACID gu…

> But the main point is that you have to really think about how to re-architect your software to take advantage of huge multi-processor machines. I appreciate the response but it doesn't address my question: given that Let's Encrypt's MySQL-family RDBMS does not implement any of the multi-core/multi-socket/cpu-affinity/lock-free/asyncIO techniques used by databases like ScyllaDB, MemSQL, and VoltDB, why were they see…

I kind of did answer a different question, but you sort of affirmed my answer & proved my point. If the "expected range" of your CPU utilization is only 25%, then you are knowingly paying for 3x more CPU than you are actually every planning to use. I suppose if you got that cash money and are willing to burn it, nothing I can say will stop you. I just question the logic of it.

As for why they suddenly dropped? I'll leave that to someone who knows this particular system far better than I do.

Re: The database servers powering Let's Encrypt

#184

Post by PeterCorless »

dastbe wrote: post #154
PeterCorless wrote: post #97

Earlier quoted context omitted.

spittakes reading the suggestion of replacing NVMe with EBS I mean, yeah, I guess you can . But a lot depends on your use case and SLA. If you need to keep ultra-low p99s — single digits — then EBS is not a real option. But if you don't mind latencies, then yeah, fine. Don't get me wrong: EBS is great. But it's not a panacea and strikes me as a mismatch for a high performance monster system. If you need NVMe, you nee…

It wasn't a suggestion of replacement, it was saying that even the closest instance still doesn't get you 24TB of on-host storage, having to use EBS instead.

This starts going down a rabbit hole of tiered storage then. Because unless you design for that, you can only be as fast as your slowest storage. It'd be like tying a 20 kg weight to the leg of a sprinter otherwise.

Re: The database servers powering Let's Encrypt

#186

Post by zinekeller »

ksec wrote: post #115

>Intel® SSD DC P4610 Interesting they decide to put in PCIE 3.0 NVMe SSD instead of PCI-E 4? Imagine having 24x Intel Optane [1]. PCI-E 5 is actually just around the corner. I would imagine next time Let's Encrypt could upgrade again and continue to use a Single DB Machine to Serve. [1] https://www.servethehome.com/new-intel-optane-p5800x-100-dwp...

Dirty secret when it comes to current PCIe 4 drives: they have optimised sequential RW speed too much that they have forgotten random RW speed (which is the main driver in databases).

Re: The database servers powering Let's Encrypt

#187

Post by zinekeller »

ddorian43 wrote: post #27
hinkley wrote: post #22

Earlier quoted context omitted.

If I had a billion dollars, I'd put a research group together to study the prospects of index sharding. That is, full table replication, but individual servers maintaining differing sets of indexes. OLAP and single request transactions could be routed to specialized replicas based on query planning, sending requests to machines that have appropriate indexes, and preferably ones where those indexes are hot.

The problem is the network. You need billion dollars to fix the network so it's as fast as local ram/nvme.

... and even if you somehow solved that, the law of physics hits you hard. Latency can be a real performance killer generally and is doubly true in database-type computing.

Re: The database servers powering Let's Encrypt

#188

Post by FlyingSnake »

jaas wrote: post #104
tyingq wrote: post #25

I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me. 64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU. If you price it out on Dell's site, you get a r…

Just so everyone here is aware re: pricing... If you buy through a VAR and/or Dell reps you don't pay the price on the website. What you actually pay is typically significantly lower. I don't think anyone actually buys servers like these by just ordering from the website. We (Let's Encrypt) certainly don't. These are expensive servers, crossing into six digits, but not $200k.

This is Anchoring, a cognitive bias where an individual depends too heavily on an initial piece of information offered to make subsequent judgments during decision making.

Re: The database servers powering Let's Encrypt

#189

Post by FlyingSnake »

robk wrote: post #181
jaas wrote: post #104

Earlier quoted context omitted.

Just so everyone here is aware re: pricing... If you buy through a VAR and/or Dell reps you don't pay the price on the website. What you actually pay is typically significantly lower. I don't think anyone actually buys servers like these by just ordering from the website. We (Let's Encrypt) certainly don't. These are expensive servers, crossing into six digits, but not $200k.

There's a good thread on the sysadmin reddit where folks share the actual prices they're paying

Link for the curious?
Post reply on HN