Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The database servers powering Let's Encrypt

letsencrypt.org

Post reply on HN
Print view View original

191–200 of 236 posts

Re: The database servers powering Let's Encrypt

#191

Post by staticassertion »

jaas wrote: post #104
tyingq wrote: post #25

I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me. 64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU. If you price it out on Dell's site, you get a r…

Just so everyone here is aware re: pricing... If you buy through a VAR and/or Dell reps you don't pay the price on the website. What you actually pay is typically significantly lower. I don't think anyone actually buys servers like these by just ordering from the website. We (Let's Encrypt) certainly don't. These are expensive servers, crossing into six digits, but not $200k.

This goes for most things. We buy Dell laptops, and the price drops like 20-40% easy.

Re: The database servers powering Let's Encrypt

#193

Post by sradman »

PeterCorless wrote: post #183
sradman wrote: post #133

Earlier quoted context omitted.

> But the main point is that you have to really think about how to re-architect your software to take advantage of huge multi-processor machines. I appreciate the response but it doesn't address my question: given that Let's Encrypt's MySQL-family RDBMS does not implement any of the multi-core/multi-socket/cpu-affinity/lock-free/asyncIO techniques used by databases like ScyllaDB, MemSQL, and VoltDB, why were they see…

I kind of did answer a different question, but you sort of affirmed my answer & proved my point. If the "expected range" of your CPU utilization is only 25% , then you are knowingly paying for 3x more CPU than you are actually every planning to use. I suppose if you got that cash money and are willing to burn it, nothing I can say will stop you. I just question the logic of it. As for why they suddenly dropped? I'll…

I think Jeffbee resolved my confusion; InnoDB uses spin locks that chew up cpu cycles when the request rate exceeds the I/O rate. This is not as bad as it sounds; it uses extra power and generates extra heat but it is not doing any extra real work. InnoDB cores run at either 25% or 90% but the 90% indicates that you need an I/O upgrade not more CPU cycles.

Optimizing for CPU efficiency in a system that is I/O bound will not save significant money. Memory and NVMe are the critical factors and with the master/slave replication used, network I/O significantly undercuts the peak performance this single server is capable of.

Re: The database servers powering Let's Encrypt

#194

Post by skylanh »

FlyingSnake wrote: post #189
robk wrote: post #181

Earlier quoted context omitted.

There's a good thread on the sysadmin reddit where folks share the actual prices they're paying

Link for the curious?

Current thread: https://www.reddit.com/r/sysadmin/comments/kxw7g5/am_i_getti...

Past threads: https://www.reddit.com/user/bad0seed/submitted/

Re: The database servers powering Let's Encrypt

#195

Post by deepstack »

paxys wrote: post #153
jaas wrote: post #104

Earlier quoted context omitted.

Just so everyone here is aware re: pricing... If you buy through a VAR and/or Dell reps you don't pay the price on the website. What you actually pay is typically significantly lower. I don't think anyone actually buys servers like these by just ordering from the website. We (Let's Encrypt) certainly don't. These are expensive servers, crossing into six digits, but not $200k.

Literally the only reason to display a retail price on the website is so sales reps can say "see we are giving you a massive XX% discount!"

Sounds about right.

Re: The database servers powering Let's Encrypt

#196

Post by kornbattery »

JohnJamesRambo wrote: post #169
ggregoire wrote: post #131

Earlier quoted context omitted.

We made so much progress in that field over the last 40 years... I sometimes get lost trying to imagine what will be the computing performance available in 100, 1000, 10000 years from now...

And I'm afraid it will still obey this trend. https://danluu.com/input-lag/

That doesen't seem like some fundamental restriction, but rather a compromise we're always willing to make. 30ms is not noticeable, so we don't try to lower it and sacrifice something else.

Even the super slow ones like on kindle, are _choices_ that have been made in favor of something else. a second to turn a page on a book isn't unbearable

Re: The database servers powering Let's Encrypt

#198

Post by tialaramex »

wahern wrote: post #180
dylan604 wrote: post #135

Earlier quoted context omitted.

They might average that, but we all know averages only work on paper. For example, AWS has tutorials that provide instructions on how to setup TLS in a LAMP stack running on Linux 2 EC2s. As part of the Let's Encrypt setup, they provide a crontab entry that runs twice a day with a copy option to paste. How many EC2s all hit the Let's Encrypt server at that exact time? Since EC2s default to UTC time, that means that s…

OpenBSD recently added the "~" random range separator to their crontab syntax. The manual for acme-client, the native Let's Encrypt tool, provides ~ * * * * acme-client example.com && rcctl reload httpd as the example crontab entry. Though, for maximum friendliness to Let's Encrypt that should probably be something like "~ ~ * * ~", which would run the command at a random time once per week. I think you could accompl…

You probably should not run it only once a week. That means it takes up to a whole week to even attempt to automatically fix a problem, which is a long time. True that's going to be four attempts before, finally, the certificate actually expires (if the problem is just intermittent service availability), but it doesn't seem worth the small additional risk.

Now, sure, in principle you should have active monitoring so you'd know immediately if there's a problem status e.g. certificate with only 14 days left until expiry; revoked, expired or otherwise bad certificate presented by server; OCSP staples missing. But we know lots of people don't have monitoring, and I guarantee at least one person reading this HN thread is mistaken and isn't monitoring everything they thought they were.

Like Certbot acme-client does a local check before taking any action, so if run once a day (or indeed once an hour) it will not normally call out to the Let's Encrypt service.

Unlike Certbot acme-client doesn't do OCSP checks, so it won't even talk to the network to get an OCSP response. Even with Certbot this (OCSP) is provided through a CDN (so it's roughly similar cost to fetching a static image for a popular web site) and so your check is negligible in terms of operating costs for Let's Encrypt.

Re: The database servers powering Let's Encrypt

#199

Post by top_sigrid »

lisper wrote: post #113

I don't understand why they are trying so hard to avoid sharding. It seems to me that this is a perfect example of an "embarassingly parallel" problem for which sharding would be borderline trivial. What am I missing?

It still works like this and they have plenty of headroom with the new solution. Chances are good that when they need to upgrade this solution, technology will have advanced far enough as well. Guessing what their performance demands are (far, far more reads than writes) this seems to work fine for them, so why make it more complicated? They do talk about read-only replicas, so they have some distribution.

Re: The database servers powering Let's Encrypt

#200

Post by midasuni »

paxys wrote: post #153
jaas wrote: post #104

Earlier quoted context omitted.

Just so everyone here is aware re: pricing... If you buy through a VAR and/or Dell reps you don't pay the price on the website. What you actually pay is typically significantly lower. I don't think anyone actually buys servers like these by just ordering from the website. We (Let's Encrypt) certainly don't. These are expensive servers, crossing into six digits, but not $200k.

Literally the only reason to display a retail price on the website is so sales reps can say "see we are giving you a massive XX% discount!"

I wonder if such “price on application” behaviours is partially what drives services like AWS

I’m looking at buying some fortigate firewalls to do some natting, looks like 200Fs will be fine, but even the fortigate sales guys won’t give me a price, I have to go to resellers who also refuse to give prices, which adds friction. Cisco exactly the same.

When looking at options, price is at the forefront of my mind, but sales guys want me to choose their company, and even commit to the specific device and numbers, before I even see the price.

Almost walked away from the fortigate option until I found avfirewalls which gave me a ballpark idea of what I could afford to implement, and what trade offs. The benefits of the fortigate over a mikrotik were worth it at that price, but it was painful getting the price out and they nearly lost the sale as I assumed it would be 10 times more.

Post reply on HN