Live data from Hacker News

The database servers powering Let's Encrypt

letsencrypt.org

221–230 of 236 posts

Re: The database servers powering Let's Encrypt

#221

Lets for the sake of argument assume that Lets Encrypt is a malicious actor. Can they easily compromise the security of the websites using their certificates?

Yes, but they better not get caught. It's a trust based model. It's actually Internet stack developers/packagers (anything from protocols in OS or libraries to browsers and devices) that are trusting Let's Encrypt among other certificate authorities.

Re: The database servers powering Let's Encrypt

#222
post #25

I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me. 64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU. If you price it out on Dell's site, you get a r…

You can build a pretty nice home 32-core/256gb threadripper system for ~ $5k. (24 nvme drives not included)

Re: The database servers powering Let's Encrypt

#223

Earlier quoted context omitted.

That doesen't seem like some fundamental restriction, but rather a compromise we're always willing to make. 30ms is not noticeable, so we don't try to lower it and sacrifice something else. Even the super slow ones like on kindle, are _choices_ that have been made in favor of something else. a second to turn a page on a book isn't unbearable

30ms latency is acceptable. But that's the best we ever had. High-end Android phones have 120+ ms latecy. That's easily noticeable and actually annoying (at least to me). My personal pet peeve is the latency of input when i'm starting new applications in KDE. E.g. I'll start a new terminal with Super+Enter, followed by a Super+Right Arrow in order to tile it to the right. But the latency is big enough, that often it'…

I think that if you had a top-to-bottom approach to a Linux distro designed entirely for latency of user interaction, you'd get something that would keep up with you.

I keep meaning to try out WindowMaker again as my Fedora window manager. I feel like it would be incredibly speedy on today's hardware.

Re: The database servers powering Let's Encrypt

#224
post #113

I don't understand why they are trying so hard to avoid sharding. It seems to me that this is a perfect example of an "embarassingly parallel" problem for which sharding would be borderline trivial. What am I missing?

It's also an example of a "mustn't fail or you break the internet" problem and a "lots of people with nation state resources have a reason to fuck with us" problem. They are prioritising simplicity as a means to security, and that makes sense to me.

Well sharding would mean that any outages affect only a small percentage of their users.

Re: The database servers powering Let's Encrypt

#225
post #25

I was, long ago, an old-school Unix sysadmin. While I was technically aware of how powerful smallish servers have become, this article really crystallized that for me. 64 cores and 24 NVME drives in a 2U spot on a rack is just insane compared to what we used to have to do to get a beefy database server. And it's not some exotic thing, just a popular mainstream Dell SKU. If you price it out on Dell's site, you get a r…

I'm surprised that's what did it for me. I admined some sun enterprise servers and was blown away when I bought a pi 3 yrs and realized the pi was probably faster than it. That enterprise server would have 100-200 engineering students logged in and working at once. :-/. We're living in the future.

I like to play a game called "A current Raspberry Pi (or mobile phone) is equivalent to what year's global computing power?"

Also "fun" is "The solid state storage I can buy for $100 is equivalent to all the world's computing storage in what year?"

I'm super fun at parties!

Re: The database servers powering Let's Encrypt

#226
post #22

Earlier quoted context omitted.

If I had a billion dollars, I'd put a research group together to study the prospects of index sharding. That is, full table replication, but individual servers maintaining differing sets of indexes. OLAP and single request transactions could be routed to specialized replicas based on query planning, sending requests to machines that have appropriate indexes, and preferably ones where those indexes are hot.

This has been done in popular commercial databases for decades, and is thoroughly researched. As far as I know, these types of architectures are no longer used at this point due to their relatively poor scalability and write performance. I don't think anyone is designing new databases this way anymore, since it only ever made sense in the context of spinning disk. The trend has been away from complex specialization o…

We pick up algorithms from 30 years ago all the time. Nobody’s as bad as the fashion industry, but we sure do try.

Part of it is short memories, but part of it is how the cost inequalities in our hardware shifts back and forth as memory or storage or network speeds fall behind or sprint ahead.

Re: The database servers powering Let's Encrypt

#227
post #22

Earlier quoted context omitted.

If I had a billion dollars, I'd put a research group together to study the prospects of index sharding. That is, full table replication, but individual servers maintaining differing sets of indexes. OLAP and single request transactions could be routed to specialized replicas based on query planning, sending requests to machines that have appropriate indexes, and preferably ones where those indexes are hot.

The problem is the network. You need billion dollars to fix the network so it's as fast as local ram/nvme.

Why would that be the case? In this case we have already accepted that multiple servers will be involved. That means the limitations of the networking are a given.

Re: The database servers powering Let's Encrypt

#228

Earlier quoted context omitted.

The problem is the network. You need billion dollars to fix the network so it's as fast as local ram/nvme.

... and even if you somehow solved that, the law of physics hits you hard . Latency can be a real performance killer generally and is doubly true in database-type computing.

Are you guys talking about single server, non redundant databases? That’s not even apples and oranges. More like watermelons and blueberries.

Re: The database servers powering Let's Encrypt

#229
post #71
post #60

Earlier quoted context omitted.

> traditional OLTP row stores are I/O bound due to contention (locking and latching). Does anyone have an explanation for this? I have seen CPU bound database servers when developers push application logic in to the database. Everything from using server-side functions like MD5() to needless triggers and stored procedures that could have been done application side.

Any MySQL with more than about 100 concurrent queries of the same InnoDB table is going to be CPU bound on locks. Their whole locking scheme doesn't scale; it's designed to look great in benchmarks with few clients.

I am very interested. Could you provide more detail or a link about this. I would love to understand what they are doing that look good on benchmark but does not actually scale.

Re: The database servers powering Let's Encrypt

#230

Earlier quoted context omitted.

I did a quick search and it looks like InnoDB implements a spin-lock. Do you see increased CPU utilization when the buffer pool is overloaded? This could explain the behavior described in the article.

Correct, it uses an aggressive spinlock without fairness features for waiters like you'd see with a proper production-quality mutex. This makes it very efficient in the absence of contention (i.e. in trivial benchmarks) and extremely poorly behaved in times of contention.

I assumed they would spin for only a short period of time only then fallback to a real lock.

This is usually done to avoid blocking the thread before the end of it's time slice if there is a chance the lock would become available.

If instead they implement a pure spin loop with sleep I can see why it does not perform well.

Post reply on HN