Building Facebook's Service Encryption Infastructure • Hacker News Board

Re: Building Facebook's Service Encryption Infastructure

#41

Post by herpderperator » Wed, May 29, 2019, 11:25 PM UTC

Shish2k wrote: post #9

> After several days, we finally narrowed down the issue to a bad Advanced Vector Extensions (AVX) instruction on a single CPU in our fleet This isn't even the first time I've heard of an issue at FB being caused by a single bad CPU instruction. Working at a scale where "Problem X is a one-in-a-million edge case" and "Problem X happens several times per day" are synonymous is weird...

What does "a single bad CPU instruction" mean here? That the CPU was faulty and reliably miscalculated AVX instructions? I don't understand.

Re: Building Facebook's Service Encryption Infastructure

#42

Post by bdd » Wed, May 29, 2019, 11:42 PM UTC

herpderperator wrote: post #41

Shish2k wrote: post #9

> After several days, we finally narrowed down the issue to a bad Advanced Vector Extensions (AVX) instruction on a single CPU in our fleet This isn't even the first time I've heard of an issue at FB being caused by a single bad CPU instruction. Working at a scale where "Problem X is a one-in-a-million edge case" and "Problem X happens several times per day" are synonymous is weird...

What does "a single bad CPU instruction" mean here? That the CPU was faulty and reliably miscalculated AVX instructions? I don't understand.

Yes. For that particular CPU, in one core (and its sibling thread (as in hyperthread)), AVX instruction is broken. It doesn't do what it is supposed to do.

Recently, again at Facebook, we found a few machines with CPUs where ADCX instruction is broken on at least one core. This is especially fatal because OpenSSL and Fizz (our TLS 1.3 impl) use these instructions for AMD64 architecture in RSA implementation.

Re: Building Facebook's Service Encryption Infastructure

#43

Post by brown9-2 » Wed, May 29, 2019, 11:48 PM UTC

sudoyear123 wrote: post #22

dirkg wrote: post #19

Earlier quoted context omitted.

yes I know they are different, but the lessons learnt/dev in both products probably end up influencing each other. My point was encryption of services is built into K8s/service mesh and wondering how it fares compared to FB's approach.

This would make a great comparison. I'm not certain whether or not K8's mutual auth supports session ticket resumptions and distribution of short lived ticket keys. The ticket rotation design would probably make a great addition to K8. There are a lot of intricate details in design which can make a major difference in not only performance but also whether or not the system wakes you up at night.

Kubernetes does not have a built-in mutual auth solution

Re: Building Facebook's Service Encryption Infastructure

#44

Post by ropman76 » Thu, May 30, 2019, 1:25 AM UTC

"After several years of trying to manage these issues with Kerberos, we decided to redesign the system from the ground up.." I am curious, is there anyone who went the opposite direction direction and implemented a new Kerberos library or setup?

Re: Building Facebook's Service Encryption Infastructure

#45

Post by sroussey » Thu, May 30, 2019, 1:57 AM UTC

bdd wrote: post #42

herpderperator wrote: post #41

Earlier quoted context omitted.

What does "a single bad CPU instruction" mean here? That the CPU was faulty and reliably miscalculated AVX instructions? I don't understand.

Yes. For that particular CPU, in one core (and its sibling thread (as in hyperthread)), AVX instruction is broken. It doesn't do what it is supposed to do. Recently, again at Facebook, we found a few machines with CPUs where ADCX instruction is broken on at least one core. This is especially fatal because OpenSSL and Fizz (our TLS 1.3 impl) use these instructions for AMD64 architecture in RSA implementation.

What kind of burn-in process is used to tease these out before deploying? Curious

Re: Building Facebook's Service Encryption Infastructure

#46

Post by bdd » Thu, May 30, 2019, 2:07 AM UTC

sroussey wrote: post #45

bdd wrote: post #42

Earlier quoted context omitted.

Yes. For that particular CPU, in one core (and its sibling thread (as in hyperthread)), AVX instruction is broken. It doesn't do what it is supposed to do. Recently, again at Facebook, we found a few machines with CPUs where ADCX instruction is broken on at least one core. This is especially fatal because OpenSSL and Fizz (our TLS 1.3 impl) use these instructions for AMD64 architecture in RSA implementation.

What kind of burn-in process is used to tease these out before deploying? Curious

Barring the specifics, we deploy new machines in batches of tens of thousands at a time and they practically don't receive any production traffic for weeks while repeatedly running burn-in tests followed by baseline daemons. If they survive, they get provisioned for production.

Infant mortality rate of CPUs--especially single socket designs--is very low compared to DIMMs and SSDs. They tend to develop these issues later in their economic lives. They are also comparatively rare to other module failures.

Re: Building Facebook's Service Encryption Infastructure

#47

Post by cryptonector » Thu, May 30, 2019, 3:47 AM UTC

ropman76 wrote: post #44

"After several years of trying to manage these issues with Kerberos, we decided to redesign the system from the ground up.." I am curious, is there anyone who went the opposite direction direction and implemented a new Kerberos library or setup?

So, several places, such as Stanford or Morgan Stanley, have sophisticated Kerberos setups using something like Russ Allbery's Wallet[0] (Stanford) or Roland Dowdeswell's OSKT[1] (Morgan Stanley, Two Sigma) stack.

For example, OSKT is a self-service toolkit that lets users build up access controls for clusters, "role accounts" (user accounts for running application automation), and what not. Users use "krb5_prestash" to indicate what hosts should have what role accounts' credentials (the user must own the hosts and role accounts) and krb5_keytab to get keys for services on hosts they are allowed to run. A nifty trick is to have wildcard DNS A RRs for hosts so that one can have HTTP/${USER}.$(uname -n) principals (and keys for them) on any host the user can login to.

All of this is high-performance and self-service. Users don't need to file JIRA tickets or whatever to get their keys for their services.

Self-service credential provisioning is absolutely essential to successful deployment at scale of any authentication system one uses, whether that be Kerberos or PKIX or DANE or anything else one might find or invent.

  [0] https://www.eyrie.org/~eagle/software/wallet/readme.html
  [1] https://oskt.secure-endpoints.com/
      https://github.com/elric1/

Re: Building Facebook's Service Encryption Infastructure

#48

Post by Boulth » Thu, May 30, 2019, 8:11 PM UTC

bdd wrote: post #20

Boulth wrote: post #10

Earlier quoted context omitted.

I'm actually interested in this subject so I'll check out your links when I'll be able to. At first sight this sounds like wrapping tokens or third party caveats in Macaroons.

We presented about CATs again in Def Con 26. It's a 21 minute talk but if you're interested in how CAT differ from Macaroons, you can skip to 16:15 mark where Yueting explains https://cryptovillage.org/cats-a-tale-of-scalable-authentica...

I've seen both videos, nice explanation.

If you don't mind I wouldn't necessarily agree with the comment about JWT by Yueting. JWT is just a format, querying backend to get a new token is not necessary (this is only how people often use them). I actually built a small PoC that mints new JWTs on client side (in the browser) signing them with a non-exportable key (through Webcrypto).

As for Macaroons I believe they could also be adjusted to resemble CATs as I understood them (with layers for different services). I do have other issues with Macaroons though (https://news.ycombinator.com/item?id=17878845)...