Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building Facebook's Service Encryption Infastructure

code.fb.com

Post reply on HN
Print view View original

11–20 of 48 posts

Re: Building Facebook's Service Encryption Infastructure

#11

Post by Diederich »

Shish2k wrote: post #9

> After several days, we finally narrowed down the issue to a bad Advanced Vector Extensions (AVX) instruction on a single CPU in our fleet This isn't even the first time I've heard of an issue at FB being caused by a single bad CPU instruction. Working at a scale where "Problem X is a one-in-a-million edge case" and "Problem X happens several times per day" are synonymous is weird...

At a previous organization I worked at that operated at this kind of scale, I came up with a couple of maxims:

1. If you're only 99% automated, you're dead.

2. Anything you can easily imagine going wrong is probably going wrong right now.

3. Everything else that can possibly go wrong will go wrong at some point in the not distant future.

Some of the problems we ran into were pretty fun and challenging. The longest substantial one I'm aware of took a couple of years to figure out.

Re: Building Facebook's Service Encryption Infastructure

#12

Post by StreamBright »

Marako wrote: post #8

It seems that they recreated a Service Mesh. Running Istio or Consul Connect takes care or the vast majority of issues listed in this post: Encryption, Identity, access control. And even trasparently for the developpers (no modification of the code...)

Only the scale that these solutions can support is different.

Re: Building Facebook's Service Encryption Infastructure

#13

Post by bogomipz »

StreamBright wrote: post #12
Marako wrote: post #8

It seems that they recreated a Service Mesh. Running Istio or Consul Connect takes care or the vast majority of issues listed in this post: Encryption, Identity, access control. And even trasparently for the developpers (no modification of the code...)

Only the scale that these solutions can support is different.

Isn't Istio implemented mostly as a sidecar container(Envoy Proxy) though? The article mentions they are running containers via their Tupperware orchestrator. If they are largely running containerized where is the scaling issue with adding sidecars to implement the service mesh? I don't have any experience with Istio but I'm genuinely curious along which axis it(or Connect, Linkerd etc) doesn't scale.

Re: Building Facebook's Service Encryption Infastructure

#15

Post by jedberg »

dpflan wrote: post #6

Is this an engineering article to help prime the pump for discussing FB's approach to taking privacy more seriously (e.g. Zuckerberg's the "future is private")? The article does not explicitly state any connection to such larger FB company and product developments, but it made me think it's connected in some way.

Looks like a pretty typical "we solve hard problems and you should come join us" recruiting blog post.

Re: Building Facebook's Service Encryption Infastructure

#16

Post by Diederich »

dirkg wrote: post #14

I wonder what the performance differences are with their approach vs using K8s + service mesh of your choice which includes all this. I imagine Google runs a bigger scale operation on top of Borg and their internal service mesh.

Borg != Kubernetes k8s is based on Borg, but they are quite different.

Companies at this scale have integrations between all different levels and layers of the 'stack' that make the use of off the shelf software difficult or impossible.

Re: Building Facebook's Service Encryption Infastructure

#17

Post by seanyesmunt »

Diederich wrote: post #11
Shish2k wrote: post #9

> After several days, we finally narrowed down the issue to a bad Advanced Vector Extensions (AVX) instruction on a single CPU in our fleet This isn't even the first time I've heard of an issue at FB being caused by a single bad CPU instruction. Working at a scale where "Problem X is a one-in-a-million edge case" and "Problem X happens several times per day" are synonymous is weird...

At a previous organization I worked at that operated at this kind of scale, I came up with a couple of maxims: 1. If you're only 99% automated, you're dead. 2. Anything you can easily imagine going wrong is probably going wrong right now. 3. Everything else that can possibly go wrong will go wrong at some point in the not distant future. Some of the problems we ran into were pretty fun and challenging. The longest su…

Can you share the problem that took a couple of years to figure out?

Re: Building Facebook's Service Encryption Infastructure

#18

Post by shereadsthenews »

bogomipz wrote: post #13
StreamBright wrote: post #12

Earlier quoted context omitted.

Only the scale that these solutions can support is different.

Isn't Istio implemented mostly as a sidecar container(Envoy Proxy) though? The article mentions they are running containers via their Tupperware orchestrator. If they are largely running containerized where is the scaling issue with adding sidecars to implement the service mesh? I don't have any experience with Istio but I'm genuinely curious along which axis it(or Connect, Linkerd etc) doesn't scale.

Envoy is so slow that deployment at this scale would be too costly, or if you could afford it would immediately present itself as a huge opportunity for cost reduction. People who are measuring their tail latency in microseconds aren't going to tolerate Envoy's marginal latency, which will be milliseconds even at the median.

Re: Building Facebook's Service Encryption Infastructure

#19

Post by dirkg »

Diederich wrote: post #16
dirkg wrote: post #14

I wonder what the performance differences are with their approach vs using K8s + service mesh of your choice which includes all this. I imagine Google runs a bigger scale operation on top of Borg and their internal service mesh.

Borg != Kubernetes k8s is based on Borg, but they are quite different. Companies at this scale have integrations between all different levels and layers of the 'stack' that make the use of off the shelf software difficult or impossible.

yes I know they are different, but the lessons learnt/dev in both products probably end up influencing each other.

My point was encryption of services is built into K8s/service mesh and wondering how it fares compared to FB's approach.

Re: Building Facebook's Service Encryption Infastructure

#20

Post by bdd »

Boulth wrote: post #10
sudoyear123 wrote: post #7

Earlier quoted context omitted.

Ya they're similar in that they are all signed blobs of data, but different in the sense that they are specifically designed to send authentication information via several layers of proxies

I'm actually interested in this subject so I'll check out your links when I'll be able to. At first sight this sounds like wrapping tokens or third party caveats in Macaroons.

We presented about CATs again in Def Con 26. It's a 21 minute talk but if you're interested in how CAT differ from Macaroons, you can skip to 16:15 mark where Yueting explains https://cryptovillage.org/cats-a-tale-of-scalable-authentica...
Post reply on HN