Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Building Facebook's Service Encryption Infastructure

code.fb.com

Post reply on HN
Print view View original

31–40 of 48 posts

Re: Building Facebook's Service Encryption Infastructure

#31

Post by lclarkmichalek »

Shish2k wrote: post #9

> After several days, we finally narrowed down the issue to a bad Advanced Vector Extensions (AVX) instruction on a single CPU in our fleet This isn't even the first time I've heard of an issue at FB being caused by a single bad CPU instruction. Working at a scale where "Problem X is a one-in-a-million edge case" and "Problem X happens several times per day" are synonymous is weird...

It's always fun to have the space in your schedule to drill down to that level of issue. TLS is pretty useful in that way, as you can catch CPU issues as transport layer errors instead of way down in your application with strange decoding issues.

Re: Building Facebook's Service Encryption Infastructure

#32

Post by Diederich »

laingc wrote: post #30
Diederich wrote: post #23

Earlier quoted context omitted.

Sure. I didn't figure this out, but was relatively close to the investigation. My team provided a lot of data that ended up being used to come to root cause. My team and extended teams managed almost 200,000 network devices, spread all over the world, most of which were Cisco, and most of which were installed in stores. And most of the switch ports were connected to customer facing Point Of Sale devices. Among these…

That’s a fantastic anecdote. I would have loved to have been a fly on the wall when the results were reported to management.

Thanks.

I suspect it would not have been received as that big a deal to management or to anyone else. In those halcyon days, we were running into and usually solving all kinds of such edgy, extreme scale problems. It was a lot of work, but a hell of a lot of fun too.

Re: Building Facebook's Service Encryption Infastructure

#33

Post by ianlevesque »

Diederich wrote: post #23
seanyesmunt wrote: post #17

Earlier quoted context omitted.

Can you share the problem that took a couple of years to figure out?

Sure. I didn't figure this out, but was relatively close to the investigation. My team provided a lot of data that ended up being used to come to root cause. My team and extended teams managed almost 200,000 network devices, spread all over the world, most of which were Cisco, and most of which were installed in stores. And most of the switch ports were connected to customer facing Point Of Sale devices. Among these…

Amusing that you were causing them physical pain when parting with their money at the store, like negative conditioning.

Re: Building Facebook's Service Encryption Infastructure

#35

Post by lioeters »

laingc wrote: post #30
Diederich wrote: post #23

Earlier quoted context omitted.

Sure. I didn't figure this out, but was relatively close to the investigation. My team provided a lot of data that ended up being used to come to root cause. My team and extended teams managed almost 200,000 network devices, spread all over the world, most of which were Cisco, and most of which were installed in stores. And most of the switch ports were connected to customer facing Point Of Sale devices. Among these…

That’s a fantastic anecdote. I would have loved to have been a fly on the wall when the results were reported to management.

Indeed, great story. I'd love to have seen the face of the person who finally figured it out.

Re: Building Facebook's Service Encryption Infastructure

#36

Post by wnevets »

Diederich wrote: post #23
seanyesmunt wrote: post #17

Earlier quoted context omitted.

Can you share the problem that took a couple of years to figure out?

Sure. I didn't figure this out, but was relatively close to the investigation. My team provided a lot of data that ended up being used to come to root cause. My team and extended teams managed almost 200,000 network devices, spread all over the world, most of which were Cisco, and most of which were installed in stores. And most of the switch ports were connected to customer facing Point Of Sale devices. Among these…

but at least you get to blame the users!

Re: Building Facebook's Service Encryption Infastructure

#37

Post by throwawaymath »

plainOldText wrote: post #34

> We run one of the largest microservices deployments in the world, with thousands of services that perform billions of requests per second. Thousands? Does one truly need that many microservices, even if you are the size of Facebook? That is a lot.

That actually sounds about right from my perspective (and experience). Out of curiosity, why do you think that is a lot?

I would go a bit further and say most companies should only proceed with a microservice architecture if they have sufficient scale and automation such that decomposing their architecture will result in at least a high double digit number of discrete services.

Re: Building Facebook's Service Encryption Infastructure

#38

Post by Diederich »

ianlevesque wrote: post #33
Diederich wrote: post #23

Earlier quoted context omitted.

Sure. I didn't figure this out, but was relatively close to the investigation. My team provided a lot of data that ended up being used to come to root cause. My team and extended teams managed almost 200,000 network devices, spread all over the world, most of which were Cisco, and most of which were installed in stores. And most of the switch ports were connected to customer facing Point Of Sale devices. Among these…

Amusing that you were causing them physical pain when parting with their money at the store, like negative conditioning.

Yes, that particular joke did surface after we discovered the problem. (:

Re: Building Facebook's Service Encryption Infastructure

#39

Post by Terretta »

bogomipz wrote: post #26
shereadsthenews wrote: post #18

Earlier quoted context omitted.

Envoy is so slow that deployment at this scale would be too costly, or if you could afford it would immediately present itself as a huge opportunity for cost reduction. People who are measuring their tail latency in microseconds aren't going to tolerate Envoy's marginal latency, which will be milliseconds even at the median.

Interesting, I wasn't aware that Istio has such performance issues. Isn't Google using this though as well or at least an internal version of it? Surely they are on the same scale as FB. I'm curious at to what the cause of the latency is. TLS handshakes?

That report resulted in an update to Istio docs: “Warning not to use demo profile for performance evaluation”

https://github.com/istio/istio.io/pull/4220

More here, which basically suggests, don’t stop Istio from scaling out before 500 rps, it doesn’t like that at all:

https://kinvolk.io/blog/2019/05/performance-benchmark-analys...

Re: Building Facebook's Service Encryption Infastructure

#40

Post by plainOldText »

throwawaymath wrote: post #37
plainOldText wrote: post #34

> We run one of the largest microservices deployments in the world, with thousands of services that perform billions of requests per second. Thousands? Does one truly need that many microservices, even if you are the size of Facebook? That is a lot.

That actually sounds about right from my perspective (and experience). Out of curiosity, why do you think that is a lot? I would go a bit further and say most companies should only proceed with a microservice architecture if they have sufficient scale and automation such that decomposing their architecture will result in at least a high double digit number of discrete services.

I think I was reading microservices but I was imagining services and that's why it seemed like a lot. I guess it depends on the granularity of the decomposition.

I agree, scale and automation are two important factors when decomposing architectures. I think it would be really valuable if systems could decompose themselves to some degree, based on scale and other factors, without much of an operator's intervention.

Post reply on HN