Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Meta built large-scale cryptographic monitoring

engineering.fb.com

Post reply on HN
Print view View original

21–30 of 32 posts

Re: Meta built large-scale cryptographic monitoring

#21

Post by loeg »

jamra wrote: post #14
bagels wrote: post #7

What it's like to be a Facebook engineer in this context: It's your oncall week, and all of a sudden you get an urgent task that you must resolve in the next few days: Figure out how to replace a crypto algorithm in some component your code depends on or else we tell your manager that you're not playing ball.

You’d be calling libraries that are vetted by the security team. Engineering at Meta is a lot like coloring with crayons. It’s very limiting but the infra does a lot for you.

This is true, but like, sometimes you're using them inappropriately. Or you've got a limited exception for performance reasons, and your replacement solution is, uh, unsound.

Re: Meta built large-scale cryptographic monitoring

#23

Post by yodsanklai »

pdpi wrote: post #11
bagels wrote: post #7

What it's like to be a Facebook engineer in this context: It's your oncall week, and all of a sudden you get an urgent task that you must resolve in the next few days: Figure out how to replace a crypto algorithm in some component your code depends on or else we tell your manager that you're not playing ball.

When I was there, the experience here would've been more like "you see some comms on Workplace saying this thing went live". For a small number of people, it would look like "It's your oncall week, and you unexpectedly receive a large diff stack from a complete stranger". Most engineers at Meta won't touch anything related to C++ during their tenure there, and you can safely assume all this stuff would be abstracted…

> you'd probably have known about this ages

but nobody paid attention and it falls on oncall's plate

Re: Meta built large-scale cryptographic monitoring

#24

Post by paxys »

bagels wrote: post #7

What it's like to be a Facebook engineer in this context: It's your oncall week, and all of a sudden you get an urgent task that you must resolve in the next few days: Figure out how to replace a crypto algorithm in some component your code depends on or else we tell your manager that you're not playing ball.

How it really works - security team files a task to replace a crypto algorithm in your team's code and gives you a SLA deadline of 6 months. 6 months later, no one has touched it, so they escalate. Your manager says "we are short staffed" and requests an extension. They give you another 3 months and write a detailed guide on what lines to change. 3 months later everyone they originally spoke to has left the team, and the new people don't have any context. They try and figure it out but the security person has also left the team. No one remembers what the task was originally created for. A year later the service that was using the old algorithm is itself deprecated.

Re: Meta built large-scale cryptographic monitoring

#25

Post by Mistletoe »

loeg wrote: post #21
jamra wrote: post #14

Earlier quoted context omitted.

You’d be calling libraries that are vetted by the security team. Engineering at Meta is a lot like coloring with crayons. It’s very limiting but the infra does a lot for you.

This is true, but like, sometimes you're using them inappropriately. Or you've got a limited exception for performance reasons, and your replacement solution is, uh, unsound.

Do you ever eat the crayons?

Re: Meta built large-scale cryptographic monitoring

#26

Post by pstoll »

tptacek wrote: post #17
HeatrayEnjoyer wrote: post #15

Earlier quoted context omitted.

But shouldn't that bound be astronomical?

GCM is like 4 billion messages, which is not astronomical at Facebook's scale.

Meta (we) process many billions of messages per second. Finding enough randomness is a thing at scale.

Re: Meta built large-scale cryptographic monitoring

#27

Post by bdd »

cactusplant7374 wrote: post #2

> Since there is a limit to how much data a symmetric cryptographic key can protect Is this regulatory related or self imposed?

If you want to read more about cryptographic key wear out, Soatok wrote an excellent explainer on this a few years ago https://soatok.blog/2020/12/24/cryptographic-wear-out-for-sy...

Re: Meta built large-scale cryptographic monitoring

#28

Post by bagels »

pdpi wrote: post #11
bagels wrote: post #7

What it's like to be a Facebook engineer in this context: It's your oncall week, and all of a sudden you get an urgent task that you must resolve in the next few days: Figure out how to replace a crypto algorithm in some component your code depends on or else we tell your manager that you're not playing ball.

When I was there, the experience here would've been more like "you see some comms on Workplace saying this thing went live". For a small number of people, it would look like "It's your oncall week, and you unexpectedly receive a large diff stack from a complete stranger". Most engineers at Meta won't touch anything related to C++ during their tenure there, and you can safely assume all this stuff would be abstracted…

Sure, sometimes I got a bunch of automod diffs. Sometimes it was: fix this now. It's impossible to know everything that every infra team is cooking up.

Re: Meta built large-scale cryptographic monitoring

#29

Post by paxys »

smitelli wrote: post #3

Scribe[1], Scuba[2], Hive[3], and folly[4]. It's neat for sure, but speaking as an outsider without the resources to learn (much less create!) an entire universe of data infrastructure and SDKs, it feels like I'm in the corner eating dinner at the kids' table. At any rate, I'm glad somebody out there is doing this stuff. [1]: https://engineering.fb.com/2019/10/07/core-infra/scribe/ [2]: https://research.facebook.com/…

You can absolutely learn it. Elasticsearch, Logstash, Kibana, Grafana, Prometheus. Set up a local Docker Compose deployment of the full stack and pass through some dummy events. Write an app to generate synthetic load. Play around with aggregations and visualizations. Tech blogs use a lot of buzzwords but you just need to know the fundamental concepts.

Re: Meta built large-scale cryptographic monitoring

#30

Post by bwock »

paxys wrote: post #24
bagels wrote: post #7

What it's like to be a Facebook engineer in this context: It's your oncall week, and all of a sudden you get an urgent task that you must resolve in the next few days: Figure out how to replace a crypto algorithm in some component your code depends on or else we tell your manager that you're not playing ball.

How it really works - security team files a task to replace a crypto algorithm in your team's code and gives you a SLA deadline of 6 months. 6 months later, no one has touched it, so they escalate. Your manager says "we are short staffed" and requests an extension. They give you another 3 months and write a detailed guide on what lines to change. 3 months later everyone they originally spoke to has left the team, and…

You forgot the multiple rounds of taskcreeper closing the task due to inactivity
Post reply on HN