Live data from Hacker News

Facebook engineers: we have no idea where we keep all your personal data

theintercept.com

91–100 of 137 posts

Re: Facebook engineers: we have no idea where we keep all your personal data

#91
post #11

In many ways you can think of large, long-living tech companies not unlike old cities like, say, London or Paris. The buildings and roads you see are built on top of older buildings and ruins. The streets are weirdly shaped and intersect at odd angles because they were made hundreds of years before and adapted over time as needs evolved. There are catacombs underneath sidewalks and no one genuinely understands it all…

No post body was provided.

Re: Facebook engineers: we have no idea where we keep all your personal data

#92

Earlier quoted context omitted.

If more industries did carry this governance burden, we'd see a lot less cases of companies getting hacked left and right. For way too many companies, "IT" is just another cost center that doesn't bring in any profits and so is barely kept at life support levels. And when something inevitably happens, it's rarely the C-level that gets to go to the unemployment office, it's more one poor soul who has complained for ye…

Hah, I might have agreed before I worked on a credit card processing system. There were plenty of rules alright, but it's possible to abide by all the rules without actually making a secure system. Some of the rules made it harder to create a secure system because they did not reflect the state of the art, or they made adding additional security too burdensome to implement. In the end, you can't force competence thro…

> In the end, you can't force competence through rules.

I mean, we have in all other forms of engineering. Why is software different?

Re: Facebook engineers: we have no idea where we keep all your personal data

#93
post #33
post #11

In many ways you can think of large, long-living tech companies not unlike old cities like, say, London or Paris. The buildings and roads you see are built on top of older buildings and ruins. The streets are weirdly shaped and intersect at odd angles because they were made hundreds of years before and adapted over time as needs evolved. There are catacombs underneath sidewalks and no one genuinely understands it all…

I work for a large long-living tech company. Where legal compliance and security are concerned, "it is just layering and organic growth" is not something you get to say. If we were speculating about such a thing on a coffee break, maybe we'd get the kind of answer given here, but if a single reference doesn't exist for something that is necessary for compliance , people get paged until it does. (There may be other th…

Counterpoint: the conclusions of legal, compliance and security rest upon social fictions and power structures, but fundamentally at a certain scale, all things act like the original commenter said.

In my experience with the above, people get paged until a "socially plausible fiction or resolution" can be created.

There's several versions of these, one of which is to engage in enough rituals and processes so that we all pretend the task has been completed and the requisite outputs and points of reference exist. An archaeologist knows they don't and they can't and that we lay on bedrocks of speculation, history, ignorance and myth, but we don't have to worry too much about the fiction we create because we also know that disproving such mechanically is also extremely impossible/unlikely.

And we have a host of other social functions to get out of the cases where it could, such as:

- throwing people under the bus/human sacrifice/ finger pointing

- contrition

- plausible deniabilty

- continued non-compliance and non-enforcement

- bribery

- bullshit jobs

Re: Facebook engineers: we have no idea where we keep all your personal data

#94
post #63
post #11

In many ways you can think of large, long-living tech companies not unlike old cities like, say, London or Paris. The buildings and roads you see are built on top of older buildings and ruins. The streets are weirdly shaped and intersect at odd angles because they were made hundreds of years before and adapted over time as needs evolved. There are catacombs underneath sidewalks and no one genuinely understands it all…

This is a seductive notion, but it's indeed possible for a tech company to understand how a single piece of data goes through its systems. It might take a while, and it might involve weeks of code spelunking and dozens of conversations with engineers — but it is indeed possible. A codebase is not encumbered with the physical constaints of an old city. It's not like trying to figure out the physical state of a certain…

The thing with tech though is that it's easy to add new connections, connecting anything to anything.

You might have two independent datasets that can't be stored together due to privacy or security regulations. The teams who work on that know and understand that. But some other team working on something completely unrelated might join those datasets for some non-nefarious purpose, which then creates a new dataset with those data combined. Now other teams who also don't know about the regulations might build on top of that combined dataset. They might even generate new data fields from the data that's not supposed to be joined. This stuff can propogate through layers of systems, queries, service calls, analytical engines, offline spreadsheets, etc. It's very difficult to keep track while keeping your engineering teams independent and nimble.

Re: Facebook engineers: we have no idea where we keep all your personal data

#95
post #86

Earlier quoted context omitted.

Counterpoint, in my career, I've seen a lot of Security Audits that seem on paper to be very thorough and detailed but when you dig deeper, you can see that it only touched the surface and the auditor missed a lot of things.

I mean, sure, security is complicated, lots of judgment calls, but within this mindset, if a team/org/company said "we have no idea where we're using log4j" it was then their job to have someone working around the clock until they figured that out and fixed it . Figuring out how to meet new requirements that weren't around when systems were begun is, IMO, fully just part of the job. For FB engineers to be throwing up…

as per my other comment, in most of the sufficiently big or historical cases you have to distinguish between two things:

- reality

- the socially acceptable fiction in the report

the exercise would be to create an output that allows people to pretend we know where all instances of "log4j" were used, or that sufficient depth and resources have been expended on such.

but in a sufficiently large and complex organisation, we (and by we I mean technically able, knowledgeable or sufficiently aware individuals) know that the content of such is likely to be as factually true as reading a historian's report entitled: "enumeration of all points in past century where cutlery has been used".

Re: Facebook engineers: we have no idea where we keep all your personal data

#96

Earlier quoted context omitted.

Hah, I might have agreed before I worked on a credit card processing system. There were plenty of rules alright, but it's possible to abide by all the rules without actually making a secure system. Some of the rules made it harder to create a secure system because they did not reflect the state of the art, or they made adding additional security too burdensome to implement. In the end, you can't force competence thro…

> In the end, you can't force competence through rules. I mean, we have in all other forms of engineering. Why is software different?

Sounds like a pretty good argument for 'software engineering is not engineering'.

Re: Facebook engineers: we have no idea where we keep all your personal data

#97

Earlier quoted context omitted.

Yes and it’s a good thing that most industries do not carry this governance burden.

I think specifically any industry working in personal data collection of minors should absolutely have this governance burden. If Meta knows enough about their users to know Instagram is giving teenage girls mental illnesses, Meta engineers should absolutely be hoisted out of their beds if they don't have precise, well-defined, secure data stores.

The problem with your example is the engineers have built exactly what the company wanted. Perhaps the mental illness was an unintended side effect (perhaps known). But either way a decision was made that they want X and so the engineers built X. What you're talking about is banning X, which is something a bit difference than a governance burden/compliance issue as those can generally be followed to a specification and directly measured for deficiency (eg. how to handle HIPAA data, how long to store records, etc.)

If the issue is about teenagers being too immature for the product X, then typically we ban it or age restrict it using legislation (eg tobacco, alcohol, porn, etc)

I think we as a society need to figure out where social media fits in and how we want to contain it and it's still early days for that. As much as I like the idea of an age restriction, I look around the world and see 6 month olds playing with iPads, 6 year olds on tiktok, etc. as the more 'average' way to parent and wonder if I'm in the minority. My kid is 4 and has never touched a device as entertainment, we taught him how to eat out at restaurants/be in public instead of pacifying him. But that seems like an old fashioned parenting method.

Ok, he's technically used it briefly as a viewfinder for the camera and to facetime his grandparents

Re: Facebook engineers: we have no idea where we keep all your personal data

#98
post #11

In many ways you can think of large, long-living tech companies not unlike old cities like, say, London or Paris. The buildings and roads you see are built on top of older buildings and ruins. The streets are weirdly shaped and intersect at odd angles because they were made hundreds of years before and adapted over time as needs evolved. There are catacombs underneath sidewalks and no one genuinely understands it all…

You can hire one or more data privacy people whose whole job it is to track this. They get an inventory of all the systems and talk to the managers of every system, find the answers and document them. Meta is choosing not to know or pretending not to know. They have the resources to know where your data goes.

Hiring one person to do this at a company this scale, will give you results in the next decade. BTW, when the report is complete, it will no longer be accurate. Sure you can scale the process by putting more people on it, but it won't scale linearly. Additionally you will need buy in from all the system managers to dedicate time to assist the new data privacy team. It is doable, but comes at a cost.

Re: Facebook engineers: we have no idea where we keep all your personal data

#99
post #11

In many ways you can think of large, long-living tech companies not unlike old cities like, say, London or Paris. The buildings and roads you see are built on top of older buildings and ruins. The streets are weirdly shaped and intersect at odd angles because they were made hundreds of years before and adapted over time as needs evolved. There are catacombs underneath sidewalks and no one genuinely understands it all…

> I think you'd get the same answer about many other aspects of data, code, system history, etc at any other 10+ yr old tech giant.

Processes to access personal data vary wildly among tech companies. At Apple, when a machine learning engineer wants to store and use data on the server side, they need to go through layers of approval from lawyers. Even when approved, it often comes with serious constraints about what can be done with the data. Meta is a lot more lax.

Re: Facebook engineers: we have no idea where we keep all your personal data

#100
post #63
post #11

In many ways you can think of large, long-living tech companies not unlike old cities like, say, London or Paris. The buildings and roads you see are built on top of older buildings and ruins. The streets are weirdly shaped and intersect at odd angles because they were made hundreds of years before and adapted over time as needs evolved. There are catacombs underneath sidewalks and no one genuinely understands it all…

This is a seductive notion, but it's indeed possible for a tech company to understand how a single piece of data goes through its systems. It might take a while, and it might involve weeks of code spelunking and dozens of conversations with engineers — but it is indeed possible. A codebase is not encumbered with the physical constaints of an old city. It's not like trying to figure out the physical state of a certain…

>> It might take a while, and it might involve weeks of code spelunking and dozens of conversations with engineers — but it is indeed possible

If it takes a while it will be out of date by the time it is complete.

A single request might directly impact a dozen separate systems, which in turn call other services. You won't be able to find anyone who knows much past the layer they directly interact with. All of these services are being worked on. Some older ones are in the process of being deprecated in favor of newer ones. Someone is already working on the design of the replacements for the newer ones. Changes are being pushed weekly if not daily. If you send a request and then send it again a minute later, it might go through a completely different set of services based on a feature toggle.

You can't step into the same river twice.

Post reply on HN