Live data from Hacker News

Facebook engineers: we have no idea where we keep all your personal data

theintercept.com

121–130 of 137 posts

Re: Facebook engineers: we have no idea where we keep all your personal data

#121
post #17

After reading some more of the transcript, I think the article does such a bad job of describing what was being asked for that it makes the court seem incompetent and Facebook actually rather reasonable. My original comment is below: Government contracted construction workers: We Have No Idea Where Your Tax Money Goes When tasked with answering the simple question "Which specific bricks did my tax money buy this year…

Government contracted construction workers: We Have No Idea Where Your Tax Money Goes

I think it would be a more meaningful analogy if you had posited two government accountants.

Re: Facebook engineers: we have no idea where we keep all your personal data

#122
post #63
post #11

In many ways you can think of large, long-living tech companies not unlike old cities like, say, London or Paris. The buildings and roads you see are built on top of older buildings and ruins. The streets are weirdly shaped and intersect at odd angles because they were made hundreds of years before and adapted over time as needs evolved. There are catacombs underneath sidewalks and no one genuinely understands it all…

This is a seductive notion, but it's indeed possible for a tech company to understand how a single piece of data goes through its systems. It might take a while, and it might involve weeks of code spelunking and dozens of conversations with engineers — but it is indeed possible. A codebase is not encumbered with the physical constaints of an old city. It's not like trying to figure out the physical state of a certain…

> it might involve weeks of code spelunking and dozens of conversations with engineers

It's worse than you think. First, a little background: I worked on one of Facebook's largest storage systems for two years. So let's talk about some concrete examples.

I knew the rebalancing code better than anyone by the time I left. This is a pretty essential bit of functionality, running regularly on any cluster that has been up for any amount of time and moving quite a bit of data each time. After a chunk was copied, my code would go to delete the old one. I know for a fact that the deletion could fail without my code even getting a useful error from lower layers. I found and fixed many such cases, but I'm sure more remained. That could leave an "orphan" chunk where nobody would know to look for it, and chunks were self-identifying enough that if enough such orphans existed they could be reconstructed into a whole block possibly containing user data.

I knew the data-repair code almost as well. Same problem. I knew the system-repair code less well. Similar there too. In fact I was involved in pulling back "repaired" systems and using them to recover data that would have been lost, more than once. I watched other engineers get rewarded for cleaning up petabytes' worth of no-longer-reachable data (because of cases like I'd mentioned, or bugs, or whatever) that was still taking up space on our millions of disks - again, multiple times. And that's all just one storage system. I'll bet others had similar issues. Also, the problems with truly erasing disks and particularly SSDs are pretty well known. If a machine was taken out of our system entirely and repurposed for another one, or vice versa, there could still be data on its platters/chips that could be recovered with sufficient forensic effort. (These things were physically destroyed before leaving FB, and I've even seen the impressive machines that do it, but not between "lifetimes" within the company.)

So knowing the current code is not enough. You'd have to know every past state of the code during a relevant timeframe, including what bugs it had, which is challenging to say the least. Every configuration detail, too. You'd have to know every rebalancing, reconstruction, or repair event that might have affected each disk. It really is like figuring out the physical state of an area in London. Nothing short of scanning every sector, even as more hardware enters and leaves the system almost every minute, could produce an absolute guarantee.

I'm not saying it can't be done in any system of similar size and complexity. Just that it couldn't in Facebook's, even without malice, largely because so much code was written without that in mind and it's a really hard thing to bolt on afterward. I know there are people working toward it. It's just going to take longer than most people - even technical people - think, and saying it's done would be a lie.

Re: Facebook engineers: we have no idea where we keep all your personal data

#123

Earlier quoted context omitted.

The software world evolves much faster than any of the engineering disciplines and has fewer physical constraints. The requirements for software are rarely fully specified when construction starts. Software is faster to build, deploy and fix so getting it right the first time is not as important. The environment in which it is deployed changes rapidly and in entirely unexpected ways, which is not the case for things…

> The software world evolves much faster than any of the engineering disciplines and has fewer physical constraints. The requirements for software are rarely fully specified when construction starts. Software is faster to build, deploy and fix so getting it right the first time is not as important. The environment in which it is deployed changes rapidly and in entirely unexpected ways, which is not the case for thing…

> These are all artefacts of software development as we practice it, rather than truisms.

Well, they're not laws of nature or anything. But pragmatically, how could it be otherwise? When it comes to bridges and buildings, correctness beats timeliness, but when it comes to software the opposite is usually true. There are exceptions of course (e.g. medical software)

> Maybe we should reserve the term software engineering for the applications of software development where rigour is involved?

Agree.

Re: Facebook engineers: we have no idea where we keep all your personal data

#124
post #35

Earlier quoted context omitted.

That’s an absurd analogy that doesn’t fit at all. FB uses the data it can’t find or tell you anything about to successfully sell ad space to third parties targeted back at you. Let’s make it concrete. We know FB keeps track of which webpages you visit. We also know they use that data as a way to help target ads. That data isn’t in the data export they give you, as far as I can tell from a brief search. Is there data…

Exactly. If Facebook can target ads at specific users to begin with, then how can they possibly claim they don't know where the data is or how to find it? Then how do they know the users are being targeted "correctly" or at all? It's a ridiculous argument that falls apart under any scrutiny. Their entire business model relies on them knowing how to target specific users using huge graphs of information.

Engineers know how to find a service or a data store that meets a particular need. If extraordinarily skilled, they might even know how to find the best one. Wizards know how to go exploring in the wilderness beyond an API boundary to track down particular problems. A god’s eye view of all services, all data stores, all relationships, and all flows can at best be machine generated. Only extreme zoom levels of tiny regions are of any meaning to a human reader. It’s the difference between going shopping and accounting for all the flows of goods and services in an economy.

Re: Facebook engineers: we have no idea where we keep all your personal data

#125
post #63
post #11

In many ways you can think of large, long-living tech companies not unlike old cities like, say, London or Paris. The buildings and roads you see are built on top of older buildings and ruins. The streets are weirdly shaped and intersect at odd angles because they were made hundreds of years before and adapted over time as needs evolved. There are catacombs underneath sidewalks and no one genuinely understands it all…

This is a seductive notion, but it's indeed possible for a tech company to understand how a single piece of data goes through its systems. It might take a while, and it might involve weeks of code spelunking and dozens of conversations with engineers — but it is indeed possible. A codebase is not encumbered with the physical constaints of an old city. It's not like trying to figure out the physical state of a certain…

[deleted]

Re: Facebook engineers: we have no idea where we keep all your personal data

#126
post #11

In many ways you can think of large, long-living tech companies not unlike old cities like, say, London or Paris. The buildings and roads you see are built on top of older buildings and ruins. The streets are weirdly shaped and intersect at odd angles because they were made hundreds of years before and adapted over time as needs evolved. There are catacombs underneath sidewalks and no one genuinely understands it all…

As far as code goes, it's true for many companies. As for data, it was similar in many large European companies in the pre-GDPR era. Today, one crucial question is always asked: when you work with data, is this personal data? If it is, you need to deal it with a special way. Personal data is both an asset and a liability. Most companies found a way do do it. It was a long and often very painful process, involving eve…

[deleted]

Re: Facebook engineers: we have no idea where we keep all your personal data

#127
post #11

In many ways you can think of large, long-living tech companies not unlike old cities like, say, London or Paris. The buildings and roads you see are built on top of older buildings and ruins. The streets are weirdly shaped and intersect at odd angles because they were made hundreds of years before and adapted over time as needs evolved. There are catacombs underneath sidewalks and no one genuinely understands it all…

What you appear to be saying is that large technology companies are designed from the ground up around flouting the laws about data retention.

Why is this OK?

Re: Facebook engineers: we have no idea where we keep all your personal data

#128
post #86

Earlier quoted context omitted.

I mean, sure, security is complicated, lots of judgment calls, but within this mindset, if a team/org/company said "we have no idea where we're using log4j" it was then their job to have someone working around the clock until they figured that out and fixed it . Figuring out how to meet new requirements that weren't around when systems were begun is, IMO, fully just part of the job. For FB engineers to be throwing up…

as per my other comment, in most of the sufficiently big or historical cases you have to distinguish between two things: - reality - the socially acceptable fiction in the report the exercise would be to create an output that allows people to pretend we know where all instances of "log4j" were used, or that sufficient depth and resources have been expended on such. but in a sufficiently large and complex organisation…

Then these organizations should be torn down, their executives jailed, and replaced by organizations that are not deliberately designed so compliance with data privacy and retention laws is completely impossible.

Re: Facebook engineers: we have no idea where we keep all your personal data

#129

Earlier quoted context omitted.

I think specifically any industry working in personal data collection of minors should absolutely have this governance burden. If Meta knows enough about their users to know Instagram is giving teenage girls mental illnesses, Meta engineers should absolutely be hoisted out of their beds if they don't have precise, well-defined, secure data stores.

The problem with your example is the engineers have built exactly what the company wanted. Perhaps the mental illness was an unintended side effect (perhaps known). But either way a decision was made that they want X and so the engineers built X. What you're talking about is banning X, which is something a bit difference than a governance burden/compliance issue as those can generally be followed to a specification a…

> it's still early days for that.

We've had social networks for twenty five years.

Re: Facebook engineers: we have no idea where we keep all your personal data

#130
post #11

In many ways you can think of large, long-living tech companies not unlike old cities like, say, London or Paris. The buildings and roads you see are built on top of older buildings and ruins. The streets are weirdly shaped and intersect at odd angles because they were made hundreds of years before and adapted over time as needs evolved. There are catacombs underneath sidewalks and no one genuinely understands it all…

That's just a truism at any tech company. The issue is Facebook/Meta was misleading about what data is being gathered and the "Download Your Information" didn't actually show you what data FB had gathered about you. By their own admission they don't even know yet users and regulatory agencies were knowingly misled into believing that information was accurate.
Post reply on HN