Instagram kept deleted photos and messages on its servers for more than a year • Hacker News Board

Re: Instagram kept deleted photos and messages on its servers for more than a year

#151

Post by jccooper » Fri, Aug 14, 2020, 3:56 PM UTC

gilrain wrote: post #51

imgabe wrote: post #48

I don't know how many times this is going to have to happen before people understand this: when you put something online, assume it is essentially public. Forever. If you don't want it to be public forever, don't put it online.

This is victim blaming and only reinforces the status quo. The way things are isn't the way things have to be. We can change the rules if we work together. Or we can give up and blame the victims.

You can have Instagram delete that photo when you change your mind, but how about the friend who saw it and saved it, or the already-illegal bot that crawled the site and archived it?

"Changing the rules" can reduce data availability, probably well enough for most purposes, and that's good. But it's simply strictly true that once you publish something, you cannot assure it's unpublished. And everyone should know this and act that way.

Re: Instagram kept deleted photos and messages on its servers for more than a year

#152

Post by throw_me_2020 » Fri, Aug 14, 2020, 4:00 PM UTC

ctvo wrote: post #136

holidayacct wrote: post #116

Whenever someone publishes an article like this, I want them to find out what its like when you reach the threshold of 100000 individual delete requests per second which ends up being five million actual deletes when you factor in all of the associated references to the item being deleted and its metadata. Then I want them to find out what happens when you have to propagate those deletes across geographically distrib…

Hi. I also work at this scale. It doesn't take a year to delete the data. Be upfront with your users: 'Your delete request is being processed. No one will be able to see this item though it may take up to three days for the data to be completely removed from our servers' This is not a technical challenge, it's an issue of priorities and Facebook doesn't prioritize privacy. They were able to release a TikTok clone in…

Deletion is very much a technical/organizational challenge.

Imagine some engineer writing some feature you've never heard of and they decide they need data in a different format. They write a job to do this.

Now imagine that job somehow has a bug which prevents the deletion system from understanding that copy exists. That bug could be non-obvious to find.

Re: Instagram kept deleted photos and messages on its servers for more than a year

#153

Post by jandrewrogers » Fri, Aug 14, 2020, 4:21 PM UTC

jacquesm wrote: post #77

londons_explore wrote: post #64

Earlier quoted context omitted.

The vast majority of companies I've seen consider issuing "SQL DELETE" is sufficient for GDPR compliance. One day, they will be burned when some investigation finds out that those rows are still sitting in uncompacted tables, or on now-unallocated disk blocks, or on now-remapped SSD sectors. The only way to be sure the data is deleted is to copy all the data you want to keep to a new drive and burn the old one. Anyth…

All it takes is one dataleak[+] to blow that wide open. Secure erase is hard. But what is much less hard is to encrypt data and then to limit the problem to getting rid of the decryption key. This reduces the problem in scope to one single datum rather than a whole chain of possible plaintext copies. Secure erase is a contractual requirement for many relationships, it is interesting that none of the major db vendors…

Secure delete is not implemented in databases because it is extraordinarily expensive, and destroys the performance of all other non-delete operations. When you say “encrypt data”, you are ignoring the fact that it can’t be implemented as “same data structures, just encrypted”. Encryption puts constraints on data structures and data representation that are fundamentally incompatible with and adverse to the design and functioning of most databases in existence, even ignoring the other ugly operational issues with that approach which no one thinks about. It would require radically redesigning the internals of most database engines, which is not really a practical option.

I’ve thought a lot about what it would take to design a “delete optimized” database kernel every since GDPR became a thing. In principle it is possible, but I seriously doubt anyone would use a database that is literally orders of magnitude slower and less scalable for everything except delete operations. Would it be acceptable to increase the resource intensity of databases by 10x (and environmental footprint implied) to get “real” deletes? That is the tradeoff here.

This has precedent in SQL databases designed for high-assurance applications with ultra-fine access and visibility controls. When the average software engineer understands how they work, it sounds like a great idea for having more secure data and they wonder why it doesn’t seem to exist. The reality is that they do exist but they are so abysmally slow for even elementary things that no one would ever dream of using it unless there is a narrow government requirement.

Re: Instagram kept deleted photos and messages on its servers for more than a year

#154

Post by parliament32 » Fri, Aug 14, 2020, 4:22 PM UTC

holidayacct wrote: post #116

Whenever someone publishes an article like this, I want them to find out what its like when you reach the threshold of 100000 individual delete requests per second which ends up being five million actual deletes when you factor in all of the associated references to the item being deleted and its metadata. Then I want them to find out what happens when you have to propagate those deletes across geographically distrib…

This. The amount of time quoted in the article in excessive but it's also sparse on details (maybe his data request also sent him backups related to him? a year is reasonable in this case).

Deletion is pretty much the hardest thing to do in data management, way harder than inserting or retrieving data. I know GDPR says you need to be able to delete a user's data, but how reasonable is it to trawl through your offsite backups to find their items and remove them?

Re: Instagram kept deleted photos and messages on its servers for more than a year

#155

Post by mtnGoat » Fri, Aug 14, 2020, 4:24 PM UTC

jandrewrogers wrote: post #150

dhdhhdd wrote: post #146

Earlier quoted context omitted.

If you can insert... for sure you can delete? It doesn't take days for your picture to show up. Sure, eventual consistency and replication and distributed storage is hard. Alternative approach: encrypt at rest, delete encryption key. It would require extra resources for sure, but if privacy was a concern... ;-) (and then have a reasonable expiry on your CDN if you use one). (of course then we can argue that deleting…

The cost of insert and delete is massively asymmetrical. This is an intentional and fundamental architectural decision in all of our data infrastructure and built into the data structures and algorithms that are used. Not only does “delete optimized” data infrastructure not exist, in many cases we don’t have good computer science for how you would even design such a thing while having good performance for all the oth…

I would say a system that can't effectively delete also scales poorly. No?

Re: Instagram kept deleted photos and messages on its servers for more than a year

#156

Post by areyousure » Fri, Aug 14, 2020, 4:27 PM UTC

jacquesm wrote: post #105

rco8786 wrote: post #97

Earlier quoted context omitted.

I'm responding here because we've hit the limit on the other comment stream, but I truly want you to better informed on this subject so here goes: > Whether data is an image or some other record is immaterial. This is wildly inaccurate. The content of the data is extremely important in determining how to properly store and (potentially) dispose of it. If the data contains PII, or is covered under PCI or HIPAA the pro…

As engineering manager of a large household name company you really should know better. PII isn't even a term under the GDPR. But feel free to play fast and loose with this and see where it ends up, as far as I'm concerned it can't happen fast enough that regulators crack down on companies that wilfully ignore the law. The GDPR does not say to 'soft delete', it says to delete , and that you should make every reasonab…

You're right that GDPR doesn't use the TLA "PII", but literally the first definition is "‘personal data’ means any information relating to an identified or identifiable natural person".

https://gdpr-info.eu/art-4-gdpr/

Re: Instagram kept deleted photos and messages on its servers for more than a year

#157

Post by mtnGoat » Fri, Aug 14, 2020, 4:34 PM UTC

Based on the conversation here, it would seem like deleting data is an unsolved problem in CS, or some of you just make it overly complex. Every system I have built had deletes, because it is the right thing to do... No team I've been on got away with not doing the right thing because it was hard. We put our heads down and got it done. Smh.

Some of the engineers at these companies forgot their morals as soon as something got hard I guess.

Re: Instagram kept deleted photos and messages on its servers for more than a year

#158

Post by parliament32 » Fri, Aug 14, 2020, 4:40 PM UTC

ctvo wrote: post #136

holidayacct wrote: post #116

Whenever someone publishes an article like this, I want them to find out what its like when you reach the threshold of 100000 individual delete requests per second which ends up being five million actual deletes when you factor in all of the associated references to the item being deleted and its metadata. Then I want them to find out what happens when you have to propagate those deletes across geographically distrib…

Hi. I also work at this scale. It doesn't take a year to delete the data. Be upfront with your users: 'Your delete request is being processed. No one will be able to see this item though it may take up to three days for the data to be completely removed from our servers' This is not a technical challenge, it's an issue of priorities and Facebook doesn't prioritize privacy. They were able to release a TikTok clone in…

>It doesn't take a year to delete the data

It does if you have cold backups that take a year to cycle out. They're often offsite, compressed, and incrementally hashed so finding individual items and removing them is really hard -- you're much better off just waiting for them to expire, and a year isn't an unreasonable amount of time.

Re: Instagram kept deleted photos and messages on its servers for more than a year

#159

Post by jandrewrogers » Fri, Aug 14, 2020, 4:42 PM UTC

mtnGoat wrote: post #155

jandrewrogers wrote: post #150

Earlier quoted context omitted.

The cost of insert and delete is massively asymmetrical. This is an intentional and fundamental architectural decision in all of our data infrastructure and built into the data structures and algorithms that are used. Not only does “delete optimized” data infrastructure not exist, in many cases we don’t have good computer science for how you would even design such a thing while having good performance for all the oth…

I would say a system that can't effectively delete also scales poorly. No?

It depends on the definition of “delete”. Removing a record from a data model scales just fine, and is how it has always been defined in database systems. Making that record physically unrecoverable from any hardware in the broader system is extraordinarily expensive for fundamental technical reasons. Traditionally “delete” has always meant the former in all systems, and physical deletion is deferred to a point in the future when either the an opportunity arises to do it inexpensively or the cost becomes acceptable. There is a value in recovering the resources consumed by the deleted record but it comes at a very high cost.

If you change the definition of “delete” to mean unrecoverable physical deletion, then sure, it scales poorly. But it is a bit like redefining “fast car” to “can travel faster than Mach 5” — technically a valid as a redefinition while completely ignoring the engineering realities of what a car can do.

Re: Instagram kept deleted photos and messages on its servers for more than a year

#160

Post by LunaSea » Fri, Aug 14, 2020, 4:43 PM UTC

holidayacct wrote: post #129

kuschku wrote: post #125

Earlier quoted context omitted.

They've got 30 days to delete data, there's no excuse not to manage it within 30 days. Also, you're saying they intentionally built a software stack without even thinking about how to obey the law (the 30-day deletion requirement existed even before the GDPR, since 1996 even, just with lower fines) I hope in a few years this case will be taught in school just like the Therac case is being taught right now.

It's not that simple, some of these companies have tech stacks that are run-a-way trains written by people who had good intentions but wrote their code and built their infrastructure in a way that was meant for 1/50th of the traffic they currently have. I don't know anything about instagram's infrastructure but what I can tell you for certain is that every one of these companies has legacy infrastructure and code tha…

That really isn't the user's problem.

Laws and rules need to be followed no matter what.

It's the same with GDPR.