Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Instagram kept deleted photos and messages on its servers for more than a year

theverge.com

Post reply on HN
Print view View original

161–170 of 205 posts

Re: Instagram kept deleted photos and messages on its servers for more than a year

#161

Post by areyousure »

jacquesm wrote: post #50
xwdv wrote: post #49

Earlier quoted context omitted.

Everyone thinks deletion is so simple and straight forward until you start dealing with the cascading effects and edge cases.

Yes, especially backups and log files are hard. The databases are relatively easy if they allow for in place overwrites or compaction of tables. Even then you have to be careful. Data is funny that way. It is easy to acquire, easy to lose if you want to keep it and devilishly hard to get rid of for real if that is what you want to do.

Regarding backups: what is your suggestion for backups of databases that contain individual rows that might need to be deleted?

Re: Instagram kept deleted photos and messages on its servers for more than a year

#162

Post by jacquesm »

areyousure wrote: post #161
jacquesm wrote: post #50

Earlier quoted context omitted.

Yes, especially backups and log files are hard. The databases are relatively easy if they allow for in place overwrites or compaction of tables. Even then you have to be careful. Data is funny that way. It is easy to acquire, easy to lose if you want to keep it and devilishly hard to get rid of for real if that is what you want to do.

Regarding backups: what is your suggestion for backups of databases that contain individual rows that might need to be deleted?

You have several options, none of them pretty. The first is to load, update and rewrite the backup. Time consuming, error prone and you may fuck up the backup that you need.

The second starts way further back: encrypt sensitive fields at rest, drop the keys when the user requests a deletion.

That way you never even have to touch the backups in order to have the right end result.

Re: Instagram kept deleted photos and messages on its servers for more than a year

#163

Post by jeffbee »

endtime wrote: post #144
holidayacct wrote: post #116

Whenever someone publishes an article like this, I want them to find out what its like when you reach the threshold of 100000 individual delete requests per second which ends up being five million actual deletes when you factor in all of the associated references to the item being deleted and its metadata. Then I want them to find out what happens when you have to propagate those deletes across geographically distrib…

Google has an internal standard for how long it takes to wipe out user data after a deletion request, and compliance with it is taken very seriously. It's a lot less than 1 year.

I think this points to (one of) the significant flaws in the cult of the "minimal viable product". By eliding up-front design tasks, you may implement something in which complying with the law is impossible. Data retention compliance has to be designed in from the beginning, or it might be technically infeasible to add it later.

Re: Instagram kept deleted photos and messages on its servers for more than a year

#164

Post by jacquesm »

jandrewrogers wrote: post #153
jacquesm wrote: post #77

Earlier quoted context omitted.

All it takes is one dataleak[+] to blow that wide open. Secure erase is hard. But what is much less hard is to encrypt data and then to limit the problem to getting rid of the decryption key. This reduces the problem in scope to one single datum rather than a whole chain of possible plaintext copies. Secure erase is a contractual requirement for many relationships, it is interesting that none of the major db vendors…

Secure delete is not implemented in databases because it is extraordinarily expensive, and destroys the performance of all other non-delete operations. When you say “encrypt data”, you are ignoring the fact that it can’t be implemented as “same data structures, just encrypted”. Encryption puts constraints on data structures and data representation that are fundamentally incompatible with and adverse to the design and…

The law doesn't really care about your notion of what can be done effeciently, if it can be done then you should probably do it or risk being found in violation of the law. There is some very specific language to that effect in the GDPR to make sure that it is clear that 'whatever you could reasonably do' needs to be done. You can then go and argue that you felt it wasn't reasonable but I doubt that will fly.

But you're totally right that this is a tricky problem and hard to do properly. On the plus side, field level encryption is something that we've been doing for ages and that already works quite well. If you design carefully you can even get some processing done on those fields though it will take some major breakthroughs in DB engine design before you can have your cake and eat it too, in the sense that you can be both legally compliant and do all the kinds of processing you can do today. I even doubt if that is desirable, lots of those examples of processing should probably not be done in the first place.

Re: Instagram kept deleted photos and messages on its servers for more than a year

#165

Post by will4274 »

londons_explore wrote: post #68
rco8786 wrote: post #15

Earlier quoted context omitted.

Soft deletes. This is always the case. Not just at Facebook, it is just standard practice across tech co’s.

Google takes deletes seriously. Extreme efforts go into deleting stuff within 30 days of the user requesting deletion. Imagine how hard that is when a datacenter is switched off for 14 days for maintenance, and then a fire breaks out and takes it offline for a further 20 days... When something is powered off, it's very hard to do those deletions... Yet misses of the deadline are exceedingly rare, even in cases like t…

As does Microsoft. I'm always baffled at these "soft deletes are normal" threads. Maybe at a bootstrapping startup, but most real tech companies implement hard deletes within the first couple years.

Re: Instagram kept deleted photos and messages on its servers for more than a year

#166

Post by murftown »

m4tthumphrey wrote: post #12
jermier wrote: post #8

I want to know if the photos are securely deleted. It's not enough that the mere reference to a file is gone. I want everything overwritten with zeroes, and the photo made properly irrecoverable.

Is this something you would actually expect from a media sharing/storage site?

Yes.

Re: Instagram kept deleted photos and messages on its servers for more than a year

#167

Post by toast0 »

jandrewrogers wrote: post #153
jacquesm wrote: post #77

Earlier quoted context omitted.

All it takes is one dataleak[+] to blow that wide open. Secure erase is hard. But what is much less hard is to encrypt data and then to limit the problem to getting rid of the decryption key. This reduces the problem in scope to one single datum rather than a whole chain of possible plaintext copies. Secure erase is a contractual requirement for many relationships, it is interesting that none of the major db vendors…

Secure delete is not implemented in databases because it is extraordinarily expensive, and destroys the performance of all other non-delete operations. When you say “encrypt data”, you are ignoring the fact that it can’t be implemented as “same data structures, just encrypted”. Encryption puts constraints on data structures and data representation that are fundamentally incompatible with and adverse to the design and…

> When you say “encrypt data”, you are ignoring the fact that it can’t be implemented as “same data structures, just encrypted”. Encryption puts constraints on data structures and data representation that are fundamentally incompatible with and adverse to the design and functioning of most databases in existence, even ignoring the other ugly operational issues with that approach which no one thinks about. It would require radically redesigning the internals of most database engines, which is not really a practical option.

You can't do effective per-user encryption on columns the database software needs to read (things you'll query or join on), but the database rarely needs message/post content and image content (often not even stored in the database). So encrypting those could be privacy helpful, if you can make the per-user encrypted store better at deletes than in general.

For person to person messages, if you have a separate record for the sender and the receiver, you can do some per-user transformation on the other correspondent, but that might be indexed, so data without keys would still show messaging patterns.

Re: Instagram kept deleted photos and messages on its servers for more than a year

#168

Post by notyourwork »

loriverkutya wrote: post #127
holidayacct wrote: post #116

Whenever someone publishes an article like this, I want them to find out what its like when you reach the threshold of 100000 individual delete requests per second which ends up being five million actual deletes when you factor in all of the associated references to the item being deleted and its metadata. Then I want them to find out what happens when you have to propagate those deletes across geographically distrib…

Whenever I see someone write a comment like this, I want him to understand, that don't say deleted, if it's not deleted. Also, if it not works at scale, don't tell your users that it works and pretend it works as you communicated. I find really bothering, when engineers trying to hide behind the "it's a very complicated process in the background, so we just say we have done something, when the truth is, it is in prog…

Bingo! Its the same as saying we take your username and password and let you login. Do you or do you not? Its a boolean feature that either does or doesn't do what you say it does. Deletion can be complicated and if it is provide that indication to your users. Software is about setting the correct expectations, otherwise we leave users to interpret written language which we all know leaves room for misinterpretation due to all sorts of complexities with the way humans communicate with one another.

Re: Instagram kept deleted photos and messages on its servers for more than a year

#169

Post by slg »

parliament32 wrote: post #158
ctvo wrote: post #136

Earlier quoted context omitted.

Hi. I also work at this scale. It doesn't take a year to delete the data. Be upfront with your users: 'Your delete request is being processed. No one will be able to see this item though it may take up to three days for the data to be completely removed from our servers' This is not a technical challenge, it's an issue of priorities and Facebook doesn't prioritize privacy. They were able to release a TikTok clone in…

>It doesn't take a year to delete the data It does if you have cold backups that take a year to cycle out. They're often offsite, compressed, and incrementally hashed so finding individual items and removing them is really hard -- you're much better off just waiting for them to expire, and a year isn't an unreasonable amount of time.

The article specifically states this data was downloaded using the "Download Your Information tool on Instagram". The undeleted data wasn't from a cold backup or any backup, it was still on production systems.

Re: Instagram kept deleted photos and messages on its servers for more than a year

#170

Post by [deleted] »

parliament32 wrote: post #158
ctvo wrote: post #136

Earlier quoted context omitted.

Hi. I also work at this scale. It doesn't take a year to delete the data. Be upfront with your users: 'Your delete request is being processed. No one will be able to see this item though it may take up to three days for the data to be completely removed from our servers' This is not a technical challenge, it's an issue of priorities and Facebook doesn't prioritize privacy. They were able to release a TikTok clone in…

>It doesn't take a year to delete the data It does if you have cold backups that take a year to cycle out. They're often offsite, compressed, and incrementally hashed so finding individual items and removing them is really hard -- you're much better off just waiting for them to expire, and a year isn't an unreasonable amount of time.

[deleted]
Post reply on HN