Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Apache Kafka and GDPR compliance

landoop.com

Post reply on HN
Print view View original

31–40 of 47 posts

Re: Apache Kafka and GDPR compliance

#31

Post by brians »

alexatkeplar wrote: post #25

We've been doing a lot of thinking about how to support GDPR at Snowplow (Kafka and Kinesis but plenty of other logs and stores) - for our first phase we're just going to support irreversible pseudonymization of tagged PII: https://github.com/snowplow/snowplow/issues/3472 For later phases, yes user-specific encryption of PII or hashing-with-lookup table are the way to go...

I wish you wouldn’t call it irreversible. Every large public claim of that sort has proven false. Consider the Netflix case, where the separate IMDB review dataset allowed reconstruction of pseudonymous movie watching records.

These approaches may help with compliance, but they’re the opposite of real safety.

Re: Apache Kafka and GDPR compliance

#32

Post by throwanem »

theptip wrote: post #19
numbsafari wrote: post #18

Earlier quoted context omitted.

My understanding is that the GDPR “right to be forgotten” does not cover backups. There may be some exceptions, but there are practical limits on its reach.

Interesting, if that's so, can we redefine the underlying Kafka topics as "backups" and achieve compliance by having the stream processors drop "forgotten" records when replaying a topic?

Good luck selling that one to a judge. Let us know how it goes!

Re: Apache Kafka and GDPR compliance

#34

Post by MarkMc »

throwaway2016a wrote: post #9

I'm interested in the right to be forgotten section but I'm confused as too what this article is saying... How exactly do you "forget" the data on the logs? One interesting solution that kills two birds with one stone is if you encrypt the personally identifiable information then delete the private key if there is a request to be forgotten. Has the added benefit of also effectively destroying the data in backup copie…

Dear System Administrator,

We've just hacked your server and wiped the crypto keys for your users. As you know, all your backups are now useless.

Send us $1 million in Bitcoin to get your crypto keys back.

Sincerely,

Hacker McHackface

Re: Apache Kafka and GDPR compliance

#35

Post by tscs37 »

pcl wrote: post #16
nerpderp83 wrote: post #13

Earlier quoted context omitted.

Encrypt with a user specific key when the data enters the log. You can effectively delete all the user specific data by throwing the key away. No tracking down files or reprocessing necessary.

Is that acceptable within GDPR requirements?

The GDPR contains exceptions when deleting individual user data is infeasible or if the data in question is a backup, in which case you must only keep a log somewhere so you don't forget to delete again.

Re: Apache Kafka and GDPR compliance

#36

Post by tscs37 »

sulam wrote: post #27
numbsafari wrote: post #18

Earlier quoted context omitted.

My understanding is that the GDPR “right to be forgotten” does not cover backups. There may be some exceptions, but there are practical limits on its reach.

I believe your understanding is incorrect. GDPR certainly includes storage and processing, both of which backups probably trigger. Anyway, think about the spirit of the law, and then think about how that interacts with backups. If someone asks to be deleted from your system, you do so, and then you restore a backup with their data, you have clearly violated the intent.

Keep a log of deleted users and re-delete upon restore.

The GDPR contains exceptions for data storage for which it is infeasible or outside reasonable effort to delete individual records or you have legal compliances to uphold.

Re: Apache Kafka and GDPR compliance

#37

Post by dannyw »

theptip wrote: post #17

This "right to be forgotten" requirement is quite staggering in scope. Do I need to dig out all of my offsite tape backups and re-transcribe them to edit out my user's data every time a user requests to be forgotten? Sibling comments mention a cunning scheme with encryption, but that doesn't really help an enterprise with an existing non-GDPR-compliant backup archive.

Deletion doesn’t have to be immediately complete; if you rotate backups on a six month period then that’s okay.

At least, that’s what I heard about how a BigCo is doing GDPR.

Re: Apache Kafka and GDPR compliance

#38

Post by kbart »

MarkMc wrote: post #34
throwaway2016a wrote: post #9

I'm interested in the right to be forgotten section but I'm confused as too what this article is saying... How exactly do you "forget" the data on the logs? One interesting solution that kills two birds with one stone is if you encrypt the personally identifiable information then delete the private key if there is a request to be forgotten. Has the added benefit of also effectively destroying the data in backup copie…

Dear System Administrator, We've just hacked your server and wiped the crypto keys for your users. As you know, all your backups are now useless. Send us $1 million in Bitcoin to get your crypto keys back. Sincerely, Hacker McHackface

If somebody managed to hack into your servers deep enough to access private keys, you are f*cked anyway (they can as well delete/encrypt all data), so it's not an argument against user data encryption. Actually, storing private keys safely is easier than bulk data, because you can use dedicated hardware for that - HSM.

Re: Apache Kafka and GDPR compliance

#39

Post by mk89 »

nerpderp83 wrote: post #13
Antwnis wrote: post #10

Earlier quoted context omitted.

> How exactly do you "forget" the data on the logs? If we think around the options, you can have either: i) eventual deletion (log retention policy) ii) compacted topics (and push null values) iii) expensive re-processing of the entire log iv) expensive segment re-write operation with each option bringing in a new set of challenges

Encrypt with a user specific key when the data enters the log. You can effectively delete all the user specific data by throwing the key away. No tracking down files or reprocessing necessary.

Today’s encryption is strong. Who knows about tomorrow?

Re: Apache Kafka and GDPR compliance

#40

Post by kbart »

skyisblue wrote: post #29

With GDPR do we need to get consent from users before we can set any cookies?

GPDR itself doesn't specify cookies use. "Cookie law" is defined in ePrivacy Directive (2002/58/EC) which to be replaced by ePrivacy Regulation which is an addendum to GPDR. Actually, it's going to be much saner approach than the joke the current "cookie law" is:

"Simpler rules on cookies: the cookie provision, which has resulted in an overload of consent requests for internet users, will be streamlined. The new rule will be more user-friendly as browser settings will provide for an easy way to accept or refuse tracking cookies and other identifiers. The proposal also clarifies that no consent is needed for non-privacy intrusive cookies improving internet experience (e.g. to remember shopping cart history) or cookies used by a website to count the number of visitors."[0]

To answer your question "do we need to get consent from users before we can set any cookies?"

It depends: yes for tracking cookies, no for others. How to tell them apart is another question..

0. https://en.wikipedia.org/wiki/EPrivacy_Regulation_(European_...

Post reply on HN