Apache Kafka and GDPR compliance
landoop.com
Apache Kafka and GDPR compliance
1–10 of 47 posts
Re: Apache Kafka and GDPR compliance
#2>The right to be forgotten, becomes one of the hardest challenges because of data immutability. Apache Kafka does not support deleting records, and although some eventual deletion is supported, it requires
This always seemed like an incredibly toxic decision to me. It's one that crops up in all sorts of systems, large and small. What, none of these people /ever/ foresaw the need to delete some data?
Re: Apache Kafka and GDPR compliance
#3>The right to be forgotten, becomes one of the hardest challenges because of data immutability. Apache Kafka does not support deleting records, and although some eventual deletion is supported, it requires This always seemed like an incredibly toxic decision to me. It's one that crops up in all sorts of systems, large and small. What, none of these people /ever/ foresaw the need to delete some data?
It's not that simple. For example in my business, we may give some money to help someone "once in its life" (the law says so). Therefore, if the persons asks to be deleted, then we might not apply the law anymore because it'll mean we won't remember the decision... I think GDPR is a good thing, but at some point, in my business, those who write the laws will have to be aware of it (and the legal teams is miles away from the IT stuff, sadly).
Re: Apache Kafka and GDPR compliance
#4>The right to be forgotten, becomes one of the hardest challenges because of data immutability. Apache Kafka does not support deleting records, and although some eventual deletion is supported, it requires This always seemed like an incredibly toxic decision to me. It's one that crops up in all sorts of systems, large and small. What, none of these people /ever/ foresaw the need to delete some data?
It's not that simple. For example in my business, we may give some money to help someone "once in its life" (the law says so). Therefore, if the persons asks to be deleted, then we might not apply the law anymore because it'll mean we won't remember the decision... I think GDPR is a good thing, but at some point, in my business, those who write the laws will have to be aware of it (and the legal teams is miles away f…
GDPR has an exemption related to the legal requirement to process data that might cover this (and related) scenarios.
> ...(unless) processing is necessary for compliance with a legal obligation to which the controller is subject;
Re: Apache Kafka and GDPR compliance
#5>The right to be forgotten, becomes one of the hardest challenges because of data immutability. Apache Kafka does not support deleting records, and although some eventual deletion is supported, it requires This always seemed like an incredibly toxic decision to me. It's one that crops up in all sorts of systems, large and small. What, none of these people /ever/ foresaw the need to delete some data?
It's not that simple. For example in my business, we may give some money to help someone "once in its life" (the law says so). Therefore, if the persons asks to be deleted, then we might not apply the law anymore because it'll mean we won't remember the decision... I think GDPR is a good thing, but at some point, in my business, those who write the laws will have to be aware of it (and the legal teams is miles away f…
The GDPR offers exceptions to the right to erasure, this mostly includes legal compliance (banks) or in the interest of legal claims or when data cannot be easily deleted as individual record. It also does not affect any non-digital documents which aren't filed. This is all laid out very thoroughly in the legal documents relating to this.
Re: Apache Kafka and GDPR compliance
#6>The right to be forgotten, becomes one of the hardest challenges because of data immutability. Apache Kafka does not support deleting records, and although some eventual deletion is supported, it requires This always seemed like an incredibly toxic decision to me. It's one that crops up in all sorts of systems, large and small. What, none of these people /ever/ foresaw the need to delete some data?
As everything in life, to gain something, you need to sacrifice something else. With RDBMS you get mutability; but to go 10x or 100x faster/larger you need to make hard decisions.
HDFS, S3 and other systems have immutability in-built. Immutability is not bad per-se, as it give (some) assurance that data has not been tampered with, and although it could be implemented, the system cost could be significant.
Stricking the right balance is the challenge
Re: Apache Kafka and GDPR compliance
#7>The right to be forgotten, becomes one of the hardest challenges because of data immutability. Apache Kafka does not support deleting records, and although some eventual deletion is supported, it requires This always seemed like an incredibly toxic decision to me. It's one that crops up in all sorts of systems, large and small. What, none of these people /ever/ foresaw the need to delete some data?
Where integrity matters, you never want data to be mutated with no trace. An audit trail is almost always needed - it’s not an extreme leap, then, to say “why don’t we just replay the audit trail to arrive at the current state?”
Re: Apache Kafka and GDPR compliance
#8Earlier quoted context omitted.
It's not that simple. For example in my business, we may give some money to help someone "once in its life" (the law says so). Therefore, if the persons asks to be deleted, then we might not apply the law anymore because it'll mean we won't remember the decision... I think GDPR is a good thing, but at some point, in my business, those who write the laws will have to be aware of it (and the legal teams is miles away f…
GDPR has an exemption related to the legal requirement to process data that might cover this (and related) scenarios. > ...(unless) processing is necessary for compliance with a legal obligation to which the controller is subject;
Does this mean that someone can game 1-time special offers by repeatedly signing up and then demanding to be forgotten?
There's probably no legal obligation to enforce once-only cashback sign-up offers, so the right to be forgotten would presumably have to be followed.
Re: Apache Kafka and GDPR compliance
#9I'm interested in the right to be forgotten section but I'm confused as too what this article is saying...
How exactly do you "forget" the data on the logs?
One interesting solution that kills two birds with one stone is if you encrypt the personally identifiable information then delete the private key if there is a request to be forgotten. Has the added benefit of also effectively destroying the data in backup copies too.
Re: Apache Kafka and GDPR compliance
#10I'm interested in the right to be forgotten section but I'm confused as too what this article is saying... How exactly do you "forget" the data on the logs? One interesting solution that kills two birds with one stone is if you encrypt the personally identifiable information then delete the private key if there is a request to be forgotten. Has the added benefit of also effectively destroying the data in backup copie…
> How exactly do you "forget" the data on the logs?
If we think around the options, you can have either:
i) eventual deletion (log retention policy) ii) compacted topics (and push null values) iii) expensive re-processing of the entire log iv) expensive segment re-write operation
with each option bringing in a new set of challenges