I'm wondering if anyone thought about a GDPR extension that would include machine learning extension, ie. being forgotten meant "unlearning" to the model from my data (or relearning it on dataset from which my data was removed).
Apache Kafka and GDPR compliance
21–30 of 47 posts
Re: Apache Kafka and GDPR compliance
#22Earlier quoted context omitted.
> What, none of these people /ever/ foresaw the need to delete some data? It's a performance trade off, and not a very surprising one. Hard Disk Drives have always been known to never actually delete data (if you want the data gone, you overwrite it with 0s). It's not unimaginable that this performance trade-off found its way up the stack. And just like a regular HDD, you can forcibly delete the data, it's just a ver…
>And just like a regular HDD, you can forcibly delete the data Except apparently not, because the linked article is literally saying it's not supported. I get wanting an audit trail, and I get wanting to not delete data if you don't have to for performance reasons, but neither of those things is the same as saying "it's literally not possible to delete stuff".
>Except apparently not, because the linked article is literally saying it's not supported.
I see your point, but you are mistaken (or you took the wrong impression from the article) - it is supported, you just wouldn't want to do it day-to-day, and for the context of the article it might as well not exist. In Kafka, if you want to forcibly delete the data, you could simply just force topic compaction after a delete. Depending on the size of your data, a regular delete could take hours, which would likely blow the resource usage on any decently sized deployment.
I bring this up because a lot of shiny "BigData" databases use Log Structured Merge Trees, which are immutable and deletes are mostly "soft-deletes" until a "compaction".
Re: Apache Kafka and GDPR compliance
#23Earlier quoted context omitted.
>And just like a regular HDD, you can forcibly delete the data Except apparently not, because the linked article is literally saying it's not supported. I get wanting an audit trail, and I get wanting to not delete data if you don't have to for performance reasons, but neither of those things is the same as saying "it's literally not possible to delete stuff".
What is your thought on this? > Encrypt with a user specific key when the data enters the log. You can effectively delete all the user specific data by throwing the key away. No tracking down files or reprocessing necessary. from https://news.ycombinator.com/item?id=15847674
From a technical perspective, that seems like a sensible compromise to me.
Re: Apache Kafka and GDPR compliance
#24This "right to be forgotten" requirement is quite staggering in scope. Do I need to dig out all of my offsite tape backups and re-transcribe them to edit out my user's data every time a user requests to be forgotten? Sibling comments mention a cunning scheme with encryption, but that doesn't really help an enterprise with an existing non-GDPR-compliant backup archive.
My understanding is that the GDPR “right to be forgotten” does not cover backups. There may be some exceptions, but there are practical limits on its reach.
Given the fact that backup is a must for almost any system, it would be silly for GDPR to not "cover" backup files.
Re: Apache Kafka and GDPR compliance
#25We've been doing a lot of thinking about how to support GDPR at Snowplow (Kafka and Kinesis but plenty of other logs and stores) - for our first phase we're just going to support irreversible pseudonymization of tagged PII:
https://github.com/snowplow/snowplow/issues/3472
For later phases, yes user-specific encryption of PII or hashing-with-lookup table are the way to go...
Re: Apache Kafka and GDPR compliance
#26Earlier quoted context omitted.
> What, none of these people /ever/ foresaw the need to delete some data? It's a performance trade off, and not a very surprising one. Hard Disk Drives have always been known to never actually delete data (if you want the data gone, you overwrite it with 0s). It's not unimaginable that this performance trade-off found its way up the stack. And just like a regular HDD, you can forcibly delete the data, it's just a ver…
>And just like a regular HDD, you can forcibly delete the data Except apparently not, because the linked article is literally saying it's not supported. I get wanting an audit trail, and I get wanting to not delete data if you don't have to for performance reasons, but neither of those things is the same as saying "it's literally not possible to delete stuff".
> Except apparently not, because the linked article is literally saying it's not supported.
Not entirely true. Kafka, out of the box (and as far as I know, I'm no expert) will keep the records for 7 days and delete them afterwards.
Most people I know (including myself) use Kafka to keep records for longer and a good option is to use what the article describes, which is to compact the logs. In that case the log, after a configured period of time (or when it reaches determined size) gets compacted and only the latest message with an id gets saved, all previous messages with the same id get deleted (that's why the process needs a message with a null velue to perform the "delete").
Only in the case when you want to keep the data forever and can't use compaction (compaction assumes that your messages always contain the full state of an entity, so the last message will always contain the current state and the previous can be deleted with no side effects), then there's no way to delete a specific message. I'd have to read the exceptions for backups included in GDPR, but you could make the case that, in this case, the Kafka log is maintained only as a backup of the data, to be able to replay it again in case something downstream gets broken.
Re: Apache Kafka and GDPR compliance
#27This "right to be forgotten" requirement is quite staggering in scope. Do I need to dig out all of my offsite tape backups and re-transcribe them to edit out my user's data every time a user requests to be forgotten? Sibling comments mention a cunning scheme with encryption, but that doesn't really help an enterprise with an existing non-GDPR-compliant backup archive.
My understanding is that the GDPR “right to be forgotten” does not cover backups. There may be some exceptions, but there are practical limits on its reach.
I believe your understanding is incorrect. GDPR certainly includes storage and processing, both of which backups probably trigger.
Anyway, think about the spirit of the law, and then think about how that interacts with backups. If someone asks to be deleted from your system, you do so, and then you restore a backup with their data, you have clearly violated the intent.
Re: Apache Kafka and GDPR compliance
#28This "right to be forgotten" requirement is quite staggering in scope. Do I need to dig out all of my offsite tape backups and re-transcribe them to edit out my user's data every time a user requests to be forgotten? Sibling comments mention a cunning scheme with encryption, but that doesn't really help an enterprise with an existing non-GDPR-compliant backup archive.
I asked a GDPR consultant we hired about backups. The answer was basically that the letter of the law requires data to be deleted from backups, but people aren't going to do that, and there is some language about "reasonable" effort or something like that.
This might be useful: http://www.davidfroud.com/does-right-to-erasure-include-back...
Re: Apache Kafka and GDPR compliance
#29With GDPR do we need to get consent from users before we can set any cookies?
Re: Apache Kafka and GDPR compliance
#30I'm wondering if anyone thought about a GDPR extension that would include machine learning extension, ie. being forgotten meant "unlearning" to the model from my data (or relearning it on dataset from which my data was removed).
if your PII has been incorporated into a model, let's say giving a likelihood to buy red cars based on 100 data points, then it's fairly
safe to assume your PII is anonymised - i cannot imagine a way back from model that each input