How we built a GDPR-compliant website analytics platform without using cookies • Hacker News Board

Re: How we built a GDPR-compliant website analytics platform without using cookies

#121

Post by orra » Wed, Jul 24, 2019, 12:29 AM UTC

krageon wrote: post #113

orra wrote: post #75

Earlier quoted context omitted.

I think that's a fair question. AFAICT, v1 of PECR awkwardly applies whenever the cookie is not functionally directly necessary for the service that the user is using. PECR applies even if, like here, the cookie is just for counting unique numbers of visitors, and is not used for fingerprinting individuals. The draft v2 of PECR contains an exemption for first party analytics. I think this maybe strikes a nice balance…

What makes analytics first-party? When the first party serves them, or only if the data never leaves machines under the direct and exclusive control of the first party?

I don't know the answer to that.

FWIW, my guess would be that the definition is fairly strict.

Now, I don't think that would prevent the first party using data processors, but I suspect the first party would have to exercise a lot of control over the processor. This would be in contrast to a service like Google Analytics, where the company's control and choice is simply limited to take it or leave it.

Re: How we built a GDPR-compliant website analytics platform without using cookies

#122

Post by JackWritesCode » Wed, Jul 24, 2019, 2:46 AM UTC

nybble41 wrote: post #119

JackWritesCode wrote: post #102

Earlier quoted context omitted.

So if someone had unlimited access to our servers, that would be a problem. One piece to note is that page views get deleted after around 30 minutes. So one of the reasons we posted to HN was for conversations like this. Reading what you put makes me think we need to do more when creating the PageRequestSignature and the SiteRequestSignature, because if someone had access to our server, got the hash, stopped all cron…

The case I would be (slightly) concerned about would be where an attacker has obtained limited, read-only access to your database, so they have the current salt and the hashes related to the last 30 minutes of activity but not the ability to simply log whatever data they want. At that point brute-forcing the 32-bit address space and a small set of common user agent strings would allow them to determine the IP address…

I think what’s been helpful for us with posting this here is to hear of different ideas for how someone might hack Fathom. When we came up with it, our starting point wasn’t “you have the salt and IP address”, go break a hash. It was on the assumption that you don’t have the salt or IP. I think we can improve what we’ve built. 720 salts improves resilience in a few areas but not in the scenario you are painting here. The scenario you’re painting here has made me think of additional ideas though.

If they had the salt, IP and user agent, they’d have to also brute force every possible hostname and pathname, which would be insane. But I suppose they’d only have to do a few million based on the data we have on hostnames / pathnames...

Lots of ideas for improvement are popping into my head and I love how this community keeps challenging you to improve things. We had feedback on Reddit but it was much angrier!

The next step is to take your feedback and look into how we would defend against the scenario you’ve provided. Thank you!

Re: How we built a GDPR-compliant website analytics platform without using cookies

#123

Post by JackWritesCode » Wed, Jul 24, 2019, 10:39 AM UTC

gcbw2 wrote: post #120

JackWritesCode wrote: post #101

Earlier quoted context omitted.

You're focused mostly on Recital 26, which was only a theory of mine, outside of that we are GDPR compliant anyway. I likely shouldn't have included it since that isn't our primary ground for processing. Please see: https://usefathom.com/data/ And yes the daily hash gets stored until midnight. But what are you talking about with 'search query' containing IP, user agent etc.?

If a search query on your data would contain all the components of the original hash, i don't have to walk backwards and break the hash. i just have to hash my query terms in the same way. Also I suggested you store the daily hash forever. But even if you really erase it every day, as you say, If you or an attacker makes the same request every day at a predetermined time, when you/they get your logs, you/they can use…

So that needs to be our next target point (access logs). We want to move to a position to keep no access lgos.

And a hacker could indeed "win" if they broke into our system, got the salt and exported the DB. We didn't focus on this in our article, as it's unbelievably unrealistic, but it's still possible. Our next step is to address that.

Without the hash, it's practically impossible to brute force.

Re: How we built a GDPR-compliant website analytics platform without using cookies

#124

Post by bashy » Wed, Jul 24, 2019, 3:50 PM UTC

JackWritesCode wrote: post #92

bashy wrote: post #29

Earlier quoted context omitted.

Definitely. Helps with people who disable JS as well. I've used this before https://goaccess.io

bashy from laracasts? Hey there! And I've heard the disabled JS piece before. How many people are disabling javascript?

Yes that's me. Hi there.

I use fathom on one of my sites and I like it. I was just replying to someone who said people could just use access logs instead. Some don't like adding JS to their site I guess.

Ad blockers will also block requests to the tracker.js if you use analytics.domain for example.

Re: How we built a GDPR-compliant website analytics platform without using cookies

#125

Post by labawi » Thu, Jul 25, 2019, 1:43 PM UTC

If visits expire after 30 min, why not rotate the salt every 30 min? Keep current and previous salt, update as needed.

I would have more faith in privacy, if you didn't store the salt in the DB or permanent storage. If you manage to statically load-balance the users (e.g. hash site, ip, user-agent, don't forget site), the hash could be in-memory only. Sessions would break on server restart, but that's more of a feature.

To move thing further, you might not even need to store the hashes in the DB. Keep them in server memory only and (real-time) update aggregate data in DB.

Re: How we built a GDPR-compliant website analytics platform without using cookies

#126

Post by JackWritesCode » Fri, Jul 26, 2019, 4:20 AM UTC

labawi wrote: post #125

If visits expire after 30 min, why not rotate the salt every 30 min? Keep current and previous salt, update as needed. I would have more faith in privacy, if you didn't store the salt in the DB or permanent storage. If you manage to statically load-balance the users (e.g. hash site, ip, user-agent, don't forget site), the hash could be in-memory only. Sessions would break on server restart, but that's more of a featu…

The visit expires 30 minutes after the visitor lands. The expiration isn't generic.

It's an interesting idea. We have multiple servers under the load balancers, so we'd be able to store them in Redis, but that is no better than permanent storage, since Redis could still be breached and you'd see it with ease.

Re: How we built a GDPR-compliant website analytics platform without using cookies

#127

Post by gcbw2 » Fri, Jul 26, 2019, 6:50 PM UTC

JackWritesCode wrote: post #123

gcbw2 wrote: post #120

Earlier quoted context omitted.

If a search query on your data would contain all the components of the original hash, i don't have to walk backwards and break the hash. i just have to hash my query terms in the same way. Also I suggested you store the daily hash forever. But even if you really erase it every day, as you say, If you or an attacker makes the same request every day at a predetermined time, when you/they get your logs, you/they can use…

So that needs to be our next target point (access logs). We want to move to a position to keep no access lgos. And a hacker could indeed "win" if they broke into our system, got the salt and exported the DB. We didn't focus on this in our article, as it's unbelievably unrealistic, but it's still possible. Our next step is to address that. Without the hash, it's practically impossible to brute force.

Not talking about a hacker. I am stating that the described hash dance offers no exclusion from GDPR as saying "we promise we won't look" would do.

My point about brute forcing being useless, is that you hold all the information needed to re-create the hash. All but one tiny piece that is the random number. so brute force is a very effective O(). And since it is stored in your locally available data, there is no rate constraints.

Re: How we built a GDPR-compliant website analytics platform without using cookies

#128

Post by JackWritesCode » Sat, Jul 27, 2019, 3:11 AM UTC

gcbw2 wrote: post #127

JackWritesCode wrote: post #123

Earlier quoted context omitted.

So that needs to be our next target point (access logs). We want to move to a position to keep no access lgos. And a hacker could indeed "win" if they broke into our system, got the salt and exported the DB. We didn't focus on this in our article, as it's unbelievably unrealistic, but it's still possible. Our next step is to address that. Without the hash, it's practically impossible to brute force.

Not talking about a hacker. I am stating that the described hash dance offers no exclusion from GDPR as saying "we promise we won't look" would do. My point about brute forcing being useless, is that you hold all the information needed to re-create the hash. All but one tiny piece that is the random number. so brute force is a very effective O( ). And since it is stored in your locally available data, there is no rat…

> I am stating that the described hash dance offers no exclusion from GDPR as saying "we promise we won't look" would do.

Under your logic, you would never trust us because we could just add $log->write(UserIp, UserAgent, Hostname, Path) in plain text. Trust is very important and what you do with the data is important under GDPR.

And we don't hold all the information to re-create the hash, that's the thing.

I thought a lot about "Oh but you could just do this, this and this" but, no, that argument doesn't hold. Our obligation under GDPR is what we actually do with data.

Re: How we built a GDPR-compliant website analytics platform without using cookies

#129

Post by krageon » Mon, Jul 29, 2019, 8:41 AM UTC

orra wrote: post #121

krageon wrote: post #113

Earlier quoted context omitted.

What makes analytics first-party? When the first party serves them, or only if the data never leaves machines under the direct and exclusive control of the first party?

I don't know the answer to that. FWIW, my guess would be that the definition is fairly strict. Now, I don't think that would prevent the first party using data processors, but I suspect the first party would have to exercise a lot of control over the processor. This would be in contrast to a service like Google Analytics, where the company's control and choice is simply limited to take it or leave it.

A data processor agreement is not usually negotiated all that hard and this is indeed not really possible with truly large companies (and let's face it, that's where most companies get their enterprise software). Therefore I feel that expecting a lot of control being exercised is a bit of a pipe dream.