Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Someone attacked our company

usefathom.com

Post reply on HN
Print view View original

31–40 of 112 posts

Re: Someone attacked our company

#31

Post by iampims »

JackWritesCode wrote: post #13
cperciva wrote: post #12

Earlier quoted context omitted.

Well, it's your money... I'd be happy to help make this more efficient though (at no charge of course). Offhand I'd say "web server which accumulates data and uploads it to S3 every N requests or M seconds" would probably get you what you need at a tiny fraction of the cost of "lambda which posts to SQS". Create an AMI and toss it at an autoscaling group and you really won't need to worry about scaling issues either.

I bet you could set up something way cheaper. And I'm certain you know 10x more than me regarding servers, hardening, configuration, etc. And I'm certain you enjoy servers! For us, the cost works and we have appropriate margin for it. The cost savings aren't worth the extra "we have to monitor these servers" thoughts. Our approach is 100% emotional.

ApiGateway -> kinesis stream -> lambda consumes batch of N page views. You keep the serverless approach you like, and it will cost you a fraction of what you currently spend.

Re: Someone attacked our company

#32

Post by JackWritesCode »

hartator wrote: post #28
JackWritesCode wrote: post #24

Earlier quoted context omitted.

Awesome, I'd love some advice if you're willing. So here's our situation. * We're getting hit with a huge DDoS attack, repeatedly over 3 weeks, with no sign of stopping * With zero access logs, there was no way to find patterns in the attack, and we had no way to block it * Our service was going offline during these attacks * We introduced access logs that are auto-deleted after 24 hours. We redacted all information…

> * We're getting hit with a huge DDoS attack, repeatedly over 3 weeks, with no sign of stopping I've read the full blog post, I am not convinced it's a DDoS attack. Traffic patterns for web analytics will come from over the place and will look like a DDoS when it's not. For example, a customer misplacing their analytics in a JS loop and having a moderate traffic blog will generate billions of requests from all over…

It was 100% a layer 7 DDoS, the attack was targeted and malicious. I can't say too much but the AWS team confirmed it. But I appreciate how something like you describe could happen.

I like the idea of an MD5 hash. Although I'm not too certain why an IP would be a bad thing to log for 24 hours. From a privacy law perspective, the MD5 hash is considered PII. And if we see an IP address in an access log, we know that an IP visited one of the tens of thousands of websites Fathom runs on, but we don't know which one.

Edit: Something else, with MD5 there's no way of finding patterns with the IPs, so you'd have to play whack a mole. Whereas raw IPs allow no real privacy invasion whilst allowing pattern detection

Re: Someone attacked our company

#33

Post by JackWritesCode »

ve55 wrote: post #26
JackWritesCode wrote: post #24

Earlier quoted context omitted.

Awesome, I'd love some advice if you're willing. So here's our situation. * We're getting hit with a huge DDoS attack, repeatedly over 3 weeks, with no sign of stopping * With zero access logs, there was no way to find patterns in the attack, and we had no way to block it * Our service was going offline during these attacks * We introduced access logs that are auto-deleted after 24 hours. We redacted all information…

I would probably start by looking for third-parties to help manage to problems, at least unless they all end up at the application layer. Throw it beyond Cloudflare and talk to them, and if they're not the right fit, try someone else.

They're still going to need to process IP addresses or some kind of PII though, that's the thing :(

Re: Someone attacked our company

#34

Post by JackWritesCode »

iampims wrote: post #31
JackWritesCode wrote: post #13

Earlier quoted context omitted.

I bet you could set up something way cheaper. And I'm certain you know 10x more than me regarding servers, hardening, configuration, etc. And I'm certain you enjoy servers! For us, the cost works and we have appropriate margin for it. The cost savings aren't worth the extra "we have to monitor these servers" thoughts. Our approach is 100% emotional.

ApiGateway -> kinesis stream -> lambda consumes batch of N page views. You keep the serverless approach you like, and it will cost you a fraction of what you currently spend.

I saw Kinesis stream today with DynamoDB and I will have to review it :)

Re: Someone attacked our company

#36

Post by JackWritesCode »

fargle wrote: post #35
zapstar wrote: post #3

What would the motivation be for such an attack? Any thoughts on who could be behind it?

Perhaps AWS needs to sell AWS Shield Advanced? Ha ha, I'm probably too paranoid.

You and me both, fargle. When the attacks started, I thought someone was attacking us just so I would write a technical blog post.

Re: Someone attacked our company

#37

Post by hartator »

JackWritesCode wrote: post #32
hartator wrote: post #28

Earlier quoted context omitted.

> * We're getting hit with a huge DDoS attack, repeatedly over 3 weeks, with no sign of stopping I've read the full blog post, I am not convinced it's a DDoS attack. Traffic patterns for web analytics will come from over the place and will look like a DDoS when it's not. For example, a customer misplacing their analytics in a JS loop and having a moderate traffic blog will generate billions of requests from all over…

It was 100% a layer 7 DDoS, the attack was targeted and malicious. I can't say too much but the AWS team confirmed it. But I appreciate how something like you describe could happen. I like the idea of an MD5 hash. Although I'm not too certain why an IP would be a bad thing to log for 24 hours. From a privacy law perspective, the MD5 hash is considered PII. And if we see an IP address in an access log, we know that an…

> It was 100% a layer 7 DDoS, the attack was targeted and malicious. I can't say too much but the AWS team confirmed it. But I appreciate how something like you describe could happen.

Layer 7 just means they are making ton of HTTP requests from lot of IPS however this is the nature of a web analytics.

> we know that an IP visited one of the tens of thousands of websites Fathom runs on, but we don't know which one.

How can the attackers know which websites have your analytics on? Crawling the web is super hard.

You should be able to get back which accounts are affected from your analytics call:

    https://starman.fathomdns.com/?p=%2F&h=https%3A%2F%2Fusefathom.com&r=&sid=BIABKBRK&res=1440x900
Isn't sid=BIABKBRK the account? The access log should store this `sid` allowing you to just block the account. (or ask them for ton of money if it's a legitimate use :) )

Re: Someone attacked our company

#38

Post by JackWritesCode »

hartator wrote: post #37
JackWritesCode wrote: post #32

Earlier quoted context omitted.

It was 100% a layer 7 DDoS, the attack was targeted and malicious. I can't say too much but the AWS team confirmed it. But I appreciate how something like you describe could happen. I like the idea of an MD5 hash. Although I'm not too certain why an IP would be a bad thing to log for 24 hours. From a privacy law perspective, the MD5 hash is considered PII. And if we see an IP address in an access log, we know that an…

> It was 100% a layer 7 DDoS, the attack was targeted and malicious. I can't say too much but the AWS team confirmed it. But I appreciate how something like you describe could happen. Layer 7 just means they are making ton of HTTP requests from lot of IPS however this is the nature of a web analytics. > we know that an IP visited one of the tens of thousands of websites Fathom runs on, but we don't know which one. Ho…

> Layer 7 just means they are making ton of HTTP requests from lot of IPS however this is the nature of a web analytics.

Exactly! Which is why it was so hard. There's no path pattern. Everything hits "/". So the only way to fight back is to match IP / header patterns (but even then, we have to redact sensitive headers).

> How can the attackers know which websites have your analytics on? Crawling the web is super hard.

The attacker went after some of our more high profile customers. They're known via testimonials or from Twitter.

> The access log should store this `sid` allowing you to just block the account. (or ask them for ton of money if it's a legitimate use :) )

We could certainly temporarily block traffic to a site. The problem is, without some kind of firewall (e.g. WAF), our application has to absorb so much traffic, and that's the issue. We need to block it at the edge.

Re: Someone attacked our company

#40

Post by redis_mlc »

PradeetPatel wrote: post #39

It has been proposed that having a robust monitoring and alerting system would significantly reduce the impact of a DDoS. What would be an advisable approach for a start up with limited resource in a situation like this?

Usually, no.

Monitoring and alerting will tell you of a problem, but that doesn't fix anything by itself. You either need to ask your ISP to "do something about it but don't blackhole my server(s)", or DIY.

Post reply on HN