Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Someone attacked our company

usefathom.com

Post reply on HN
Print view View original

101–110 of 112 posts

Re: Someone attacked our company

#101

Post by throwaway189262 »

JackWritesCode wrote: post #24
hartator wrote: post #21

PHP Laravel on Amazon Lambda to count pageviews? Are you sure it's not just a regular customer and not a DDOS? > privacy-first analytics solution [...] The only downside of this is that we need to keep access logs (IP & User-Agent, no browsing history) for 24 hours Keeping IPs don't make you super privacy friendly.

Awesome, I'd love some advice if you're willing. So here's our situation. * We're getting hit with a huge DDoS attack, repeatedly over 3 weeks, with no sign of stopping * With zero access logs, there was no way to find patterns in the attack, and we had no way to block it * Our service was going offline during these attacks * We introduced access logs that are auto-deleted after 24 hours. We redacted all information…

API spam is a big problem in Google Analytics too, you might just be big enough now to have become a target. I think the answer is to build a spam filter. You could take the ones made for email and adapt them for analytics requests possibly. This seems like a real place where ML pattern recognition models would shine. Any realistic spam filter will involve sampling a fraction of traffic for deep analysis and IP banning badly behaving subnets for ~24h. And it's definitely better to shadow ban, but smart assholes will have an account to see if their spam is getting through as well.

I wouldn't worry about short-term IP caching. AWS's upstream load balancers and your own servers are probably doing it anyways to maintain TCP state tables. Linux kernel's "conntrack". If you don't want to cache IP's you can you a probabilistic data structure like a bloom filter synched between instances. If has a small false positive rate but is very fast and doesn't store whole IP. Bloom filter based IP filtering is used in every big DDOS prevention system I know of.

As much as you like lambda, I would ditch it. And the queues. My general advice is that any time you need to add a work queue to something, it's not fast enough.

Your analytics endpoint data ingestion should be something lightning fast like Go, Rust, or an async Java back end. Analytics is a lossy process, you lose traces because of browser behavior and plugins all the time anyways so I wouldn't prioritize 100% accuracy.

I would focus on power/dollar over reliability. If I was you, my ingest boxes would be load balanced with DNS round robin and sitting at various Colocation providers. Get a fat 40 gig unlimited data pipe. Build some stupid fast Rust/Go/Java backend that can saturate that pipe. And do all your filtering/spam analysis here.

I don't think lambda, SQS queues, PHP are the best technologies for this kind of mass data ingestion. I don't even think your ingestion layer should be on AWS. I would follow the lead of other companies doing mass data ingestion and build your own machines. That's how CloudFlare, Netflix etc are able to handle so much traffic without going bankrupt.

I would consider yourself lucky that your first DDOS was so small. 10k requests/sec is tiny. ~400k/sec can be generated on a regular desktop with fiber internet connection. Right now, a single user could knock you offline by messing around with JMeter. I think it's a wake up call that you're in the infrastructure business whether you like it or not, and you need to massively beef up your data ingestion layer. Realistically, you should be able to handle ~50 gigabit attack with 10 million requests/sec. I think that's achievable with a couple boxes colocated on 40 gig lines running fast software

Re: Someone attacked our company

#102

Post by throwaway189262 »

jusssi wrote: post #48
hartator wrote: post #28

Earlier quoted context omitted.

> * We're getting hit with a huge DDoS attack, repeatedly over 3 weeks, with no sign of stopping I've read the full blog post, I am not convinced it's a DDoS attack. Traffic patterns for web analytics will come from over the place and will look like a DDoS when it's not. For example, a customer misplacing their analytics in a JS loop and having a moderate traffic blog will generate billions of requests from all over…

> You can one-way hash the IP. So you can still look for pattern but you've lost the actual IP. And same one-way hash IP can block whichever IP seems devious in your firewall. The plaintext space (amount of possible IPs, even more so for IP blocks) is so small that that you can try all the possible plaintexts within seconds, essentially reversing the hash.

You can store "possible attacking" IP's in a counting Bloom filter and start blocking when it saturates. Random false positives are probably not an issue for an analytics service that's inherently lossy anyways.

By using Bloom filter you avoid storing IP's, and save tons of space. They're basically required for rate limiting in IPV6 where the address space is so large an attacker can just run you out of ram by using so many different IP's.

It's also pretty common to have several thresholds of blocking based on individual IP vs subnets. So if more than 2 ips get blocked in a small subnet, you block the whole subnet.

Re: Someone attacked our company

#103

Post by throwaway189262 »

JackWritesCode wrote: post #38
hartator wrote: post #37

Earlier quoted context omitted.

> It was 100% a layer 7 DDoS, the attack was targeted and malicious. I can't say too much but the AWS team confirmed it. But I appreciate how something like you describe could happen. Layer 7 just means they are making ton of HTTP requests from lot of IPS however this is the nature of a web analytics. > we know that an IP visited one of the tens of thousands of websites Fathom runs on, but we don't know which one. Ho…

> Layer 7 just means they are making ton of HTTP requests from lot of IPS however this is the nature of a web analytics. Exactly! Which is why it was so hard. There's no path pattern. Everything hits "/". So the only way to fight back is to match IP / header patterns (but even then, we have to redact sensitive headers). > How can the attackers know which websites have your analytics on? Crawling the web is super hard…

> The problem is, without some kind of firewall (e.g. WAF), our application has to absorb so much traffic, and that's the issue. We need to block it at the edge.

I think you need some high performance ingestion code.

One day somebody is going to load test or misconfigure their site. Or you'll get a really big client. Or somebody will get Reddit frontpaged or slashdotted. Blocking huge volumes of traffic isn't always going to be an option. You should be able to handle it.

I would get off all this "sexy" stuff like lamdba and SQS and build some old school battle boxes at a co-lo. Run some extreme performance framework like Actix or Vert.X built to handle giant piles of traffic. A single box with those frameworks and a 40 gig line can handle millions of requests a second.

Use counting bloom filters synced between boxes for IP blocking. This avoids saving IP's and is needed to prevent ram exhaustion attacks anyways. Use two layers, one for individual IP's and a second for blocking subnets with lots of bad actors in them. Block for ~24h by using dual sets of bloom filters in a sliding window with 12h overlap where IP's get added to both. When a filter is 24h old, discard it. This needs to be done because you can't remove items from Bloom filters, they saturate over time.

On these super boxes you can do filter, blocking, batching, and dedup. Then send the data off to wherever for further processing.

You mentioned not wanting to use Cloudflare because they're a competitor. That's fine but I would take a look at their blog. They go over the tech they use for mass data ingestion and filtering, and it's basically what I just described.

Re: Someone attacked our company

#104

Post by miyuru »

throwaway189262 wrote: post #101
JackWritesCode wrote: post #24

Earlier quoted context omitted.

Awesome, I'd love some advice if you're willing. So here's our situation. * We're getting hit with a huge DDoS attack, repeatedly over 3 weeks, with no sign of stopping * With zero access logs, there was no way to find patterns in the attack, and we had no way to block it * Our service was going offline during these attacks * We introduced access logs that are auto-deleted after 24 hours. We redacted all information…

API spam is a big problem in Google Analytics too, you might just be big enough now to have become a target. I think the answer is to build a spam filter. You could take the ones made for email and adapt them for analytics requests possibly. This seems like a real place where ML pattern recognition models would shine. Any realistic spam filter will involve sampling a fraction of traffic for deep analysis and IP banni…

yes, they really need to architecture their service. they don't seem to have a reliable scaling up plan.

I've had a similar ddos attack which manged to bypass the cloudflare protection and our app handled it without $0 increase in bill.

Re: Someone attacked our company

#105

Post by knolax »

exolymph wrote: post #96
toyg wrote: post #78

> We will not let a lonely nerd attack our business How do they even know it was "a lonely nerd"? In fact, it's much more likely that it was carried out by a well-socialized team of shady professionals, on a commission from competitors. It's 2020, people, can we drop it with the "Hack3rs" stereotypes...?

> It's 2020, people, can we drop it with the "Hack3rs" stereotypes...? Well, stereotype accuracy is one of the most robust and replicable findings from psychology research...

Citing the same field of study that came up with and prescribed lobotomies doesn't help your point.

Re: Someone attacked our company

#106

Post by throwaway189262 »

miyuru wrote: post #104
throwaway189262 wrote: post #101

Earlier quoted context omitted.

API spam is a big problem in Google Analytics too, you might just be big enough now to have become a target. I think the answer is to build a spam filter. You could take the ones made for email and adapt them for analytics requests possibly. This seems like a real place where ML pattern recognition models would shine. Any realistic spam filter will involve sampling a fraction of traffic for deep analysis and IP banni…

yes, they really need to architecture their service. they don't seem to have a reliable scaling up plan. I've had a similar ddos attack which manged to bypass the cloudflare protection and our app handled it without $0 increase in bill.

Yeah. A single big client or even a viral video on a single page will knock out their systems. 10k requests/sec when there's 3 billion people surfing the web isn't enough. I would hate to be in their position right now. It's gonna be tough to engineer this with a 2 man team

Re: Someone attacked our company

#107

Post by whalesalad »

parliament32 wrote: post #95
whalesalad wrote: post #4

Interesting that they won’t use Cloudflare due to competition in the analytics space. It makes me realize that there actually isn’t a legitimate competitor to Cloudflare and they are dominating right now.

There isn't any competition in the free-ish space, no. But there are some big-boy options: Incapsula/Imperva, DOSarrest, Akamai, SiteLock, Radware, and a few other smaller players. These are complex setups where you at minimum need to be running an AS and own your IP space and infrastructure -- you get all sorts of cool solutions like sending (router) flows to them to monitor your traffic, and when an attack is detec…

You had me until the last sentence. I would urge you to do an nslookup on the top 10,000 alexa domains. You would be surprised at how many point to Cloudflare DNS.

Re: Someone attacked our company

#108

Post by joshmanders »

JackWritesCode wrote: post #85
hartator wrote: post #81

Earlier quoted context omitted.

You have to consider OP might have been misled by the Amazon Sales team to buy their $3k per month DDoS protection.

Never expected to see someone defending me on Hacker News. I appreciate it. To clear things up: * I found AWS Shield Advanced organically * It was a DDoS attack * I am incredibly happy with the personalized service. It’s like hiring someone to handle it, except you have people on call 24x7

> It’s like hiring someone to handle it, except you have people on call 24x7

This is what confuses me out of a lot of these replies, saying you're being swindled for paying for this "absurd service" when you should just "own your infra yourself and hire people to manage this"

Do they think owning all this yourself and hiring specialized DDoS people is gonna cost less than $36k/yr? Because I don't think it will.

Re: Someone attacked our company

#109

Post by matsur »

parliament32 wrote: post #95
whalesalad wrote: post #4

Interesting that they won’t use Cloudflare due to competition in the analytics space. It makes me realize that there actually isn’t a legitimate competitor to Cloudflare and they are dominating right now.

There isn't any competition in the free-ish space, no. But there are some big-boy options: Incapsula/Imperva, DOSarrest, Akamai, SiteLock, Radware, and a few other smaller players. These are complex setups where you at minimum need to be running an AS and own your IP space and infrastructure -- you get all sorts of cool solutions like sending (router) flows to them to monitor your traffic, and when an attack is detec…

I work at Cloudflare. We offer protection for the "complex setups" you describe with BYO ASN, IP packet filtering + forwarding, and GRE (or PNI) for traffic return. And we're very good at it.

https://www.cloudflare.com/case-studies/wikimedia-foundation is an example deployment covering Wikimedia's datacenters.

Re: Someone attacked our company

#110

Post by tatersolid »

hartator wrote: post #80
jusssi wrote: post #48

Earlier quoted context omitted.

> You can one-way hash the IP. So you can still look for pattern but you've lost the actual IP. And same one-way hash IP can block whichever IP seems devious in your firewall. The plaintext space (amount of possible IPs, even more so for IP blocks) is so small that that you can try all the possible plaintexts within seconds, essentially reversing the hash.

Then a longer to run hash function, like 256 times sha256 with a secret salt. It's reversible in theory but it will take a lot of resources.

You can’t afford to do that to every 1500-byte packet at 10-100 Gbps.
Post reply on HN