Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Incident Report – DDoS Attack

blog.dnsimple.com

Post reply on HN
Print view View original

31–40 of 40 posts

Re: Incident Report – DDoS Attack

#31

Post by rpug »

aeden wrote: post #10
kator wrote: post #9

> A new customer signed up for our service and brought in multiple domains that were already facing a DDoS attack. The customer had already tried at least 2 other providers before DNSimple. Once the domains were delegated to us, we began receiving the traffic from the DDoS. I'm curious did they know this in advance or discovered it after the fact? I often wonder about business models where the core expense is "unlimi…

Anthony from DNSimple here. We discovered it after the fact, via a tip from other DNS providers.

As someone who has been down this road many times before - I can't stress this enough: DDoS mitigation solutions don't solve the problem of an app-specific layer7 attack and it is important to do some testing of how well your mitigation service responds (and that it isn't a silver bullet.) Additionally, you need to make sure your team has tested and proven procedures for engaging the service, respond to attacks, etc. Services like NimbusDDoS (www.nimbusddos.com) are good because you can do some real scenario testing and make sure your team and infrastructure is prepared. There are other services out there too that I am less familiar with, but either way really good stuff to do.

Re: Incident Report – DDoS Attack

#32

Post by b0k »

latch wrote: post #2

I need to learn to let things go, but: https://news.ycombinator.com/item?id=4280515 I've been a DnsMadeEasy customer for a while (they had an outage ~4 years ago from a 50Gbps attack), but once my year is up, I'm switching to Route53. The addition of the Geo DNS Queries was key for me. It isn't clear to me why I shouldn't pick Route53. DnsSimple's unlimited queries seems nice, but I kinda like having actual scaling c…

I really don't understand why some of these low-grade DNS hosting services are so popular when Route53 is available. With Route53 you get a top-grade DNS service that is equivalent, if not better, than the enterprise hosted DNS solutions but at the price of the low-end consumer style services.

I swear by Route53, it is the only service I use on AWS and I have moved a lot of my clients over to it.

Re: Incident Report – DDoS Attack

#33

Post by b0k »

kator wrote: post #9

> A new customer signed up for our service and brought in multiple domains that were already facing a DDoS attack. The customer had already tried at least 2 other providers before DNSimple. Once the domains were delegated to us, we began receiving the traffic from the DDoS. I'm curious did they know this in advance or discovered it after the fact? I often wonder about business models where the core expense is "unlimi…

"unlimited" plans are subsidised by low utilization users who are getting less than what they paid for.

to pull it off properly as a service provider, you really need to have a solid understanding of user usage patterns.

one of the big problems that tips the low/high utilization ratio unfavorably is that unlimited plans that are primarily marketed for being unlimited tend to attract users in the high utilization bracket.

so the challenge for service providers is not just understanding users and understanding that ratio but figuring out how you are going to market to, and signup, those users who will be in the low utilization bracket and will essentially be paying for something they won't be user (which is hard to do)

it isn't hard to find case studies of companies that launch optimistically with one pricing plan around unlimited, to then only go back and revise their pricing and break promises because they didn't understand their users and were unable to market to and signup low utilization users.

one recent example is Bitcasa

Re: Incident Report – DDoS Attack

#34

Post by llasram »

brownbat wrote: post #22
iancarroll wrote: post #18

Earlier quoted context omitted.

I was looking at http://map.ipviking.com earlier and it was apparent it was a botnet, most likely innocent home users with a virus.

It'd be nice if IPs involved in botnet DDoS's could go into a public registry, then get a banner from Google saying, "Hey, you might have a virus, someone reported you to this list." Abuse would be tricky, you might be able to limit it by letting only a few DDoS mitigation providers populate the list.

This is actually one of the main uses for the ISP/telco product appliance sold by my employer, Damballa. The appliance reports client IPs which appear to be infected with malware to the ISP, who then reports this their affected customers by whatever mechanism the ISP prefers.

This particular DDoS I actually believe is _not_ due to a botnet, or at least believe there is insufficient evidence either way. The attack appears to be using a technique/infrastructure I’ve been passively tracking for nearly a year, wherein the attack DNS requests are spoofed to appear from seemingly-random clients and sent to open recursive DNS servers across the Internet. This makes the attack look like a botnet to superficial analysis on the target side, but this isn’t necessarily the case. In the small amount of time I’ve so-far invested in trying to track down the origin, I have yet to observe generation of the initial query packets.

Re: Incident Report – DDoS Attack

#35

Post by bobbykjack »

coldpie wrote: post #21
iancarroll wrote: post #18

Earlier quoted context omitted.

I was looking at http://map.ipviking.com earlier and it was apparent it was a botnet, most likely innocent home users with a virus.

Remember to keep any machine under your control up to date! I'm looking at you, XP die-hards. If you're able to, monitor your network traffic periodically as well.

It's easy to toss out statements like "monitor your network traffic"; do you have any good suggestions for how an average developer with relatively little understanding of networking can go about doing so?

Re: Incident Report – DDoS Attack

#36

Post by fapjacks »

b0k wrote: post #32
latch wrote: post #2

I need to learn to let things go, but: https://news.ycombinator.com/item?id=4280515 I've been a DnsMadeEasy customer for a while (they had an outage ~4 years ago from a 50Gbps attack), but once my year is up, I'm switching to Route53. The addition of the Geo DNS Queries was key for me. It isn't clear to me why I shouldn't pick Route53. DnsSimple's unlimited queries seems nice, but I kinda like having actual scaling c…

I really don't understand why some of these low-grade DNS hosting services are so popular when Route53 is available. With Route53 you get a top-grade DNS service that is equivalent, if not better, than the enterprise hosted DNS solutions but at the price of the low-end consumer style services. I swear by Route53, it is the only service I use on AWS and I have moved a lot of my clients over to it.

I agree, but there are some low-end DNS providers which have good services that give you more "domains" for less than what it costs with Route53. I use Route53 for a lot of my sites, but for tiny client sites (and personal stuff which has lots of domains), it's hard to beat $60 a year for 25 domains at DNSMadeEasy. That's less than half of what it costs to use AWS for the same number of zones. Granted, the price drops after those 25 on Route53, so if you have thousands of zones in one account, best to use Route53. Just as an example of an edge case.

Re: Incident Report – DDoS Attack

#37

Post by jamespo »

bobbykjack wrote: post #35
coldpie wrote: post #21

Earlier quoted context omitted.

Remember to keep any machine under your control up to date! I'm looking at you, XP die-hards. If you're able to, monitor your network traffic periodically as well.

It's easy to toss out statements like "monitor your network traffic"; do you have any good suggestions for how an average developer with relatively little understanding of networking can go about doing so?

Glasswire for Windows or Little Snitch for Mac?

Re: Incident Report – DDoS Attack

#38

Post by thaumaturgy »

brownbat wrote: post #22
iancarroll wrote: post #18

Earlier quoted context omitted.

I was looking at http://map.ipviking.com earlier and it was apparent it was a botnet, most likely innocent home users with a virus.

It'd be nice if IPs involved in botnet DDoS's could go into a public registry, then get a banner from Google saying, "Hey, you might have a virus, someone reported you to this list." Abuse would be tricky, you might be able to limit it by letting only a few DDoS mitigation providers populate the list.

> banner from Google saying, "Hey, you might have a virus, someone reported you to this list."

Unfortunately this is already in use with some malicious ads as well as phone scams to get people to give remote access to overseas tech centers that then scam them into paying good money for nothing.

To date the only tech line about this is, "nobody legitimate will ever contact you to tell you you're infected with a virus."

So I don't know how you could develop trust in that environment.

Re: Incident Report – DDoS Attack

#39

Post by coldpie »

bobbykjack wrote: post #35
coldpie wrote: post #21

Earlier quoted context omitted.

Remember to keep any machine under your control up to date! I'm looking at you, XP die-hards. If you're able to, monitor your network traffic periodically as well.

It's easy to toss out statements like "monitor your network traffic"; do you have any good suggestions for how an average developer with relatively little understanding of networking can go about doing so?

Well, my router with dd-wrt just gives me a traffic diagram. I don't check it super often, admittedly. I wonder if it could be modified to signal a warning somehow?

Re: Incident Report – DDoS Attack

#40

Post by crystaln »

ad_hominem wrote: post #25
crystaln wrote: post #6

Earlier quoted context omitted.

Hopefully not. CloudFlare is remarkably unreliable for a service that claims to improve uptime.

[citation needed] Last I checked CloudFlare routinely handles[1] 10Gbps to 65Gbps attacks, and has successfully handled attacks as large as 300Gbps and 400Gbps. According to this report DNSSimple crumbled under 25Gbps. [1]: https://support.cloudflare.com/hc/en-us/articles/200170216-H...

We moved off of CloudFlare because of repeated outages and bugs. Our uptime improved significantly without CloudFlare.

Granted we are probably more vulnerable to DoS, but our general uptime is far better now.

Post reply on HN