Live data from Hacker News

My first DDoS attack for a $200 ransom

ghirardotti.fr

51–60 of 67 posts

Re: My first DDoS attack for a $200 ransom

#51
post #11
post #8

Roughly, a somewhat lackluster response to a somewhat lackluster DDoS attempt. They tried blocking specific ip addresses, which didn't work, because the attack was somewhat distributed. They then just turned on some caching, which allowed the site to function, albeit with an unknown excess bandwidth charge pending. And, the DDoS itself can't of been terribly impressive, as all it took to mitigate was a bit of caching…

Thinking on this some more, this story makes even less sense. He first mentions having to change Apache to recognize X-Forwarded-For, because there is Amazon Elastic Load Balancing between his site and the internet. This means, of course, that the "attacking ips" aren't making direct connections to his EC2 instance. They are proxied connections, all from the internal ELB service. So later, when he mentions trying to…

I think you're absolutely right for iptables, as I didn't changed anything to use the X-Forwarded-For IP, so this part might have been fully useless. About the EC2, it's because it's managing the SSL for us, and we used to have two servers behind it.

Everything could have been planed way better (cached, written with a fancy language...), we could have had 10 mil requests/µsec... the main idea was just to get tell how we tried to manage the situation, with the website and skill we have.

I also think my testimony is nearer to what most of web dev can be confronted to, in contrary to one Cloudflare/Gihub BS press release written by 10 experts to increase valuation :p

Re: My first DDoS attack for a $200 ransom

#52
post #26
post #11

Earlier quoted context omitted.

Thinking on this some more, this story makes even less sense. He first mentions having to change Apache to recognize X-Forwarded-For, because there is Amazon Elastic Load Balancing between his site and the internet. This means, of course, that the "attacking ips" aren't making direct connections to his EC2 instance. They are proxied connections, all from the internal ELB service. So later, when he mentions trying to…

You might use ELB + one EC2 to serve an SSL certificate. That takes the encryption load off of EC2 and is durable. AWS has a new SSL service though, but this was a recommended way until recently.

Exactly our situation :)

Re: My first DDoS attack for a $200 ransom

#53
post #8

Roughly, a somewhat lackluster response to a somewhat lackluster DDoS attempt. They tried blocking specific ip addresses, which didn't work, because the attack was somewhat distributed. They then just turned on some caching, which allowed the site to function, albeit with an unknown excess bandwidth charge pending. And, the DDoS itself can't of been terribly impressive, as all it took to mitigate was a bit of caching…

I was shocked that 12 requests/second could take down any site. I use async logic (previously OpenResty, more recently NodeJS and Go) and largely pregenerated sites, so 2500 requests/second is a minimum baseline -- on a much lower end instance than an m4.xlarge. There's a reason I don't use PHP (or any primarily synchronous language like Ruby) any more.

But can you do pregenerated sites all the time? Like in my case I have a search page, and some dynamic things, I should generate a huge amount of pages (but while writing this I'm thinking that it could be feasible though a bit complicated to regenerate in numerous situations)

Re: My first DDoS attack for a $200 ransom

#54

This is an amazingly weak DDoS, put your site behind CloudFlare or similar free service and go take a nap. They'll tank this without raising an eyebrow.

Yep, true, it's planned. But sometimes their captcha page tend to block some legitimate trafic...

It's not that impressive because we read everyday articles about crazy DDoS big companies are able to mitigate. But when it's the website your responsible for, whatever the number of requests/sec, you just need to find way to manage it, and CloudFlare can have some weird side effects.

Re: My first DDoS attack for a $200 ransom

#55

Ummmm.... A cache layer for any web application is a must have, perhaps he could have avoided the attack all along if it were present on the system since day one?... At least for this kind of attack, a more serious DDoS won't be tamed by "just adding cache"

Well, when your DDOS is '10 requests a second', your site is probably not the most sophisticated.

Haha, true! But please share your personal DDoS experience is mine is not large enough :p

Re: My first DDoS attack for a $200 ransom

#56

Earlier quoted context omitted.

[ Edit: I'm almost certain it's an ipv6 vs ipv4 issue. the ipv4 addresses resolve to github pages land, the ipv6 address resolves to somewhere inside OVH - the issue being that if the viewer's network infrastructure prefers ipv6, they will get a holding page from OVH stating that that "Site not installed / The site ghirardotti.fr is not yet installed" ] Dig: dig ghirardotti.fr ;; ->>HEADER Both ip's belong to Github,…

If they're using OVH they already have free DDoS protection. I use OVH myself and I've had a few mitigated DDoS attacks that I barely noticed. Perhaps 10 requests/sec is below OVH's detection threshold. My last one was 8.4Gbps @ 1 million packets per sec. 10 requests/sec would be difficult to even notice :)

Nope, OVH is just for our domain name. Also is talking about my blog, not https://www.villa-bali.com website, which is on AWS.

Re: My first DDoS attack for a $200 ransom

#57

> 40 cores [m4.10xlarge], but still unable to process 10 requests/sec my goodness.

That's php for you. Although I use php myself quite often, it can be a resource hog if you're lazy about optimization. A customer I was working with was using wordpress, and their homepage took about 5 seconds to load due to a hideously inefficient wordpress module that was doing the exact same sql query thousands of times! With a little bit of optimization I managed to get it down to about 1 or 2 seconds. For my own…

I came to make the same point - but PHP itself is not directly the problem.

There are many, very popular Wordpress plugins that take hundreds of SQL queries just to render a landing page. 10 hits/sec legitimately is "DDoS" territory for many businesses running such things.

Re: My first DDoS attack for a $200 ransom

#58
post #47

Earlier quoted context omitted.

Strawman. For tasks that are primarily IO bound, async architectures can scale more than synchronous languages. Period. It would take intentionally (or newb/cluelessly) bad design to end up with a Node server that DOSed at 12 connections per second. In PHP, if you run 4 threads, it just takes a backend with a 333ms latency (on all queries performed serially) to limit you to 12 connections per second. If you only run…

>>It would take intentionally (or newb/cluelessly) bad design to end up with a Node server that DOSed at 12 connections per second. Single threaded makes for it's own pitfalls. I assume you can imagine some cpu bound tasks that would have node.js at 12 connections/sec or less.

>I assume you can imagine some cpu bound tasks that would have node.js at 12 connections/sec or less.

Certainly. That would fall under "newb/clueless" design, though. Anyone who would throw a CPU-bound task into a primary Node server shouldn't be allowed near architectural design decisions. Whereas code in PHP written using best practices can easily end up with a server that can barely hit 100 queries per second.

Imagine, for instance, a situation where the client needs to do 50 requests to the server to render a page [1][2], and each query ends up with 20ms of latency on the PHP side; assuming you're running 8 threads (and the client makes 8 concurrent requests), a single page query could block your server for 125ms. A slow client or network might even block your PHP threads for longer. Node could crunch through ten thousand requests like that per second when running on four CPUs, meaning 125 of these bloated pages rendered per second, compared to ... 8 or less.

Even with decent client pages that can render with ONE server query, a couple dozen database lookups are par for the course, sometimes including an authentication lookup on a third-party OAUTH server. That could be 125ms all by itself, and in PHP your thread is blocked while the lookup happens. With the async model, once the query is off, the server is doing work on other requests until the query has returned data.

Many CPU-bound tasks like "convert an image" are already coded in Node to happen in the background, triggering a callback when they're done so you can then send the result to the client. And in Node it's absolutely trivial to offload any likely CPU-bound task to a microservice, where the NodeJS server just queries the microservice and waits for the result. Which you'd want to do, of course, if a task is CPU-bound, because you would want a faster server than V8 running it anyway. Go would be a likely candidate, and Go handles threading either through light/async threads or via actual threading, as necessary. It's quite awesome.

And if you really can't trust your developer to write code without extensive time-consuming calculations, then make them use Elixir or Erlang. It will use preemptive multitasking at the VM level if a thread takes up too much time, and even if they foolishly write a task that takes hundreds of milliseconds to complete, it will still task swap and serve other clients.

But arguing that pathologically bad code in Node can make it perform as badly as PHP does all the time isn't exactly a ringing endorsement for the language.

[1] In 2014 the average number of objects a web page requested was 112, and seemed to continue to be going up, though I'm assuming a lot of those are static resources and third party requests, like for analytics and ads. http://www.websiteoptimization.com/speed/tweak/average-web-p... I've personally seen pages with 70-80 requests against a PHP backend to render one page.

[2] And I wouldn't call a client page needing 50 requests a best practice, but I'm assuming that we're talking about the server side here, and that we are being forced to deal with an existing client that behaves that way. So call it "best practices on the server."

Re: My first DDoS attack for a $200 ransom

#59
post #48

Earlier quoted context omitted.

I was shocked that 12 requests/second could take down any site. I use async logic (previously OpenResty, more recently NodeJS and Go) and largely pregenerated sites, so 2500 requests/second is a minimum baseline -- on a much lower end instance than an m4.xlarge. There's a reason I don't use PHP (or any primarily synchronous language like Ruby) any more.

Put a database behind it with complicated enough requests, and you'll run out of connections to the db regardless if you have async io. You'll have to add a connection pooler or multiple master/slave dbs - and even then you will run into problems with enough traffic. Not everything is sending back some json from something that is readily available.

To really scale a complex architecture, certainly there are other things you need to do, like setting up database replication and/or sharding, caching, and other scaling strategies.

That can typically be done from Node by simply adding the "use pooling" option in your database library (or sometimes by switching to the NPM for that database that enables pooling), and when you have additional slave databases, adding those to the database init call.

As far as Node is concerned, it really is that easy. Scaling PHP, though, pretty much requires that you add more threads, which means (after a point) you need a lot of RAM, or just adding more instances and load balancing between them.

Node won't prevent database scaling issues, but it will keep the part that it does handle a lot easier to maintain.

Re: My first DDoS attack for a $200 ransom

#60
post #45

Earlier quoted context omitted.

That 2500 is after taking into account relevant database queries. With async servers (which, as far as I know, PHP can't be, at least with the way it typically integrates into Apache) you can accept all the requests, forward off the database requests each depends on, and still send back the pages to everyone who requested one. Of course I've avoided WordPress and similar frameworks for years for a reason as well; I'v…

node.js does have it's advantages, but it's possible to be very fast with php. For example, the advantage of the "warm app" with node is often approximated by using a shared memory kv store in php. And, while an async io approach scales in a simpler way, you can typically find an optimum tuning for fastcgi that scales very well. Looking at this benchmark: https://www.techempower.com/benchmarks/#section=data-r12&hw=..…

>benchmarks are sometimes bullshit, but the idea that node's approach is somehow light years ahead just doesn't pan out in the real world.

It's in the real world where you have database connections that can't always scale to 186k/second with low latency on all responses, which is what's required to get the performance out of HHVM in the linked benchmark. In a typical architecture you may not have the database local to the PHP server, meaning latency will be much higher. And it's upstream latency that kills a synchronous connection.

And it's in the real world where a number of third party queries may be involved with a request, and those queries may take 100ms to resolve, during which time your thread is blocked in PHP whereas Node can be busy handling other requests or even other aspects of the same request (a Promise.all of a half dozen simultaneous database queries, for instance).

Async approaches are light years ahead in the real world. I keep seeing references to HHVM when people try to defend PHP, so I did a Google search to find out if it supports async, and the answer is "a little bit." [1] Basically it looks like, within a single request, it can execute several queries in parallel, like my Promise.all example above. So it looks like Hack, at least, has that feature. But as far as I can tell it doesn't mix multiple connections in a single thread. And you have to be careful to use only async-aware operations or you lose the benefits; Node is designed around async behavior, so everything you're likely to use supports it by default.

Those benchmarks are in fact unrealistic. And the Node implementation in that benchmark uses a single connection to MySQL, so the 20 queries are actually executed in series instead of in parallel. [0] If they used a connection pool instead, they could all execute in parallel. Look at the numbers:

Queries 1 5 10 15 20 nodejs 85,490 22,917 12,083 8,250 6,254 hhvm 12,369 12,428 10,056 10,394 9,322

The NodeJS results shouldn't drop that fast off of the single query unless they're all using the same MySQL connection; even using just 4 connections from a pool should speed it up by a lot. Also, the HHVM source uses stored procedures while the Node version recompiles the query every time. Finally, Node can be faster when driven from Nginx, while it's using the internal Node server instead.

Here's an article I just stumbled across that talks about the limits of HHVM to accelerate PHP: [2] It touches on some of the same points. Async is the important way to speed up real world apps, and it's not the default "PHP Way."

Aside from that, probably 90% of the people using PHP are using it "normal" PHP on a hosted server in Apache and not HHVM at all. So you're basically arguing hypotheticals: "IF they use HHVM, and IF they write the code exactly right, and IF their upstream servers have really low latency all the time, then PHP isn't much slower than Node."

Node is usually faster in the real world given common coding patterns. And Node gives you Socket.IO, which pretty much kills the relevance of PHP no matter how you slice it. Even long polling would slaughter a PHP server; you'd be able to support at most one concurrent user per thread. Async servers are a good at supporting long running connections.

[0] https://github.com/TechEmpower/FrameworkBenchmarks/blob/mast...

[1] https://docs.hhvm.com/hack/async/introduction

[2] http://www.infoworld.com/article/2948132/app-servers/hhvm-38...

Post reply on HN