Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

HTTP/2 rapid reset attack impacting Nginx products

nginx.com

Post reply on HN
Print view View original

51–60 of 66 posts

Re: HTTP/2 rapid reset attack impacting Nginx products

#51

Post by kuekacang »

tedunangst wrote: post #43
j16sdiz wrote: post #38

Earlier quoted context omitted.

I guess the parent post want to know if there are any _specific_ and _effective_ change for this kind of attacks. In this case, simplifying the protocol won't help -- the vulnerability is: 1. Backend servers can't cancel immediately (this is no protocol problem) 2. The client can make concurrent request in a connection (This is the goal of Http/2) 3. The concurrency is pre determined, there is no way for the server t…

You should be able to cancel a request, but you don't get it back (as in, can't send another) until the server acks the cancel.

In agent-like app sure. But how would you detect connections from multiple client instance?

Re: HTTP/2 rapid reset attack impacting Nginx products

#52

Post by knorker »

bullen wrote: post #47

Just use HTTP/1.1, it's the final protocol. Nothing Google or Microsoft does will dethrone it. Forget the browser; use a C or Java client and HTTP. If they block port 80, just use another port. They cannot win.

Who is "they"?

Is there some kind of evil back room conspiracy to make the web faster, using open standards?

Re: HTTP/2 rapid reset attack impacting Nginx products

#53

Post by crote »

j16sdiz wrote: post #38
kelthan wrote: post #34

Earlier quoted context omitted.

Yes, because it's exploitable. Systems that require very high cognitive load on their human operators (whether machines, programming languages, etc.) are alway destined to fail. Human beings are not good at doing boring, repetitious work that requires them to stay focused: lapses will occur. And hackers are going to find and exploit those gaps. So the best way to avoid those problems is to build solutions or specific…

I guess the parent post want to know if there are any _specific_ and _effective_ change for this kind of attacks. In this case, simplifying the protocol won't help -- the vulnerability is: 1. Backend servers can't cancel immediately (this is no protocol problem) 2. The client can make concurrent request in a connection (This is the goal of Http/2) 3. The concurrency is pre determined, there is no way for the server t…

If I understand it correctly, a big part of the problem is that 1) requests which are in the process of being cancelled are not counted towards the concurrency limit, and 2) you can create and cancel a request in the same package.

1 allows you to have more pending requests than intended, making some form of DDoS possible. 2 allows it to trivially scale to hundreds of requests per packet rather than just the pending-stream limit, limited only by the packet size.

Disallowing 2 should be fairly trivial, as there is no valid reason to cancel a request in the same packet you started it. I'd consider it more of an implementation bug than a protocol problem.

Issue 1 is definitely a protocol problem though, and it's going to be a bit trickier to fix as it would require nontrivial changes to the request state machine. A fix would require subtracting a request from the pending-stream count not when it is cancelled, but when its resources have been fully cleaned up - and ideally you'd even add some sort of throttling on that to make it even harder to abuse.

Re: HTTP/2 rapid reset attack impacting Nginx products

#54

Post by crote »

gshulegaard wrote: post #30
hsbauauvhabzb wrote: post #23

Earlier quoted context omitted.

I think the important distinction is ‘a user may plausibly have this non default configure’ vs ‘this config is obscure almost nobody will be running it this way’

I am not sure I understand how stream limit configuration between two L4/L7 load balancers is meaningfully different. In my mind, either the configuration of stream limits is a vulnerability for all L4/L7 load balancers that offer that configuration or it's not for all of them. If one doesn't _offer_ configuration of stream limits and therefore is not susceptible to user misconfiguration, then I would get the distinc…

Nginx and HAProxy work around the issue in different ways.

Nginx by default simply kills the entire connection after 1000 requests. With this attack, that's two packets. This severely limits its amplification and basically makes the bypass of the concurrent stream limit a moot point - unless you manually increased the requests-until-killed count.

HAProxy avoids the issue by deviating from the specification. You are supposed to only count active requests towards the concurrent stream limit and ignore cancelled ones, but HAProxy does count cancelled requests and only removes them from the stream count once their resources are fully released. In practice this means the attack isn't any worse than regular http/2 requests.

The protocol-level bug still exists, but in both cases it just can't be used to launch an attack anymore.

Re: HTTP/2 rapid reset attack impacting Nginx products

#55

Post by ziddoap »

gshulegaard wrote: post #18
ziddoap wrote: post #15

Earlier quoted context omitted.

CVEs aren't restricted to only consider default configs.

There is a difference between an application being innately vulnerable and a user configuration exposing a vulnerability. Interestingly enough, HAProxy seems to have the same mitigation: > Until HAProxy dips below the configured stream limit again, new stream creation remains pending—regular timeouts eventually apply and the stream is cut if the situation does not resolve itself. This can occur during an attack. http…

I'm not saying there isn't a difference.

OP mentioned they didn't find Nginx listed on the CVE, and the reply said

>If you read the article, you'll see that the default configuration is not affected.

Which, in the context of OPs comment, implies that the CVE wouldn't be associated because the default config is not affected.

Hence my reply that CVEs don't care whether its default config or not. If there is a CVE associated with the program, there is a CVE associated with the program, rare config or not.

Re: HTTP/2 rapid reset attack impacting Nginx products

#56

Post by gshulegaard »

crote wrote: post #54
gshulegaard wrote: post #30

Earlier quoted context omitted.

I am not sure I understand how stream limit configuration between two L4/L7 load balancers is meaningfully different. In my mind, either the configuration of stream limits is a vulnerability for all L4/L7 load balancers that offer that configuration or it's not for all of them. If one doesn't _offer_ configuration of stream limits and therefore is not susceptible to user misconfiguration, then I would get the distinc…

Nginx and HAProxy work around the issue in different ways. Nginx by default simply kills the entire connection after 1000 requests. With this attack, that's two packets . This severely limits its amplification and basically makes the bypass of the concurrent stream limit a moot point - unless you manually increased the requests-until-killed count. HAProxy avoids the issue by deviating from the specification. You are…

Thanks for taking the time to explain the nuanced implementation difference.

Re: HTTP/2 rapid reset attack impacting Nginx products

#57

Post by bullen »

knorker wrote: post #52
bullen wrote: post #47

Just use HTTP/1.1, it's the final protocol. Nothing Google or Microsoft does will dethrone it. Forget the browser; use a C or Java client and HTTP. If they block port 80, just use another port. They cannot win.

Who is "they"? Is there some kind of evil back room conspiracy to make the web faster, using open standards?

They are those that disable HTTP.

Like it was not enough to make HTTPS default, they need to eradicate the opposition.

The list is too long to enumerate but they all have one thing in common; they profit from root certificates.

The web is not faster, it's bloated. It's only open if you can understand it and implement it.

HTTP/1.1 is the fastest, most open, web you'll ever have; because it is small.

Re: HTTP/2 rapid reset attack impacting Nginx products

#58

Post by knorker »

bullen wrote: post #57
knorker wrote: post #52

Earlier quoted context omitted.

Who is "they"? Is there some kind of evil back room conspiracy to make the web faster, using open standards?

They are those that disable HTTP. Like it was not enough to make HTTPS default, they need to eradicate the opposition. The list is too long to enumerate but they all have one thing in common; they profit from root certificates. The web is not faster, it's bloated. It's only open if you can understand it and implement it. HTTP/1.1 is the fastest, most open, web you'll ever have; because it is small.

> They are those that disable HTTP.

Who's disabling HTTP?

> Like it was not enough to make HTTPS default, they need to eradicate the opposition.

I think you need to elaborate your world view by many paragraphs before I can understand what you're trying to say.

You're against HTTPS? Plain TLS wrapping HTTP?

> they profit from root certificates.

The (web) root certificate industry has never been weaker than it is today, thanks to free root CAs like LetsEncrypt.

> HTTP/1.1 is the fastest, most open, web you'll ever have; because it is small.

So run it. Nobody's stopping you. I won't. Aside from firesheep, the fact that you thanks to Snowden we no longer have to be called a conspiracy theorist to believe in Echelon, and with all the webpage ad and bitcoin mining injectors, or just plain DPI middleboxes, I don't think HTTP is a reasonable default.

Google used to provide "Conncetion: Close" (with the typo), to work around middle boxes that did shenanigans to headers, but luckily only compared some bad checksum of the header, not string comparison.

HTTP/1.1 is open and simple, sure, but it's also being interfered with on a massive scale. Companies out there are selling ad injector boxes to ISPs. They only work on HTTP. You don't have ads on your blog? Well, you do now, for the visitors from some ISPs.

Now HTTP/2 and HTTP/3, I see more of your point. Encryption is just table stakes to get a working website, at this point, but nobody's obligated to race to eliminate every RTT. Plain HTTPS is fine.

CSS sprites is still (last I checked) a bit faster than individual resources over HTTP/2 or 3. But if I'm making a photo album showing 50 thumbnails at once, then I'm unlikely to use plain simple HTTPS with individual resources. I'd at least have to choose between CSS sprites or HTTP/2/3. It would just make for a poor user experience otherwise. And if you're not making a website for your users, then what is it for?

Re: HTTP/2 rapid reset attack impacting Nginx products

#60

Post by bullen »

knorker wrote: post #58
bullen wrote: post #57

Earlier quoted context omitted.

They are those that disable HTTP. Like it was not enough to make HTTPS default, they need to eradicate the opposition. The list is too long to enumerate but they all have one thing in common; they profit from root certificates. The web is not faster, it's bloated. It's only open if you can understand it and implement it. HTTP/1.1 is the fastest, most open, web you'll ever have; because it is small.

> They are those that disable HTTP. Who's disabling HTTP? > Like it was not enough to make HTTPS default, they need to eradicate the opposition. I think you need to elaborate your world view by many paragraphs before I can understand what you're trying to say. You're against HTTPS? Plain TLS wrapping HTTP? > they profit from root certificates. The (web) root certificate industry has never been weaker than it is today…

All major browsers have now removed HTTP available as default = you need to change a setting to even be able to access a HTTP url.

Anti-virus software blocks native apps that try to connect on port 80 and you cannot make them open the port even if the setting is available.

I will always use HTTP/1.1 on port 80 but my customers wont be able to connect even if they try, my only option is to tell them to uninstall their anti-virus and hope that works.

To force a certificate that is gate kept by root-certificates and forces you to identify yourself is the largest censorship humanity has had so far: It protects the consumer from those that don't have a root-cert., but hurts a producer that is not complying with the authority.

And HTTP/2 & 3 tries to insert the cert. as a base function meaning you wont even be able to connect without it.

Pseudonymity was always the most important feature of the internet.

Watch them come after TCP and UDP soon: They will say that to use unencrypted protocols you need a license from your government.

Post reply on HN