Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

How my Apache server became a malicious free internet proxy

blog.atrament.net

Post reply on HN
Print view View original

11–20 of 42 posts

Re: How my Apache server became a malicious free internet proxy

#11

Post by tomschlick »

Canada wrote: post #5
burnte wrote: post #3

Earlier quoted context omitted.

I ran a machine in a colo for almost four years, public facing web server for a dedicated webapp. I managed to escape any attacks or infections, but I'm positive it was 80% of what I did and 20% that it's a low volume app that never got attention by malicious actors. And I've been in the industry for a long time, not a noob.

Everything is attacked. It's all automated, not personal.

Yup install something like fail2ban & setup email alerts. You'll get a new notice every 5 minutes or so for the first month from all the china based bots trying to login via common ssh credentials.

Re: How my Apache server became a malicious free internet proxy

#12

Post by e12e »

jawshie wrote: post #4

Any idea what the actual vulnerability was?

From part two[p2]: "After stumbling back into my custom Apache configurations directory, I found a file I made to redirect all the proxy settings to /etc/httpd/conf.d/proxy.conf. I can't believe I missed this file."

(...)

"I changed ProxyRequests On to ProxyRequests Off and restarted Apache sudo service httpd restart. My blog & my websites loaded. I finally came to the solution after a few hours of looking at configs."

(...)

"I ran top and noticed fail2ban was consuming 98-99% of my allocated CPU. [Note: As mentioned by the original author in part1, fail2ban was set up to track Apache httpd access logs, and that's (presumably) why it was consuming so much CPU. -e12e] Holy shit. This culprit was running in the background and I did not even know that it was such an intensive resource hog on my machine. I turned fail2ban's service off sudo service fail2ban stop and I removed it from2 auto-starting on system boots with chkconfig fail2ban off."

Apache is a bit of a complicated beast, and it probably doesn't help that way back when, one didn't set up proxies to web application servers, one ran code in the server (mod_php, mod_perl and even mod_python). Java/tomcat got their own proxy module (mod_jk), and after a while, as more (hw) resources became available, it started to make more sense for everyone to follow the good practice of breaking up services by user (either actual (human) user, or at least service user, like "php" or "cgi-bin" etc). And it became more common to use mod_proxy to forward requests to backends (like php-fpm).

For those new to Apache, it's still easy to miss that Apache can also work as a full http proxy -- and it's easier than it probably should be to set up an open http proxy without intending to. But you generally do have to type in a setting of "ProxyRequests On" -- which kind of does give a hint of what's going on.

[p2] http://blog.atrament.net/how-my-apache-server-became-a-malic...

Re: How my Apache server became a malicious free internet proxy

#13

Post by codezero »

bognition wrote: post #2

This is a perfect example of why most people should not run their own hardware. Don't get me wrong its really fun to build and configure your own server and I openly encourage people to learn but I also remind them that its extremely difficult (for a novice) to do securely. Additionally connecting a misconfigured server to the internet doesn't just hurt the server owner but the entire network is affected, as you are…

I was going to say the opposite. It's awesome that we can spin up boxes and host our own servers, and on top of that, learn from our mistakes. I doubt there's a person here who's never been host to malware, spam or some other malady as a result of some of the experimentation they've done as they learned more about computer systems.

Re: How my Apache server became a malicious free internet proxy

#14

Post by scintill76 »

Sanddancer wrote: post #9
jawshie wrote: post #4

Any idea what the actual vulnerability was?

It's really, really easy to misconfigure mod_proxy and set yourself up as an open proxy. The ProxyRequests directive sounds like it should be needed for any sort of proxying, but is only really needed if you're allowing your apache instance to act as a forward proxy, not as a reverse proxy. For reverse proxying, which is what you want most of the time, you really want ProxyPass and ProxyPassReverse .

The phrase "[my blog] was being hosted on another port because apache was taking up the internet http port 80" sounds like the reason they were trying to set up a reverse-proxy.

Apache docs have an obvious warning about ProxyRequests and security: https://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxyre... .

This config snippet looks like it was copied/modified without understanding:

        
                AddDefaultCharset off
                Order deny,allow
                Allow from .example.com
Example.com? If you read the docs on Order (https://httpd.apache.org/docs/2.2/mod/mod_authz_host.html#or...), you see that Deny,Allow defaults to allow, so that's why it's an open proxy.

Above that, there is a comment "turning ProxyRequests on and allowing proxying from all may allow spammers to use your proxy to send email", so I guess it was somewhat safe originally, until ProxyRequests was changed to On without reading and understanding the comment.

Re: How my Apache server became a malicious free internet proxy

#15

Post by A010 »

tomschlick wrote: post #11
Canada wrote: post #5

Earlier quoted context omitted.

Everything is attacked. It's all automated, not personal.

Yup install something like fail2ban & setup email alerts. You'll get a new notice every 5 minutes or so for the first month from all the china based bots trying to login via common ssh credentials.

I run snort and from its alerts, the bots come from all around the world not all from CN. Though the CN sources are largest.

Re: How my Apache server became a malicious free internet proxy

#16

Post by userbinator »

As someone who's used open proxies to get around geo-IP-tracking/restrictions/censoring, I get the point about excessive bandwidth usage (you can apply per-IP ratelimiting for that), but it does make me a bit sad that open proxies are now considered "malicious"...

Re: How my Apache server became a malicious free internet proxy

#17

Post by MariuszGalus »

scintill76 wrote: post #14
Sanddancer wrote: post #9

Earlier quoted context omitted.

It's really, really easy to misconfigure mod_proxy and set yourself up as an open proxy. The ProxyRequests directive sounds like it should be needed for any sort of proxying, but is only really needed if you're allowing your apache instance to act as a forward proxy, not as a reverse proxy. For reverse proxying, which is what you want most of the time, you really want ProxyPass and ProxyPassReverse .

The phrase "[my blog] was being hosted on another port because apache was taking up the internet http port 80" sounds like the reason they were trying to set up a reverse-proxy. Apache docs have an obvious warning about ProxyRequests and security: https://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxyre... . This config snippet looks like it was copied/modified without understanding: AddDefaultCharset off Order d…

I made the mistake of thinking it was harmless to enable. Also, with the solutions I've found online for enabling 'ghost blog with apache virtualhosts'. I guess someone trolled me.

Re: How my Apache server became a malicious free internet proxy

#18

Post by MariuszGalus »

pki wrote: post #6
jawshie wrote: post #4

Any idea what the actual vulnerability was?

my guess would be mod_proxy that didn't explicitly only define a single ip port to proxy to (ala upstream) therefore allowing a http connect?

I know that they can spoof the referrer. I have mine set to allow '.atrament.net'. I was thinking of using my IP in the setting, but I did not have any problems with '.atrament.net' just yet.

Re: How my Apache server became a malicious free internet proxy

#19

Post by MariuszGalus »

userbinator wrote: post #16

As someone who's used open proxies to get around geo-IP-tracking/restrictions/censoring, I get the point about excessive bandwidth usage (you can apply per-IP ratelimiting for that), but it does make me a bit sad that open proxies are now considered "malicious"...

I used the word malicious because I saw people and have a list of credentials now from compromised accounts and spambots. Mainly from Russia. Also, all the sport betting websites that were being hit. I think there must have been something shady there. I was also used for ad-click fraud. :|

Re: How my Apache server became a malicious free internet proxy

#20

Post by vacri »

scintill76 wrote: post #14
Sanddancer wrote: post #9

Earlier quoted context omitted.

It's really, really easy to misconfigure mod_proxy and set yourself up as an open proxy. The ProxyRequests directive sounds like it should be needed for any sort of proxying, but is only really needed if you're allowing your apache instance to act as a forward proxy, not as a reverse proxy. For reverse proxying, which is what you want most of the time, you really want ProxyPass and ProxyPassReverse .

The phrase "[my blog] was being hosted on another port because apache was taking up the internet http port 80" sounds like the reason they were trying to set up a reverse-proxy. Apache docs have an obvious warning about ProxyRequests and security: https://httpd.apache.org/docs/2.2/mod/mod_proxy.html#proxyre... . This config snippet looks like it was copied/modified without understanding: AddDefaultCharset off Order d…

> If you read the docs on Order

That's yet another example of apache config violating POLS (Principle Of Least Astonishment). You have a set of Allow rules and a set of Deny rules. If a request does not match a rule in either set, then what happens to the request depends on the ordering of these non-matching rulesets (!!) instead of a reasonable default with an explicitly configured alternate option.

It's also a bad name - there's no hint that this affects the default action; you just have to know ahead of time.

Post reply on HN