No post body was provided.
Ask HN: Someone is proxy-mirroring my website, can I do anything?
291–300 of 310 posts
Re: Ask HN: Someone is proxy-mirroring my website, can I do anything?
#292Earlier quoted context omitted.
This... there are definitely aspects of the proxy that they aren't configuring or are unaware of. i.e. ssl_cipher, http_x_requested_with, http_accept... and the order of all headers supplied... the casing of all headers supplied... TLS client HELO. It is relatively easy, if you have enough signals, to essentially create a fingerprint that they won't understand how it works. Yet it will be effective at blocking it reg…
This is some really cool anti-scraping inside baseball. Is it safe to say that Cloudflare uses these techniques for weeding out bots?
It's safe to say that if you have enough signals from every possible layer (of which the above a barely a few) that it becomes trivial to build a model that can identify the majority of bots.
However, then you're left with the really hard problem of when real browsers are used. But hey, you went a long way before you had to actually look at traffic patterns and in the meantime you've significantly raised the costs for those operating the bots.
It's also worth noting that if you really get enough signals, that bot writers cannot control them all. Everyone can rewrite a HTTP header, but can you pick the right HTTP headers in the right order with the right TLS cipher and TLS HELO to appear to be the same as Chrome on Windows? Good luck.
Re: Ask HN: Someone is proxy-mirroring my website, can I do anything?
#293Earlier quoted context omitted.
> 5. Instead of banning them, return fake content (fake titles and fake images etc) if proxy is detected OR the ip is flagged. > 6. Don't ban the flagged ip's. She/He's gonna find another one. Make them angry and their user's angry so they give up on you. There's a popular blog that no longer gets linked on HN. The author didn't like the discussions HN had around his writing, so any visitors with HN as the referer ar…
Does anyone not have their referer header supressed or faked?
I strip the referrer generally via https://wiki.mozilla.org/Security/Referrer, unfortunately it breaks a small number of sites very badly, such as web.archive.org and a few others. some of them claiming it was done to combat scraping.
Re: Ask HN: Someone is proxy-mirroring my website, can I do anything?
#2941. Create fake url endpoint. And go to that endpoint in the adversary's website, when your server gets request, flag the ip. Do this nonstop with a script. 2. Create fake html elements and put unique strings inside. And you can search that string in search engines for finding similar fake sites on different domains. 3. Create fake html element and put all request details in encrypted format. Visit adversary's website…
I remember years ago there was a way to DDoS a server by opening the connection and sending data REALLY slow, like 1 byte a second. I wonder if there is a way to do the opposite of that, where ever request is handed off to a worker which slow enough to keep the connection alive. I doubt this can scale well, but just a thought.
you can have some fun with nginx if you can identify on your backend whether the request is coming from a malicious source, e.g. with X-Accel-Limit-Rate
Re: Ask HN: Someone is proxy-mirroring my website, can I do anything?
#295Earlier quoted context omitted.
Might I suggest a spin on this: instead of blocking the IPs, consider serving up different content to those IPs. You could make a page that shames their domain name for stealing content. You could make a redirect page that redirects people to your website. Or you could make a page with absolutely disgusting content. I think it would discourage them from playing the cat and mouse game with you and fixing it by getting…
I've tried this with zip bombs, but I can't tell how well it worked out.
Wait what? Care to follow on this hypothetical topic please?
Re: Ask HN: Someone is proxy-mirroring my website, can I do anything?
#296Earlier quoted context omitted.
Also for (2), any worries that your own providers might imagine you're trying to mount some half-baked DOS campaign?
Wasn't really worried about that. I didn't do it as a super quick burst, but in a space of multiple hours. First because the proxy servers were super slow and second - I couldn't automate it - their servers had some kind of bot detection which would catch me calling the URLs through script. Instead, I installed a browser extension which would automatically reload a browser tab after specified timeout (I've set it to…
Look out as this is not optimal.
Since they will fingerprint your browser. But it looks like they were people with low IQ, so you were fine.
Re: Ask HN: Someone is proxy-mirroring my website, can I do anything?
#2971. Create fake url endpoint. And go to that endpoint in the adversary's website, when your server gets request, flag the ip. Do this nonstop with a script. 2. Create fake html elements and put unique strings inside. And you can search that string in search engines for finding similar fake sites on different domains. 3. Create fake html element and put all request details in encrypted format. Visit adversary's website…
> 1. Create fake big css files (10MB etc). And repeatedly download that from the adversary's website. This should cost them too much money on proxies.
Nope, since anybody doing this and it has at least minimum intelligence are using residential botnets as proxies.
Re: Ask HN: Someone is proxy-mirroring my website, can I do anything?
#298Earlier quoted context omitted.
I've tried this with zip bombs, but I can't tell how well it worked out.
Wait what? Care to follow on this hypothetical topic please?
zip bombs are files that when unzipped expand to enormous sizes. I'm not sure if OP put one to be downloaded for the offender to kill their disk space, or if you could stream one hoping the client browser/scraper would attempt to decompress and crash for memory or disk outages?
That's my read on it anyway.
Re: Ask HN: Someone is proxy-mirroring my website, can I do anything?
#299My networking knowledge isn't great, so apologies if this is wrong. But if it's not wrong, it could help. FIND THE IP FOR THE DOMAIN PS > ping sukuns.us.to Pinging sukuns.us.to [45.86.61.166] with 32 bytes of data: Reply from 45.86.61.166: bytes=32 time=319ms TTL=39 ... REVERSE DNS TO FIND HOST https://dnschecker.org/ip-whois-lookup.php?query=45.86.61.166 Apparently it's "Dedipath". And that WHOIS lookup gives an abu…
This is not a bad idea, though i would guess that if these guys change IPs, then it will be annoying to spend your time sneding emails, etc. But, then i thought: why not automate this with some simple scripts? You have al;ready outlined your recipe, so simply automate the steps...But the more i thought of the automation around this, you need to be creful not to turn into a "spammer of sorts, constantly sending emails…
Aha, some more good ideas there! But you're right, there's tradeoffs and dependencies and uncertainties throughout, so it's not easy to even guess in advance what would work or be worthwhile. Plus as you say there could be negative consequences from a kind of arms-race, with the solution becoming a problem in itself.
It's not the same thing, but I'm reminded now of email in the past, when you would usually get an undeliverable message if something went wrong. But later that was almost entirely stopped - because of spam. Massive volumes of spam was sent from forged addresses, and much of it led to those replies. So that made things worse by doubling the volume, plus the innocents whose addresses had been forged got deluges of confusing undeliverable messages!
I think you're right in that changing IPs would be easy for them. But, changing hosts would be significantly more work and hassle. So if the abuse reporting worked, that could have much more of an impact...
Re: Ask HN: Someone is proxy-mirroring my website, can I do anything?
#300My networking knowledge isn't great, so apologies if this is wrong. But if it's not wrong, it could help. FIND THE IP FOR THE DOMAIN PS > ping sukuns.us.to Pinging sukuns.us.to [45.86.61.166] with 32 bytes of data: Reply from 45.86.61.166: bytes=32 time=319ms TTL=39 ... REVERSE DNS TO FIND HOST https://dnschecker.org/ip-whois-lookup.php?query=45.86.61.166 Apparently it's "Dedipath". And that WHOIS lookup gives an abu…
Abuse contacts never work. I've never had any success hounding them about malicious sites they host.
I have almost no experience of this, and nothing recent, so I don't know. But I'm not surprised at what you say, given the amount of abusive stuff that happens online nowadays.