Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Tell HN: Cloudflare Is Blocking Piped

news.ycombinator.com

Post reply on HN
Print view View original

111–120 of 127 posts

Re: Tell HN: Cloudflare Is Blocking Piped

#111

Post by prdonahue »

jart wrote: post #106
solardev wrote: post #101

Earlier quoted context omitted.

How did the attackers get your origin ip to begin with? I thought cloudflare was supposed to shield it at the DNS level, and in theory your origin should be dropping all connections not coming from an authenticated Cloudflare proxy?

They weren't able to talk to my origin IP, because when I was using Cloudflare, I blocked at the firewall all IPs that weren't Cloudflare. The problem is that they would DDOS my server through Cloudflare. And because the traffic was being proxied, I couldn't block the attackers without blocking Cloudflare. Unless of course I wanted to fill out a form on their website 9,000 times. It's an awesome website by the way. I…

Do you have a support ticket # you can email me w/details (pat at cloudflare)?

We take every reported false negative as an opportunity to improve our DDoS mitigations, and these reports are very helpful.

As of a few weeks ago, you can now report FNs/FPs for Bot Mitigation directly in the dashboard, and we'll be expanding this pattern for use with DDoS Mitigation as well.

Re: Tell HN: Cloudflare Is Blocking Piped

#112

Post by kevincox »

prdonahue wrote: post #109
kevincox wrote: post #90

Earlier quoted context omitted.

Cloudflare's default settings are very hostile to RSS feeds in general. They block these as part of bot blocking. Which of course is silly because these are intended to be accessed by bots. Even Cloudflare's blog RSS feed is affected by this.

I agree this is a problem, and we're actively working to fix it. Specifically, there's a ticket in progress to improve how bot mitigation handles requests for certain types of static content (including RSS feeds).

Good to hear. It seems to me that more or less all static content should be exampt because it can be cached so serving it doesn't cost the origin anything.

Of course there are ways to bypass the cache with bogus URL parameters that make this difficult and some customers that are concerned with scraping even if that content is "static".

Re: Tell HN: Cloudflare Is Blocking Piped

#113

Post by cmeacham98 »

nikisweeting wrote: post #18

I used to love Cloudflare but their argument for free speech absolutism went out the window when they started making judgement calls about which sites to block and which to keep. Now I'm just disappointed but not surprised. Will probably move off entirely once Tailscale funnels allow for custom termination CNAMEs. If this particular instance is them getting DMCA'd then it's not really their fault, but I'm confirmatio…

"I know that Cloudflare is legally required by the US government to abide by the DMCA, but this supports my theory they are censoring the web more and more on purpose!"

?????

Re: Tell HN: Cloudflare Is Blocking Piped

#114

Post by ricardo81 »

AtNightWeCode wrote: post #85
ricardo81 wrote: post #69

Earlier quoted context omitted.

Not sure if you're being sarcastic! In the end for any scraping they're just raising the barrier of entry. Automated browsers, residential proxies, captcha services just make it more involved for those determined to hit a URL successfully. Not necessarily a bad thing, but the line and grey area and the definition of a 'legitimate' request varies, and one entity as a middle-man deciding that is less than ideal.

Not sure where you coming from but let us go back 10-15 years when there was an open market for commercial crawlers and IP ranges to be used for it. You sold shoes and scraped all other competitors for instance. That era is over. For legitimate interests today including search engines and services for price comparison that data is often provided for free. There are design patterns used today that does among other thi…

>For legitimate interests today including search engines and services for price comparison that data is often provided for free.

Can you explain that further in the context of search engines, new or existing, need to crawl websites and Cloudflare are a barrier to entry? You seem to contradict yourself.

>There are design patterns used today that does among other things provide incorrect prices to scrapers.

If you say so, and hopefully they do it 100% correctly.

Re: Tell HN: Cloudflare Is Blocking Piped

#115

Post by josephcsible »

capableweb wrote: post #108
josephcsible wrote: post #103

Earlier quoted context omitted.

DoH is 100% a good thing. It makes surveillance of your Internet traffic harder, not easier. If you don't trust Cloudflare, then pick a different DoH provider that you do trust.

Nothing is 100% "a good thing", everything has tradeoffs. In this case, you're moving the trust you put in your ISP or anyone who resolves your DNS queries to Cloudflare. Depending on where you are in the world, or how your threat profile looks, this might be good or bad, or degrees of good/bad. That everyone is starting to tunnel more and more of their traffic to one single entity (Cloudflare or not) is overall not…

> In this case, you're moving the trust you put in your ISP or anyone who resolves your DNS queries to Cloudflare.

Not necessarily:

> If you don't trust Cloudflare, then pick a different DoH provider that you do trust.

Re: Tell HN: Cloudflare Is Blocking Piped

#117

Post by 6c737133 »

vdfs wrote: post #110
nine_k wrote: post #54

Earlier quoted context omitted.

But wait, AI models will help bots looks like real humans accessing a site! They'll try hard to will fool the AI models that check if a site is browsed by a human. Ha-ha, only serious.

No need for AI, browser can easily be automated and captcha can be solved using cheap services

> captcha can be solved using cheap services

call it what it is - you're using slave labor in a 3rd world country to solve rudimentary puzzles for you

Re: Tell HN: Cloudflare Is Blocking Piped

#119

Post by wefarrell »

bjord wrote: post #73
codedokode wrote: post #43

Cloudflare might end a golden era of scraping, when it was trivial to scrape data from any site. Now Cloudflare helps site owners to make sure than only humans can read their contents manually. As more site owners switch to similar services, web will become less and less machine readable. No automated data processing, no archiving.

this won't stop the overall trend, but it can help you get around cloudflare's effective scraping blocking (copying my comment from a previous thread): If you're scraping with Python, try cloudscraper—among other things(!), it supports JS rendering (basically the bare-minimum check cloudflare does), without needing to run a full browser in the background. It's built on requests, so integration was pretty easy. https:…

JS rendering is not enough. Cloudflare monitors UI interactions and browser footprints to assess whether it’s a human or a bot.

Re: Tell HN: Cloudflare Is Blocking Piped

#120

Post by mst »

yencabulator wrote: post #94
lillecarl wrote: post #92

Earlier quoted context omitted.

You can fill the pipes to the server(s) you're targeting, it doesn't have to be application layer.

These days, Cloudflare lets you serve your origin via a tunnel from a host that doesn't even have a public IP. And if you run that in a cloud, the NAT isn't your problem -> your attacker will have to DoS that cloud as a whole.

That's an extremely smart approach that I sincerely doubt the site operators would have been capable of dealing with.

Part of the art of consultancy is, sometimes to my great annoyance, optimising for "within the customer's budget" and "within the customer's capacity to maintain it after I'm no longer involved" over "best possible solution."

Plus in this particular case I was working pro bono because (a) I quite liked the site in question continuing to exist (b) a Shadowcat alumnus asked me nicely (c) I take great pleasure in ruining a griefer's entire week. So lightest possible touch was strongly indicated.

The end result was not remotely clever, but it's been in production for a while now and has not to my knowledge caused financial or uptime issues, so I'm going to call it a win even if the inelegance of -how- I won continues to irritate me ;)

Post reply on HN