At Amazon, it's not uncommon for both ycombinator and twitter to be unreachable because of our IPs being blocked.
Ask PG: Are there IP blocks?
21–30 of 30 posts
Re: Ask PG: Are there IP blocks?
#22Our work got blocked as well this week, it seems. I can still browse HN via https though. I don't know if that's intentional or not. :) Have you tried loading: https://news.ycombinator.com/ ?
That sounds more like your work added some new rules to their filter that bans some of the the content here. Using https would encrypt that content and keep the filter from scanning it. Now you just have to hope that they don't ban it by url.
Re: Ask PG: Are there IP blocks?
#23Our work got blocked as well this week, it seems. I can still browse HN via https though. I don't know if that's intentional or not. :) Have you tried loading: https://news.ycombinator.com/ ?
That sounds more like your work added some new rules to their filter that bans some of the the content here. Using https would encrypt that content and keep the filter from scanning it. Now you just have to hope that they don't ban it by url.
No, I can assure you that's not the case, unless Comcast decided to do it. It appears to be fine now, but for this week we were getting 0 byte responses from port 80, but everything working fine over HTTPS.
Re: Ask PG: Are there IP blocks?
#24Yes, we block IPs that seem to be crawlers ignoring robots.txt. We've always blocked abusive IPs, but I tightened up the blocking a few weeks ago. A lot of people were crawling HN, most of them unnecessarily because they were doing things they could have done more efficiently through HNSearch's API. Some users may remember that the site had gotten really slow a few weeks ago. One of the reasons it's faster now is tha…
I think it's time I shut down http://ihackernews.com . It's getting impossible to keep the IP address from being blocked. Really unfortunate, because a lot of people like to use the site.
Anyone know if this what the WP7 HN App uses? I've noticed that the app no longer works which is a shame, I used it all the time.
Re: Ask PG: Are there IP blocks?
#25Yes, we block IPs that seem to be crawlers ignoring robots.txt. We've always blocked abusive IPs, but I tightened up the blocking a few weeks ago. A lot of people were crawling HN, most of them unnecessarily because they were doing things they could have done more efficiently through HNSearch's API. Some users may remember that the site had gotten really slow a few weeks ago. One of the reasons it's faster now is tha…
I think it's time I shut down http://ihackernews.com . It's getting impossible to keep the IP address from being blocked. Really unfortunate, because a lot of people like to use the site.
I had to stop using it because of that. And I started to think of re-writing it as a self-hosted solution to hopefully avoid IP bans.
I don't know if you'd consider releasing it for anyone to install on their own server, and if pg would be fine with that solution. (assuming that each page load would dynamically scrape a single page on ycombinator.com)
edit: of course, it might still have its issues if a lot of people were to host it with the same host…
Another solution would be for pg to add a mobile stylesheet. That's honestly the only reason why I would need it.
Re: Ask PG: Are there IP blocks?
#26Earlier quoted context omitted.
I think it's time I shut down http://ihackernews.com . It's getting impossible to keep the IP address from being blocked. Really unfortunate, because a lot of people like to use the site.
Anyone know if this what the WP7 HN App uses? I've noticed that the app no longer works which is a shame, I used it all the time.
Appears so: https://github.com/jpf/hacker-news-wp7/blob/master/README
If HN could give back json instead of html I'm guessing a lot of the crawling could be mitigated to client side. Kind of like appending .json to everything on reddit
Re: Ask PG: Are there IP blocks?
#27Yes, we block IPs that seem to be crawlers ignoring robots.txt. We've always blocked abusive IPs, but I tightened up the blocking a few weeks ago. A lot of people were crawling HN, most of them unnecessarily because they were doing things they could have done more efficiently through HNSearch's API. Some users may remember that the site had gotten really slow a few weeks ago. One of the reasons it's faster now is tha…
I think it's time I shut down http://ihackernews.com . It's getting impossible to keep the IP address from being blocked. Really unfortunate, because a lot of people like to use the site.
If you tell me the ip addr I can whitelist it. pg@ycombinator.com
Re: Ask PG: Are there IP blocks?
#28Earlier quoted context omitted.
I think it's time I shut down http://ihackernews.com . It's getting impossible to keep the IP address from being blocked. Really unfortunate, because a lot of people like to use the site.
If you tell me the ip addr I can whitelist it. pg@ycombinator.com
Wow, thanks! Email sent.
Re: Ask PG: Are there IP blocks?
#29Our work got blocked as well this week, it seems. I can still browse HN via https though. I don't know if that's intentional or not. :) Have you tried loading: https://news.ycombinator.com/ ?
If you tell me the ip I'll whitelist it. pg@ycombinator.com
Re: Ask PG: Are there IP blocks?
#30Our primary ISP (BSNL NIB) at work is entirely blocked as well. Bharti Airtel is able to get access.