11–20 of 30 posts
I also get empty responses when I try to browse HN over Tor. I assume this is because my IP address looks like a spammer's. I, too, would like to know if there are IP blocks, and what (if anything) legitimate users can do to get around them.
Here is the extension: https://chrome.google.com/webstore/detail/hhedbplnihmkekhgma...
Next day, no HN, so I spent the next week browsing HN on Firefox with a proxy setup through an EC2 instance. Thankfully, my IP changed or the ban is gone.
For what I was doing the HNSearch API wouldn't have helped, but if there was an API like the one at ihackernews.com that's running and live, that'd be great.
Yes, we block IPs that seem to be crawlers ignoring robots.txt. We've always blocked abusive IPs, but I tightened up the blocking a few weeks ago. A lot of people were crawling HN, most of them unnecessarily because they were doing things they could have done more efficiently through HNSearch's API. Some users may remember that the site had gotten really slow a few weeks ago. One of the reasons it's faster now is tha…
Beware of a Chrome extension called "Hacker News Sidebar", it presumably got me IP banned this week. It cross checks every page visited with HN to see if it has a thread and if so, displays the thread. Here is the extension: https://chrome.google.com/webstore/detail/hhedbplnihmkekhgma...
We use ipban but that is not what we want: we want a system which can easily detected "bad" crawler or "abusing" user and ban them for some time.
As of now, we have a simple script going thru apache logs and sending list us list of IP and their activity.
Have you tried loading: https://news.ycombinator.com/?
Question: Is there are software which can be easily installed on apache or inside app to detect crawlers? We use ipban but that is not what we want: we want a system which can easily detected "bad" crawler or "abusing" user and ban them for some time. As of now, we have a simple script going thru apache logs and sending list us list of IP and their activity.
Question: Is there are software which can be easily installed on apache or inside app to detect crawlers? We use ipban but that is not what we want: we want a system which can easily detected "bad" crawler or "abusing" user and ban them for some time. As of now, we have a simple script going thru apache logs and sending list us list of IP and their activity.
Yes, we block IPs that seem to be crawlers ignoring robots.txt. We've always blocked abusive IPs, but I tightened up the blocking a few weeks ago. A lot of people were crawling HN, most of them unnecessarily because they were doing things they could have done more efficiently through HNSearch's API. Some users may remember that the site had gotten really slow a few weeks ago. One of the reasons it's faster now is tha…