Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Scraperr – A Self Hosted Webscraper

github.com

Post reply on HN
Print view View original

61–70 of 96 posts

Re: Scraperr – A Self Hosted Webscraper

#61

Post by whazor »

lucb1e wrote: post #19

Funny, I saw this HN headline just after banning another scraper's IP range You're welcome to scrape my sites but please do it ethically. Idk how to define that but some examples of things I consider not cool: - Scraping without a contact method, or at least some unique identifier (like your project's codename), in the user agent string. This is common practice, see e.g.: https://en.wikipedia.org/wiki/User-Agent_head…

I find it unethical for a website robots.txt to allow-list particular search engines and ban all others. Essentially you are colluding with established search providers.

Re: Scraperr – A Self Hosted Webscraper

#62

Post by reconnecting »

edoceo wrote: post #22
lucb1e wrote: post #19

Funny, I saw this HN headline just after banning another scraper's IP range You're welcome to scrape my sites but please do it ethically. Idk how to define that but some examples of things I consider not cool: - Scraping without a contact method, or at least some unique identifier (like your project's codename), in the user agent string. This is common practice, see e.g.: https://en.wikipedia.org/wiki/User-Agent_head…

What do you have for log analytics and ban automation? Could you say more about how to identify these bad-bots?

We use tirreno [1] to manually and automatically analyze traffic and block unwanted bots. Although bot management is not currently listed as an official feature, it works well and is particularly helpful in complex bot hunting.

[1] https://github.com/TirrenoTechnologies/tirreno

Re: Scraperr – A Self Hosted Webscraper

#63

Post by Loic »

whazor wrote: post #61
lucb1e wrote: post #19

Funny, I saw this HN headline just after banning another scraper's IP range You're welcome to scrape my sites but please do it ethically. Idk how to define that but some examples of things I consider not cool: - Scraping without a contact method, or at least some unique identifier (like your project's codename), in the user agent string. This is common practice, see e.g.: https://en.wikipedia.org/wiki/User-Agent_head…

I find it unethical for a website robots.txt to allow-list particular search engines and ban all others. Essentially you are colluding with established search providers.

Not necessarily, I have a website with 95% (maybe even more) of the traffic generated by crawlers. If some of them are behaving badly, it is fair to exclude them with my robots.txt.

But of course, the ones behaving badly tend to not respect the robots.txt, so you end up banning the IP or IP block.

And here, I am a nice guy, the crawler must really be a piece of crap for me to start to block.

Re: Scraperr – A Self Hosted Webscraper

#64

Post by reconnecting »

lucb1e wrote: post #29
edoceo wrote: post #22

Earlier quoted context omitted.

What do you have for log analytics and ban automation? Could you say more about how to identify these bad-bots?

There is no automation, I use `tail -f access.log` I just look at what's happening on my server every now and then. Sometimes not for months, but then when I set up a project like that caching proxy, I'm currently keeping a more regular eye to see that crawlers aren't bothering the upstream via me. Most respect the robots policy, most of the ones that don't set a user agent string that include the word 'bot' and so I…

Interesting. Our open-source platform [1] has the capacity to help with all of this through a GUI and rule engine, but I'm still concerned about whether we should present this way of bot hunting as a feature. I worry that this approach may be irrelevant in today's context.

[1] https://github.com/TirrenoTechnologies/tirreno

Re: Scraperr – A Self Hosted Webscraper

#65

Post by fredoliveira »

andrethegiant wrote: post #37

Shameless plug: prefix any URL with https://pure.md/ to get the pure markdown of that page. Useful for direct piping into an LLM. Has bot detection avoidance, proxy rotation, and headless JS rendering built in.

that looks fantastic - well done!

Re: Scraperr – A Self Hosted Webscraper

#66

Post by tough »

reconnecting wrote: post #64
lucb1e wrote: post #29

Earlier quoted context omitted.

There is no automation, I use `tail -f access.log` I just look at what's happening on my server every now and then. Sometimes not for months, but then when I set up a project like that caching proxy, I'm currently keeping a more regular eye to see that crawlers aren't bothering the upstream via me. Most respect the robots policy, most of the ones that don't set a user agent string that include the word 'bot' and so I…

Interesting. Our open-source platform [1] has the capacity to help with all of this through a GUI and rule engine, but I'm still concerned about whether we should present this way of bot hunting as a feature. I worry that this approach may be irrelevant in today's context. [1] https://github.com/TirrenoTechnologies/tirreno

I mean if anything with AI data and main sources are becoming the actual precious resource again.

So i'd expect an uptick in bots as everyone races to try and compete with google on data hoarding

Re: Scraperr – A Self Hosted Webscraper

#67

Post by tough »

therein wrote: post #55
nomilk wrote: post #26

Earlier quoted context omitted.

That's awesome. Thanks for sharing. First time hearing of the fetch() approach! If I understand correctly, regular browser automation might typically involve making separate GET requests for each page. Whereas the fetch() strategy involves making a GET for the first page (just as with regular browser automation), then after satisfying cloudflare, rather than going on to the next GET request, use fetch( ) to retrieve…

Almost. I mean it's not like fetch(..) is going to lead to some esoteric kind of HTTP request method. I am guessing parent comment is saying what it is saying because fetch will utilize the cookies and other crumbs set by the successful completion of the captcha. If you can take all those crumbs and include it in your next GET request, you don't need to resort to utilizing fetch.

Scammers will use fingerprints from their victims browser/IP/geolocation to try and impersonate them, you basically can buy not only stolen credentials but also the environment in which to run them -safely- from such vendors

Re: Scraperr – A Self Hosted Webscraper

#68

Post by tough »

dotancohen wrote: post #44
anxman wrote: post #21

Earlier quoted context omitted.

Pack ad blockers into your containers. They can be loaded into Chrome and help immensely in suppressing popovers while crawling.

Thank you, I'll experiment with that. Tips and advice welcome!

Another cool trick is to deny all the content types you don't care about in your playwright. so if you only want text why bother allowing requests for fonts, css, svgs, images, videos, etc

Just request the html and cap down all the other stuff

PS: I also think this has the nice side-effect of you consuming less resources (that you didnt care about/need anyways) from the server, so win win

Re: Scraperr – A Self Hosted Webscraper

#69

Post by ToucanLoucan »

whazor wrote: post #61
lucb1e wrote: post #19

Funny, I saw this HN headline just after banning another scraper's IP range You're welcome to scrape my sites but please do it ethically. Idk how to define that but some examples of things I consider not cool: - Scraping without a contact method, or at least some unique identifier (like your project's codename), in the user agent string. This is common practice, see e.g.: https://en.wikipedia.org/wiki/User-Agent_head…

I find it unethical for a website robots.txt to allow-list particular search engines and ban all others. Essentially you are colluding with established search providers.

This rather bluntly runs up against the fact that permitting crawling is an expense the web operator is taking on, ergo, receiving that content is by definition a privilege not a right.

Re: Scraperr – A Self Hosted Webscraper

#70

Post by reconnecting »

tough wrote: post #66
reconnecting wrote: post #64

Earlier quoted context omitted.

Interesting. Our open-source platform [1] has the capacity to help with all of this through a GUI and rule engine, but I'm still concerned about whether we should present this way of bot hunting as a feature. I worry that this approach may be irrelevant in today's context. [1] https://github.com/TirrenoTechnologies/tirreno

I mean if anything with AI data and main sources are becoming the actual precious resource again. So i'd expect an uptick in bots as everyone races to try and compete with google on data hoarding

As I can see, there is already a heavy wave of new AI/startup/VC etc. data companies that goes beyond the data consumption expectations of websites in the pre-AI era.

However, I see the development of new bot types that tackle security in more aggressive ways. It's not just simple SQL injection as it was before, but more sophisticated and custom bots that not only request but also push a lot.

Or just a couple of days ago, I found a new type of bot that "brute-forces" website folder structure. ~205,000 requests in a couple of days.

These new bots are probably not directly the work of AI, but they seem to be a consequence of it.

Post reply on HN