From the repo (clipped):
When using Scraperr, please remember to:
Respect robots.txt
Terms of Service
Rate Limiting
Kudos for promoting ethical usage; makes a change from some of the grifters selling borderline ddos-bots as crawlers.
Scraperr – A Self Hosted Webscraper
81–90 of 96 posts
Re: Scraperr – A Self Hosted Webscraper
#82Earlier quoted context omitted.
What have you had success doing with this? Curious to test it
I mostly use it to aggregate event calendars for all the concert/sport/etc venues, meetups, and clubs in my area and do some other scraping tasks. I host a little wrapper around llm-scraper on a DigitalOcean droplet that I call from Val.town scripts I only check most places once a week so I use the LLM to do the scraping but there are a few cases where I have to scrape thousands of pages very frequently so I use the…
Oh Im interested in doing something similiar, is it hard to do?
Re: Scraperr – A Self Hosted Webscraper
#83Earlier quoted context omitted.
I find it unethical for a website robots.txt to allow-list particular search engines and ban all others. Essentially you are colluding with established search providers.
Not necessarily, I have a website with 95% (maybe even more) of the traffic generated by crawlers. If some of them are behaving badly, it is fair to exclude them with my robots.txt. But of course, the ones behaving badly tend to not respect the robots.txt, so you end up banning the IP or IP block. And here, I am a nice guy, the crawler must really be a piece of crap for me to start to block.
Deny-listing/banning bad crawlers is fine. Especially if they ignore the robots.txt.
But allow-listing particular crawlers only is collusion.
Re: Scraperr – A Self Hosted Webscraper
#84Funny, I saw this HN headline just after banning another scraper's IP range You're welcome to scrape my sites but please do it ethically. Idk how to define that but some examples of things I consider not cool: - Scraping without a contact method, or at least some unique identifier (like your project's codename), in the user agent string. This is common practice, see e.g.: https://en.wikipedia.org/wiki/User-Agent_head…
I find it unethical for a website robots.txt to allow-list particular search engines and ban all others. Essentially you are colluding with established search providers.
I don't know if that's a reply at me or a general remark, but yes, I never understood why you'd include a few big names and ban the rest for example. That's just screaming for anticompetitiveness. I don't know if my mention of robots.txt sounded like I do this, but I do not
Re: Scraperr – A Self Hosted Webscraper
#85Funny, I saw this HN headline just after banning another scraper's IP range You're welcome to scrape my sites but please do it ethically. Idk how to define that but some examples of things I consider not cool: - Scraping without a contact method, or at least some unique identifier (like your project's codename), in the user agent string. This is common practice, see e.g.: https://en.wikipedia.org/wiki/User-Agent_head…
> - Scraping without a contact method, or at least some unique identifier (like your project's codename), in the user agent string. This is a very effective way to make sure you won't get any scraping done!
Tell that to Googlebot, Bingbot, Petalbot, SemrushBot, MJ12bot, MojeekBot, DotBot, YandexBot, SeznamBot, Barkrowler, AhrefsBot, DuckDuckBot, AcademicBotRTU, Bytespider, Applebot, ZoominfoBot, TelegramBot, TwitterBot, SemanticScholarBot, redditbot, Pinterestbot... From a quick peek at my access log, all include either a link (most) or an email address (zoom, tiktok/bytedance, dotbot, and that academic bot)
Very few individual bots don't follow this good practice. Most of the IP ranges of violating bots are owned by Huawei (a few is Huawei Cloud so it could be anyone, but the majority seems to be Huawei themselves) and the remainder is all small beans as far as I remember (few thousand accesses in a day and then disappear forever, for example)
Re: Scraperr – A Self Hosted Webscraper
#86Earlier quoted context omitted.
> It sounds like poorly designed database queries with unindexed joins Neither of those assumptions are correct. As an example, one page needs to look through 2.5 million records to find where the world record holder changed because it provides stats on who held the most records, held them for the greatest cumulative time, etc. The only thing to do would be introducing caching layers for parts of the computation, but…
Like I said then, you need indexes on those columns which you filter on in this table. Search a table of 2.5 million records for a value is still blazing fast if you use indexes correctly. I’m talking about 0.01 seconds or less. Even with tables much larger. I agree about Google being shit. However, my website makes my living, and feeds and clothes my children, so I have to play along to their rules, or suffer. Pleas…
Right...
In case it helps to understand: compare it to something like weather models. You can't "just add indexes" to make it fast, but you can store the result of an hourly run and serve that to people in milliseconds. In my case, nobody's paying me to serve them that 'weather report' so it is what it is
> Like I said then, you need indexes on those columns
Dude, like I said ("that's not a correct assumption" in response to "It sounds like ... unindexed joins"), I have indexes on the relevant columns in the correct order
Believe me I've benchmarked and SQL-EXPLAINed everything. All substantial queries have a code comment saying what index it uses as a way to make sure that changes to one query (and its corresponding index) don't affect another. I've learned a lot in this project about how everything from the different Where parts to the Order By clause to cardinality estimates to explicit index hints affect which indexes it can use as well as chooses to use. I enjoy learning about it, but now that I know the things relevant for this project, I'm also just done with the project and would rather spend my spare time on something new rather than adding code and introducing code and/or infrastructure complexity for storing parts of the computation that don't frequently change for example. Or if it was a popular site with frequent new visitors, that could be worth it as well. It's not. That doesn't make it necessarily a poor design if it's simply a lot of data
> Search a table of 2.5 million records for a value is still blazing fast
If you read what I write then you'd know it's not about looking up a single record
Re: Scraperr – A Self Hosted Webscraper
#87Earlier quoted context omitted.
Like I said then, you need indexes on those columns which you filter on in this table. Search a table of 2.5 million records for a value is still blazing fast if you use indexes correctly. I’m talking about 0.01 seconds or less. Even with tables much larger. I agree about Google being shit. However, my website makes my living, and feeds and clothes my children, so I have to play along to their rules, or suffer. Pleas…
You need to drive and fine tune a Ferrari because it feeds your family. The OP just drive a nice little car, because it is fun to drive and he enjoys it. He could extract another +5% of torque by fine tuning, but he does not care, this is not where his joy is and where he wants to spend his time.
To be completely fair to the person you're responding to, they're talking about pages that take 20 seconds to load. On a regular website that hopes to get visitors from search engines, say, that is indeed insane and the fix is not to squeeze out 5% by fine tuning, the fix is to re-architect the thing
I don't mind people asking why it is this slow (whether I can't or why I don't re-architect it) or suggest fixes, but as the sibling comment to yours (from u/monooso) put it well, it would be nice if one does not assume that I'm an incompetent fool. The person also doesn't seem to read what I explained before suggesting more of the same in another reply. Thanks for adding your comment as well, I appreciate the sentiment. Even if I'm not sure if it applies in this case, it definitely applies to other things I do (I may have too many hobby projects running on that server.. ^^)
Re: Scraperr – A Self Hosted Webscraper
#88Earlier quoted context omitted.
You need to drive and fine tune a Ferrari because it feeds your family. The OP just drive a nice little car, because it is fun to drive and he enjoys it. He could extract another +5% of torque by fine tuning, but he does not care, this is not where his joy is and where he wants to spend his time.
OP is driving with handbrake engaged.
How so?
Re: Scraperr – A Self Hosted Webscraper
#89Earlier quoted context omitted.
> - Scraping without a contact method, or at least some unique identifier (like your project's codename), in the user agent string. This is a very effective way to make sure you won't get any scraping done!
Tell that to Googlebot, Bingbot, Petalbot, SemrushBot, MJ12bot, MojeekBot, DotBot, YandexBot, SeznamBot, Barkrowler, AhrefsBot, DuckDuckBot, AcademicBotRTU, Bytespider, Applebot, ZoominfoBot, TelegramBot, TwitterBot, SemanticScholarBot, redditbot, Pinterestbot... From a quick peek at my access log, all include either a link (most) or an email address (zoom, tiktok/bytedance, dotbot, and that academic bot) Very few in…
None of the institutional market intelligence products I’ve ever worked on in nearly a decade of doing this do. Why? Cause they wouldn’t otherwise work.
Many APIs require specific user agents. Tools like curl impersonate require specific user agents.
Re: Scraperr – A Self Hosted Webscraper
#90Earlier quoted context omitted.
Wow, it's written in Pascal! That surely brings me to memory lane.
With Pascal being my first "adult" language, not used in 20 years ... it is surprising how readable that code is. Makes me wish for such simpler times.
that fits, I wrote the first code for xidel almost 20 years ago
and it still uses Pascal because I didn't plan to change it but just wanted to show people what I programmed 20 years ago