Scraperr – A Self Hosted Webscraper
github.com
Scraperr – A Self Hosted Webscraper
1–10 of 96 posts
Re: Scraperr – A Self Hosted Webscraper
#2Interesting, wish it had markdown output like firecrawl for embedding/llm use cases
Re: Scraperr – A Self Hosted Webscraper
#3My preferred "self-hosted" webscraper is a local, single binary called xidel [1]. The feature I really like is that it can also follow links.
Re: Scraperr – A Self Hosted Webscraper
#4Is there a reason for using Selenium over something like Playwright? I haven't had very many positive experiences with selenium, and playwright I found is easier to use and more flexible.
Also, for stuff like this:
`modified_value = original_value.replace("HeadlessChrome", "Chrome")`
There's quite a few ways to figure out that a browser is a bot, and I don't think replacing a few values like this does much. Not asking you to reveal any tricks, just saying that if you're using something like Playwright, you can e.g. run scripts in the browser to adjust your fingerprint more easily.
Re: Scraperr – A Self Hosted Webscraper
#5Not a web scraper, but a web crawler software. Allows to specify method of crawling, selenium, and others. Returns data in JSON (status code, text contents, etc).
Re: Scraperr – A Self Hosted Webscraper
#6Is there a reason for using Selenium over something like Playwright? I haven't had very many positive experiences with selenium, and playwright I found is easier to use and more flexible. Also, for stuff like this: `modified_value = original_value.replace("HeadlessChrome", "Chrome")` There's quite a few ways to figure out that a browser is a bot, and I don't think replacing a few values like this does much. Not askin…
I am quite aware, but I actually built most of the scraping logic a long time ago, before I even knew that playwright was a thing.
I am looking to refactor a lot of this, and switching over to playwright is a high priority, using something like camoufox for scraping, instead of just chromium.
Most of my work on this the past month has been simple additions that are nice to haves
Re: Scraperr – A Self Hosted Webscraper
#7Is there a reason for using Selenium over something like Playwright? I haven't had very many positive experiences with selenium, and playwright I found is easier to use and more flexible. Also, for stuff like this: `modified_value = original_value.replace("HeadlessChrome", "Chrome")` There's quite a few ways to figure out that a browser is a bot, and I don't think replacing a few values like this does much. Not askin…
With the custom headers, you can actually trick a lot of sites with bot protection to let you load their sites (even big sites like youtube, which I have found success in)
Re: Scraperr – A Self Hosted Webscraper
#8My preferred "self-hosted" webscraper is a local, single binary called xidel [1]. The feature I really like is that it can also follow links. [1] https://github.com/benibela/xidel
Wow, it's written in Pascal! That surely brings me to memory lane.
Re: Scraperr – A Self Hosted Webscraper
#9Is there a reason for using Selenium over something like Playwright? I haven't had very many positive experiences with selenium, and playwright I found is easier to use and more flexible. Also, for stuff like this: `modified_value = original_value.replace("HeadlessChrome", "Chrome")` There's quite a few ways to figure out that a browser is a bot, and I don't think replacing a few values like this does much. Not askin…
I am quite aware, but I actually built most of the scraping logic a long time ago, before I even knew that playwright was a thing. I am looking to refactor a lot of this, and switching over to playwright is a high priority, using something like camoufox for scraping, instead of just chromium. Most of my work on this the past month has been simple additions that are nice to haves
I was in a similar boat with my scrapers. Started with Selenium 5-6 years ago and only discovered Playwright 2 years ago. Spend a month or so swapping the two, which was well worth it. Cleaner API, async support.
Re: Scraperr – A Self Hosted Webscraper
#10Earlier quoted context omitted.
I am quite aware, but I actually built most of the scraping logic a long time ago, before I even knew that playwright was a thing. I am looking to refactor a lot of this, and switching over to playwright is a high priority, using something like camoufox for scraping, instead of just chromium. Most of my work on this the past month has been simple additions that are nice to haves
I was in a similar boat with my scrapers. Started with Selenium 5-6 years ago and only discovered Playwright 2 years ago. Spend a month or so swapping the two, which was well worth it. Cleaner API, async support.
Luckily, I have some experience with playwright, so swapping shouldn't take me too long.
Currently working on a PR to swap over