11–20 of 60 posts
Do you have any plans to monetize this? How are you supporting development?
We provide Apify platform to publish your scrapers as Actors for the developer community, and developers earn money through it. You can use Crawlee for Python as well :)
tldr; Crawlee is and always will be free to use and open sourced.
The code example on the front page has this:
`const data = await crawler.get_data()`
That looks like Javascript? Is there a missing underscore?
Looks nice, and modern python. The code example on the front page has this: `const data = await crawler.get_data()` That looks like Javascript? Is there a missing underscore?
Earlier quoted context omitted.
Usually refers to “evading bot detection”. Detecting when blocked and switching proxy/“browser fingerprint”.
Is this a good feature to include? Shouldn't we respect the host's settings on this?
The fact is - many many people want to scrape these sites and there is massive demand for tools to help them do that, so if APIFY/Crawlee decide to take the moral ground and not offer a way around bot detection, someone else will.
Earlier quoted context omitted.
Is this a good feature to include? Shouldn't we respect the host's settings on this?
It’s a fair and totally reasonable question but clashes with reality. Many hosts have data that others want/like to scrape (eBay, Amazon, Google, airlines, etc.) and they setup anti-scraping mechanisms to try and prevent scraping. Whether or not to respect those desires is a bigger question but not one for the scraping library - it’s one for those doing the scraping and their lawyers. The fact is - many many people w…
I found the api a lot better than any python scraping api till date. However I am tempted to try out python with Crawlee.
The playwright integration with gotScraping makes the entire programming experience a breeze. My crawling and scraping involves all kinds of frontend rendered websites with a lot of modified XHR responses to be captured. And IT JUST WORKS!
Thanks a ton . I will definitely use the Apify platform to scale given the integration.
Earlier quoted context omitted.
It’s a fair and totally reasonable question but clashes with reality. Many hosts have data that others want/like to scrape (eBay, Amazon, Google, airlines, etc.) and they setup anti-scraping mechanisms to try and prevent scraping. Whether or not to respect those desires is a bigger question but not one for the scraping library - it’s one for those doing the scraping and their lawyers. The fact is - many many people w…
Ah yes, the old 'if I don't build the bombs for them, someone else will'. I don't think this is taking the moral high ground, this is saying we don't care whether it's moral, there's demand and we'll build it.
Can it be used to obtain RSS contents? Most of examples focus on html
- RSS feed is transferred via HTTP. - BeatifulSoup can parse both HTML & XML.
(RSS uses XML format)