Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Crawlee for Python – a web scraping and browser automation library

crawlee.dev

Post reply on HN
Print view View original

1–10 of 60 posts

Show HN: Crawlee for Python – a web scraping and browser automation library

#1

Post by jancurn »

Hey all,

This is Jan, the founder of Apify (https://apify.com/) — a full-stack web scraping platform. After the success of Crawlee for JavaScript (https://github.com/apify/crawlee/) and the demand from the Python community, we're launching Crawlee for Python today!

The main features are:

- A unified programming interface for both HTTP (HTTPX with BeautifulSoup) & headless browser crawling (Playwright)

- Automatic parallel crawling based on available system resources

- Written in Python with type hints for enhanced developer experience

- Automatic retries on errors or when you’re getting blocked

- Integrated proxy rotation and session management

- Configurable request routing - direct URLs to the appropriate handlers

- Persistent queue for URLs to crawl

- Pluggable storage for both tabular data and files

For details, you can read the announcement blog post: https://crawlee.dev/blog/launching-crawlee-python

Our team and I will be happy to answer here any questions you might have.

Show HN: Crawlee for Python – a web scraping and browser automation library
crawlee.dev

Re: Show HN: Crawlee for Python – a web scraping and browser automation library

#4

Post by jancurn »

marban wrote: post #3

Nice list, but what would be the arguments for switching over from other libraries? I’ve built my own crawler over time, but from what I see, there’s nothing truly unique.

The main advantage (for now) is that the library has a single interface for both HTTP and headless browsers, and bundled auto scaling. You can write your crawlers using the same base abstraction, and the framework takes care of this heavy lifting. Developers of scrapers shouldn't need to reinvent the wheel, and just focus on building the "business" logic of their scrapers. Having said that, if you wrote your own crawling library, the motivation to use Crawlee might be lower, and that's fair enough.

Please note that this is the first release, and we'll keep adding many more features as we go, including anti-blocking, adaptive crawling, etc. To see where this might go, check https://github.com/apify/crawlee

Re: Show HN: Crawlee for Python – a web scraping and browser automation library

#6

Post by sauain »

intev wrote: post #5

How is this different from Scrapy?

hey intev,

- Crawlee has out-of-the-box support for headless browser crawling (Playwright). You don't have to install any plugin or set up the middleware. - Crawlee has a minimalistic & elegant interface - Set up your scraper with fewer than 10 lines of code. You don't have to care about what middleware, settings, and anything are or need to be changed, on the top that we also have templates which makes the learning curve much smaller. - Complete type hint coverage. Which is something Scrapy hasn't completed yet. - Based on standard Asyncio. Integrating Scrapy into a classic asyncio app requires integration of Twisted and asyncio. Which is possible, but not easy, and can result in troubles.

Re: Show HN: Crawlee for Python – a web scraping and browser automation library

#8

Post by robertlagrant »

jancurn wrote: post #4
marban wrote: post #3

Nice list, but what would be the arguments for switching over from other libraries? I’ve built my own crawler over time, but from what I see, there’s nothing truly unique.

The main advantage (for now) is that the library has a single interface for both HTTP and headless browsers, and bundled auto scaling. You can write your crawlers using the same base abstraction, and the framework takes care of this heavy lifting. Developers of scrapers shouldn't need to reinvent the wheel, and just focus on building the "business" logic of their scrapers. Having said that, if you wrote your own craw…

Can I ask - what is anti-blocking?

Re: Show HN: Crawlee for Python – a web scraping and browser automation library

#10

Post by fullspectrumdev »

robertlagrant wrote: post #8
jancurn wrote: post #4

Earlier quoted context omitted.

The main advantage (for now) is that the library has a single interface for both HTTP and headless browsers, and bundled auto scaling. You can write your crawlers using the same base abstraction, and the framework takes care of this heavy lifting. Developers of scrapers shouldn't need to reinvent the wheel, and just focus on building the "business" logic of their scrapers. Having said that, if you wrote your own craw…

Can I ask - what is anti-blocking?

Usually refers to “evading bot detection”.

Detecting when blocked and switching proxy/“browser fingerprint”.

Post reply on HN