Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Crawlee for Python – a web scraping and browser automation library

crawlee.dev

Post reply on HN
Print view View original

41–50 of 60 posts

Re: Show HN: Crawlee for Python – a web scraping and browser automation library

#41

Post by arresin »

mnmkng wrote: post #37
mdaniel wrote: post #31

You'll want to prioritize documenting the existing features, since it's no good having a super awesome full stack web scraping platform if only you can use it. I ordinarily would default to a "read the source" response but your cutesy coding style makes that a non-starter As a concrete example: command-f for "tier" on https://crawlee.dev/python/docs/guides/proxy-management and tell me how anyone could possibly know w…

Sorry about the confusion. Some features, like the tiered proxies, are not documented properly. You’re absolutely right. Updates will come soon. We wanted to have as many features in the initial release as possible, because we have a local Python community conference coming up tomorrow and we wanted to have the library ready for that. More docs will come soon. I promise. And thanks for the shout.

I literally had to go through the entire codebase the documentation is that lacking. It’s boring to document but imo it’s the lowest hanging fruit to get people moving down that crawlee -> appify funnel.

Re: Show HN: Crawlee for Python – a web scraping and browser automation library

#42

Post by mnmkng »

localfirst wrote: post #39

in one sentence, what does this do that existing web scraping and browser automation doesn't do?

In one word. Nothing.

But I personally think it does some things a little easier, a little faster and little more conveniently than the other libraries and tools out there.

Although there’s one thing that the JS version of Crawlee has which unfortunately isn’t in Python yet, but it will be there soon. AFAIK it’s unique among all libraries. It’s automatically detecting whether a headless browser is needed or if HTTP will suffice and using the most performant option.

Re: Show HN: Crawlee for Python – a web scraping and browser automation library

#45

Post by localfirst »

mnmkng wrote: post #42
localfirst wrote: post #39

in one sentence, what does this do that existing web scraping and browser automation doesn't do?

In one word. Nothing. But I personally think it does some things a little easier, a little faster and little more conveniently than the other libraries and tools out there. Although there’s one thing that the JS version of Crawlee has which unfortunately isn’t in Python yet, but it will be there soon. AFAIK it’s unique among all libraries. It’s automatically detecting whether a headless browser is needed or if HTTP w…

is there anything that uses a computer vision model/ocr locally to extract data?

I find some dynamic sites purposefully make it extremely difficult to parse and they obfuscate the XHR calls to their API

I've also seen some websites pollute the data when it detects scraping which results in garbage data but you don't know until its verified

Re: Show HN: Crawlee for Python – a web scraping and browser automation library

#46

Post by nurettin »

BiteCode_dev wrote: post #29
thebytefairy wrote: post #16

Earlier quoted context omitted.

Ah yes, the old 'if I don't build the bombs for them, someone else will'. I don't think this is taking the moral high ground, this is saying we don't care whether it's moral, there's demand and we'll build it.

Google and Amazon where built on scrapped data, who are you kidding?

I make sure to enroll in projects which scrape Google/Amazon en-masse just for the satisfaction.

Re: Show HN: Crawlee for Python – a web scraping and browser automation library

#47

Post by robertlagrant »

BiteCode_dev wrote: post #29
thebytefairy wrote: post #16

Earlier quoted context omitted.

Ah yes, the old 'if I don't build the bombs for them, someone else will'. I don't think this is taking the moral high ground, this is saying we don't care whether it's moral, there's demand and we'll build it.

Google and Amazon where built on scrapped data, who are you kidding?

There's a bidirectional benefit to Google at least. That's why SEO exists. People want to appear in search results.

Re: Show HN: Crawlee for Python – a web scraping and browser automation library

#48

Post by mnmkng »

localfirst wrote: post #45
mnmkng wrote: post #42

Earlier quoted context omitted.

In one word. Nothing. But I personally think it does some things a little easier, a little faster and little more conveniently than the other libraries and tools out there. Although there’s one thing that the JS version of Crawlee has which unfortunately isn’t in Python yet, but it will be there soon. AFAIK it’s unique among all libraries. It’s automatically detecting whether a headless browser is needed or if HTTP w…

is there anything that uses a computer vision model/ocr locally to extract data? I find some dynamic sites purposefully make it extremely difficult to parse and they obfuscate the XHR calls to their API I've also seen some websites pollute the data when it detects scraping which results in garbage data but you don't know until its verified

We tried a self hosted OCR model a few years ago, but the quality and speed wasn’t great. From experience, it’s usually better to reverse engineer the APIs. The more complicated they are, the less they change. So it can sometimes be painful to set up the scrapers, but once they work, they tend to be more stable than other methods.

Data pollution is real. Also location specific results, personalized results, A/B testing, and my favorite, badly implemented websites are real as well.

When you encounter this, you can try scraping the data from different locations, with various tokens, cookies, referrers etc. and often you can find a pattern to make the data consistent. Websites hate scraping, but they hate showing wrong data to human users even more. So if you resemble a legit user, you’ll most likely get correct data. But of course, there are exceptions.

Re: Show HN: Crawlee for Python – a web scraping and browser automation library

#49

Post by mnmkng »

manishsharan wrote: post #35

Can this work on intranet sites like sharepoint or confluence , which require employee SSO ? I was trying to build a small Langchain based RAG based on internal documents but getting the documents from sharepoint/confluence (we have both) is very painful.

Technically it can. You can log in with the PlaywrightCrawler class without issue. The question is if there’s 2FA as well and how that’s handled. Crawlee does not have any abstraction for handling 2FA as it depends a lot on what verification options are supported on the SSO side. So that part would need a custom implementation within Crawlee.
Post reply on HN