Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: Crawlee for Python – a web scraping and browser automation library

crawlee.dev

Post reply on HN
Print view View original

31–40 of 60 posts

Re: Show HN: Crawlee for Python – a web scraping and browser automation library

#31

Post by mdaniel »

You'll want to prioritize documenting the existing features, since it's no good having a super awesome full stack web scraping platform if only you can use it. I ordinarily would default to a "read the source" response but your cutesy coding style makes that a non-starter

As a concrete example: command-f for "tier" on https://crawlee.dev/python/docs/guides/proxy-management and tell me how anyone could possibly know what `tiered_proxy_urls: list[list[str]] | None = None` should contain and why?

Re: Show HN: Crawlee for Python – a web scraping and browser automation library

#33

Post by beeboobaa3 »

jancurn wrote: post #22
thebytefairy wrote: post #16

Earlier quoted context omitted.

Ah yes, the old 'if I don't build the bombs for them, someone else will'. I don't think this is taking the moral high ground, this is saying we don't care whether it's moral, there's demand and we'll build it.

There are many legitimate and legal use cases where one might want to circumvent blocking of bots. We believe that everyone has the moral right to access and fairly use non-personal publicly available data on the web the way they want, not just the way the publishers want them to. This is the core founding principle of the open web, which allowed the web to become what it is today. BTW we continuously update this exh…

Thoughts on this law? https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A...

Re: Show HN: Crawlee for Python – a web scraping and browser automation library

#35

Post by manishsharan »

Can this work on intranet sites like sharepoint or confluence , which require employee SSO ?

I was trying to build a small Langchain based RAG based on internal documents but getting the documents from sharepoint/confluence (we have both) is very painful.

Re: Show HN: Crawlee for Python – a web scraping and browser automation library

#37

Post by mnmkng »

mdaniel wrote: post #31

You'll want to prioritize documenting the existing features, since it's no good having a super awesome full stack web scraping platform if only you can use it. I ordinarily would default to a "read the source" response but your cutesy coding style makes that a non-starter As a concrete example: command-f for "tier" on https://crawlee.dev/python/docs/guides/proxy-management and tell me how anyone could possibly know w…

Sorry about the confusion. Some features, like the tiered proxies, are not documented properly. You’re absolutely right. Updates will come soon.

We wanted to have as many features in the initial release as possible, because we have a local Python community conference coming up tomorrow and we wanted to have the library ready for that.

More docs will come soon. I promise. And thanks for the shout.

Re: Show HN: Crawlee for Python – a web scraping and browser automation library

#38

Post by mnmkng »

beeboobaa3 wrote: post #33
jancurn wrote: post #22

Earlier quoted context omitted.

There are many legitimate and legal use cases where one might want to circumvent blocking of bots. We believe that everyone has the moral right to access and fairly use non-personal publicly available data on the web the way they want, not just the way the publishers want them to. This is the core founding principle of the open web, which allowed the web to become what it is today. BTW we continuously update this exh…

Thoughts on this law? https://eur-lex.europa.eu/legal-content/EN/TXT/?uri=CELEX%3A...

It’s an “old” law that did not consider many intricacies of internet and the platforms that exist on it and it’s mostly made obsolete by EU case law, which has shrunk the definition of a protected database under this law so much that it’s practically inapplicable to web scraping.

(Not my opinion. I visited a major global law firm’s seminar on this topic a month ago and this is what they said.)

Re: Show HN: Crawlee for Python – a web scraping and browser automation library

#40

Post by mnmkng »

holoduke wrote: post #36

Does it have event listeners to wait for specific elements based on certain pattern matches. One reason i am still using phantomjs is because it simulates the entire browser and you can compile your own webkit in it.

It uses Playwright under the hood, so yes, it can do all of that, and more.
Post reply on HN