Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

RoboBrowser: Your friendly neighborhood web scraper

github.com

Post reply on HN
Print view View original

31–40 of 61 posts

Re: RoboBrowser: Your friendly neighborhood web scraper

#31

Post by jakubbalada »

pcr0 wrote: post #8

Hmm, I can see why I'd want to use this library over piecing together requests and BS4 myself for every project. I love how simple the examples look. I have a project I'm working on that will involve scraping many different websites on a daily basis. My only scraping experience so far is using cheerio[0] to scrape a single page with a 1,000 row HTML table. Should I start with something BS-based like this or should I…

If you want to scrape many websites on a daily basis, have a look at https://www.apifier.com as an alternative.

Disclaimer: I'm a cofounder there

Re: RoboBrowser: Your friendly neighborhood web scraper

#32

Post by [deleted] »

pcr0 wrote: post #8

Hmm, I can see why I'd want to use this library over piecing together requests and BS4 myself for every project. I love how simple the examples look. I have a project I'm working on that will involve scraping many different websites on a daily basis. My only scraping experience so far is using cheerio[0] to scrape a single page with a 1,000 row HTML table. Should I start with something BS-based like this or should I…

[deleted]

Re: RoboBrowser: Your friendly neighborhood web scraper

#33

Post by Dolores12 »

aexaey wrote: post #25

I'm surprised nobody has mentioned WWW::Mechanize - classic perl library [1] or python port of it [2], which is much closer to RoboBrowser than selenium/phantomjs/horseman. [1] http://search.cpan.org/~ether/WWW-Mechanize-1.75/lib/WWW/Mec... [2] https://pypi.python.org/pypi/mechanize/

Mechanize is outdated and python 2 only. We tried it and switched to RoboBrowser.

Re: RoboBrowser: Your friendly neighborhood web scraper

#34

Post by kmike84 »

pkmishra wrote: post #7

What benefit does it provide in comparison to Scrapy?

I think the main difference is that Scrapy is async - it downloads pages in parallel by default, so it is more efficient. But async APIs can be harder to use - you need callbacks or generators everywhere, so sync packages (like RoboBrowser) can be easier to get started.

Re: RoboBrowser: Your friendly neighborhood web scraper

#35

Post by guilamu »

tsergiu wrote: post #10
est wrote: post #3

I hope scrapers could be in a form of Chrome extensions, it would record my webpage actions as macros, then execute the macros on a remote headless server without downtime with periodic revisits. No need to program or config anything.

Shameless plug, check out https://parsehub.com

Parsehub is pretty great, the free tier is interesting and the support is excellent.

Re: RoboBrowser: Your friendly neighborhood web scraper

#36

Post by simula67 »

PhasmaFelis wrote: post #19
tlrobinson wrote: post #5

Earlier quoted context omitted.

Really? It's right there on the main Github page, a 3 sentence description and 6 code examples.

I know, I read it. It's for "browsing the web without a standalone web browser," and I'm sure that if that was something I had needed, I would have said "Oh! How lovely!" But, since I didn't have that need already, I'm not clear why someone would want that. And I'd like to know! So could you give me a couple of practical use cases? "User stories," if you're into that?

Here are a couple of use cases :

* Lets say you are Google and you want to test if the site is working correctly every day. You could code up a Python script that opens up www.google.com, searches for "facebook" and makes sure that the first result points to www.facebook.com. This script can be configured to run everyday and if someone accidentally pushes an update to the site that causes www.facebook.com to not show up as the top result, the script automatically reverts the site back to its original state. This means users continue to get best search results even if an engineer made a mistake with the ranking algorithm.

* Lets say you are Ebay and you want to make sure that the prices for products on your site is competitive with those at Amazon. You can code up a Python script which searches for some products that customers regularly buy, like an iPhone, and extract the lowest offered price at Amazon. It can then compare them with the lowest offered price of an iPhone on Ebay. If the lowest offered price on Ebay is much larger than that at Amazon, you can offer a discount. This convinces the customer that they are getting competitive offers from Ebay and stops them from writing off Ebay when they want to shop online.

Re: RoboBrowser: Your friendly neighborhood web scraper

#37

Post by itsyogesh »

est wrote: post #3

I hope scrapers could be in a form of Chrome extensions, it would record my webpage actions as macros, then execute the macros on a remote headless server without downtime with periodic revisits. No need to program or config anything.

I would urge you to try nightmare[1], its a web scraper that uses electron as a headless browser, and they have a plugin called daydream[2] that records your webpage actions and convert it into a nightmare script. You have to do a couple of retries, but it does work.

[1] https://github.com/segmentio/nightmare [2] https://github.com/segmentio/daydream

Re: RoboBrowser: Your friendly neighborhood web scraper

#38

Post by gkst »

I've used robobrowser for a project, where I needed to log in to a website and subsequently access pages as a logged in user. It worked well and I like the API. For "simple" scrapers that require authentication or some form of user interaction this is a good tool. If I need to scrape many pages from a site as fast as possible, I'd probably go for Scrapy though.

Re: RoboBrowser: Your friendly neighborhood web scraper

#40

Post by Benfromparis »

Interesting for unprotected websites but it's easy to detect and to block: no valid js, no valid meta header, no valid cookie, suspect behavior...

Selenium is a much "elaborated" solution, but still, can be detected most of the time.

Disclosure: I'm DataDome co-founder. If you want to detect bad bots and scrapers on your website, don't hesitate to try out for free and to share your feedback with us https://datadome.co

Post reply on HN