Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What are best tools for web scraping?

news.ycombinator.com

Post reply on HN
Print view View original

91–100 of 243 posts

Re: Ask HN: What are best tools for web scraping?

#91

Post by sklarsa »

ddorian43 wrote: post #49
sharmi wrote: post #4

If you are a programmer, scrapy[0] will be a good bet. It can handle robots.txt, request throttling by ip, request throttling by domain, proxies and all other common nitty-gritties of crawling. The only drawback is handling pure javascript sites. We have to manually dig into the api or add a headless browser invocation within the scrapy handler. Scrapy also has the ability to pause and restart crawls [1], run the cra…

Would you recommend it for scalable projects ? Like, crawl twitter or tumblr ?

I've used it for some larger scrapes (nothing at the scale you're talking about, but still sizeable) and scrapy has very tight integration with scrapinghub.com to handle all of the deployment issues (including worker uptime, result storage, rate-limiting, etc). Not affiliated with them in any way, just have had a good experience using them in the past.

Re: Ask HN: What are best tools for web scraping?

#93

Post by toutenrab »

ivansavz wrote: post #47
Bromskloss wrote: post #39

Earlier quoted context omitted.

> BeautifulSoup / lxml When should one use one or the other, would you say?

You can use the BeautifulSoup API with the `lxml` parser: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#insta... I've heard that `lxml` can choke on certain badly-formed markup, but it's very fast. Personally has never failed on me.

It may not be related but I also noted that processing HTML with lxml (e.g. update every URL of a HTML document with a different domain for instance) was producing malformed HTML with duplicated tags. So I would recommend to use lxml only as a data extraction tool.

Re: Ask HN: What are best tools for web scraping?

#94

Post by ngneer »

https://github.com/featurist/coypu is nice for browser automation. A related question: what are good tools for database scraping, meaning replicating a backend database via a web interface (not referring to compromising the application, rather using allowed queries to fully extract the database).

Re: Ask HN: What are best tools for web scraping?

#95

Post by ddorian43 »

sklarsa wrote: post #91
ddorian43 wrote: post #49

Earlier quoted context omitted.

Would you recommend it for scalable projects ? Like, crawl twitter or tumblr ?

I've used it for some larger scrapes (nothing at the scale you're talking about, but still sizeable) and scrapy has very tight integration with scrapinghub.com to handle all of the deployment issues (including worker uptime, result storage, rate-limiting, etc). Not affiliated with them in any way, just have had a good experience using them in the past.

Every `hosted/cloud/saas/paas` goes into bazillions $$$ for anything largescale. Starting from aws bandwidth and including nearly every service on this earth.

Re: Ask HN: What are best tools for web scraping?

#96

Post by jdc0589 »

indescions_2017 wrote: post #21

Headless Chrome, Puppeteer, NodeJS (jsdom), and MongoDB. Fantastic stack for web data mining. Async based using promises for explicit user input flow automation.

I had a ton of issues with JsDom historically. They could have been fixed, but Cheerio always worked out better for me.

Re: Ask HN: What are best tools for web scraping?

#97

Post by djaychela »

cjsuk wrote: post #54
djaychela wrote: post #37

Earlier quoted context omitted.

Any details on this anywhere, or is it not for public consumption? I'm just getting started in Python and want to do something with Gumtree and eBay as an idea to help me in a different sphere.

It's not really for public consumption because it's embarrassingly badly written :) It's pretty dumb really. Just figured out the search URLs and then parse the list responses. It then stores the auctions/ad IDs it has seen in a tiny redis instance with 60 days' expiry on each ID it inserts. If there are any items it hasn't seen each time it runs, it compiles them in a list and emails them to me via AWS SNS. Runs eve…

Thanks for the info - I'm sure mine will be of lower quality when I do write it - hoping to compile real-world info on sold vehicles by scraping info from eBay and Gumtree, but that will take time and more skills than I currently possess. Good to hear someone's made something out of a similar idea, though.

Re: Ask HN: What are best tools for web scraping?

#98

Post by pwaai »

hey I'm working on this thing called BAML (browser automation markup language) and it looks something like this:

    OPEN http://asdf.com
    CRAWL a
    EXTRACT {'title': '.title'}
It's meant to be super simple and built from ground up to support crawling Single Page Applications.

Also, creating a terminal client (early ver: https://imgur.com/a/RYx5g) for it which will launch a Chrome browser and scrape everything. http://export.sh is still very early in the works, I'd appreciate any feedback (email in profile, contact form doesn't work).

Re: Ask HN: What are best tools for web scraping?

#99

Post by vsupalov »

That really depends on your project and tech stack. If you're into Python and are going to deal with relatively static HTML, then the Python modules Scrapy [1], BeautifulSoup [2] and the whole Python data crunching ecosystem are at your disposal. There's lots of great posts about getting such a stack off the ground and using it in the wild [3]. It can get you pretty darn far, the architecture is solid and there are lots of services and plugins which probably do everything you need.

Here's where I hit the limit with that setup: dynamic websites. If you're looking at something like discourse-powered communities or similar, and don't feel a bit too lazy to dig into all the ways requests are expected to look, it's no fun anymore. Luckily, there's lots of js-goodness which can handle dynamic website, inject your javascript for convenience and more [4].

The recently published Headless Chrome [5] and puppeteer [6] (a Node API for it), are really promising for many kinds of tasks - scraping among them. You can get a first impression in this article [7]. The ecosystem does not seem to be as mature yet, but I think this will be foundation of the next go-to scraping tech stack.

If you want to try it yourself, I've written a brief intro [8] and published a simple dockerized development environment [9], so you can give it a go without cluttering your machine or find out what dependencies you need and how the libraries are called.

[1] https://scrapy.org/

[2] https://www.crummy.com/software/BeautifulSoup/bs4/doc/

[3] http://sangaline.com/post/advanced-web-scraping-tutorial/

[4] https://franciskim.co/dont-need-no-stinking-api-web-scraping...

[5] https://developers.google.com/web/updates/2017/04/headless-c...

[6] https://github.com/GoogleChrome/puppeteer

[7] https://blog.phantombuster.com/web-scraping-in-2017-headless...

[8] https://vsupalov.com/headless-chrome-puppeteer-docker/

[9] https://github.com/vsupalov/docker-puppeteer-dev

Re: Ask HN: What are best tools for web scraping?

#100

Post by mrskitch »

mrskitch wrote: post #70

I’d recommend puppeteer or some other Chrome driver. It’s fast and resilient even on single page apps. If you’re looking to run it on a Linux machine also take a look at https://browserless.io (full disclosure I’m the creator of that site).

I should note that this doesn't lock you into any particular lib, just solves the problem of running on Chrome in a service like fashion.
Post reply on HN