Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What are best tools for web scraping?

news.ycombinator.com

Post reply on HN
Print view View original

151–160 of 243 posts

Re: Ask HN: What are best tools for web scraping?

#151

Post by danso »

mfontani wrote: post #128

Whatever you end up using for scraping, I beg you to pick a unique user-agent which allows a webmaster to understand which crawler is it, to better allow it to pass through (or be banned, depending). Don't stick with the default "scrapy" or "Ruby" or "Jakarta Commons-HttpClient/...", which end up (justly) being banned more easily than unique ones, like "ABC/2.0 - https://example.com/crawler" or the like.

Note that for some libraries, the agent is set to empty or whatever the default is for the tool (e.g. `curl/7.43.0` for curl). It's always worth setting it to something.

As a frequent scraper of government sites, and sometimes commercial sites for research purposes, I avoid as much as possible as faking a User Agent, i.e. copying the default strings for popular browsers:

`Mozilla/5.0 (Windows NT 6.1) AppleWebKit/537.36 (KHTML, like Gecko) Chrome/41.0.2228.0 Safari/537.36`

Almost always, if a site rejects my scraper on the basis of agent, they're doing a regex for "curl", "wget" or for an empty string. Setting a user-agent to something unique and explicit, i.e. "Dan's program by danso@myemail.com" works fine without feeling shady.

Maybe for old government sites that break on anything but IE, you'll have to pretend to be IE, but that's very rare.

Re: Ask HN: What are best tools for web scraping?

#152

Post by tmuir »

kzisme wrote: post #109

So in general what do most people use web scraping for? Is it building up their on database of things not available via an API or something? It always sounds interesting, but the need for it is what confuses me.

I've generally used it to sort data in some way that's not available on the original webpage. Either into a csv file, making large lists easier to view, or to determine some optimum, such as the best price.

- Which squares have historically hit the most often in Superbowl Squares (http://www.picks.org/nfl/super-bowl-squares)

- Search a job website for a search term and list of locations, collecting each job title, company, location, and link, to view as one large spreadsheet, instead of having to navigate through 10 results per page.

- Collect cost of living indices in a list of cities

Re: Ask HN: What are best tools for web scraping?

#153

Post by danso »

upofadown wrote: post #150
danso wrote: post #102

Always fascinated by how diverse the discussion and answers is for HN threads on web-scraping. Goes to show that "web-scraping" has a ton of connotations, everything from automated-fetching of URLs via wget or cURL, to data management via something like scrapy. Scrapy is a whole framework that may be worthwhile, but if I were just starting out for a specific task, I would use: - requests http://docs.python-requests.o…

>Python 3, AFAIK, doesn't have anything as handy as Ruby/Perl's Mechanize. Did the version of Mechanize written in Py2 stop being supported?

Looks like it's recently been updated but no big announcement that it's Python 3 ready: https://github.com/python-mechanize/mechanize

I've also seen these alternatives:

- https://robobrowser.readthedocs.io/en/latest/

- https://github.com/MechanicalSoup/MechanicalSoup

MechanicalSoup seems well updated but the last time I tried these libraries, they were either buggy (and/or I was ignorant) and I just couldn't get things to work as I was used to in Ruby and Mechanize.

Re: Ask HN: What are best tools for web scraping?

#154

Post by bootcat »

One of the important avenues to scrape AJAX heavy and phantomjs avoiding websites is using the google chrome extension support. They can mirror the dom and send it to an external server for processing where we can use python lxml to xpath to appropriate nodes. This worked for me to scrape Google, before we hit the capatcha. If anyone is interested, i can share code i wrote to scrape websites !

If you can scrape findthecompany database ? I have done it successfully !!

Re: Ask HN: What are best tools for web scraping?

#155

Post by grad_ml »

ajimix wrote: post #106

I've been doing scraping for many years and at the end it's always the same, you build a lot of stuff to bypass site restrictions and finally, once you are done, you can start scraping. It all goes fine until the site you are scrapping bans you... So what I do now? - proxycrawl https://proxycrawl.com - node http://nodejs.org With proxycrawl I don't need to worry about bans or blocks and I can crawl sites like amazon,…

To me, it seems proxycrawl is very expensive! If I may ask, can you talk little about your crawl volume and cost?

Re: Ask HN: What are best tools for web scraping?

#156

Post by ajimix »

grad_ml wrote: post #155
ajimix wrote: post #106

I've been doing scraping for many years and at the end it's always the same, you build a lot of stuff to bypass site restrictions and finally, once you are done, you can start scraping. It all goes fine until the site you are scrapping bans you... So what I do now? - proxycrawl https://proxycrawl.com - node http://nodejs.org With proxycrawl I don't need to worry about bans or blocks and I can crawl sites like amazon,…

To me, it seems proxycrawl is very expensive! If I may ask, can you talk little about your crawl volume and cost?

I’m crawling around 80-120M per month and the price for me fits my needs. But I suggest that you contact them if you have special needs or requirements.

Also you have to consider the amount of work, time and money that you will save by not maintaining your own system to avoid blocks and bans from the websites you are trying to crawl. With them you just call an API endpoint and you don't have to care about all that

Re: Ask HN: What are best tools for web scraping?

#158

Post by samtc »

CGamesPlay wrote: post #118
samtc wrote: post #67

I maintain ~30 different crawlers. Most of them are using Scrapy. Some are using PhantomJS/CasperJS but they are called from Scrapy via a simple web service. All data (zip files, pdf, html, xml, json) we collect are stored as-is (/path/to/ / / ) and processed later using a Spark pipeline. lxml.html is WAY faster than beautifulsoup and less prone to exception. We have cronjob (cron + jenkins) that trigger dataset upda…

I have a similar set up! How do you monitor for failures and deal with the scrape target changing?

We monitor exceptions with Sentry. We store raw data so we don't have to hurry to fix the ETL, we only have to fix navigation logic and we keep crawling.

Re: Ask HN: What are best tools for web scraping?

#159

Post by samtc »

frik wrote: post #105
samtc wrote: post #67

I maintain ~30 different crawlers. Most of them are using Scrapy. Some are using PhantomJS/CasperJS but they are called from Scrapy via a simple web service. All data (zip files, pdf, html, xml, json) we collect are stored as-is (/path/to/ / / ) and processed later using a Spark pipeline. lxml.html is WAY faster than beautifulsoup and less prone to exception. We have cronjob (cron + jenkins) that trigger dataset upda…

> We use Redis to send task (update / discovery) to our crawlers. Some kind of queue implemented with Redis? How does it work?

It's a simple redis list containing JSON task. We have a custom Scrapy Spider hooked to next_request and item_scraped [1]. It check (lpop) for update/discovery tasks in the list and build a Request [2]. We only crawl max ~1 request per second, so performance is not an issue.

For every website we crawl we implement a custom discovery/update logic.

Discovery can be, for example, crawl a specific date range, seq number, postal code.... We usually seed discovery based on the actual data we have, like highest_company_number + 1000, so we get the newly registered companies.

Update is to update a single document. Like crawl document for company number 1234. We generate a Request [2] to crawl only that document.

[1] https://doc.scrapy.org/en/latest/topics/signals.html

[2] https://doc.scrapy.org/en/latest/topics/request-response.htm...

Re: Ask HN: What are best tools for web scraping?

#160

Post by maxisme »

stoneridge wrote: post #58
sharmi wrote: post #4

If you are a programmer, scrapy[0] will be a good bet. It can handle robots.txt, request throttling by ip, request throttling by domain, proxies and all other common nitty-gritties of crawling. The only drawback is handling pure javascript sites. We have to manually dig into the api or add a headless browser invocation within the scrapy handler. Scrapy also has the ability to pause and restart crawls [1], run the cra…

Haven't tried this[0] yet, but Scrapy should be able to handle JavaScript sites with the JavaScript rendering service Splash[1]. scrapy-splash[2] is the plugin to integrate Scrapy and Splash. [0] https://blog.scrapinghub.com/2015/03/02/handling-javascript-... [1] https://splash.readthedocs.io/en/stable/index.html [2] https://github.com/scrapy-plugins/scrapy-splash

Reading this from my phone looked like you meant there was a web scraping tool actually called “this[0]” which would be a cracking name.
Post reply on HN