Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What are best tools for web scraping?

news.ycombinator.com

Post reply on HN
Print view View original

51–60 of 243 posts

Re: Ask HN: What are best tools for web scraping?

#52

Post by adventured »

giarc wrote: post #28

For non-coders, import.io is great. However, they used to have a generous free plan that has since went away (you are limited to 500 records now). Still a great product, problem is they don't have a small plan (starts at $299/month and goes up to $9,999).

I was looking at services in this area a few weeks ago to automate a small need I had and ran across these guys. They offer a free 5,000 monthly request basic plan. I gave it a try, worked fine (I ended up building my own solution for greater control). It's just for scraping open graph (with some fall-back capability) tags though.

https://www.opengraph.io/

Re: Ask HN: What are best tools for web scraping?

#54

Post by cjsuk »

djaychela wrote: post #37
cjsuk wrote: post #8

Earlier quoted context omitted.

Using this as well with Requests to automate eBay/gumtree/craigslist. Works very well

Any details on this anywhere, or is it not for public consumption? I'm just getting started in Python and want to do something with Gumtree and eBay as an idea to help me in a different sphere.

It's not really for public consumption because it's embarrassingly badly written :)

It's pretty dumb really. Just figured out the search URLs and then parse the list responses. It then stores the auctions/ad IDs it has seen in a tiny redis instance with 60 days' expiry on each ID it inserts. If there are any items it hasn't seen each time it runs, it compiles them in a list and emails them to me via AWS SNS. Runs every 5 minutes from cron on a Raspberry Pi Zero plugged into the back of my XBox 360 as a power supply and my router via a USB/ethernet cable.

The main bulk of the work went into the searches to run which are a huge list of typos on things with a high return. I tend to buy, test, then reship them for profit. Not much investment gives a very good return - pays for the food bill every month :)

Re: Ask HN: What are best tools for web scraping?

#55

Post by sharmi »

Bromskloss wrote: post #36
sharmi wrote: post #4

If you are a programmer, scrapy[0] will be a good bet. It can handle robots.txt, request throttling by ip, request throttling by domain, proxies and all other common nitty-gritties of crawling. The only drawback is handling pure javascript sites. We have to manually dig into the api or add a headless browser invocation within the scrapy handler. Scrapy also has the ability to pause and restart crawls [1], run the cra…

Would you still recommend Scrapy if the task wasn't specifically crawling?

Nope. It is very specifically tailored to crawling. If you just need something distributed why not check out RQ [0], Gearman [1] or Celery [2]? RQ and Celery are python specific.

[0] : http://python-rq.org/docs/

[1] : http://gearman.org/

[2] : http://docs.celeryproject.org

Re: Ask HN: What are best tools for web scraping?

#56

Post by luckystarr »

Bromskloss wrote: post #36
sharmi wrote: post #4

If you are a programmer, scrapy[0] will be a good bet. It can handle robots.txt, request throttling by ip, request throttling by domain, proxies and all other common nitty-gritties of crawling. The only drawback is handling pure javascript sites. We have to manually dig into the api or add a headless browser invocation within the scrapy handler. Scrapy also has the ability to pause and restart crawls [1], run the cra…

Would you still recommend Scrapy if the task wasn't specifically crawling?

I once used it to automate the, well, scraping of statistics from an affiliate network account. So you can do pretty specific stuff, as long as it involves HTTP/HTTPS requests.

Re: Ask HN: What are best tools for web scraping?

#58

Post by stoneridge »

sharmi wrote: post #4

If you are a programmer, scrapy[0] will be a good bet. It can handle robots.txt, request throttling by ip, request throttling by domain, proxies and all other common nitty-gritties of crawling. The only drawback is handling pure javascript sites. We have to manually dig into the api or add a headless browser invocation within the scrapy handler. Scrapy also has the ability to pause and restart crawls [1], run the cra…

Haven't tried this[0] yet, but Scrapy should be able to handle JavaScript sites with the JavaScript rendering service Splash[1]. scrapy-splash[2] is the plugin to integrate Scrapy and Splash.

[0] https://blog.scrapinghub.com/2015/03/02/handling-javascript-...

[1] https://splash.readthedocs.io/en/stable/index.html

[2] https://github.com/scrapy-plugins/scrapy-splash

Re: Ask HN: What are best tools for web scraping?

#60

Post by gros_roberts »

ivansavz wrote: post #47
Bromskloss wrote: post #39

Earlier quoted context omitted.

> BeautifulSoup / lxml When should one use one or the other, would you say?

You can use the BeautifulSoup API with the `lxml` parser: https://www.crummy.com/software/BeautifulSoup/bs4/doc/#insta... I've heard that `lxml` can choke on certain badly-formed markup, but it's very fast. Personally has never failed on me.

lxml.etree.HTMLParser(recover=True) should work for bad HTML. A few times I had to replace characters before giving the page to lxml, but it was more of an encoding issue.
Post reply on HN