Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introduction to web scraping with Python

datawhatnow.com

Post reply on HN
Print view View original

31–40 of 63 posts

Re: Introduction to web scraping with Python

#31

Post by masklinn »

dmn001 wrote: post #23
masklinn wrote: post #7

Earlier quoted context omitted.

> 1. lxml is way faster than BeautifulSoup - this may not matter if all you're waiting for is the network. But if you're parsing something on disk, this may be significant. Caveat: lxml's HTML parser is garbage, so is BS's, they will parse pages in non-obvious ways which do not reflect what you see in your browser, because your browser follows HTML5 tree building. html5lib fixes that (and can construct both lxml and…

On the contrary, I have found lxml suitable for all of my scraping projects where the objective is to write some XPath to parse or extract some data from some element.

LXML itself is, the problem is that its HTML parser (libxml's really) is an ad-hoc "HTML4" parser which means the tree it builds routinely diverges from a proper HTML5 tree as you'd find in e.g. your browser's developer tools and the way it fixes (or whether it fixes it at all) markup is completely ad-hoc and hard to predict.

Re: Introduction to web scraping with Python

#33

Post by wootie512 »

qrybam wrote: post #15
tekkk wrote: post #14

Yeah this might be handy for small stuff but it's way too naive for anything bigger than couple pages. I recently had to scrape some pictures and meta-data from a website and while scripts like these seemed cool they really didn't scale up at all. Consider navigation, following URLs and downloading pictures all while remaining in the limits what's considered non-intrusive. My first attempt, similar to this, failed mi…

I've been through the rigmarole of writing my own crawlers and and find Scrapy very powerful. I've run into roadblocks with dynamic/Javascript heavy sites; for those parts selenium+chromedriver works really well. As parent and others have said: this is a grey area so make sure to read the terms of use and/or gain permission before scraping.

I am just getting into web scraping and have also been using Selenium with either Firefox or PhantomJS. Is there a better way to handle the javascript heavy sites? I found one library called dryscraping but haven't had the time to look too deep into it.

Re: Introduction to web scraping with Python

#34

Post by dmn001 »

ptero wrote: post #28
dmn001 wrote: post #24

Earlier quoted context omitted.

There is no issue with parsing and scraping in the same loop as long as there is caching in there as well. You don't want to be hitting the server repeatedly whilst you're debugging. A project like Scrapy should have caching on by default, but it seems to be an afterthought. Repeatable and reproducible parsing of cached websites is necessary, e.g. if you find additional data fields that you want to parse without down…

I think the bigger point is the benefit of storing pulled data as is for the future, not so much about hitting the server multiple times. If so, I agree with this 100% -- being able to re-run your algorithms later on a local dataset is a powerful capability. Later time, different computer, new software version -- no problem, you have a local copy of the data. With caching, you are at the mercy of whatever third party…

By caching, I just mean storing of data locally so you don't have to request it again under a certain timeframe. I use my own caching scripts written in Python, if you use a 3rd party library then data deletion does not matter too much either if you configure it properly and backup the data - html/json data compresses really well using lzma2 in 7-zip.

Re: Introduction to web scraping with Python

#36

Post by jackschultz »

Loic wrote: post #19

It is making one mistake, it is parsing and scraping in the same loop. You should pull the data, store them and have another process accessing the data store and perform the parsing and understanding of the data. A "quick" parsing can be done to pull the links and build your frontier, but the data should be pulled and stored for the main parsing. This allows you to test your parsing routines independently of the targ…

1000% percent this. I write about Python web scraping a lot and the big one is that there's two parts. First is gathering the pages you need to scrape locally, and the second is scraping the pages you've saved. You need need to separate those two to avoid hitting their servers over and over when you're tying to debug the scraping code. My way is to write the first in a file called gather.py, and then the other in scrape.py. Have that in mind before doing and heavy scraping.

Since my scrapes aren't always the biggest, feel free to just save the html in a local folder and then scrape from there. This part of the project depends on how many pages you need to scrape, the size of the files, whether you need to store the data, whether it's a one time scrape or croned, etc. Either way, save the files, and scrape from there.

Here are some of the posts I've done on the subject if people reading the comments want to see more about scraping.

- https://bigishdata.com/2017/05/11/general-tips-for-web-scrap... - https://bigishdata.com/2017/06/06/web-scraping-with-python-p... - https://bigishdata.com/2017/05/11/general-tips-for-web-scrap...

Re: Introduction to web scraping with Python

#37

Post by RhodesianHunter »

wootie512 wrote: post #33
qrybam wrote: post #15

Earlier quoted context omitted.

I've been through the rigmarole of writing my own crawlers and and find Scrapy very powerful. I've run into roadblocks with dynamic/Javascript heavy sites; for those parts selenium+chromedriver works really well. As parent and others have said: this is a grey area so make sure to read the terms of use and/or gain permission before scraping.

I am just getting into web scraping and have also been using Selenium with either Firefox or PhantomJS. Is there a better way to handle the javascript heavy sites? I found one library called dryscraping but haven't had the time to look too deep into it.

Take a look at the Google team's Puppeteer

https://github.com/GoogleChrome/puppeteer

Re: Introduction to web scraping with Python

#39

Post by dmn001 »

victor106 wrote: post #35

How do you guys manage masking the IP address when you want to scrape using your python script?

I find there is really no need to hide or mask the IP address when web scraping. The use of proxies or Tor to do so is completely unnecessary and maybe prohibitive e.g. try using Google in Tor.

Re: Introduction to web scraping with Python

#40

Post by brianwawok »

wootie512 wrote: post #33
qrybam wrote: post #15

Earlier quoted context omitted.

I've been through the rigmarole of writing my own crawlers and and find Scrapy very powerful. I've run into roadblocks with dynamic/Javascript heavy sites; for those parts selenium+chromedriver works really well. As parent and others have said: this is a grey area so make sure to read the terms of use and/or gain permission before scraping.

I am just getting into web scraping and have also been using Selenium with either Firefox or PhantomJS. Is there a better way to handle the javascript heavy sites? I found one library called dryscraping but haven't had the time to look too deep into it.

Splash runs in docker and does a decent job. From the scrapinghub team.
Post reply on HN