Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introduction to web scraping with Python

datawhatnow.com

Post reply on HN
Print view View original

21–30 of 63 posts

Re: Introduction to web scraping with Python

#21

Post by dmn001 »

gmac wrote: post #11
laktek wrote: post #4

I found a lot of use cases for web scraping is kinda ad-hoc and usually, occurs as part of another task (eg. a research project or enhancing a record). I ended up releasing a simple hosted API service called Page.REST ( https://page.rest ) for people who would like to save that extra dev effort and infrastructure cost.

I agree, and I find scripting a web browser via the developer console a really productive approach. First, it's completely interactive. Second, it's the browser, so absolutely everything works. It doesn't matter if the data you want is only loaded by an obscure JS function when a hidden form is submitted on a button click. Just find the button, .click() it, and wait for a mutation event. I have a write up on this[1],…

That may be fine for javascript heavy websites for a site with a few pages, but for anything with more than say 1,000 pages it is much more efficient to scrape using requests with lxml. The requests can be made concurrently, are scalable and there is no browser overhead with page rendering.

Re: Introduction to web scraping with Python

#22

Post by akvadrako »

gmac wrote: post #11
laktek wrote: post #4

I found a lot of use cases for web scraping is kinda ad-hoc and usually, occurs as part of another task (eg. a research project or enhancing a record). I ended up releasing a simple hosted API service called Page.REST ( https://page.rest ) for people who would like to save that extra dev effort and infrastructure cost.

I agree, and I find scripting a web browser via the developer console a really productive approach. First, it's completely interactive. Second, it's the browser, so absolutely everything works. It doesn't matter if the data you want is only loaded by an obscure JS function when a hidden form is submitted on a button click. Just find the button, .click() it, and wait for a mutation event. I have a write up on this[1],…

You can even control Chrome remotely via Python with a pretty simple web sockets api:

https://github.com/marty90/PyChromeDevTools

That's really the best of both worlds.

Re: Introduction to web scraping with Python

#23

Post by dmn001 »

masklinn wrote: post #7
drej wrote: post #2

I love requests+lxml, use it fairly regularly, just a few quick notes: 1. lxml is way faster than BeautifulSoup - this may not matter if all you're waiting for is the network. But if you're parsing something on disk, this may be significant. 2. Don't forget to check the status code of r (r.status_code or less generally r.ok) 3. Those with a background in coding might prefer the .cssselect method available in whatever…

> 1. lxml is way faster than BeautifulSoup - this may not matter if all you're waiting for is the network. But if you're parsing something on disk, this may be significant. Caveat: lxml's HTML parser is garbage, so is BS's, they will parse pages in non-obvious ways which do not reflect what you see in your browser, because your browser follows HTML5 tree building. html5lib fixes that (and can construct both lxml and…

On the contrary, I have found lxml suitable for all of my scraping projects where the objective is to write some XPath to parse or extract some data from some element.

Re: Introduction to web scraping with Python

#24

Post by dmn001 »

Loic wrote: post #19

It is making one mistake, it is parsing and scraping in the same loop. You should pull the data, store them and have another process accessing the data store and perform the parsing and understanding of the data. A "quick" parsing can be done to pull the links and build your frontier, but the data should be pulled and stored for the main parsing. This allows you to test your parsing routines independently of the targ…

There is no issue with parsing and scraping in the same loop as long as there is caching in there as well. You don't want to be hitting the server repeatedly whilst you're debugging.

A project like Scrapy should have caching on by default, but it seems to be an afterthought. Repeatable and reproducible parsing of cached websites is necessary, e.g. if you find additional data fields that you want to parse without downloading the entire site over again.

Re: Introduction to web scraping with Python

#25

Post by dorfsmay »

tekkk wrote: post #14

Yeah this might be handy for small stuff but it's way too naive for anything bigger than couple pages. I recently had to scrape some pictures and meta-data from a website and while scripts like these seemed cool they really didn't scale up at all. Consider navigation, following URLs and downloading pictures all while remaining in the limits what's considered non-intrusive. My first attempt, similar to this, failed mi…

Python Requests has a notion if "session" which takes care of cookies etc... Use it all the time when needing to automate tasks that require to sign in.

Re: Introduction to web scraping with Python

#26

Post by anewhnaccount2 »

qrybam wrote: post #15
tekkk wrote: post #14

Yeah this might be handy for small stuff but it's way too naive for anything bigger than couple pages. I recently had to scrape some pictures and meta-data from a website and while scripts like these seemed cool they really didn't scale up at all. Consider navigation, following URLs and downloading pictures all while remaining in the limits what's considered non-intrusive. My first attempt, similar to this, failed mi…

I've been through the rigmarole of writing my own crawlers and and find Scrapy very powerful. I've run into roadblocks with dynamic/Javascript heavy sites; for those parts selenium+chromedriver works really well. As parent and others have said: this is a grey area so make sure to read the terms of use and/or gain permission before scraping.

Notice how it's not a grey area when Google do it. The usually double standard apply I guess.

Re: Introduction to web scraping with Python

#27

Post by Alex3917 »

Loic wrote: post #19

It is making one mistake, it is parsing and scraping in the same loop. You should pull the data, store them and have another process accessing the data store and perform the parsing and understanding of the data. A "quick" parsing can be done to pull the links and build your frontier, but the data should be pulled and stored for the main parsing. This allows you to test your parsing routines independently of the targ…

Agreed on saving the files first. Here is a code snippet that implements something similar but saves each URL response first, albeit not using WARC:

https://pastebin.com/6F962RVJ

Re: Introduction to web scraping with Python

#28

Post by ptero »

dmn001 wrote: post #24
Loic wrote: post #19

It is making one mistake, it is parsing and scraping in the same loop. You should pull the data, store them and have another process accessing the data store and perform the parsing and understanding of the data. A "quick" parsing can be done to pull the links and build your frontier, but the data should be pulled and stored for the main parsing. This allows you to test your parsing routines independently of the targ…

There is no issue with parsing and scraping in the same loop as long as there is caching in there as well. You don't want to be hitting the server repeatedly whilst you're debugging. A project like Scrapy should have caching on by default, but it seems to be an afterthought. Repeatable and reproducible parsing of cached websites is necessary, e.g. if you find additional data fields that you want to parse without down…

I think the bigger point is the benefit of storing pulled data as is for the future, not so much about hitting the server multiple times. If so, I agree with this 100% -- being able to re-run your algorithms later on a local dataset is a powerful capability. Later time, different computer, new software version -- no problem, you have a local copy of the data.

With caching, you are at the mercy of whatever third party caching scheme is used under the hood and raw pulled data can disappear any time without your explicit command (e.g., if some library gets updated and decides that this invalidates the caching scheme).

Re: Introduction to web scraping with Python

#29

Post by WhitneyLand »

Could CSS selectors, with a few minor extensions, be just as good at XPath for this kind of thing?

I guess a lot of the reason I find xpath frustrating is my usage frequency corresponds exactly to the time needed to forget the syntax and have to relearn/refresh it in my head.

If CSS selectors needed only a few enhancements to compete with XPath, it might be worth enhancing a selector library to enable quick ramp up speed for more web people.

Re: Introduction to web scraping with Python

#30

Post by TACIXAT »

WhitneyLand wrote: post #29

Could CSS selectors, with a few minor extensions, be just as good at XPath for this kind of thing? I guess a lot of the reason I find xpath frustrating is my usage frequency corresponds exactly to the time needed to forget the syntax and have to relearn/refresh it in my head. If CSS selectors needed only a few enhancements to compete with XPath, it might be worth enhancing a selector library to enable quick ramp up s…

In the Chrome console you can right click elements in the sources tab and select Copy > Copy XPath.

For example, your comment:

//*[@id="15541111"]/td/table/tbody/tr/td[3]/div[2]

Post reply on HN