Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Introduction to web scraping with Python

datawhatnow.com

Post reply on HN
Print view View original

51–60 of 63 posts

Re: Introduction to web scraping with Python

#51

Post by newlyretired »

dmn001 wrote: post #21
gmac wrote: post #11

Earlier quoted context omitted.

I agree, and I find scripting a web browser via the developer console a really productive approach. First, it's completely interactive. Second, it's the browser, so absolutely everything works. It doesn't matter if the data you want is only loaded by an obscure JS function when a hidden form is submitted on a button click. Just find the button, .click() it, and wait for a mutation event. I have a write up on this[1],…

That may be fine for javascript heavy websites for a site with a few pages, but for anything with more than say 1,000 pages it is much more efficient to scrape using requests with lxml. The requests can be made concurrently, are scalable and there is no browser overhead with page rendering.

I've done a lot of scraping in my day, and I've found that lxml/requests is 2-3 OOM more resource efficient than a Selenium based browser. That JS/rendering engine is HEAVY!

Re: Introduction to web scraping with Python

#53

Post by staticautomatic »

ivansavz wrote: post #44
Loic wrote: post #19

It is making one mistake, it is parsing and scraping in the same loop. You should pull the data, store them and have another process accessing the data store and perform the parsing and understanding of the data. A "quick" parsing can be done to pull the links and build your frontier, but the data should be pulled and stored for the main parsing. This allows you to test your parsing routines independently of the targ…

For a simple caching solution that works well with requests, you can look at cachecontrol: from cachecontrol import CacheControl sess = requests.session() cached_sess = CacheControl(sess) response = cached_sess.get('http://google.com') Very good for interactive debugging when you have to make multiple GET requests. First time you'll hit the webserver, after that it's all served from cache.

requests.session() --> requests.Session()

Re: Introduction to web scraping with Python

#54

Post by staticautomatic »

masklinn wrote: post #7
drej wrote: post #2

I love requests+lxml, use it fairly regularly, just a few quick notes: 1. lxml is way faster than BeautifulSoup - this may not matter if all you're waiting for is the network. But if you're parsing something on disk, this may be significant. 2. Don't forget to check the status code of r (r.status_code or less generally r.ok) 3. Those with a background in coding might prefer the .cssselect method available in whatever…

> 1. lxml is way faster than BeautifulSoup - this may not matter if all you're waiting for is the network. But if you're parsing something on disk, this may be significant. Caveat: lxml's HTML parser is garbage, so is BS's, they will parse pages in non-obvious ways which do not reflect what you see in your browser, because your browser follows HTML5 tree building. html5lib fixes that (and can construct both lxml and…

Are you talking about etree.HTML() being garbage? And what are your thoughts on parsing it as xml (e.g. etree.fromstring(), etree.parse() )?

Re: Introduction to web scraping with Python

#55

Post by masklinn »

staticautomatic wrote: post #54
masklinn wrote: post #7

Earlier quoted context omitted.

> 1. lxml is way faster than BeautifulSoup - this may not matter if all you're waiting for is the network. But if you're parsing something on disk, this may be significant. Caveat: lxml's HTML parser is garbage, so is BS's, they will parse pages in non-obvious ways which do not reflect what you see in your browser, because your browser follows HTML5 tree building. html5lib fixes that (and can construct both lxml and…

Are you talking about etree.HTML() being garbage? And what are your thoughts on parsing it as xml (e.g. etree.fromstring(), etree.parse() )?

> Are you talking about etree.HTML() being garbage?

Yes.

> And what are your thoughts on parsing it as xml (e.g. etree.fromstring(), etree.parse() )

No problem there. XML is much stricter and thus easier to "get right" so to speak. lxml's html parser is built upon libxml's HTML parser[0], which predates HTML5, has not been updated to handle it, and is as its documentation notes

> an HTML 4.0 non-verifying parser

This means it harks back to an era where every parser did its thing and tried its best on the garbage it was given without necessarily taking in account the neighbour.

[0] http://xmlsoft.org/html/libxml-HTMLparser.html

Re: Introduction to web scraping with Python

#56

Post by robattila128 »

dmn001 wrote: post #39
victor106 wrote: post #35

How do you guys manage masking the IP address when you want to scrape using your python script?

I find there is really no need to hide or mask the IP address when web scraping. The use of proxies or Tor to do so is completely unnecessary and maybe prohibitive e.g. try using Google in Tor.

When you are hitting sites thousands of times you have to make yourself as human like and anonymous as possible. Which isn't even as hard as it sounds. Just your address, user agent and random timers are the three most important things in botting.

http://www.blackhatunderground.net/forum/the-deep-web/9-blac...

Re: Introduction to web scraping with Python

#57

Post by robattila128 »

chinathrow wrote: post #46

I wonder how many folks using this will obey the robots.txt as explained nicely within the article: "Robots Web scraping is powerful, but with great power comes great responsibility. When you are scraping somebody’s website, you should be mindful of not sending too many requests. Most websites have a “robots.txt” which shows the rules that your web scraper should obey (which URLs are allowed to be scraped, which ones…

Not many. Browser testing libraries are widely repurposed as automation tools by black hats.

Re: Introduction to web scraping with Python

#59

Post by tycho01 »

WhitneyLand wrote: post #29

Could CSS selectors, with a few minor extensions, be just as good at XPath for this kind of thing? I guess a lot of the reason I find xpath frustrating is my usage frequency corresponds exactly to the time needed to forget the syntax and have to relearn/refresh it in my head. If CSS selectors needed only a few enhancements to compete with XPath, it might be worth enhancing a selector library to enable quick ramp up s…

> If CSS selectors needed only a few enhancements to compete with XPath

You may want to try ParslePy, it combines CSS/XPath functionality, allowing you to declaratively specify the selector paths in a JSON file. I just made a PR to allow YAML over JSON, but not sure if Pip picked up on it yet.

Re: Introduction to web scraping with Python

#60

Post by davvolun »

anewhnaccount2 wrote: post #26
qrybam wrote: post #15

Earlier quoted context omitted.

I've been through the rigmarole of writing my own crawlers and and find Scrapy very powerful. I've run into roadblocks with dynamic/Javascript heavy sites; for those parts selenium+chromedriver works really well. As parent and others have said: this is a grey area so make sure to read the terms of use and/or gain permission before scraping.

Notice how it's not a grey area when Google do it. The usually double standard apply I guess.

I don't understand what this comment is referring to. Google's spider respects robots.txt, just block all paths and google will not crawl your site. So too for Bing, Yahoo, Baidu (some complications though, I think), Yandex.... Most of the major spiders respect robots.txt.

Is there some major Google web scraping effort I'm not aware of?

Post reply on HN