Introduction to web scraping with Python
datawhatnow.com
Introduction to web scraping with Python
1–10 of 63 posts
Re: Introduction to web scraping with Python
#2I love requests+lxml, use it fairly regularly, just a few quick notes:
1. lxml is way faster than BeautifulSoup - this may not matter if all you're waiting for is the network. But if you're parsing something on disk, this may be significant.
2. Don't forget to check the status code of r (r.status_code or less generally r.ok)
3. Those with a background in coding might prefer the .cssselect method available in whatever object the parsed document results in. That's obviously a tad slower than find/findall/xpath, but it's oftentimes too convenient to pass upon.
4. Kind of automatic, but I'll say it anyway - scraping is a gray area, always make sure that what you're doing is legitimate.
Re: Introduction to web scraping with Python
#3Anyone knows good repo to do the same in Go?
Re: Introduction to web scraping with Python
#4I found a lot of use cases for web scraping is kinda ad-hoc and usually, occurs as part of another task (eg. a research project or enhancing a record). I ended up releasing a simple hosted API service called Page.REST (https://page.rest) for people who would like to save that extra dev effort and infrastructure cost.
Re: Introduction to web scraping with Python
#5Anyone knows good repo to do the same in Go?
goquery [1] is pretty nice.
Re: Introduction to web scraping with Python
#6I love requests+lxml, use it fairly regularly, just a few quick notes: 1. lxml is way faster than BeautifulSoup - this may not matter if all you're waiting for is the network. But if you're parsing something on disk, this may be significant. 2. Don't forget to check the status code of r (r.status_code or less generally r.ok) 3. Those with a background in coding might prefer the .cssselect method available in whatever…
I had reason to gather news articles and extract keywords and authors - can't remember why I didn't use BeatifulSoup, because that was my first choice. In the end I used lxml with its html5parser:
https://github.com/vonj/scraping
Regarding legality - something frowned upon is putting load on servers. You may get blocked, rate limited or worse if you put too much strain on servers. Especially when you as I did, experimented with different query options to get the data I needed, and had to rerun scraping a number of times.
A neat trick I found, was to configure a web proxy locally (squid, in my case) to aggressively cache EVERYTHING. This was, new runs only went out to the news sites for new queries I had never run before. Very helpful, it also speeded up the development to access files locally (cached in squid) instead of having to go out to the internet all the time.
Re: Introduction to web scraping with Python
#7I love requests+lxml, use it fairly regularly, just a few quick notes: 1. lxml is way faster than BeautifulSoup - this may not matter if all you're waiting for is the network. But if you're parsing something on disk, this may be significant. 2. Don't forget to check the status code of r (r.status_code or less generally r.ok) 3. Those with a background in coding might prefer the .cssselect method available in whatever…
> 1. lxml is way faster than BeautifulSoup - this may not matter if all you're waiting for is the network. But if you're parsing something on disk, this may be significant.
Caveat: lxml's HTML parser is garbage, so is BS's, they will parse pages in non-obvious ways which do not reflect what you see in your browser, because your browser follows HTML5 tree building.
html5lib fixes that (and can construct both lxml and bs trees, and both libraries have html5lib integration), however it's slow. I don't know that there is a native compatible parser (there are plenty of native HTML5 parsers e.g. gumbo or html5ever but I don't remember them being able to generate lxml or bs trees).
> 2. Don't forget to check the status code of r (r.status_code or less generally r.ok)
Alternatively (depending on use case) `r.raise_for_status()`. I'm still annoyed that there's no way to ask requests to just check it outright.
> Those with a background in coding might prefer the .cssselect method available in whatever object the parsed document results in. That's obviously a tad slower than find/findall/xpath, but it's oftentimes too convenient to pass upon.
FWIW cssselect simply translates CSS selectors to XPath, and while I don't know for sure I'm guessing it has an expression cache, so it should not be noticeably slower than XPath (CSS selectors are not a hugely complex language anyway)
Re: Introduction to web scraping with Python
#8I love requests+lxml, use it fairly regularly, just a few quick notes: 1. lxml is way faster than BeautifulSoup - this may not matter if all you're waiting for is the network. But if you're parsing something on disk, this may be significant. 2. Don't forget to check the status code of r (r.status_code or less generally r.ok) 3. Those with a background in coding might prefer the .cssselect method available in whatever…
> 1. lxml is way faster than BeautifulSoup - this may not matter if all you're waiting for is the network. But if you're parsing something on disk, this may be significant. Caveat: lxml's HTML parser is garbage, so is BS's, they will parse pages in non-obvious ways which do not reflect what you see in your browser, because your browser follows HTML5 tree building. html5lib fixes that (and can construct both lxml and…
Re: Introduction to web scraping with Python
#9Re: Introduction to web scraping with Python
#10Earlier quoted context omitted.
> 1. lxml is way faster than BeautifulSoup - this may not matter if all you're waiting for is the network. But if you're parsing something on disk, this may be significant. Caveat: lxml's HTML parser is garbage, so is BS's, they will parse pages in non-obvious ways which do not reflect what you see in your browser, because your browser follows HTML5 tree building. html5lib fixes that (and can construct both lxml and…
https://github.com/kovidgoyal/html5-parser
Nice. Seems to only do lxml tree building?