Web Scraping 101 with Python • Hacker News Board

Re: Web Scraping 101 with Python

#21

Post by whoisburbansky » Wed, Feb 10, 2021, 5:19 PM UTC

Tistel wrote: post #8

It’s fun to combine jupyter notebooks and py scraping. If you are working 15 pages/screens deep, you can “stay at the coal face” and not have to rerun the whole script after making a change to the latest step.

I love the imagery of this being "at the coal face" thanks for that

Re: Web Scraping 101 with Python

#22

Post by lemagedurage » Wed, Feb 10, 2021, 5:19 PM UTC

toolslive wrote: post #2

fetching html and then parsing it navigating the parsed result (or with regexp) is what used to work 20 years ago. These days, with all these reactive javascript frameworks you better skip to item number 5: headless browsing. Also mind that Facebook, Instagram, ... will have anti-scraping measures in place. It's a race ;)

It's not all bad, many modern sites just expose a JSON API that can be used. It really depends on how protective and large the company behind it is.

Re: Web Scraping 101 with Python

#23

Post by lemagedurage » Wed, Feb 10, 2021, 5:22 PM UTC

mikece wrote: post #16

Aside from the Beautiful Soup library, is there something about Python that makes it a better choice for web scraping than languages such as Java, JavaScript, Go, Perl or even C#?

I think Python makes sense, at least for the prototyping phase. There's a lot of trial and error involved, and Python is quick to write.

Re: Web Scraping 101 with Python

#24

Post by edmundsauto » Wed, Feb 10, 2021, 5:24 PM UTC

ackbar03 wrote: post #14

I've always had pretty bad experiences with web scrapping, it's such a pain in the ass and frequently breaks. I'm not sure if I'm doing it wrong or if that's how it's supposed to be.

It’s heavily dependent on the site you’re scraping. If they put in active counter measures, have a complex structure, or update their templates frequently, it’s going to be an uphill battle.

Most sites IME are pretty easy.

Re: Web Scraping 101 with Python

#25

Post by freedomben » Wed, Feb 10, 2021, 5:24 PM UTC

mikece wrote: post #16

Aside from the Beautiful Soup library, is there something about Python that makes it a better choice for web scraping than languages such as Java, JavaScript, Go, Perl or even C#?

I find javascript (node) to be best suited to web scraping personally. Using the same language to scrape/process as you use to develop those interfaces seems most natural.

Re: Web Scraping 101 with Python

#26

Post by monkeybutton » Wed, Feb 10, 2021, 5:25 PM UTC

mikece wrote: post #16

Aside from the Beautiful Soup library, is there something about Python that makes it a better choice for web scraping than languages such as Java, JavaScript, Go, Perl or even C#?

I like python for the ease of use and scraping is I/O bound anyways so there's no pressure to switch to a more performant language.

Re: Web Scraping 101 with Python

#27

Post by kruchone » Wed, Feb 10, 2021, 5:26 PM UTC

In my career I found several reasons not to use regular expressions for parsing an HTML response, but the largest was the fact that it may work for 'properly formed' documents, but you would be surprised how lax all browsers are about requiring the document to be well-formed. Your regex, unless particularly handled, will not be able to handle sites like this (and there are a lot, at least from my career experience). And you may be able to work 'edge cases' into your RegEx, but good luck finding anyone but the expression author who fully understands and can confidently change it as time goes on. It is also a PITA to debug when groupings/etc. aren't working (and there will be a LOT of these cases with HTML/XML documents).

It is honestly almost never worth it unless you have constraints on what packages you can use and you MUST use regular expressions. Just do your future-self a favor and use BeautifulSoup or some other package designed to parse the tree-like structure of these documents.

One way it can be used appropriately is just finding a pattern in the document- without caring where it is w.r.t. the rest of the document. But even then, do you really want to match: --> ?

Re: Web Scraping 101 with Python

#28

Post by VBprogrammer » Wed, Feb 10, 2021, 5:28 PM UTC

One tip I would pass on when trying to scrape data from a website, start by using wget in mirror mode to download the useful pages. It's much faster to iterate on scraping the data once you have it locally. Also, less likely to accidentally kill the site or attract the attention of the host.

Re: Web Scraping 101 with Python

#29

Post by strin » Wed, Feb 10, 2021, 5:29 PM UTC

VBprogrammer wrote: post #28

One tip I would pass on when trying to scrape data from a website, start by using wget in mirror mode to download the useful pages. It's much faster to iterate on scraping the data once you have it locally. Also, less likely to accidentally kill the site or attract the attention of the host.

That works only for static page though. Many modern pages would require you to run a selenium or puppetteer to scrape the content.

Re: Web Scraping 101 with Python

#30

Post by jbergstroem » Wed, Feb 10, 2021, 5:29 PM UTC

mikece wrote: post #16

Aside from the Beautiful Soup library, is there something about Python that makes it a better choice for web scraping than languages such as Java, JavaScript, Go, Perl or even C#?

The Scrapy library written in Python (https://scrapy.org) is excellent for writing and deploying scrapers.