Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping 101 with Python

scrapingbee.com

Post reply on HN
Print view View original

11–20 of 134 posts

Re: Web Scraping 101 with Python

#11

Post by tyingq »

daolf wrote: post #10
tyingq wrote: post #6

PyPpeteer might be worth a look as well. Basically a port of the JS puppeteer project that drives headless Chrome via the Devtools API. As mentioned elsewhere, using anything other than headless isn't useful beyond a fairly narrow scope these days. https://github.com/pyppeteer/pyppeteer

I think you'd be surprised by the amount of website you can scrape without an headless browser. Even Google SERP can be scraped with a simple HTTP client.

That's what I meant by narrow...a known set of sites and data you want to extract.

I imagine, for example, building on the SERP example might hit a wall if you added logged in vs not logged SERPS, iterating over carousel data, reading advertisement data etc.

Re: Web Scraping 101 with Python

#12

Post by JackC »

tyingq wrote: post #6

PyPpeteer might be worth a look as well. Basically a port of the JS puppeteer project that drives headless Chrome via the Devtools API. As mentioned elsewhere, using anything other than headless isn't useful beyond a fairly narrow scope these days. https://github.com/pyppeteer/pyppeteer

There's an official Python library for Playwright as well: https://github.com/microsoft/playwright-python

Re: Web Scraping 101 with Python

#13

Post by daolf »

tyingq wrote: post #11
daolf wrote: post #10

Earlier quoted context omitted.

I think you'd be surprised by the amount of website you can scrape without an headless browser. Even Google SERP can be scraped with a simple HTTP client.

That's what I meant by narrow...a known set of sites and data you want to extract. I imagine, for example, building on the SERP example might hit a wall if you added logged in vs not logged SERPS, iterating over carousel data, reading advertisement data etc.

log in wall can easily be bypassed with an HTTP client by setting the correct auth header.

From what I can observe, 2/3 websites can be scraped without using a headless browser.

Re: Web Scraping 101 with Python

#15

Post by toolslive »

iagovar wrote: post #5
A4ET8a8uTh0 wrote: post #3

Earlier quoted context omitted.

This. Even relatively simple websites are much harder to parse today. I did a minor side project for a customer scraping some info and anti-scraping measures were in full force. It feels like an all out war.

Such as? I've never encounter anything I wasn't able to overcome.

please click on all the traffic lights you see below.

Re: Web Scraping 101 with Python

#16

Post by mikece »

Aside from the Beautiful Soup library, is there something about Python that makes it a better choice for web scraping than languages such as Java, JavaScript, Go, Perl or even C#?

Re: Web Scraping 101 with Python

#17

Post by diarrhea »

mikece wrote: post #16

Aside from the Beautiful Soup library, is there something about Python that makes it a better choice for web scraping than languages such as Java, JavaScript, Go, Perl or even C#?

Don't know about large scales, but just today I threw together a script using selenium, imported pandas to mangle the scraped data and quickly exported to json. For quick and dirty, possibly one-off jobs like that, Python is a great choice.

Re: Web Scraping 101 with Python

#18

Post by monkeybutton »

almost wrote: post #4
toolslive wrote: post #2

fetching html and then parsing it navigating the parsed result (or with regexp) is what used to work 20 years ago. These days, with all these reactive javascript frameworks you better skip to item number 5: headless browsing. Also mind that Facebook, Instagram, ... will have anti-scraping measures in place. It's a race ;)

I've found that a lot of the time that's not needed. Often you'll find the data as a JSON blob in the page and can just read it directly from there. Or find that there's an API endpoint that the javascript reads.

I find this method works best. Skip looking at the page and instead watch all the network requests as the page loads.

Re: Web Scraping 101 with Python

#19

Post by CapriciousCptl »

Tistel wrote: post #8

It’s fun to combine jupyter notebooks and py scraping. If you are working 15 pages/screens deep, you can “stay at the coal face” and not have to rerun the whole script after making a change to the latest step.

Oh! That's a good idea. My goto has always been pipelining along a series of functions, but never thought of just using Jupyter for some reason.

Re: Web Scraping 101 with Python

#20

Post by diarrhea »

Tistel wrote: post #8

It’s fun to combine jupyter notebooks and py scraping. If you are working 15 pages/screens deep, you can “stay at the coal face” and not have to rerun the whole script after making a change to the latest step.

`ipython -i ` also works similarly for debugging, by having the powerful interpreter open after running the script, without the jupyter overhead.
Post reply on HN