Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping 101 with Python

scrapingbee.com

Post reply on HN
Print view View original

51–60 of 134 posts

Re: Web Scraping 101 with Python

#51

Post by sdfsrrte543 »

lysecret wrote: post #44
cubano wrote: post #41

My last contract job was to build a 100% perfect website mirroring program for a group of lawyers who were interested in building class action lawsuits against some of the more henious scammers out there. I ended up building like 8 versions of it, literally using every PHP and Python library and resource I could find. I tried httrack, php-ultimate-web-scraper (from github), headless chromium. headless selenium, and a…

What stack did you end up using ?

>Selenium with python turned out to be the winning combination, and of course, it was the last one I tried.

Re: Web Scraping 101 with Python

#52

Post by dewey »

thaumasiotes wrote: post #50
dewey wrote: post #47

Earlier quoted context omitted.

How are you going to deal with values generated by JS and used to sign requests?

If they're really being generated client-side, you're free to generate them yourself by any means you want. But also, that's a strange thing for the website to do, since it's applying a security feature (signatures) in a way that prevents it from providing any security. If they're generated server-side like you would expect, and sent to the client, you'd get them the same way you get anything else, by asking for them…

I'm not sure what's your point. Of course you can replicate every request in your scraper / with curl if you want to if you know all the input variables.

Doing that for web scraping purposes where everything is changing all the time and you have more than one target website is just not feasible if you have to reverse engineer some custom JS for every site. Using some kind of headless browser for modern websites will be way easier and more reliable.

Re: Web Scraping 101 with Python

#53

Post by pknerd »

tyingq wrote: post #6

PyPpeteer might be worth a look as well. Basically a port of the JS puppeteer project that drives headless Chrome via the Devtools API. As mentioned elsewhere, using anything other than headless isn't useful beyond a fairly narrow scope these days. https://github.com/pyppeteer/pyppeteer

I am often contacted by people who ask me to scrape a dynamic/JS rendered websites. You might be surprised to know that many of such dynamic websites are actually depending on some API end-point which is being accessed via some AJAX like functionality which you can access directly and get the required data. I often faced the situation where data was not fetched via some external source was already available either in data-field or some JSON like structure hence no need to use Selenium with headless browser.

Re: Web Scraping 101 with Python

#54

Post by jjice »

bityard wrote: post #43

Is web scraping going to continue to be a viable thing, now that the web is mainly an app delivery platform rather than a content delivery platform? Can you scrape a webasm site?

I'm not 100% sure what you mean by a 'webasm' site (web assembly powered?), but the article describes scarping via headless browsers which actually render the page and allow you to select elements that are client rendered.

Re: Web Scraping 101 with Python

#55

Post by WrtCdEvrydy »

monkeybutton wrote: post #18
almost wrote: post #4

Earlier quoted context omitted.

I've found that a lot of the time that's not needed. Often you'll find the data as a JSON blob in the page and can just read it directly from there. Or find that there's an API endpoint that the javascript reads.

I find this method works best. Skip looking at the page and instead watch all the network requests as the page loads.

ZAP HUD Proxy is the best option here...

Load the page on it and it shows you all the request being made along with the payload as it happens.

Find the one you need, copy the data, endpoint and HTTP verb and recreate it in your language of choice :D

Re: Web Scraping 101 with Python

#56

Post by tyingq »

pknerd wrote: post #53
tyingq wrote: post #6

PyPpeteer might be worth a look as well. Basically a port of the JS puppeteer project that drives headless Chrome via the Devtools API. As mentioned elsewhere, using anything other than headless isn't useful beyond a fairly narrow scope these days. https://github.com/pyppeteer/pyppeteer

I am often contacted by people who ask me to scrape a dynamic/JS rendered websites. You might be surprised to know that many of such dynamic websites are actually depending on some API end-point which is being accessed via some AJAX like functionality which you can access directly and get the required data. I often faced the situation where data was not fetched via some external source was already available either in…

Sure. This one happens not be Selenium.

Re: Web Scraping 101 with Python

#58

Post by andreilys »

philshem wrote: post #46

Before jumping into frameworks, if your data is lucky enough to be stored in an html table: import pandas as pd dfs = pd.read_html(url) Where ‘dfs’ is an array of dataframes - one item for each html table on the page. https://pandas.pydata.org/pandas-docs/stable/reference/api/p...

+1 this has saved me countless of hours

Re: Web Scraping 101 with Python

#59

Post by psychomugs »

Tistel wrote: post #8

It’s fun to combine jupyter notebooks and py scraping. If you are working 15 pages/screens deep, you can “stay at the coal face” and not have to rerun the whole script after making a change to the latest step.

I write scrapers for fun and notebooks for work but never thought to combine the two. Great idea!

Re: Web Scraping 101 with Python

#60

Post by fudged71 »

I really appreciate the tips in the comments here.

As a beginner it makes a lot of sense to iterate on a local copy with jupyter rather than fetching resources over and over until you get it right. I wish more tutorials focused on this workflow.

Post reply on HN