Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping 101 with Python

scrapingbee.com

Post reply on HN
Print view View original

81–90 of 134 posts

Re: Web Scraping 101 with Python

#81

Post by thaumasiotes »

tester756 wrote: post #66
thaumasiotes wrote: post #50

Earlier quoted context omitted.

If they're really being generated client-side, you're free to generate them yourself by any means you want. But also, that's a strange thing for the website to do, since it's applying a security feature (signatures) in a way that prevents it from providing any security. If they're generated server-side like you would expect, and sent to the client, you'd get them the same way you get anything else, by asking for them…

>If they're really being generated client-side, you're free to generate them yourself by any means you want. But also, that's a strange thing for the website to do what?? Page loads -> Javascript sends request to backend -> it returns data -> javascript does stuff with it and renders it.

Sure, that's the model from several comments up. It doesn't involve signing anything.

Re: Web Scraping 101 with Python

#82

Post by isbvhodnvemrwvn »

freedomben wrote: post #25
mikece wrote: post #16

Aside from the Beautiful Soup library, is there something about Python that makes it a better choice for web scraping than languages such as Java, JavaScript, Go, Perl or even C#?

I find javascript (node) to be best suited to web scraping personally. Using the same language to scrape/process as you use to develop those interfaces seems most natural.

Especially with stuff like Puppeteer which allows you to execute JS in context of the browser (which admittedly can lead to weird bugs as the functions are serialized and lose context)

Re: Web Scraping 101 with Python

#83

Post by edmundsauto »

strin wrote: post #29
VBprogrammer wrote: post #28

One tip I would pass on when trying to scrape data from a website, start by using wget in mirror mode to download the useful pages. It's much faster to iterate on scraping the data once you have it locally. Also, less likely to accidentally kill the site or attract the attention of the host.

That works only for static page though. Many modern pages would require you to run a selenium or puppetteer to scrape the content.

For these sites, I crawl using a JS powered engine, and just save the relevant page content to disk.

Then I can craft my regex/selectors/etc., once I have the data stored locally.

This helps if you get caught and shut down - it won't turn off your development effort, and you can create a separate task to proxy requests.

Re: Web Scraping 101 with Python

#84

Post by spsphulse »

VBprogrammer wrote: post #28

One tip I would pass on when trying to scrape data from a website, start by using wget in mirror mode to download the useful pages. It's much faster to iterate on scraping the data once you have it locally. Also, less likely to accidentally kill the site or attract the attention of the host.

Definitely! Scrape the disk, not the web.

Re: Web Scraping 101 with Python

#86

Post by JosephRedfern »

philshem wrote: post #46

Before jumping into frameworks, if your data is lucky enough to be stored in an html table: import pandas as pd dfs = pd.read_html(url) Where ‘dfs’ is an array of dataframes - one item for each html table on the page. https://pandas.pydata.org/pandas-docs/stable/reference/api/p...

Woah. I’ve used pandas a fair amount and had no idea about this. Thank you!

Re: Web Scraping 101 with Python

#88

Post by NDizzle »

silicon2401 wrote: post #74
NDizzle wrote: post #35

I've been involved in many web scraper jobs over the past 25 years or so. The most recent one, which was a long time ago at this point, was using scrapy. I went with XML tools for controlling the DOM. It's worked unbelievably well. It's been running for roughly 5 years at this point. I send a command at a random time between 11pm and 4am to wake up an ec2 instance. It checks its tags to see if it should execute the s…

why can't you just keep using python2? surely some people out there are interested enough to keep updating and maintaining it?

I certainly can keep using it. There have been so many efforts to get people to update Python 2 code to Python 3 code that it's on my backlog to do it. Will I get to it this year? Probably not.

Re: Web Scraping 101 with Python

#89

Post by ddorian43 »

jamra wrote: post #76
cameroncairns wrote: post #75

I think this article does an OK job covering how to scrape websites rendered serverside, but I strongly discourage people from scraping SPAs using a headless browser unless they absolutely have to. The article's author touches on this briefly, but you're far better off using the network tab in your browser's debug tools to see what AJAX requests are being made and figuring out how those APIs work. This approach resul…

How would you deal with authentication?

There was (is?) a DARPA project called "Memex" that was built to crawl the hidden web that has many tools like crawling with authentication, automatic registration, machine-learning to detect search-forms, auto detecting pagination etc etc etc etc https://github.com/darpa-i2o/memex-program-index

Re: Web Scraping 101 with Python

#90

Post by ddorian43 »

nr2x wrote: post #73
tluyben2 wrote: post #65

I wanted to do some larger distributed scraping jobs recently and although it was easy to get everything running on one machine (with different tools including Scrapy), I was surprised how hard it was to do at scale. The open source ones I could find was hard/impossible to get working, overly complex, badly documented etc. The services I found to be reasonably priced for small jobs, but at scale they quickly become v…

It gets more complicated when you need to leverage real browser engines (eg Chrome). I've got jobs spread across ~ 20 machines/ 140 concurrent browser instances, it's non-trivial.

How many pages can you render in a a second per vcpu core ?
Post reply on HN