Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping 101 with Python

scrapingbee.com

Post reply on HN
Print view View original

61–70 of 134 posts

Re: Web Scraping 101 with Python

#61

Post by tluyben2 »

bityard wrote: post #43

Is web scraping going to continue to be a viable thing, now that the web is mainly an app delivery platform rather than a content delivery platform? Can you scrape a webasm site?

Maybe it will turn out to be that way, but this is far from reality at the moment. There are not many sites that cannot be scraped statically and there definitely are very few sites/apps that are webasm.

It'll change, but who knows how much. At least currently, most scraping professionals are not even using headless browsers as their targets are statically rendered.

Re: Web Scraping 101 with Python

#62

Post by tluyben2 »

monkeybutton wrote: post #42
RhodesianHunter wrote: post #31

Earlier quoted context omitted.

I'd say that really depends on your scale and what you're doing with the content you scrape. In my experience with large scale scraping you're much better off using something like Java where you can more easily have a thread pool with thousands of threads (or better yet, Kotlin coroutines) handling the crawling itself and a *NUM CORES thread pool handling CPU bound tasks like parsing.

Could you give a ballpark figure for what you mean by large scale scraping? I've only worked on a couple projects, one was a broad (100K to 500K domains) and shallow (root + 1 level of page depth, also with a low cap on the number of children pages). The other just a single domain but scraping around 50K pages from it.

I would say millions of domains regularly. That's where the pricing of most 'scraping services' falls down too compared to just doing it yourself.

Re: Web Scraping 101 with Python

#63

Post by amenod »

iagovar wrote: post #5
A4ET8a8uTh0 wrote: post #3

Earlier quoted context omitted.

This. Even relatively simple websites are much harder to parse today. I did a minor side project for a customer scraping some info and anti-scraping measures were in full force. It feels like an all out war.

Such as? I've never encounter anything I wasn't able to overcome.

Recaptcha comes to mind.

That said, there are quite a few services which battle these systems for you nowadays (such as scraperapi - not affiliated, not a user). They are not always successful, but they have an advantage of maaany residential proxies (no doubt totally ethically obtained /s, but that's another story).

Re: Web Scraping 101 with Python

#64

Post by mikesholiu »

philshem wrote: post #46

Before jumping into frameworks, if your data is lucky enough to be stored in an html table: import pandas as pd dfs = pd.read_html(url) Where ‘dfs’ is an array of dataframes - one item for each html table on the page. https://pandas.pydata.org/pandas-docs/stable/reference/api/p...

what does this do

Re: Web Scraping 101 with Python

#65

Post by tluyben2 »

I wanted to do some larger distributed scraping jobs recently and although it was easy to get everything running on one machine (with different tools including Scrapy), I was surprised how hard it was to do at scale. The open source ones I could find was hard/impossible to get working, overly complex, badly documented etc.

The services I found to be reasonably priced for small jobs, but at scale they quickly become vastly more expensive than setting this up yourself. Especially when you need to run these jobs every month or so. Even if you have to write some code to make the open source solutions actually work.

Re: Web Scraping 101 with Python

#66

Post by tester756 »

thaumasiotes wrote: post #50
dewey wrote: post #47

Earlier quoted context omitted.

How are you going to deal with values generated by JS and used to sign requests?

If they're really being generated client-side, you're free to generate them yourself by any means you want. But also, that's a strange thing for the website to do, since it's applying a security feature (signatures) in a way that prevents it from providing any security. If they're generated server-side like you would expect, and sent to the client, you'd get them the same way you get anything else, by asking for them…

>If they're really being generated client-side, you're free to generate them yourself by any means you want. But also, that's a strange thing for the website to do

what??

Page loads -> Javascript sends request to backend -> it returns data -> javascript does stuff with it and renders it.

Re: Web Scraping 101 with Python

#67

Post by RhodesianHunter »

monkeybutton wrote: post #42
RhodesianHunter wrote: post #31

Earlier quoted context omitted.

I'd say that really depends on your scale and what you're doing with the content you scrape. In my experience with large scale scraping you're much better off using something like Java where you can more easily have a thread pool with thousands of threads (or better yet, Kotlin coroutines) handling the crawling itself and a *NUM CORES thread pool handling CPU bound tasks like parsing.

Could you give a ballpark figure for what you mean by large scale scraping? I've only worked on a couple projects, one was a broad (100K to 500K domains) and shallow (root + 1 level of page depth, also with a low cap on the number of children pages). The other just a single domain but scraping around 50K pages from it.

My experience was with e-commerce scraping. Not many domains, but a massive catalogue.

Re: Web Scraping 101 with Python

#68

Post by SirSourdough »

mikesholiu wrote: post #64
philshem wrote: post #46

Before jumping into frameworks, if your data is lucky enough to be stored in an html table: import pandas as pd dfs = pd.read_html(url) Where ‘dfs’ is an array of dataframes - one item for each html table on the page. https://pandas.pydata.org/pandas-docs/stable/reference/api/p...

what does this do

It reads HTML and returns the tables contained in the HTML as pandas dataframes. It’s a simple way to scrape tabular data from websites.

Re: Web Scraping 101 with Python

#70

Post by Topgamer7 »

NDizzle wrote: post #35

I've been involved in many web scraper jobs over the past 25 years or so. The most recent one, which was a long time ago at this point, was using scrapy. I went with XML tools for controlling the DOM. It's worked unbelievably well. It's been running for roughly 5 years at this point. I send a command at a random time between 11pm and 4am to wake up an ec2 instance. It checks its tags to see if it should execute the s…

Using `2to3` might get you 80% of the way there. Although cases like this make tests really valuable.
Post reply on HN