Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in Python – The Complete Guide

proxiesapi.com

Post reply on HN
Print view View original

131–140 of 151 posts

Re: Web Scraping in Python – The Complete Guide

#131

Post by mdaniel »

65 wrote: post #4

I'm not sure why Python web scraping is so popular compared to Node.js web scraping. npm has some very well made packages for DOM parsing, and since it's in Javascript we have more native feeling DOM features (e.g. node-html-parser using querySelector instead of select - it just feels a lot more intuitive). It's super easy to scrape with Puppeteer or just regular html parsers on a Lambda.

> I'm not sure why Python web scraping is so popular compared to Node.js web scraping

Take this with a grain of salt, since I am fully cognizant that I'm the outlier in most of these conversations, but Scrapy is A++ the no-kidding best framework for this activity that has been created thus far. So, if there was scrapyjs maybe I'd look into it, but there's not (that I'm aware of) so here we are. This conversation often comes up in any such "well, I just use requests & ..." conversation and if one is happy with main.py and a bunch of requests invocations, I'm glad for you, but I don't want to try and cobble together all the side-band stuff that Scrapy and its ecosystem provide for me in a reusable and predictable way

Also, often those conversations conflate the server side language with the "scrape using headed browser" language which happens to be the same one. So, if one is using cheerio https://github.com/cheeriojs/cheerio> then sure node can be a fine thing - if the blog post is all "fire up puppeteer, what can go wrong?!" then there is the road to ruin of doing battle with all kinds of detection problems since it's kind of a browser but kind of not

I, under no circumstances, want the target site running their JS during my crawl runs. I fully accept responsibility for reproducing any XHR or auth or whatever to find the 3 URLs that I care about, without downloading every thumbnail and marketing JS and beacon and and and. I'm also cognizant that my traffic will thus stand out since it uniquely does not make the beacon and marketing calls, but my experience has been that I get the ban hammer less often with my target fetches than trying to pretend to be a browser with a human on the keyboard/mouse but is not

Re: Web Scraping in Python – The Complete Guide

#132

Post by NortySpock »

NortySpock wrote: post #130
greenie_beans wrote: post #107

Earlier quoted context omitted.

this is what i try to do but i want to learn more about approaches like this, do you know any good resources about how to design ETL pipelines?

Disclaimer: previous job had a lot of cases where CSVs were dropped by SFTP, your milage may vary..., JSON APIs are said to be a different flavor of crazy... Haven't heard much beyond "ask the Old Ones", but "Murphy's law strikes again", "eventually someone will want that data even though they swore it was unnecessary", "eventually someone will ask for a backfill/replay", "eventually someone will give you a duplicate…

In conclusion...

I find, in data engineering ,the goal is not to prevent everything, it's to be flexible and prepared to handle lots of change, even silly changes, and be able to audit it, observe it, maneuver around it, and keep the mean-time-to-resolution low.

Re: Web Scraping in Python – The Complete Guide

#133

Post by appplication »

dragonwriter wrote: post #109
ethbr1 wrote: post #54

Earlier quoted context omitted.

It's unfortunate that "ETL" stuck in mindshare, as afaik almost all use cases are better with "ELT" I.e. first preserve your raw upstream via a 1:1 copy, then transform/materialize as makes sense for you, before consuming Which makes sense, as ELT models are essentially agile for data... (solution for not knowing what we don't yet know)

I think ETL is right from the perspective where E refers to “from the source of data” and L refers to “to the ultimate store of data”. But the ETL functionality should itself lives in a (sub)system that has its own logical datastore (which may or may not be physically separate from the destination store), and things should be ELT where the L is with respect to that store. So, its E(LTE)L, in a sense.

For those confused as to whether ETL or ELT is ultimately more appropriate for you… almost everyone is really just doing ETLTLTLT or ELTLTLTL anyways. The distinction is really moot.

Re: Web Scraping in Python – The Complete Guide

#134

Post by fireant »

SinjonSuarez wrote: post #44

Check out the cloudscraper library if are having speed/cpu issues with sites that require js/have cloudfare defending them. That plus a proxy list plus threading allows me to make 300 requests a minute across 32 different proxies. Recently implemented it for a project: https://github.com/rezaisrad/discogs/tree/main/src/managers

I've found myself writing the same session/proxy/rate limiting/header faking management code over and over for my scrapers. I've extracted it into it's own service that runs in docker and acts as a MITM proxy between you and target. It is client language agnostic, so you can write scrapers in python, node or whatever and still have great performance.

Highly recommend this approach, it allows you to separate infrastructure code, that gets highly complex as you need more requests, from actual spider/parser code that is usually pretty straightforward and project specific.

https://github.com/jkelin/forward-proxy-manager

Re: Web Scraping in Python – The Complete Guide

#135

Post by nkko »

jot wrote: post #79
zopper wrote: post #48

This guide (and most other guides) are missing a massive tip: Separate the crawling (finding urls and fetching the HTML content) from the scraping step (extracting structured data out of the HTML). More than once, I wrote a scraper that did both of these steps together. Only later I realized that I forgot to extract some information that I need and had to do the costly task of re-crawling and scraping everything. If…

This is how I do it. I send the URLs I want scraped to Urlbox[0] it renders the pages saves HTML (and screenshot and metadata) to my S3 bucket[1]. I get a webhook[2] when it's ready for me to process. I prefer to use Ruby so Nokogiri[3] is the tool I use for scraping step. This has been particularly useful when I've want to scrape some pages live from a web app and don't want to manage running Puppeteer or Playwright…

Does it save the whole page or just the viewport? Just checked the landing page it looks targeted to a specific case of saving “screenshots” and this is also obvious from limitations in the pricing page so it would be unfeasible for larger projects?

Re: Web Scraping in Python – The Complete Guide

#137

Post by OlikEten »

DoodahMan wrote: post #128
daolf wrote: post #66

Earlier quoted context omitted.

Hi Thomas, really sorry you had a bad experience with ScrapingBee. Would you mind sending me the account you used as I wasn't able to find anything under Thomas Isaac or Tillypa and couldn't see what was going wrong then. I'm sure your comment has nothing to do with the fact that you share the same investor as ScraperAPI but I just wanted be sure.

any HN discount by chance? ;) i'm testing y'all out now for a time-sensitive scrape job that must be done by Mar 1st.

I am surprised nobody mentioned https://apify.com/ and they even offer discount for YC startups as ex-graduate from the YC Combinator program

Re: Web Scraping in Python – The Complete Guide

#138

Post by labaron »

f311a wrote: post #89

>BeautifulSoup > Features: Excellent HTML/XML parser, easy web scraping interface, flexible navigation and search. It does not feature any parser. It’s basically a wrapper over lxml. >lxml > Features: Very fast XML and HTML parser. It’s fast, but there are alternatives that are literally 5x faster. This article is just another rewrite of a basic introduction. It’s not a guide, since it does mot describe any issues th…

lxml is written in Cython and is very efficient in my tests. Much faster than BeautifulSoup, which is pure Python.

What alternatives are 5x faster?

Re: Web Scraping in Python – The Complete Guide

#139

Post by SinjonSuarez »

fireant wrote: post #134
SinjonSuarez wrote: post #44

Check out the cloudscraper library if are having speed/cpu issues with sites that require js/have cloudfare defending them. That plus a proxy list plus threading allows me to make 300 requests a minute across 32 different proxies. Recently implemented it for a project: https://github.com/rezaisrad/discogs/tree/main/src/managers

I've found myself writing the same session/proxy/rate limiting/header faking management code over and over for my scrapers. I've extracted it into it's own service that runs in docker and acts as a MITM proxy between you and target. It is client language agnostic, so you can write scrapers in python, node or whatever and still have great performance. Highly recommend this approach, it allows you to separate infrastru…

This is great, was totally in the back of my mind as a next step.

Re: Web Scraping in Python – The Complete Guide

#140

Post by deanishe »

sakisv wrote: post #126
mrtimo wrote: post #24

Earlier quoted context omitted.

would love to learn more about what you are doing with supermarket prices

My main drive was to document the crazy price hikes that's been going on in my home country, Greece, so I'm scraping its 3 biggest supermarkets and keep track of the prices of their products* Had a lot of fun building and automating the scraping, especially in order to get around some bot catching rules that they have. For example one of them blocks all the requests originating from non-residential IPs, so I had to u…

I can't get prices on my local supermarket's website (in Germany) without selecting a specific branch.

Seems kinda suspicious to me.

Post reply on HN