Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in Python – The Complete Guide

proxiesapi.com

Post reply on HN
Print view View original

51–60 of 151 posts

Re: Web Scraping in Python – The Complete Guide

#51

Post by jawerty »

accidbuddy wrote: post #22
caesil wrote: post #17

Earlier quoted context omitted.

Not only is scraping not dead but it has won the arms race. There are ways around every defense, and this will only accelerate as AI advances. The CAPTCHAs and walls are more of a desperate, doomed retreat.

Some months ago, I had problems with captcha. I tried to write an application to access many drugstores and compare the price, but captcha with login system fail the mission. Do you have any piece of advice for me?

Try 2captcha https://2captcha.com/2captcha-api

Re: Web Scraping in Python – The Complete Guide

#53

Post by iamacyborg »

jjice wrote: post #50
zopper wrote: post #48

This guide (and most other guides) are missing a massive tip: Separate the crawling (finding urls and fetching the HTML content) from the scraping step (extracting structured data out of the HTML). More than once, I wrote a scraper that did both of these steps together. Only later I realized that I forgot to extract some information that I need and had to do the costly task of re-crawling and scraping everything. If…

I've found this to be a good practice for ETL in general. Separate the steps, and save the raw data from "E" if you can because it makes testing and verifying "T" later much easier.

I realise from working a few places that this isn't entirely common practice, but when we built the data warehouse at a startup I worked at, we engaged with a consultancy who taught us the fundamentals of how to do it properly.

One of those fundamentals was separating out the steps of landing the data vs subsequent normalisation and transformation steps.

Re: Web Scraping in Python – The Complete Guide

#54

Post by ethbr1 »

iamacyborg wrote: post #53
jjice wrote: post #50

Earlier quoted context omitted.

I've found this to be a good practice for ETL in general. Separate the steps, and save the raw data from "E" if you can because it makes testing and verifying "T" later much easier.

I realise from working a few places that this isn't entirely common practice, but when we built the data warehouse at a startup I worked at, we engaged with a consultancy who taught us the fundamentals of how to do it properly. One of those fundamentals was separating out the steps of landing the data vs subsequent normalisation and transformation steps.

It's unfortunate that "ETL" stuck in mindshare, as afaik almost all use cases are better with "ELT"

I.e. first preserve your raw upstream via a 1:1 copy, then transform/materialize as makes sense for you, before consuming

Which makes sense, as ELT models are essentially agile for data... (solution for not knowing what we don't yet know)

Re: Web Scraping in Python – The Complete Guide

#55

Post by photochemsyn »

zopper wrote: post #48

This guide (and most other guides) are missing a massive tip: Separate the crawling (finding urls and fetching the HTML content) from the scraping step (extracting structured data out of the HTML). More than once, I wrote a scraper that did both of these steps together. Only later I realized that I forgot to extract some information that I need and had to do the costly task of re-crawling and scraping everything. If…

I've found this approach works really well using JavaScript and puppeteer for the first stage, and then Python for the second stage (the re module for regular expressions is nice here IMO).

JS/puppeter seems a bit easier for things like rotating user agents, from article:

> "Websites often block scrapers via blocked IP ranges or blocking characteristic bot activity through heuristics. Solutions: Slow down requests, properly mimic browsers, rotate user agents and proxies."

Re: Web Scraping in Python – The Complete Guide

#56

Post by naiv »

hubraumhugo wrote: post #29

I got so annoyed by this kind of tedious web scraping work (maintenance, proxies, etc.) that I'm now trying to fully automate it with LLMs. AI should automate repetitive and un-creative work, and web scraping definitely fits this description. It's a boring but challenging problem. I've started using LLMs to generate web scrapers and data processing steps on the fly that adapt to website changes. Using an LLM for ever…

Minimum extraction cost 100 credits , so only 250 pages could be parsed with the regular plan?

Re: Web Scraping in Python – The Complete Guide

#57

Post by thomasisaac »

screye wrote: post #25
croemer wrote: post #18

Earlier quoted context omitted.

I've used ScrapingBee which has similar pricing and has worked well, can't say which one is better: https://www.scrapingbee.com/

We used to be on ScraperAPI, but moved to ScrapingBee after more frequent failures from ScraperAPI. If your scraping needs have realtime requirements, then I'd recommend ScrapingBee.

Weird, we found the exact opposite - what were you scraping?

ScrapingBee really struggles on so many domains - ScraperAPI is almost as good as Brightdata when it comes to hard to beat sites.

Re: Web Scraping in Python – The Complete Guide

#58

Post by BiteCode_dev »

zopper wrote: post #48

This guide (and most other guides) are missing a massive tip: Separate the crawling (finding urls and fetching the HTML content) from the scraping step (extracting structured data out of the HTML). More than once, I wrote a scraper that did both of these steps together. Only later I realized that I forgot to extract some information that I need and had to do the costly task of re-crawling and scraping everything. If…

The problem is crawling is generally optimized with info you find in the page.

Re: Web Scraping in Python – The Complete Guide

#59

Post by nlunbeck »

thijsvandien wrote: post #16

I thought scraping is kind of dead given all the CAPTCHAs and auth walls everywhere. The article does mention proxies and rate limiting, but could anyone with (recent) practical experience elaborate on dealing with such challenges?

I've noticed many smaller and medium sites only use client-side CAPTCHAs/paywalls

Re: Web Scraping in Python – The Complete Guide

#60

Post by bbkane »

zopper wrote: post #48

This guide (and most other guides) are missing a massive tip: Separate the crawling (finding urls and fetching the HTML content) from the scraping step (extracting structured data out of the HTML). More than once, I wrote a scraper that did both of these steps together. Only later I realized that I forgot to extract some information that I need and had to do the costly task of re-crawling and scraping everything. If…

Yes!! https://beepb00p.xyz/unnecessary-db.html really changed how I think about data manipulation, mostly with this principle.
Post reply on HN