Web Scraping in Python – The Complete Guide • Hacker News Board

Re: Web Scraping in Python – The Complete Guide

#41

Post by spaniard89277 » Tue, Feb 20, 2024, 5:20 PM UTC

thrdbndndn wrote: post #21

65 wrote: post #4

I'm not sure why Python web scraping is so popular compared to Node.js web scraping. npm has some very well made packages for DOM parsing, and since it's in Javascript we have more native feeling DOM features (e.g. node-html-parser using querySelector instead of select - it just feels a lot more intuitive). It's super easy to scrape with Puppeteer or just regular html parsers on a Lambda.

To me it's mainly the following three reasons, but take it with a grain of salt since my JS is not as fluent as Python. 1. the async nature of JS is surprisingly detrimental when writing scraping script. It's hard to describe, but it makes have a mental image of the whole code base or workflow harder. Writing mostly sync code and only use things like ThreadPoolExecutor (not even Threading directly) when necessary has…

I've had some experiences with selenium and now I'm using puppeteer, and I honestly don't see the problem with JS. It's true that I have not much experience coding but it seems to me that Pupeteer + Flask serving ML to extract data is the cake. Also, being able to play around evaluating expressions in pupeteer, etc, makes it manageable.

Maybe I lack experience but I don't see JS being a barrier.

I would like to know what kind of string work are you doing. I can't imagine being dependent on parsing strings and such, that looks very easy to break, even easier that css selector dance.

Re: Web Scraping in Python – The Complete Guide

#42

Post by mr_00ff00 » Tue, Feb 20, 2024, 5:20 PM UTC

cnqso wrote: post #12

Any modern web scraping set up is going to require browser agents. You will probably have to build your own tools to get anything from a major social media platform, or even NYT articles.

May be misunderstanding what you mean by “browser agents” but I’ve done some web scraping that had dynamic content and it was easy with a simple chrome driver / gecko driver + scraper crate in Rust

Re: Web Scraping in Python – The Complete Guide

#43

Post by PaulHoule » Tue, Feb 20, 2024, 5:21 PM UTC

simonw wrote: post #15

thrdbndndn wrote: post #13

Earlier quoted context omitted.

Kinda tangent, but Playwright's doc (specifically, the intro https://playwright.dev/python/docs/intro ) confuses me. It asks you to write a test and then run `pytest`, instead of just letting you to use the library directly (which exists, but is buried in the main text: https://playwright.dev/python/docs/library ). I understand that using Playwright in tests is probably the most common use case (it's even in their ta…

Hah yeah that's confusing. https://playwright.dev/python/docs/intro is actually the documentation for pytest-playwright - their pytest plugin. https://playwright.dev/python/docs/library is the documentation for their automation library. I just filed an issue pointing out that this is confusing. https://github.com/microsoft/playwright/issues/29579

Back in the day I used to use HTMLUnit

https://htmlunit.sourceforge.io/

to crawl Javascript-based sites from Java. I think it was originally intended for integration tests but it sure works well for webcrawlers.

I just wrote a Python-based webcrawler this weekend for a small set of sites that is connected to a bookmark manager (you bookmark a page, it crawls related pages, builds database records, copies images, etc.) and had a very easy time picking out relevant links, text and images w/ CSS selectors and beautifulsoup. This time I used a database to manage the frontier because the system is interactive (you add a new link and it ought to get crawled quickly) but for a long time my habit was writing crawlers that read the frontier for pass N from a text file which is one URL per line and then write the frontier for pass N+1 to another text file because this kind of crawler is not only simple to write but it doesn't get stuck in web traps.

I have a few of these systems that do very heterogenous processing of mostly scraped content and something think about setting up a celery server to break work up into tasks .

Re: Web Scraping in Python – The Complete Guide

#44

Post by SinjonSuarez » Tue, Feb 20, 2024, 5:22 PM UTC

Check out the cloudscraper library if are having speed/cpu issues with sites that require js/have cloudfare defending them. That plus a proxy list plus threading allows me to make 300 requests a minute across 32 different proxies. Recently implemented it for a project: https://github.com/rezaisrad/discogs/tree/main/src/managers

Re: Web Scraping in Python – The Complete Guide

#45

Post by nico » Tue, Feb 20, 2024, 5:23 PM UTC

accidbuddy wrote: post #22

caesil wrote: post #17

Earlier quoted context omitted.

Not only is scraping not dead but it has won the arms race. There are ways around every defense, and this will only accelerate as AI advances. The CAPTCHAs and walls are more of a desperate, doomed retreat.

Some months ago, I had problems with captcha. I tried to write an application to access many drugstores and compare the price, but captcha with login system fail the mission. Do you have any piece of advice for me?

Not sure about the currently available tools, given the break-neck speed of AI progress, but a couple of years ago I built a scraper that used a captcha-solving service, they sell something like 1000 solutions for $10, it was super cheap. The process was a bit slow because they were using humans to solve the captchas, but it worked really well

Re: Web Scraping in Python – The Complete Guide

#46

Post by tnolet » Tue, Feb 20, 2024, 5:29 PM UTC

simonw wrote: post #2

I strongly recommend adding Playwright to your set of tools for Python web scraping. It's by far the most powerful and best designed browser automation tool I've ever worked with. I use it for my shot-scraper CLI tool: https://shot-scraper.datasette.io/ - which lets you scrape web pages directly from the command line by running JavaScript against pages to extract JSON data: https://shot-scraper.datasette.io/en/stable…

100%. Playwright (which does have Python support) is completely owning this scene. The robustness is amazing.

Re: Web Scraping in Python – The Complete Guide

#47

Post by thundergolfer » Tue, Feb 20, 2024, 5:31 PM UTC

simonw wrote: post #2

I strongly recommend adding Playwright to your set of tools for Python web scraping. It's by far the most powerful and best designed browser automation tool I've ever worked with. I use it for my shot-scraper CLI tool: https://shot-scraper.datasette.io/ - which lets you scrape web pages directly from the command line by running JavaScript against pages to extract JSON data: https://shot-scraper.datasette.io/en/stable…

We use shot-scraper internally to automate keeping screenshots in our documentation up-to-date. Thanks for the tool![1]

Agree that Playwright is great. It's super easy to run on Modal.[2]

1. https://modal.com/docs/guide/workspaces#dashboard

2. https://modal.com/docs/examples/web-scraper#a-simple-web-scr...

Re: Web Scraping in Python – The Complete Guide

#48

Post by zopper » Tue, Feb 20, 2024, 5:32 PM UTC

This guide (and most other guides) are missing a massive tip: Separate the crawling (finding urls and fetching the HTML content) from the scraping step (extracting structured data out of the HTML).

More than once, I wrote a scraper that did both of these steps together. Only later I realized that I forgot to extract some information that I need and had to do the costly task of re-crawling and scraping everything.

If you do this in two steps, you can always go back, change the scraper and quickly rerun it on historical data instead of re-crawling everything from scratch.

Re: Web Scraping in Python – The Complete Guide

#49

Post by jonatron » Tue, Feb 20, 2024, 5:37 PM UTC

annowiki wrote: post #20

caesil wrote: post #17

Earlier quoted context omitted.

Not only is scraping not dead but it has won the arms race. There are ways around every defense, and this will only accelerate as AI advances. The CAPTCHAs and walls are more of a desperate, doomed retreat.

How do you get around 403/401's from WSJ/Reuters/Axios? Because I've tried user agent manipulation and it seems like I'd have to use selenium and headless to deal with them.

If curl-impersonate works, it's probably TLS fingerprinting.

Re: Web Scraping in Python – The Complete Guide

#50

Post by jjice » Tue, Feb 20, 2024, 5:37 PM UTC

zopper wrote: post #48

This guide (and most other guides) are missing a massive tip: Separate the crawling (finding urls and fetching the HTML content) from the scraping step (extracting structured data out of the HTML). More than once, I wrote a scraper that did both of these steps together. Only later I realized that I forgot to extract some information that I need and had to do the costly task of re-crawling and scraping everything. If…

I've found this to be a good practice for ETL in general. Separate the steps, and save the raw data from "E" if you can because it makes testing and verifying "T" later much easier.