Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in Python – The Complete Guide

proxiesapi.com

Post reply on HN
Print view View original

81–90 of 151 posts

Re: Web Scraping in Python – The Complete Guide

#81

Post by simonw »

philippta wrote: post #14

Shameless plug: Flyscrape[0] eliminates a lot of boilerplate code that is otherwise necessary when building a scraper from scratch, while still giving you the flexibility to extract data that perfectly fit your needs. It comes as a single binary executable and runs small JavaScript files without having to deal with npm or node (or python). You can have a collection of small and isolated scraping scripts, rather than…

Does Flyscrape execute JavaScript that is on the page (e.g. by running a headless browser) or is it just parsing HTML and using CSS selectors to extract code from a static DOM?

Re: Web Scraping in Python – The Complete Guide

#82

Post by 0xDEADFED5 »

thrdbndndn wrote: post #13
simonw wrote: post #2

I strongly recommend adding Playwright to your set of tools for Python web scraping. It's by far the most powerful and best designed browser automation tool I've ever worked with. I use it for my shot-scraper CLI tool: https://shot-scraper.datasette.io/ - which lets you scrape web pages directly from the command line by running JavaScript against pages to extract JSON data: https://shot-scraper.datasette.io/en/stable…

Kinda tangent, but Playwright's doc (specifically, the intro https://playwright.dev/python/docs/intro ) confuses me. It asks you to write a test and then run `pytest`, instead of just letting you to use the library directly (which exists, but is buried in the main text: https://playwright.dev/python/docs/library ). I understand that using Playwright in tests is probably the most common use case (it's even in their ta…

agreed, playwright is great. it even has device emulation profiles built in, so you can for instance use an iphone device with the right screen size/browser/metadata automatically

Re: Web Scraping in Python – The Complete Guide

#83

Post by justinzollars »

moritonal wrote: post #77
justinzollars wrote: post #72

I've tried this for the first time recently in 10 years - it's really become a miserable chore. There are so many countermeasures deployed to web scraping. The best path forward I could imagine is utilizing LLMs, taking screenshots and having the AI tell me what it sees on the page; but even gathering links is difficult. xml site maps for the win.

Literally step for step what I spent my weekend putting together. Here's the preview blog I wrote on it. https://blog.bonner.is/using-ai-to-find-fencing-courses-in-l... Only step you missed was embeddings to avoid all the privacy pages, and a cookie banner blocker (which arguably the AI could navigate if I cared).

Awesome! Things have gotten so bad this is the only alternative. I tried building a hobby search engine then quickly gave up, but did imagine how I would do the scraping!

Re: Web Scraping in Python – The Complete Guide

#84

Post by generalizations »

bilater wrote: post #75

I'm convinced there is a gold mine sitting right in front of us ready to be picked by someone who can intelligently combine web scraping knowledge with LLMs e.g. scrape data, feed it into LLMs do get insights in an automated fashion. I don't know exactly what the final manifestation looks like but its there and will be super obvious when someone does it.

I tried that. Turns out that LLM-generated regex is still better (and a lot faster) than using an LLM directly.

Re: Web Scraping in Python – The Complete Guide

#85

Post by nathell »

zopper wrote: post #48

This guide (and most other guides) are missing a massive tip: Separate the crawling (finding urls and fetching the HTML content) from the scraping step (extracting structured data out of the HTML). More than once, I wrote a scraper that did both of these steps together. Only later I realized that I forgot to extract some information that I need and had to do the costly task of re-crawling and scraping everything. If…

Yes!

My Clojure scraping framework [0] facilitates that kind of workflow, and I’ve been using it to scrape/restructure massive sites (millions of pages). I guess I’m going to write a blog post about scraping with it at scale. Although it doesn’t really scale much above that – it’s meant for single-machine loads at the moment – it could be enhanced to support that kind of workflow rather easily.

[0]: https://github.com/nathell/skyscraper

Re: Web Scraping in Python – The Complete Guide

#86

Post by pocket_cheese »

thijsvandien wrote: post #16

I thought scraping is kind of dead given all the CAPTCHAs and auth walls everywhere. The article does mention proxies and rate limiting, but could anyone with (recent) practical experience elaborate on dealing with such challenges?

A few different techniques -

1. use mobile phone proxies. Because of how mobile phone networks do NAT, basically it means that thousands of people share IPs and are much less like to get blocked.

2. Reverse engineer APIs if the data you want is returned in an ajax call.

3. Use a captcha solving service to defeat captchas. There's many and they are cheap.

4. Use an actual phone or get really good at convincing the server you are a mobile phone.

5. Buy 1000s of fake emails to simulate multiple accounts.

6. Experiment. Experiment. Experiment. Get some burner accounts. Figure out if they have request per min/hour/day throttling. See what behavior triggers a cloudflare captchas. Check if different variables such as email domain, useragent, voip vs non-voip sms based 2fa. your goal is to simulate a human. So if you sequentially enumerate through every document - that might be what get's you flagged.

Best of luck and happy scraping!

Re: Web Scraping in Python – The Complete Guide

#87

Post by zffr »

Here are some tips not mentioned:

1. /robots.txt can sometimes have useful info for scraping a website. It will often include links to sitemaps that let you enumerate all pages on a site. This is a useful library for fetching/parsing a sitemap (https://github.com/mediacloud/ultimate-sitemap-parser)

2. Instead of parsing HTML tags, sometimes you can extract the data you need through structured metadata. This is a useful library for extracting it into JSON (https://github.com/scrapinghub/extruct)

Re: Web Scraping in Python – The Complete Guide

#89

Post by f311a »

>BeautifulSoup

> Features: Excellent HTML/XML parser, easy web scraping interface, flexible navigation and search.

It does not feature any parser. It’s basically a wrapper over lxml.

>lxml

> Features: Very fast XML and HTML parser.

It’s fast, but there are alternatives that are literally 5x faster.

This article is just another rewrite of a basic introduction. It’s not a guide, since it does mot describe any issues that you face in practice.

Re: Web Scraping in Python – The Complete Guide

#90

Post by calf »

I've been writing rudimentary Python scripts to scrape online recipe websites for my hobby cooking purposes, and I wish there was some general software that could do this more simply. One of the websites has started making their images unclickable, so measures like that make me think it might become harder to automatically fetch such content.
Post reply on HN