Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in Python – The Complete Guide

proxiesapi.com

Post reply on HN
Print view View original

111–120 of 151 posts

Re: Web Scraping in Python – The Complete Guide

#111

Post by kalev »

nico wrote: post #40
hubraumhugo wrote: post #29

I got so annoyed by this kind of tedious web scraping work (maintenance, proxies, etc.) that I'm now trying to fully automate it with LLMs. AI should automate repetitive and un-creative work, and web scraping definitely fits this description. It's a boring but challenging problem. I've started using LLMs to generate web scrapers and data processing steps on the fly that adapt to website changes. Using an LLM for ever…

Kadoa looks great. For tool discovery/usage, are you using LangChain or something else? Also, do you support scraping private sites, ie. sites that require a login/password to access the data to scrape? Thank you!

+1 on the question about scraping behind authentication. One huge use case we have as an ecommerce store is to crawl data from our vendors, which do not have (or incomplete) export files

Re: Web Scraping in Python – The Complete Guide

#112

Post by black3r »

photochemsyn wrote: post #55
zopper wrote: post #48

This guide (and most other guides) are missing a massive tip: Separate the crawling (finding urls and fetching the HTML content) from the scraping step (extracting structured data out of the HTML). More than once, I wrote a scraper that did both of these steps together. Only later I realized that I forgot to extract some information that I need and had to do the costly task of re-crawling and scraping everything. If…

I've found this approach works really well using JavaScript and puppeteer for the first stage, and then Python for the second stage (the re module for regular expressions is nice here IMO). JS/puppeter seems a bit easier for things like rotating user agents, from article: > "Websites often block scrapers via blocked IP ranges or blocking characteristic bot activity through heuristics. Solutions: Slow down requests, p…

If you're using JS in the first step just because you need puppeteer, check out playwright. It's what the original authors of puppeteer are working on now and it's been more actively developed in the past few years, very similar in usage and features, but it also has an official python wrapper package.

Re: Web Scraping in Python – The Complete Guide

#113

Post by black3r »

3abiton wrote: post #97
simonw wrote: post #2

I strongly recommend adding Playwright to your set of tools for Python web scraping. It's by far the most powerful and best designed browser automation tool I've ever worked with. I use it for my shot-scraper CLI tool: https://shot-scraper.datasette.io/ - which lets you scrape web pages directly from the command line by running JavaScript against pages to extract JSON data: https://shot-scraper.datasette.io/en/stable…

How does it compare to selenium or puppeteer?

Playwright is a rewrite of puppeteer by people who worked on puppeteer before, but now under Microsoft instead of Github. Not sure if it reached feature parity yet, but all the things we used to do with puppeteer work with playwright, and it seems to be more actively developed.

Re: Web Scraping in Python – The Complete Guide

#114

Post by [deleted] »

daolf wrote: post #66
thomasisaac wrote: post #57

Earlier quoted context omitted.

Weird, we found the exact opposite - what were you scraping? ScrapingBee really struggles on so many domains - ScraperAPI is almost as good as Brightdata when it comes to hard to beat sites.

Hi Thomas, really sorry you had a bad experience with ScrapingBee. Would you mind sending me the account you used as I wasn't able to find anything under Thomas Isaac or Tillypa and couldn't see what was going wrong then. I'm sure your comment has nothing to do with the fact that you share the same investor as ScraperAPI but I just wanted be sure.

[deleted]

Re: Web Scraping in Python – The Complete Guide

#115

Post by hubraumhugo »

nico wrote: post #40
hubraumhugo wrote: post #29

I got so annoyed by this kind of tedious web scraping work (maintenance, proxies, etc.) that I'm now trying to fully automate it with LLMs. AI should automate repetitive and un-creative work, and web scraping definitely fits this description. It's a boring but challenging problem. I've started using LLMs to generate web scrapers and data processing steps on the fly that adapt to website changes. Using an LLM for ever…

Kadoa looks great. For tool discovery/usage, are you using LangChain or something else? Also, do you support scraping private sites, ie. sites that require a login/password to access the data to scrape? Thank you!

We found LangChain and other agentic frameworks to have too much overhead, so we built our own tailored orchestration layer. Authenticated scraping is currently in beta, could you email me your use case (see my profile)?

Re: Web Scraping in Python – The Complete Guide

#116

Post by thomasisaac »

hubraumhugo wrote: post #29

I got so annoyed by this kind of tedious web scraping work (maintenance, proxies, etc.) that I'm now trying to fully automate it with LLMs. AI should automate repetitive and un-creative work, and web scraping definitely fits this description. It's a boring but challenging problem. I've started using LLMs to generate web scrapers and data processing steps on the fly that adapt to website changes. Using an LLM for ever…

Incredible product, will give it a spin soon. How do you do under volume? I tried it out with Google but it was quite slow.

Re: Web Scraping in Python – The Complete Guide

#117

Post by jjice »

greenie_beans wrote: post #107
jjice wrote: post #50

Earlier quoted context omitted.

I've found this to be a good practice for ETL in general. Separate the steps, and save the raw data from "E" if you can because it makes testing and verifying "T" later much easier.

this is what i try to do but i want to learn more about approaches like this, do you know any good resources about how to design ETL pipelines?

I wish I did. I currently work at a startup with our core offering being ETL, so I've learned along the way as we've continued. If anyone has any, I'd love to hear as well.

Keeping raw data when possible has been huge. We keep some in our codebase for quick tests during development and then we keep raws from production runs that we can evaluate with each change, giving us an idea of the production impact of the change.

Re: Web Scraping in Python – The Complete Guide

#118

Post by rashkov »

bfeynman wrote: post #78
bilater wrote: post #75

I'm convinced there is a gold mine sitting right in front of us ready to be picked by someone who can intelligently combine web scraping knowledge with LLMs e.g. scrape data, feed it into LLMs do get insights in an automated fashion. I don't know exactly what the final manifestation looks like but its there and will be super obvious when someone does it.

I feel that the more immediate and impactful opportunity that people are doing is instead of scraping to get/understand content. LLM agents can just interactively navigate websites and perform actions. Parsing/Scraping can be brittle with changes, but an LLM agent to perform an action can just follow steps to search, click on results, and navigate like a human would

Are you aware of any projects for this? I began to build my own but quickly saw that the context window is not large enough to hold the DOM of many websites. I began to strip unnecessary things from the DOM but it became a bit of a slog. L

Re: Web Scraping in Python – The Complete Guide

#119

Post by SinjonSuarez »

mndgs wrote: post #108
SinjonSuarez wrote: post #44

Check out the cloudscraper library if are having speed/cpu issues with sites that require js/have cloudfare defending them. That plus a proxy list plus threading allows me to make 300 requests a minute across 32 different proxies. Recently implemented it for a project: https://github.com/rezaisrad/discogs/tree/main/src/managers

Nicely written scraper, btw. Good code.

appreciate that! as a few mentioned here, there’s a lot of useful scraping tools/libraries to leverage these days. headless selenium no longer seems to make sense to me for most use cases

Re: Web Scraping in Python – The Complete Guide

#120

Post by sam2426679 »

3abiton wrote: post #97
simonw wrote: post #2

I strongly recommend adding Playwright to your set of tools for Python web scraping. It's by far the most powerful and best designed browser automation tool I've ever worked with. I use it for my shot-scraper CLI tool: https://shot-scraper.datasette.io/ - which lets you scrape web pages directly from the command line by running JavaScript against pages to extract JSON data: https://shot-scraper.datasette.io/en/stable…

How does it compare to selenium or puppeteer?

Ime playwright is selenium plus some, e.g. you can inspect network activity without having a separately configured proxy.
Post reply on HN