Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping 101 with Python

scrapingbee.com

Post reply on HN
Print view View original

111–120 of 134 posts

Re: Web Scraping 101 with Python

#111

Post by concreteblock »

cubano wrote: post #41

My last contract job was to build a 100% perfect website mirroring program for a group of lawyers who were interested in building class action lawsuits against some of the more henious scammers out there. I ended up building like 8 versions of it, literally using every PHP and Python library and resource I could find. I tried httrack, php-ultimate-web-scraper (from github), headless chromium. headless selenium, and a…

Can you explain why the lawyers needed mirror sites? Does mirroring a site mean making a local copy of it?

Re: Web Scraping 101 with Python

#112

Post by zelphirkalt »

Personally I have not needed Beautifulsoup a single time, when web scraping. People say it is better for unclean HTML, which I cannot confirm, because I never needed it and always were able to get my result using LXML + etree with XPath and CSS selectors. Once I also used Scrapy, but still not Beautifulsoup. I am glad there is a guide, that starts with LXML, instead of immediately jumping to Beautifulsoup.

Re: Web Scraping 101 with Python

#113

Post by rinze »

craigmi wrote: post #107
NDizzle wrote: post #35

I've been involved in many web scraper jobs over the past 25 years or so. The most recent one, which was a long time ago at this point, was using scrapy. I went with XML tools for controlling the DOM. It's worked unbelievably well. It's been running for roughly 5 years at this point. I send a command at a random time between 11pm and 4am to wake up an ec2 instance. It checks its tags to see if it should execute the s…

"I send a command at a random time between 11pm and 4am to wake up an ec2 instance." Any chance you could tell me your setup for this?

Not my project, but if I had to do it I'd try something like the following:

* Set an autoscaling group with your instance template, max instances 1, min instances 0, desired instances 0 (nothing is running).

* Set up a Lambda function that sets the autoscaling group desired instances to 1.

* Link that function to an API Gateway call, give it an auth key, etc.

* From any machine you have, set up your cron with a random sleep and a curl call to the API.

And that should do the trick, I think.

Re: Web Scraping 101 with Python

#114

Post by antisthenes »

philshem wrote: post #46

Before jumping into frameworks, if your data is lucky enough to be stored in an html table: import pandas as pd dfs = pd.read_html(url) Where ‘dfs’ is an array of dataframes - one item for each html table on the page. https://pandas.pydata.org/pandas-docs/stable/reference/api/p...

Holy crap, is there anything pandas can't do?

Re: Web Scraping 101 with Python

#115

Post by wp381640 »

NDizzle wrote: post #35

I've been involved in many web scraper jobs over the past 25 years or so. The most recent one, which was a long time ago at this point, was using scrapy. I went with XML tools for controlling the DOM. It's worked unbelievably well. It's been running for roughly 5 years at this point. I send a command at a random time between 11pm and 4am to wake up an ec2 instance. It checks its tags to see if it should execute the s…

the tl;dr for all web scraping is to just use scrapy (and scrapyd) - otherwise you end up just writing a poorer implementation of what has already been built

My only recent change is that we no longer use Items and ItemLoaders from scrapy - we've replaced it with a custom pipeline of Pydantic schemas and objects

Re: Web Scraping 101 with Python

#116

Post by hakube »

daolf wrote: post #10
tyingq wrote: post #6

PyPpeteer might be worth a look as well. Basically a port of the JS puppeteer project that drives headless Chrome via the Devtools API. As mentioned elsewhere, using anything other than headless isn't useful beyond a fairly narrow scope these days. https://github.com/pyppeteer/pyppeteer

I think you'd be surprised by the amount of website you can scrape without an headless browser. Even Google SERP can be scraped with a simple HTTP client.

How do you deal with JavaScript then?

Re: Web Scraping 101 with Python

#117

Post by jj_jaq »

The biggest struggle I had while building web scrappers is scaling Selenium. If you need to launch Selenium hundred of thousands times per month, you need a lot of computer power which is really expensive on EC2.

A couple years ago, I discovered browserless.io which does this job for you and it's amazing. I really don't know how they made this but it just scales without any limit.

Re: Web Scraping 101 with Python

#118

Post by ses1984 »

rinze wrote: post #113
craigmi wrote: post #107

Earlier quoted context omitted.

"I send a command at a random time between 11pm and 4am to wake up an ec2 instance." Any chance you could tell me your setup for this?

Not my project, but if I had to do it I'd try something like the following: * Set an autoscaling group with your instance template, max instances 1, min instances 0, desired instances 0 (nothing is running). * Set up a Lambda function that sets the autoscaling group desired instances to 1. * Link that function to an API Gateway call, give it an auth key, etc. * From any machine you have, set up your cron with a rando…

>From any machine you have, set up your cron with a random sleep and a curl call to the API.

You might as well just call the ASG API directly.

Re: Web Scraping 101 with Python

#119

Post by Frost1x »

antisthenes wrote: post #114
philshem wrote: post #46

Before jumping into frameworks, if your data is lucky enough to be stored in an html table: import pandas as pd dfs = pd.read_html(url) Where ‘dfs’ is an array of dataframes - one item for each html table on the page. https://pandas.pydata.org/pandas-docs/stable/reference/api/p...

Holy crap, is there anything pandas can't do?

Ingest bamboo? Sorry, couldn't resist.

Re: Web Scraping 101 with Python

#120

Post by logn »

kruchone wrote: post #27

In my career I found several reasons not to use regular expressions for parsing an HTML response, but the largest was the fact that it may work for 'properly formed' documents, but you would be surprised how lax all browsers are about requiring the document to be well-formed. Your regex, unless particularly handled, will not be able to handle sites like this (and there are a lot, at least from my career experience).…

I think scraping is just inherently brittle whether you go by the DOM traversal or by regex. AI may have the best potential. Regex can be slightly more brittle like you point out with commented html or myriad other problems, but it can also be less brittle than DOM if you craft more lenient patterns. The main problem I found was regex's not being performant due to recursiveness and stack overflows (Google's RE2 lib addresses this). My favorite performance trick is to use negated character classes rather than a dot, /]*>/
Post reply on HN