Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping 101 with Python

scrapingbee.com

Post reply on HN
Print view View original

121–130 of 134 posts

Re: Web Scraping 101 with Python

#121

Post by scalableUnicon »

jj_jaq wrote: post #117

The biggest struggle I had while building web scrappers is scaling Selenium. If you need to launch Selenium hundred of thousands times per month, you need a lot of computer power which is really expensive on EC2. A couple years ago, I discovered browserless.io which does this job for you and it's amazing. I really don't know how they made this but it just scales without any limit.

For browserless.io, the developer behind it talks about the tech stack in this podcast: https://runninginproduction.com/podcast/62-browserless-gives...

Re: Web Scraping 101 with Python

#122

Post by Yenrabbit »

philshem wrote: post #46

Before jumping into frameworks, if your data is lucky enough to be stored in an html table: import pandas as pd dfs = pd.read_html(url) Where ‘dfs’ is an array of dataframes - one item for each html table on the page. https://pandas.pydata.org/pandas-docs/stable/reference/api/p...

Handy! I'm also a big fan of pd.read_clipboard() for specific selections.

Re: Web Scraping 101 with Python

#123

Post by philshem »

kortex wrote: post #110
philshem wrote: post #46

Before jumping into frameworks, if your data is lucky enough to be stored in an html table: import pandas as pd dfs = pd.read_html(url) Where ‘dfs’ is an array of dataframes - one item for each html table on the page. https://pandas.pydata.org/pandas-docs/stable/reference/api/p...

Sometimes it's also helpful to use beautiful soup to isolate the elements you want, feed the text of the elements into StringIO and give that to read_html.

Yes, this is a good idea for more complicated cases.

Re: Web Scraping 101 with Python

#124

Post by warsheep »

rinze wrote: post #113
craigmi wrote: post #107

Earlier quoted context omitted.

"I send a command at a random time between 11pm and 4am to wake up an ec2 instance." Any chance you could tell me your setup for this?

Not my project, but if I had to do it I'd try something like the following: * Set an autoscaling group with your instance template, max instances 1, min instances 0, desired instances 0 (nothing is running). * Set up a Lambda function that sets the autoscaling group desired instances to 1. * Link that function to an API Gateway call, give it an auth key, etc. * From any machine you have, set up your cron with a rando…

Why use autoscaling and not just launch the instance directly from lambda? The run time is short so there's no danger of two instances running in parallel

Re: Web Scraping 101 with Python

#125

Post by alephu5 »

strin wrote: post #29
VBprogrammer wrote: post #28

One tip I would pass on when trying to scrape data from a website, start by using wget in mirror mode to download the useful pages. It's much faster to iterate on scraping the data once you have it locally. Also, less likely to accidentally kill the site or attract the attention of the host.

That works only for static page though. Many modern pages would require you to run a selenium or puppetteer to scrape the content.

I did web-scraping professionally for two years, in the order of 10M pages per day. The performance with a browser is abysmal and requires tonnes of memory so not financially viable. We used them for some jobs, but rendered content isn't a problem, you can also simulate the API calls (common) and read the JSON, or regex the script and try to do something with that.

I'd say 99% of the time you can get by without a browser.

Re: Web Scraping 101 with Python

#126

Post by anyfactor »

I have been websraping for almost 4 years now. That is my entire niche.

The problem with web scraping is that you really don't know the ethical point of scraping ends. These days I will reverse engineer a website to minimize the request load and only target specific API endpoints. But then again I am breaching some security measures they have while doing that.

Re: Web Scraping 101 with Python

#127

Post by a1sabau »

isbvhodnvemrwvn wrote: post #82
freedomben wrote: post #25

Earlier quoted context omitted.

I find javascript (node) to be best suited to web scraping personally. Using the same language to scrape/process as you use to develop those interfaces seems most natural.

Especially with stuff like Puppeteer which allows you to execute JS in context of the browser (which admittedly can lead to weird bugs as the functions are serialized and lose context)

I'm using this exact strategy to scrape content directly from DOM using APIs like document.querySelectorAll. You can use the same code in both headless browser clients like Puppeteer or Playwright and DOM clients like cheerio or jsdom (assuming you have a wrapper over document API). Depending on the way a web page was fetched (opened in a browser tab or fetched via nodejs http/https requests), ExtractHtmlContentPlugin, ExtractUrlsPlugin use different DOM wrappers (native, cheerio, jsdom) to scrape the content.

[1] https://github.com/get-set-fetch/scraper

Re: Web Scraping 101 with Python

#128

Post by [deleted] »

pocket_cheese wrote: post #102
dewey wrote: post #52

Earlier quoted context omitted.

I'm not sure what's your point. Of course you can replicate every request in your scraper / with curl if you want to if you know all the input variables. Doing that for web scraping purposes where everything is changing all the time and you have more than one target website is just not feasible if you have to reverse engineer some custom JS for every site. Using some kind of headless browser for modern websites will…

As someone who has done a good bit of scraping, how a website is designed dictates how I scrape. If it's a static website that has consistently structured HTML and is easy to enumerate through all the webpages I'm looking for, then simple python requests code will work. The less clear case is when to use a headless browser vs reverse engineering JS/server side APIs. Typically, I will do like a 10 minute dive into the…

[deleted]

Re: Web Scraping 101 with Python

#129

Post by listenallyall »

zelphirkalt wrote: post #112

Personally I have not needed Beautifulsoup a single time, when web scraping. People say it is better for unclean HTML, which I cannot confirm, because I never needed it and always were able to get my result using LXML + etree with XPath and CSS selectors. Once I also used Scrapy, but still not Beautifulsoup. I am glad there is a guide, that starts with LXML, instead of immediately jumping to Beautifulsoup.

lxml is terrific. I agree, never really understood what beautifulsoup added that lxml couldn't just handle on its own. On the JVM, jsoup is excellent but doesn't support XPath.

Re: Web Scraping 101 with Python

#130

Post by inovica »

alephu5 wrote: post #125
strin wrote: post #29

Earlier quoted context omitted.

That works only for static page though. Many modern pages would require you to run a selenium or puppetteer to scrape the content.

I did web-scraping professionally for two years, in the order of 10M pages per day. The performance with a browser is abysmal and requires tonnes of memory so not financially viable. We used them for some jobs, but rendered content isn't a problem, you can also simulate the API calls (common) and read the JSON, or regex the script and try to do something with that. I'd say 99% of the time you can get by without a bro…

Fully agree. It takes some thought :)
Post reply on HN