Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping 101 with Python

scrapingbee.com

Post reply on HN
Print view View original

41–50 of 134 posts

Re: Web Scraping 101 with Python

#41

Post by cubano »

My last contract job was to build a 100% perfect website mirroring program for a group of lawyers who were interested in building class action lawsuits against some of the more henious scammers out there.

I ended up building like 8 versions of it, literally using every PHP and Python library and resource I could find.

I tried httrack, php-ultimate-web-scraper (from github), headless chromium. headless selenium, and a few others

By far the biggest problem was dealing with JS links...you wouldn't think from the start it would be such a big deal but yet..it was.

Selenium with python turned out to be the winning combination, and of course, it was the last one I tried. Also, this is an ideal project to implement recursion altho you have to be careful about exit conditions.

One thing that was VERY important for performance was not visiting any page more then once because, obviously, certain links in headers and footers are duped sometimes 100s of times.

JS links often made it very difficult to discover the linked page, are certain library calls that were supposed to get this info for you often didn't work.

It was a super fun project, and in the end considering I only worked for 2 months, I shipped some decent code that was getting like 98.6% of the pages perfectly.

The final presentation was interesting...for some reason my client I think got in his head that I wasn't very good programmer or something, and as we ran thru his list of sample sites expecting my program to error out or incorrectly mirror the site, but it handled all 10 of the sites about perfectly and he was rather flabbergasted because he told me it would have taken him a week hand clicking the site for the mirror but instead the program did them all in under an hour.

Re: Web Scraping 101 with Python

#42

Post by monkeybutton »

RhodesianHunter wrote: post #31
monkeybutton wrote: post #26

Earlier quoted context omitted.

I like python for the ease of use and scraping is I/O bound anyways so there's no pressure to switch to a more performant language.

I'd say that really depends on your scale and what you're doing with the content you scrape. In my experience with large scale scraping you're much better off using something like Java where you can more easily have a thread pool with thousands of threads (or better yet, Kotlin coroutines) handling the crawling itself and a *NUM CORES thread pool handling CPU bound tasks like parsing.

Could you give a ballpark figure for what you mean by large scale scraping? I've only worked on a couple projects, one was a broad (100K to 500K domains) and shallow (root + 1 level of page depth, also with a low cap on the number of children pages). The other just a single domain but scraping around 50K pages from it.

Re: Web Scraping 101 with Python

#44

Post by lysecret »

cubano wrote: post #41

My last contract job was to build a 100% perfect website mirroring program for a group of lawyers who were interested in building class action lawsuits against some of the more henious scammers out there. I ended up building like 8 versions of it, literally using every PHP and Python library and resource I could find. I tried httrack, php-ultimate-web-scraper (from github), headless chromium. headless selenium, and a…

What stack did you end up using ?

Re: Web Scraping 101 with Python

#45

Post by gazelle21 »

cubano wrote: post #41

My last contract job was to build a 100% perfect website mirroring program for a group of lawyers who were interested in building class action lawsuits against some of the more henious scammers out there. I ended up building like 8 versions of it, literally using every PHP and Python library and resource I could find. I tried httrack, php-ultimate-web-scraper (from github), headless chromium. headless selenium, and a…

Ha! I am currently building something very similar at work, and JS links are driving me up a wall. Its interesting because you would think its super simple, but I still haven't found a good solution. Luckily my boss is understanding.

Re: Web Scraping 101 with Python

#47

Post by dewey »

thaumasiotes wrote: post #40
dewey wrote: post #34

Earlier quoted context omitted.

> You can do the same thing in your scraper Rendering the page in Puppeteer / Selenium and then scraping it from there sounds like a lot easier than somehow trying to replicate that in your scraper?

Sure. How does that relate to the claim that your scraper is actually unable to make the same requests your browser does?

How are you going to deal with values generated by JS and used to sign requests?

Re: Web Scraping 101 with Python

#48

Post by pknerd »

I have been developing scrapers and crawlers and writing[1] about them for many years and used many Python based libs so far including Selenium. I have write such scrapers for individuals and startups for several purposes. The biggest issue I faced was rendering of dynamic sites and blocking of IPs due to absence of proxies which are not cheap at all, especially for individuals.

Services like Scrapingbee and ScraperAPI are serving quite good for such problems. I personally liked ScraperAPI for rendering dynamic websites due to the better response time.

Shameless Plug: In case if anyone is interested, long time back, I had written about it on my blog which you can read here[2]. Now you do not need to setup remote Chrome instance or anything. What all is required is to hit an API endpoint to fetch content from a dyanmic JS rendered websites.

[1] http://blog.adnansiddiqi.me/tag/scraping/

[2] http://blog.adnansiddiqi.me/scraping-dynamic-websites-using-...

Re: Web Scraping 101 with Python

#49

Post by johtso »

VBprogrammer wrote: post #28

One tip I would pass on when trying to scrape data from a website, start by using wget in mirror mode to download the useful pages. It's much faster to iterate on scraping the data once you have it locally. Also, less likely to accidentally kill the site or attract the attention of the host.

Or just use scrapy's caching functionality. Super convenient.

Re: Web Scraping 101 with Python

#50

Post by thaumasiotes »

dewey wrote: post #47
thaumasiotes wrote: post #40

Earlier quoted context omitted.

Sure. How does that relate to the claim that your scraper is actually unable to make the same requests your browser does?

How are you going to deal with values generated by JS and used to sign requests?

If they're really being generated client-side, you're free to generate them yourself by any means you want. But also, that's a strange thing for the website to do, since it's applying a security feature (signatures) in a way that prevents it from providing any security.

If they're generated server-side like you would expect, and sent to the client, you'd get them the same way you get anything else, by asking for them.

Post reply on HN