Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in 2016

franciskim.co

Post reply on HN
Print view View original

31–40 of 402 posts

Re: Web Scraping in 2016

#32

Post by franciskim »

dchuk wrote: post #4

Scraping with Selenium in Docker is pretty great, especially because you can use the Docker API itself to spin up/shut down containers at will. So you can spin up a container to hit a specific URL in a second, scrape whatever you're looking for, then kill the container. This can be done via a job queue (sidekiq if you're using Ruby) to do all sorts of fun stuff. That aside, hitting Insta like this is playing with fir…

True that, I hope Zuck sues me so I'll get extra famous

Re: Web Scraping in 2016

#34

Post by franciskim »

elchief wrote: post #21

To fight scrapers, we show some values as images that look like text (but not all the time) And we insert random (non-visible) html and css classes in our site to screw with em, and use randomized css classnames. This fucks with xpaths and css selectors. You can't stop them, but you can make their lives painful.

Haven't come across those yet but yes I guess it could be painful.

Re: Web Scraping in 2016

#35

Post by franciskim »

mosburger wrote: post #12

> AngelList even detects PhamtomJS (have not seen other sites do this). I run a site that aggregates/crawls job boards for remote job postings, and AngelList has been VERY difficult to crawl for various reasons, but you easily get PhantomJS to work (I have). Having said that, I've never felt very good about the fact that I'm defeating their attempts to block me (even though I feel like I'm doing them a favor) and wil…

I think PhantomJS has a bit of a giveaway in the headers where 2 lines are reversed compared to a normal Chrome, I always thought AngelList detected this flaw. Although I have heard there are builds of PhantomJS where this flaw does not exist.

Re: Web Scraping in 2016

#36

Post by [deleted] »

elchief wrote: post #21

To fight scrapers, we show some values as images that look like text (but not all the time) And we insert random (non-visible) html and css classes in our site to screw with em, and use randomized css classnames. This fucks with xpaths and css selectors. You can't stop them, but you can make their lives painful.

[deleted]

Re: Web Scraping in 2016

#38

Post by franciskim »

writeslowly wrote: post #5

Have you run into any issues from running all of your scrapers off of AWS, or just from sites detecting that you're accessing large numbers of pages in some sort of obvious pattern? I guess I was hoping there would be sites with more interesting ways to screw with web scrapers (rearranging certain page elements or something) than just throwing up a CAPTCHA.

I'm not actually doing a lot of hits, so it's generally been ok. I can just rotate my IP or solve the CAPTCHA.
Post reply on HN