Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in 2016

franciskim.co

Post reply on HN
Print view View original

61–70 of 402 posts

Re: Web Scraping in 2016

#61

Post by hchasestevens »

cookiecaper wrote: post #50
muglug wrote: post #23

Earlier quoted context omitted.

> If you throttle your scraper in proportion to the size of their site, you aren't really harming them. And do you understand their site infrastructure to know whether you're doing harm? It's perfectly possible that your script somehow bypasses safeguards they had in place to deal with heavy usage, and now their database is locking unnecessarily.

Eh, this is pretty weak. Scrapers are no different from other browsing devices. The web speaks HTTP. There's no reason that using another HTTP browser would cause any disparate impact just by virtue of not being a conventional desktop browser -- you've thrown out a pretty absurd hypothetical. In fact, scrapers usually cause less impact because they usually don't download images or execute JavaScript. I did an analysi…

A small counterpoint to this -- in the airline industry, it's relatively commonplace for seat reservations to be made for a user _before_ payment has occurred. In this case, if you're mirroring normal browser activity, you can (temporarily) reduce availability on a flight, potentially even bumping up the price for other, legitimate users, and almost certainly causing the airline to incur costs beyond normal bandwidth and server costs. I'm sure there are many other domains for which this is also the case, however rare.

Re: Web Scraping in 2016

#62

Post by nathancahill »

XCSme wrote: post #19

Tbh I didn't enjoy the article, it just seems like someone who has just learned about Node.js tried to explain (and mostly failed) how to use some packages to scrape a page. I was expecting to learn some new techniques, but all it explained was how to make a few API calls in order to solve a very specific problem. Also, there was the overall arrogant tone: "I found their interview approach a bit of a turn off so I di…

Part of the turnoff for me was the middle-schooler tone and vocabulary. Good walkthrough with good code examples though, obviously written by a very smart JS dev.

Re: Web Scraping in 2016

#63

Post by franciskim »

spikej wrote: post #53
dchuk wrote: post #4

Scraping with Selenium in Docker is pretty great, especially because you can use the Docker API itself to spin up/shut down containers at will. So you can spin up a container to hit a specific URL in a second, scrape whatever you're looking for, then kill the container. This can be done via a job queue (sidekiq if you're using Ruby) to do all sorts of fun stuff. That aside, hitting Insta like this is playing with fir…

Serious question: What do you gain from having an extra layer like docker?

Selenium grid runs in docker, so it's easy to have multiple instances running. Better control.

Re: Web Scraping in 2016

#64

Post by emodendroket »

elchief wrote: post #21

To fight scrapers, we show some values as images that look like text (but not all the time) And we insert random (non-visible) html and css classes in our site to screw with em, and use randomized css classnames. This fucks with xpaths and css selectors. You can't stop them, but you can make their lives painful.

This also hurts accessibility for disabled users.

Re: Web Scraping in 2016

#65

Post by elchief »

elmigranto wrote: post #52
elchief wrote: post #21

To fight scrapers, we show some values as images that look like text (but not all the time) And we insert random (non-visible) html and css classes in our site to screw with em, and use randomized css classnames. This fucks with xpaths and css selectors. You can't stop them, but you can make their lives painful.

> To fight scrapers, we show some values as images that look like text You are fighting screen readers more than anything; as well as legitimate plugins, form autofills, etc. If this is for captcha, you are fighting all the users as well. > And we insert random (non-visible) html and css classes in our site to screw with em, and use randomized css classnames. Legitimate browser plugins, etc. I'd just use electron or…

Except traffic from known scrapers (or what appear to be) is down 20%

Sure, xpath and css selector experts can figure it out, but that's not everyone

Re: Web Scraping in 2016

#66

Post by wongarsu »

Raphmedia wrote: post #55
Nicksil wrote: post #44

Earlier quoted context omitted.

I don't follow your argument. How does one get their scraper access to data they would otherwise not be able to access through 'normal' browsing techniques?

Example I know of: You can scrape your competitor's Facebook pages since their creation and output nice graphs of which posts generated what kind of likes and suscriptions. This data is usually limited to the owner of the page.

But I can get the same data with manual browsing. And I can just as well pay some chinese to collect the data and input it in my database.

Automated scraping is just a way to drastically reduce labor costs for information collection. Sure, it's a competitive advantage, but I think disallowing it or calling it unethical is a pretty big can of worms. Why is it ok if something is done by humans but not ok if a computer does it by himself?

Re: Web Scraping in 2016

#67

Post by detaro »

cookiecaper wrote: post #60
detaro wrote: post #48

Earlier quoted context omitted.

It depends on what you are doing. The CFAA is very far-reaching, but of course many aspects were the US answer is "CFAA" are covered by other laws. [EDIT: removed outdated information superseded by european decisions, which make the situation a lot less clear] anti-scraping: If somebody were to offer a telephone book database online and you created a copy of that to sell on your own, you'd almost certainly loose in t…

>Pro-scraping: Last big case I remember here was a flight-search site that did flight search and booking(!) via a scraper and Ryanair lost when they tried to sue them for that, since they couldn't argue convincingly how that was damaging them. Every case I've seen wrt Ryanair (they sue a lot of people) has resulted in a win for Ryanair. Do you have details on the case you're describing? >anti-scraping: [...] Scraping…

I was refering to a BGH decision (30.04.2014 (Az. I ZR 224/12)), but it seems like newer decisions from european courts kill that argument :/

I edited my original comment to reflect that.

Re: Web Scraping in 2016

#68

Post by cookiecaper »

DashRattlesnake wrote: post #57
cookiecaper wrote: post #50

Earlier quoted context omitted.

Eh, this is pretty weak. Scrapers are no different from other browsing devices. The web speaks HTTP. There's no reason that using another HTTP browser would cause any disparate impact just by virtue of not being a conventional desktop browser -- you've thrown out a pretty absurd hypothetical. In fact, scrapers usually cause less impact because they usually don't download images or execute JavaScript. I did an analysi…

> I did an analysis and a session browsed with my specialized browser would always consume less than 100K of bandwidth (and often far less), whereas a session browsed with a conventional desktop browser would consume at least 1.2 MB, even if everything was cached, and sometimes up to 5 MB. In addition, on the desktop, a JavaScript heartbeat was sent back every few seconds, so all of that data was saved too. Bandwidth…

Yeah, I just have a hard time buying that a scraper that does less than a conventional desktop browser is going to accidentally stumble across something that causes the server-side to flip out. I'm not really sure in what case your hypothetical is plausible.

Scrapers are usually used to get publicly-available data more efficiently. What you're describing would basically require the scraper to hammer an invisible endpoint somewhere, but there's no reason the scraper would do that -- it just wants to get the data displayed by the site in a more efficient manner. I suppose the browser could enforce a cooldown on an expensive callback via JavaScript, which a scraper would circumvent, but IMO that's not a fair reason to say scrapers are disallowed; cooldowns should be enforced server-side. There's no way to ensure that a user is going to execute your script. That's just part of the deal.

Everything about scrapers means less server load; no images, no wandering around the site trying to find the right place, no heavy JavaScript callbacks that invoke server-side application load, etc. Scrapers are just highly-optimized browsing devices targeting specific pieces of data; it's logical that they would be cheaper to serve than a desktop user who's concerned about aesthetics and the like.

In our specific case, those JavaScripts we didn't download included instructions to make over 100 AJAX requests on every page load. No wonder users were looking for something more efficient.

So I agree that a scraper isn't necessarily bypassing some load-heavy operations, but I find it highly implausible that a non-malicious scraper would be invoking operations that cause extra load (beyond just hitting the site too often). Frankly, I'd be surprised if there was a functional scraper that regularly invoked more resource cost per-session than a typical desktop browsing session to get equivalent data.

Re: Web Scraping in 2016

#69

Post by freshhawk »

writeslowly wrote: post #5

Have you run into any issues from running all of your scrapers off of AWS, or just from sites detecting that you're accessing large numbers of pages in some sort of obvious pattern? I guess I was hoping there would be sites with more interesting ways to screw with web scrapers (rearranging certain page elements or something) than just throwing up a CAPTCHA.

A surprisingly small number of sites care. There are some really fun things one can do with random class/id/order variations. It's also fun to feed garbage data to scrapers when you can identify them with very high probability.

But there seems to be little demand for these kinds of systems and just throttling/blocking/CAPTCHA solutions are much simpler.

Re: Web Scraping in 2016

#70

Post by franciskim »

nathancahill wrote: post #62
XCSme wrote: post #19

Tbh I didn't enjoy the article, it just seems like someone who has just learned about Node.js tried to explain (and mostly failed) how to use some packages to scrape a page. I was expecting to learn some new techniques, but all it explained was how to make a few API calls in order to solve a very specific problem. Also, there was the overall arrogant tone: "I found their interview approach a bit of a turn off so I di…

Part of the turnoff for me was the middle-schooler tone and vocabulary. Good walkthrough with good code examples though, obviously written by a very smart JS dev.

Ok, I'll try to explain to this thread. I actually thought about removing the Facebook part, but I kept it in there because that is kind of how I felt and it is real. The middle-schooler tone and vocab is probably because I don't read a lot of books, and English is my 2nd language.

In reply to XCSme - no I am not new to Node and my point of the post is to illustrate some of the techniques that I haven't seen published anywhere to HN and the community. My focus is quite different from what you think it is, so maybe it is my bad for bad writing skills, I'm still new to writing and learning.

Post reply on HN