Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in 2016

franciskim.co

Post reply on HN
Print view View original

121–130 of 402 posts

Re: Web Scraping in 2016

#121

Post by cookiecaper »

muglug wrote: post #108
cookiecaper wrote: post #50

Earlier quoted context omitted.

Eh, this is pretty weak. Scrapers are no different from other browsing devices. The web speaks HTTP. There's no reason that using another HTTP browser would cause any disparate impact just by virtue of not being a conventional desktop browser -- you've thrown out a pretty absurd hypothetical. In fact, scrapers usually cause less impact because they usually don't download images or execute JavaScript. I did an analysi…

> you've thrown out a pretty absurd hypothetical Not even remotely absurd. Where is the data your scraper consuming coming from? It's almost always served from some sort of data repository (SQL or otherwise). That data costs far more per MB to serve up quickly than JS/CSS/images. Suppose, for example, you host a blogging platform that has one very popular user. Most accounts on your site don't get a ton of visitors,…

There are already 6-8 major scrapers that do this constantly, across the whole internet, called search engines. You can't handle that?

What if you get a normal user who says "Hey, I wanna see some of the lesser known authors on this platform" and opens up a hundred tabs with rarely-read blogs? What if you get 10 users who decide to do that on the same day? Is it reasonable to sue them? Should there be a legal protection to punish them for making your site slow?

Don't blame the user for your scaling issues. If the optimized browser ("scraper") isn't hammering your site at a massively unnatural interval, it's clean. And if it is, you should have server-side controls that prevent one client from asking for too much data.

These are just normal problems that are part of being on the web. It's not fair to pin it on non-malicious users, even if they're not using a conventional desktop browser.

Re: Web Scraping in 2016

#124

Post by kough »

fake-name wrote: post #49

I do a significant amount of scraping for hobby projects, albeit mostly open websites. As a result, I've gotten pretty good a circumventing rate-limiting and most other controls. I suspect I'm one of those bad people your parents tell you to avoid - by that I mean I completely ignore robots.txt. At this point, my architecture has settled on a distributed RPC system with a rotating swarm of clients. I use RabbitMQ for…

Yet another incredible technical achievement due to someone's quest for more porn (https://github.com/fake-name/AutoTriever/blob/master/setting...).

Re: Web Scraping in 2016

#125

Post by rch »

franciskim wrote: post #84
rch wrote: post #79

There is so much that's missing from this. What about gathering tokens from customers vs. paying for social data feeds? How about canned services like 80legs?

Hmm yeah there are a lot of other things that I could write about. 80legs seem like another Scrapy type of SaaS? Not sure what you mean about gathering tokens from customers.

I've heard of companies that scrape on behalf of customers who will walk marketing people through the process of creating an API token to help mitigate rate limiting.

Re: Web Scraping in 2016

#126

Post by DashRattlesnake »

cookiecaper wrote: post #68
DashRattlesnake wrote: post #57

Earlier quoted context omitted.

> I did an analysis and a session browsed with my specialized browser would always consume less than 100K of bandwidth (and often far less), whereas a session browsed with a conventional desktop browser would consume at least 1.2 MB, even if everything was cached, and sometimes up to 5 MB. In addition, on the desktop, a JavaScript heartbeat was sent back every few seconds, so all of that data was saved too. Bandwidth…

Yeah, I just have a hard time buying that a scraper that does less than a conventional desktop browser is going to accidentally stumble across something that causes the server-side to flip out. I'm not really sure in what case your hypothetical is plausible. Scrapers are usually used to get publicly-available data more efficiently. What you're describing would basically require the scraper to hammer an invisible endp…

> What you're describing would basically require the scraper to hammer an invisible endpoint somewhere

That wasn't my point. My point was: a lot of a website's costs are hidden from a web scraper (e.g. database load), so a scraper can't claim, based on the variables they can observe (bandwidth), that they're costing the website less than normal traffic.

I was basically responding to statements like this:

> In fact, scrapers usually cause less impact because they usually don't download images or execute JavaScript.

There's really no way for a scraper to know that unless the website tells them. Their usage pattern is different than typical users and raw bandwidth (for stuff like static images) may not matter to the website.

Re: Web Scraping in 2016

#127

Post by namelezz »

XCSme wrote: post #19

Tbh I didn't enjoy the article, it just seems like someone who has just learned about Node.js tried to explain (and mostly failed) how to use some packages to scrape a page. I was expecting to learn some new techniques, but all it explained was how to make a few API calls in order to solve a very specific problem. Also, there was the overall arrogant tone: "I found their interview approach a bit of a turn off so I di…

> "I found their interview approach a bit of a turn off so I did not proceed to the next interview and ignored her emails ", this just shows a lot of immaturity.

I believe you should treat others how you want to be treated. FYI, recruiters do not usually follow up with rejected candidates and many are unresponsive. It's their way of telling candidates they are no longer interested.

Re: Web Scraping in 2016

#128

Post by gnud »

cookiecaper wrote: post #111
comex wrote: post #109

Earlier quoted context omitted.

"Clickwrap" refers to situations where you have to click through before using the service, hence the name. Agreements which are simply a passive notice in a footer somewhere are called "browse-wrap", and are much less likely to be considered enforceable: https://en.m.wikipedia.org/wiki/Browse_wrap

The line is blurred between clickwrap and browsewrap -- those are colloquial terms to describe ToS notices, not legal terms. Is it still browsewrap if you say "By clicking any of the links on this site, you agree to the ToS"? How far away from the clickable buttons must the statement be to be browsewrap instead of clickwrap? The distinction is really only a technicality in the wording, not anything substantive. In pr…

That notice is typically in the footer, and a screen reader will reach the nav-bar before mentioning the TOS notice.

Even for sighted people, the notice is often easy to miss - and this is by design.

Re: Web Scraping in 2016

#129

Post by user5994461 »

slig wrote: post #11

I wonder how effective the CloudFlare anti-scrapper protection is against this approach of breaking captchas. Also, I find it interesting that big websites don't just block all traffic from AWS IPs as they do with Tor.

There can be legitimate traffic coming from AWS, if not the site itself.

It's especially true when the site provides an API and is meant to be integrated by people/companies. In which case, the AWS traffic is likely to include major and/or important and/or paying customers. You really don't want to block that.

On the other hand, Tor is likely to be 90% evil. When in doubt, just block it. (That makes me think, I should run some proper stats and maybe publish a blog post about that. )

Re: Web Scraping in 2016

#130

Post by ysavir »

minimaxir wrote: post #82
twa927 wrote: post #73

Earlier quoted context omitted.

How can TOS have legal power for the case scraping? A website is a public property. If I'm visiting it without logging in, I don't have a chance to accept TOS. Imagine a hotel that makes guests sign a document saying they will not make photographs of the building. If I'm not a guest, I can take photographs of it and I can't even know that would be illegal.

That analogy is not equitable. If you take photographs of a building while on the building's property , they have the right to tell you to stop, or call the police to escort you off if you refuse to do so.

Let's go with a more apt analogy:

If you're entering a country, do its laws not apply to you until you've seen a copy of them? "Oh, sorry, no one told me theft is illegal here. Where does it say that? Oh, I see. Okay. I'll stop now. Thanks for letting me know."

If you cross the border without necessary documents, does that country have no right to detain you, simply because you haven't checked the laws?

Just because a website is visible and public doesn't mean its content is public domain. It just means that your first order of business as a user should be to check the terms of service. Sure, most people using a website probably don't need to--same as not needing to check a country's stance on murder--and so can just use the website as intended without violating the terms. But when you plan on using it in a way that might not be intended, and you don't check the terms of service, well, that's on you.

Post reply on HN