Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping in 2016

franciskim.co

Post reply on HN
Print view View original

161–170 of 402 posts

Re: Web Scraping in 2016

#161

Post by BrandonMarc »

elmigranto wrote: post #52
elchief wrote: post #21

To fight scrapers, we show some values as images that look like text (but not all the time) And we insert random (non-visible) html and css classes in our site to screw with em, and use randomized css classnames. This fucks with xpaths and css selectors. You can't stop them, but you can make their lives painful.

> To fight scrapers, we show some values as images that look like text You are fighting screen readers more than anything; as well as legitimate plugins, form autofills, etc. If this is for captcha, you are fighting all the users as well. > And we insert random (non-visible) html and css classes in our site to screw with em, and use randomized css classnames. Legitimate browser plugins, etc. I'd just use electron or…

> What you effectively doing is wasting time on useless stuff. This is even more useless than trying to prevent copying of DVDs or pirating games.

Can you be so sure? The Union blockade of the Confederacy had plenty of holes, and smugglers / privateers / blockade-runners made good money getting through (when they survived) ... but that doesn't mean the blockade wasn't effective all the same at weakening the Confederate military and economy.

Re: Web Scraping in 2016

#162

Post by DashRattlesnake »

cookiecaper wrote: post #131
DashRattlesnake wrote: post #126

Earlier quoted context omitted.

> What you're describing would basically require the scraper to hammer an invisible endpoint somewhere That wasn't my point. My point was: a lot of a website's costs are hidden from a web scraper (e.g. database load), so a scraper can't claim, based on the variables they can observe (bandwidth), that they're costing the website less than normal traffic. I was basically responding to statements like this: > In fact, s…

It's true that there's no way to know that for sure, but it doesn't make sense that a scraper, by virtue of its being a scraper, is incurring additional load. A scraper is only making requests that a person with a desktop browser or any other appliance that speaks HTTP could make. What's the difference between a user clicking the same button on the page 50 times or holding down F5 and a scraper that pings a page once…

> but it doesn't make sense that a scraper, by virtue of its being a scraper, is incurring additional load

It does, because scrapers don't have normal usage patterns. They're robots and behave like robots.

> What's the difference between a user clicking the same button on the page 50 times or holding down F5 and a scraper that pings a page once a minute?

Typical users aren't usually in the habit of mashing F5, especially not for robotically long periods of time. It's basically the difference between a theoretical activity and an actual activity.

Basically, scraping is not regular usage, and I don't think it's correct to pretend that they're equivalent (or more extremely, that scraping is less costly to the website).

Re: Web Scraping in 2016

#163

Post by Bjartr »

stupidcar wrote: post #120

I wrote a fairly complex spidering and scraping script in Node a few months ago. I found downcache[1] to be absolutely invaluable, particularly as I was debugging my parsing scripts, a I was able to rerun them relatively quickly over the cached responses. However, when the network was no longer a bottleneck, I found that the speed and single-threaded nature of Node became one. It wasn't really that slow, relatively s…

Wonder how difficult it would have been to pull the JS portion into Java by way of Rhino.

Re: Web Scraping in 2016

#164

Post by siegecraft »

fake-name wrote: post #49

I do a significant amount of scraping for hobby projects, albeit mostly open websites. As a result, I've gotten pretty good a circumventing rate-limiting and most other controls. I suspect I'm one of those bad people your parents tell you to avoid - by that I mean I completely ignore robots.txt. At this point, my architecture has settled on a distributed RPC system with a rotating swarm of clients. I use RabbitMQ for…

I'm sure there'd be a ton of people that would love to pay to use your platform (who cares if the source is available, I don't want to run my own because once the code is written, it's ops thats hard). But then I suppose it would be hard to stay unnoticed.

Re: Web Scraping in 2016

#165

Post by fake-name »

kough wrote: post #124
fake-name wrote: post #49

I do a significant amount of scraping for hobby projects, albeit mostly open websites. As a result, I've gotten pretty good a circumventing rate-limiting and most other controls. I suspect I'm one of those bad people your parents tell you to avoid - by that I mean I completely ignore robots.txt. At this point, my architecture has settled on a distributed RPC system with a rotating swarm of clients. I use RabbitMQ for…

Yet another incredible technical achievement due to someone's quest for more porn ( https://github.com/fake-name/AutoTriever/blob/master/setting... ).

That's a separate project:

- https://github.com/fake-name/ExHentai-Archival

- https://github.com/fake-name/PatreonArchiver

- https://github.com/fake-name/xA-Scraper

- https://github.com/fake-name/DanbooruScraper

Or... well, 4 separate projects. Whoops?

At one point, a friend and I were looking at trying to basically replicate the google deep-dream neural net thing, only with a training set of porn. It turns out getting a well tagged dataset for training is somewhat challenging.

Well-tagged hentai is trivially accessible, though. I think there's probably a paper or two in there about the demographics of the two fan groups. People are fascinating.

Next up, automate the consumption too!

Re: Web Scraping in 2016

#166

Post by muglug »

cookiecaper wrote: post #121
muglug wrote: post #108

Earlier quoted context omitted.

> you've thrown out a pretty absurd hypothetical Not even remotely absurd. Where is the data your scraper consuming coming from? It's almost always served from some sort of data repository (SQL or otherwise). That data costs far more per MB to serve up quickly than JS/CSS/images. Suppose, for example, you host a blogging platform that has one very popular user. Most accounts on your site don't get a ton of visitors,…

There are already 6-8 major scrapers that do this constantly, across the whole internet, called search engines. You can't handle that? What if you get a normal user who says "Hey, I wanna see some of the lesser known authors on this platform" and opens up a hundred tabs with rarely-read blogs? What if you get 10 users who decide to do that on the same day? Is it reasonable to sue them? Should there be a legal protect…

Search engines respect robots.txt – not sure many scrapers do.

Re: Web Scraping in 2016

#167

Post by mthoms »

cookiecaper wrote: post #119
niftich wrote: post #113

Earlier quoted context omitted.

Not the discount the validity of your experience, but the usual counterpoint to this is Google, who (like mentioned elsewhere in the thread) has been continuously scraping since the very beginning and in fact built their entire business model on doing so. They are also responsible for advancing the state-of-the-art of scraping (albeit mostly internally), through the development of V8 and headless Chromium so that the…

Yeah, Google violates the CFAA and infringes on copyright as a matter of course. Their service would be impossible if they weren't doing so. The main difference when Google was small was that Google was not dependent on any data source in particular, so even if someone denied their robot or sued them, they could cease and desist without affecting the overall value of their offering. This is different if you are getti…

Google's crawling and caching has been largely found to be fair use and thus is not considered to be infringing copyrights.

https://en.wikipedia.org/wiki/Field_v._Google,_Inc.

There are similar rulings for thumbnail images:

https://en.wikipedia.org/wiki/Perfect_10,_Inc._v._Amazon.com....

And of course books:

https://en.wikipedia.org/wiki/Authors_Guild,_Inc._v._Google,....

Re: Web Scraping in 2016

#168

Post by rckclmbr »

hk__2 wrote: post #135
fareesh wrote: post #112

Earlier quoted context omitted.

Is it common for developers in the eCommerce space to use scrapers as a means to aggressively push automated price-match algorithms? I've been asked to do this a number of times, was just curious as to how prevalent it is.

Yes, everybody scrapes the prices of the others.

Most companies will use resources not in their datacenter and not identifiable. Executives know it's sketchy, but they do it anyway.

Re: Web Scraping in 2016

#169

Post by siegecraft »

spikej wrote: post #53
dchuk wrote: post #4

Scraping with Selenium in Docker is pretty great, especially because you can use the Docker API itself to spin up/shut down containers at will. So you can spin up a container to hit a specific URL in a second, scrape whatever you're looking for, then kill the container. This can be done via a job queue (sidekiq if you're using Ruby) to do all sorts of fun stuff. That aside, hitting Insta like this is playing with fir…

Serious question: What do you gain from having an extra layer like docker?

Well it does make it extra easy to deploy a scrape node to any type of machine you might encounter (and having a diverse set of source IPs is extra important for scraping; that means you might need to deploy to AWS, Azure, Google Cloud, rackspace, digitalocean, random vps provider X and so on). So instead of having to have custom provisioning profiles for every hosting provider/image combination, you just need to get docker running on a host and you're good to go.

Re: Web Scraping in 2016

#170

Post by wtracy »

jhwhite wrote: post #118
XCSme wrote: post #19

Tbh I didn't enjoy the article, it just seems like someone who has just learned about Node.js tried to explain (and mostly failed) how to use some packages to scrape a page. I was expecting to learn some new techniques, but all it explained was how to make a few API calls in order to solve a very specific problem. Also, there was the overall arrogant tone: "I found their interview approach a bit of a turn off so I di…

>Also, there was the overall arrogant tone: "I found their interview approach a bit of a turn off so I did not proceed to the next interview and ignored her emails ", this just shows a lot of immaturity. I'm not saying you're one of these people, but it's frustrating when companies do this to potential employees and the potential is told by friends and other management type people, "well that's the company you just h…

>When someone flips it on the company then it's immature.

First, it's hard to know when companies are doing this intentionally versus when things just get lost in the shuffle. (Never attribute to malice what can be explained by incompetence, and all that.) Meanwhile, the author was clearly ignoring the interviewer intentionally.

Second, the fact that Company A treated you rudely doesn't give you license to treat unrelated Company B rudely. For that matter, I'm not sure that the fact that Employee 1 at Company A treated you rudely gives you moral license to treat Employee 2 at Company A rudely. Show a little compassion for someone trapped in a dead-end job trying to put food on their family's table, for crying out loud.

Post reply on HN