Live data from Hacker News

Web Scraping in 2016

franciskim.co

391–400 of 402 posts

Re: Web Scraping in 2016

#391
post #225

Earlier quoted context omitted.

Whether or not it technically qualifies as a "double standard," in practice I don't see anything inherently unfair about it. If a stranger enters my house without my permission, that's trespassing. But there's nothing unfair about letting in someone who I invite over.

That's a terrible analogy. Your home is private, websites are not. The fact is that websites are posted online for all to see, so it's more like saying certain people at a park may take pictures while others are not allowed. That's unfair. If everyone could take pictures, it would be fair. Yes, someone with an old bright bulb camera might be annoying people, but nobody said "fair" meant all players would be nice or t…

You can still deal with the old bright bulb cameras: you can have rules which apply to everyone. So you can make a rule at the park that pictures are allowed, but only without flash, or that only digital cameras are allowed, or only digital cameras with the fake-shutter noises turned off, etc. As long as the rule applies to everyone equally, it's fair, even if you think the rule is silly.

For websites, it's not fair to have different rules for Google than others. What would be fair is some kind of rule about how often visitors can visit, how much they're allowed to download, etc.

Personally, though, I think all this is total BS. Sites are open to the public, but they also serve the whims of their owners. If the site wants to prevent access to people from a certain IP range, that should be their right. If they don't want any scrapers, that should be their right too, or if they want to allow Google and not anyone else, that should also be their right. What isn't right is that they can use the government to enforce these arbitrary rules. If they want to block my scraper, that's fine, if they can do it on their end technologically. If they want to block my IP, they can do that too. But suing me or having the cops come to my door because they're too incompetent or lazy to do these things technologically is unacceptable. The role of government is not to enforce arbitrary policies made up by business owners.

Re: Web Scraping in 2016

#392
post #316

Earlier quoted context omitted.

Maybe we (scrappers) just need to market ourselves as search engines. _Indexing_ is what we're doing. :)

Exactly, market scrappers as search engines. And all smart websites should include a ToS that says you are not allowed to access their data, so they can sue for trespassing anyone that they don't like selectively. The far reaching of government into this, and also the pirating stuff (which I do not condone but think that arresting people for that is waay too much) is what makes me want for the system to collapse unde…

Your idea about Oracle winning and society coming crashing to a halt is ridiculous and wouldn't have happened. Your flaw is believing that the law and the government will work with logical precision, so that a flaw in the law will, like an infinite recursive loop in programming code, cause complete disaster. It doesn't work that way. There's plenty of cases where the law is clearly broken (see civil forfeiture vs. the 4th Amendment to the US Constitution), yet nothing is done. That's because the government is run by humans, and they'll enforce things the way they want. Double standards happen all the time with law, and it takes big, expensive court cases to sort them out, and of course that only happens when some moneyed interest wants to fix it (which is why civil forfeiture is still a big thing--they're not going after extremely wealthy people or corporations with it). While IBM is certainly large enough to bring a big case like you suggested, the US government is far bigger and can simply invent a legal way of ignoring them, just as was done when the SCOTUS decided to rule in favor of using Eminent Domain to seize private property to hand over to commercial interests.

Re: Web Scraping in 2016

#393
post #152

Earlier quoted context omitted.

It seems like you could easily hit those scaling issues by manually browsing the website. While I agree that it sucks to take down a site by scraping, in that specific case it sounds like the performance issues are their fault and not yours. That said, once I realized the effect my scraping had, I would (hopefully) cease my scraping.

So the thing is, I could totally believe they never saw this traffic pattern under normal load. I'd expect bar trivia scores in a certain mid-sized US city are one of those niche things where you have a very low number of uniques but each unique then pokes around on 9 or 10 pages while they're there. The fact that the site didn't crash during normal browsing was what originally led me to speculate they were maintaini…

Ok, then why couldn't you revise your scraper so that it did everything in a single session, to avoid this problem?

To me, for private, personal use, a scraper should emulate a normal human browser as much as possible to avoid causing site problems and to avoid detection. If what you're doing can be done in the background, or by a cron process at some odd hour, it doesn't have to be fast at all, and you can set the timings to be similar to a normal human.

Re: Web Scraping in 2016

#394
post #13

Earlier quoted context omitted.

This post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will. Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread.

I disagree. DOSing a site is bad behaviour, regardless of how you do this. But accessing it in an automated way instead of a browser? Not really. The deal on the Internet is like this: a website owner can provide whatever they want, and a visitor can read it however they want. Discriminating visitors based on whether or not they seem to be bots instead of people is going beyond what the site provider should do. So is…

I agree with your assertion about "the deal on the Internet", but I disagree with your conclusions. IMO, site owners should be able to discriminate all they want. However, when someone's browser (or other software...) makes a request to that site, and the site serves them some data, the user should be able to do with that what they will: either honor or decline the requests for them to download various ads or JavaScript, for instance. It should be up to the site owner to craft their site to follow their policies and whims. What I'm completely against is the idea of using the government and law enforcement to enforce some site owner's policies. The only exception I can see for this is extreme cases where this general principle falls down: DOS attacks, for instance.

If I can modify my web browser to view a site, but skip the ads, that should be my right. If the site owner codes their site to detect this and then blocks my request to see their site, that should be their right. If I modify my ad-blocker to get around their ad-blocker-block, that should be my right, and so on. As long as we don't get into something like DDOS territory where a reasonable web site has no good technological way of avoiding the problem caused by a user, this isn't something for government to get involved in.

Re: Web Scraping in 2016

#395

Earlier quoted context omitted.

I disagree. DOSing a site is bad behaviour, regardless of how you do this. But accessing it in an automated way instead of a browser? Not really. The deal on the Internet is like this: a website owner can provide whatever they want, and a visitor can read it however they want. Discriminating visitors based on whether or not they seem to be bots instead of people is going beyond what the site provider should do. So is…

I agree with your assertion about "the deal on the Internet", but I disagree with your conclusions. IMO, site owners should be able to discriminate all they want. However, when someone's browser (or other software...) makes a request to that site, and the site serves them some data, the user should be able to do with that what they will: either honor or decline the requests for them to download various ads or JavaScr…

Hmm... yeah, I guess what you (and the other commenter) describe is fairer than what I wrote. Thanks!

Re: Web Scraping in 2016

#396
post #124

Earlier quoted context omitted.

Yet another incredible technical achievement due to someone's quest for more porn ( https://github.com/fake-name/AutoTriever/blob/master/setting... ).

That's a separate project: - https://github.com/fake-name/ExHentai-Archival - https://github.com/fake-name/PatreonArchiver - https://github.com/fake-name/xA-Scraper - https://github.com/fake-name/DanbooruScraper Or... well, 4 separate projects. Whoops? At one point, a friend and I were looking at trying to basically replicate the google deep-dream neural net thing, only with a training set of porn. It turns out getti…

Oh my god. Can you share any results?

Re: Web Scraping in 2016

#397
post #21

To fight scrapers, we show some values as images that look like text (but not all the time) And we insert random (non-visible) html and css classes in our site to screw with em, and use randomized css classnames. This fucks with xpaths and css selectors. You can't stop them, but you can make their lives painful.

It would be better to invest that time in making an API so they don't need to scrape.

Re: Web Scraping in 2016

#398
post #288

> But if you are automating your exact actions that happen via a browser, can this be blocked? Yes, by checking times between actions and number of actions in a time period, and blocking atypical activity. I was IP banned from a site once for a few months, after trying to scrape it too much and hitting links on the site that were hidden from humans. The random wait settings specified in the post are better than nothi…

It really depends on the data you are scraping. My main business relies on scraping and my data mining application has been running for over 5 years. If you have enough IP addresses available to you, it becomes almost impossible to distinguish it from normal users hitting the site...and bandwidth has gotten so cheap, the overhead is very affordable.

I've noticed that most sites actually don't change that often. I deal with changes once or twice every 3 months.

"If you scrape for a living, only do it as a side job."

This is true if you are scraping the low hanging fruit. I scrape 40+ sources (I do have access to a few APIs as well) and then have to extract the patterns/data I need to then integrate it into my business model. This is all automatic now and I only work on upgrading for speed and efficiency.

If you have to scan millions of urls daily from 1 site, it's probably not going to work out. You need to figure out clever ways of getting the data and using it without breaking any laws or pissing off the site owner.

Re: Web Scraping in 2016

#399

Earlier quoted context omitted.

At least Ex supports torrents and also has some custom p2p software which you can run (serves content) from which data can be siphoned off. And what is served through their website is resized. So web-scraping is an inferior approach.

You seem to be assuming 1. I'm scraping the resized galleries. 2. I don't have the Hath perk that makes the galleries full sized. 3. I don't have a phash-based fuzzy image deduplication system on top of all this (see https://github.com/fake-name/IntraArchiveDeduplicator ). It's main purpose is to deduplicate manga ( https://github.com/fake-name/MangaCMS ).

Jesus, your projects are massive. Does your job involve working on these or are these just side things?

Re: Web Scraping in 2016

#400

Earlier quoted context omitted.

> What you effectively doing is wasting time on useless stuff. This is even more useless than trying to prevent copying of DVDs or pirating games. Can you be so sure? The Union blockade of the Confederacy had plenty of holes, and smugglers / privateers / blockade-runners made good money getting through (when they survived) ... but that doesn't mean the blockade wasn't effective all the same at weakening the Confedera…

Do you really not see the difference between military blockade and randomizing CSS classes?

Honestly, no. This one time I was pissed off at Egypt for undercutting me in cotton prices, so I tried to set up a blockade to prevent merchant ships going in and out of Cairo.

... and it would have worked, too, except for my naval vessels were all CSS classes. I even tried to name them cleverly, a la "USS hero unit" or "USS datatable table-consensed span9", but my plan was foiled.

Post reply on HN