Does anybody know what the author means by "lead" (noun)? I don't think it's any of the regular meanings: http://www.ldoceonline.com/search/?q=Lead But it doesn't seem to be any of these slang terms either: http://www.urbandictionary.com/define.php?term=lead
Web Scraping in 2016
351–360 of 402 posts
Re: Web Scraping in 2016
#352Earlier quoted context omitted.
With stuff like Facebook opengraph (e.g. og:price) and other meta tags meant to help search engines and social networks get this sort of data to display inline, do you think it's inevitable that complex scraping will no longer be needed in a practical sense since everyone will be inadvertently optimizing their markup in a way that you could write a really simple parser to grab the data?
Given that FB's bot identifies itself, no, eventually some websites will present og: markup only to FB's bot.
Couldn't you just spoof the user agent though? Or is there some other mechanism that you can use to verify the bot is Facebook's?
Re: Web Scraping in 2016
#353Earlier quoted context omitted.
Worse is that Google tries to stop scraping. It's like they don't want anyone to see past the first page of results. They could scrape your website and then they prevent you form scraping your own data back. The whole process is silly; it reflects the duct tape and chicken wire nature of the www. No one should have to "scrape" or "crawl". Data should be put into a open universal format (no tags) and submitted when ne…
That's the antithesis of the world wide web because you've just centralised data storage, which makes someone 'own' the www.
I do not understand your argument.
To give an example, there is a lot of free open source software mirrored all over the internet, mostly on ftp servers, but also on http, rsync, etc.
If you use Linux or BSD you probably are using some of this software. If you use the www, then you are probably accessing computers that use this software. If you drive a new Mercedes you are probably using some of this software. There are a lot of copies of this code in a lot of places.
Is that centralized? Does anyone hosting a mirror ("repository") "own" the software? Is it the same person or entity hosting every mirror?
Compare Google's copies of everyone else's data, also replicated in a lot of places around the world. Who "owns" this data?
Re: Web Scraping in 2016
#354Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…
This post is kind of crazy, aggrandizing bad behavior and misuse of other's resources against their will. Scraping against the TOS is super bad netizen stuff, and I dont think people should be posting positive reviews of people doing this. Breaking captchas and the like is basically blackhat work and should be looked down upon, not congratulated as I see in this thread.
'Bad netizen stuff'? Is this a comment from 1997? Breaking captchas is 'blackhat'? What cozy hippy internet alternative reality does this come from?
These are the same websites and companies that are loading evercookies and doing browser fingerprinting, that break as much as possible the anonymity citizens should enjoy, with Real Name policies, using network analysis to find who your friends are and what your politics and buying habbits are, that routinely rip private information from you cell phone and share it with oppressive regimes.
You're not in Kansas anymore Toto.
Re: Web Scraping in 2016
#355Earlier quoted context omitted.
Example I know of: You can scrape your competitor's Facebook pages since their creation and output nice graphs of which posts generated what kind of likes and suscriptions. This data is usually limited to the owner of the page.
If someone was to sit down and use paper + pencil + time to accomplish the same thing, would you still have issue with it? It's publicly available data. Should you also not watch your competitors television ads or walk in to their physical store and browse around?
The difference being that I can scrape the data of ALL my competitors since 2008, get the number of likes and comments of every single comment and automatically generate graphs of the data. Thousands of man hours in one minute.
This gives an unfair advantage to the tech-savvy "hackers". Facebook terms protects against this. Thus scraping it is disallowed.
I couldn't say if it would be moral or immoral to do this. Personally, I'm more concerned about the well being of poor scraper program that has to scrape through an entire decade of Facebook posts. Poor thing.
Re: Web Scraping in 2016
#356Earlier quoted context omitted.
Putting it behind a signup page with terms that don't allow sharing is not "making it public". And while in the US that may "just" be treated as unauthorized access, in the EU, if you make the data public it's also a violation of the Data Protection Directive, putting you at risk of prosecution in every EU country from which you have included data. You may be right from a risk minimisation perspective. But for a lot…
Eh, not really. The Data Protection Directive doesn’t even apply here – if the first party (OKCupid) made it available to a third party (the scraper), then the first party can be held in violation, but not the third party.
If you have control of personally identifiable data, it's likely that at least some of the EU data protection rules will apply to you regardless of how you got it.
Re: Web Scraping in 2016
#357Earlier quoted context omitted.
Eh, this is pretty weak. Scrapers are no different from other browsing devices. The web speaks HTTP. There's no reason that using another HTTP browser would cause any disparate impact just by virtue of not being a conventional desktop browser -- you've thrown out a pretty absurd hypothetical. In fact, scrapers usually cause less impact because they usually don't download images or execute JavaScript. I did an analysi…
A small counterpoint to this -- in the airline industry, it's relatively commonplace for seat reservations to be made for a user _before_ payment has occurred. In this case, if you're mirroring normal browser activity, you can (temporarily) reduce availability on a flight, potentially even bumping up the price for other, legitimate users, and almost certainly causing the airline to incur costs beyond normal bandwidth…
If you have ever tried to buy hot tickets online and not been able to get any, this is due to bots. Bots are the scalpers friend.
This can easily be prevented by requiring ID matching the ticket on entry, but the ticket sellers often don't seem to care.
Re: Web Scraping in 2016
#358Earlier quoted context omitted.
> misuse of other's resources against their will Nonsense, there is no implication that this activity is illicit. Many sites (I have worked with hundreds) are happy to be included in my service, but don't have the technical ability to provide a data feed. They were delighted when I told them I could aggregate their content without any extra work on their part. We respect TOS, we respect robots.txt and so on. Just bec…
"Don't shoot the messenger for pointing out the lock on your door can be picked"
"Don't shoot the messenger for pointing out that robots are capable of opening doors."
Re: Web Scraping in 2016
#359Earlier quoted context omitted.
So if instead I scrape their site (like they are scraping others) I don't have any opportunity to agree to their terms? Much like their scrapers on other sites? I'm honestly wondering about the double standard. There is a rational way to discuss morality/ethics and subsequent laws regarding most technical aspects, that often mirrors real world (read: offline/analog) scenarios. It's unfortunate that the legal system h…
>> It's unfortunate that the legal system has instead been appropriated by lawyers. omg, really? It's unfortunate that the internet has instead been appropriated by hackers. It's unfortunate that the stock market has instead been appropriated by traders. It's unfortunate that the asylum has instead been appropriated by inmates.
Yeah, really. Law is not an end in itself, it's meant to serve a purpose instead. When people, whose jobs is instrumental to the goal, start deciding about what the goal is, bad things happen. The same is with MBAs and businesses.
Re: Web Scraping in 2016
#360Earlier quoted context omitted.
It is not the same flavor of violation as trespassing on land, more like riding a bicycle on a sidewalk. Fortunately, I'm not in the business of scraping sites, but I still find this legal precedent abhorable, and I hope it gets struck down in court when push comes to shove. I would certainly vote that way if given the chance.
It's like riding a bike in a skate-park with a small sign saying "No bikes".
It's like riding a red mountain bike on a bike path with a small sign saying "only black road bikes allowed".