Earlier quoted context omitted.
How can I get ahold of you directly?
connorw at imaginaryindustries dot com
Thanks. Email sent.
Web Scraping in 2016
231–240 of 402 posts
Re: Web Scraping in 2016
#232How do you push a button like hit next on a paginated page?
Right click on the 'next' button in chrome and use 'inspect element' to find its id/class/css selector and then:
browser.findElement(webdriverio.By.id('#Next')).click();
Re: Web Scraping in 2016
#233Keep in mind that companies have sued for scraping not through the API, for example LinkedIn, which explicitly prevents scraping via the ToS: http://www.informationweek.com/software/social/linkedin-sues... OKCupid did a DMCA takedown for researchers releasing scraped data: https://www.engadget.com/2016/05/17/publicly-released-okcupi... Since both of these incidents, I now only scrape if it's a) through the API follow…
I am no expert, but I always thought you could scrape without consequence provided you never distribute your scrapings?
Re: Web Scraping in 2016
#234Earlier quoted context omitted.
The line is blurred between clickwrap and browsewrap -- those are colloquial terms to describe ToS notices, not legal terms. Is it still browsewrap if you say "By clicking any of the links on this site, you agree to the ToS"? How far away from the clickable buttons must the statement be to be browsewrap instead of clickwrap? The distinction is really only a technicality in the wording, not anything substantive. In pr…
That notice is typically in the footer, and a screen reader will reach the nav-bar before mentioning the TOS notice. Even for sighted people, the notice is often easy to miss - and this is by design.
It's by design because the vast majority of people don't care about that information and it makes the website worse for them to have a big ToS banner at the top of your page.
I don't think many websites have a secret ToS that they hope you won't read, I think most of them don't even know what their own ToS say. I signed my lease on a site with an explicit checkmark for ToS that said I agreed I would only use exactly IE7 to use their site.
Re: Web Scraping in 2016
#235Earlier quoted context omitted.
How can TOS have legal power for the case scraping? A website is a public property. If I'm visiting it without logging in, I don't have a chance to accept TOS. Imagine a hotel that makes guests sign a document saying they will not make photographs of the building. If I'm not a guest, I can take photographs of it and I can't even know that would be illegal.
The UK has a database law: https://en.wikibooks.org/wiki/UK_Database_Law#Database_Right If you scrape, and effectively reconstitute a database, then so long as the database originally had a "substantial investment" in it's "obtaining, verifying or presenting the contents" then yup... you have breached the database right, which is a modified form of copyright. You may access said database (via the web), but as soon as…
What is the definition of "reconstituting a database"? Aren't googles indexes doing that?
Re: Web Scraping in 2016
#236Earlier quoted context omitted.
Have run into exactly this before. Wrote a scraper that retrieved results from a trivia league website. Tried to be a polite scraper (<1 request per second) but the site still crashed - even with 5 seconds of sleep between requests. They were doing something weird with DB connection management (maybe just forgetting to close it and letting it timeout? I remember figuring it out but it's been quite a while) and so aft…
Now that I think about it a bit more, I think my hypothesis was that DB connections were allocated at the session level and that without cookies enabled each request initiated a new session. I'd consider that a bug not a feature but I still think it's incumbent on me, the guy scraping the website, not to trigger it.
>Now that I think about it a bit more, I think my hypothesis was that DB connections were allocated at the session level and that without cookies enabled each request initiated a new session.
Could also be something like storing hibernates second level cache in session. Unfortunately I've seen this, a significant chunk of the database was being copied into each users session.
Re: Web Scraping in 2016
#237Earlier quoted context omitted.
With a risk of stating the obvious, this is a double standard simply because there are two standards - one for Google and one for others. I can't speak for the poster you were replying to, but whilst I see it as logical self-interested behaviour by site owners, it still feels unfair.
Whether or not it technically qualifies as a "double standard," in practice I don't see anything inherently unfair about it. If a stranger enters my house without my permission, that's trespassing. But there's nothing unfair about letting in someone who I invite over.
Using the law to block crawlers is more like saying:
1. Google can come in
2. Other Americans can't come in
3. Chinese people can come in (or anywhere else where US laws don't apply)
It might not be unfair, but it is certainly pointless and arbitrary.
Re: Web Scraping in 2016
#238Re: Web Scraping in 2016
#239Earlier quoted context omitted.
Whether or not it technically qualifies as a "double standard," in practice I don't see anything inherently unfair about it. If a stranger enters my house without my permission, that's trespassing. But there's nothing unfair about letting in someone who I invite over.
Using the law to block crawlers is more like saying: 1. Google can come in 2. Other Americans can't come in 3. Chinese people can come in (or anywhere else where US laws don't apply) It might not be unfair, but it is certainly pointless and arbitrary.
To be fair, many companies which take anti-scraping seriously will also take inputs like geographic origin of a request into consideration when applying request throttling and filtering.
Re: Web Scraping in 2016
#240Not wanting to thread hijack, but just going to post an article I wrote a few years back as it covers a few other things that are still relevant; and often still gets referenced. May it'll help some people out in combination with OP's post. http://jakeaustwick.me/python-web-scraping-resource/
I was surprised to not see Scrapy listed, but then I saw there were some comments about it - but seriously, doing by hand what Scrapy has spent years perfecting is highly suboptimal.
I guess the distinction is between whether one wants to just "toy around" or run the spider for-real.