41–50 of 232 posts
[1] https://jsoup.org/ [2] https://github.com/mattes/rotating-proxy
Earlier quoted context omitted.
It's almost always illegal in the United States. It's prohibited by a combination of the CFAA, copyright law, and contractual obligations imposed by Terms of Use, which are usually considered applicable if you load more than one page ("browsewrap"). The CFAA makes it a crime to access any computer network without authorization or in excess of granted authorization. The Terms of Use will usually prohibit "any automate…
Doesn't publishing information on a public web server equals to granting authorization to download it? Why publish it otherwise? And copyright laws are supposed to protect only creative works, not every page on the web.
This is the argument that there is an implied license. The counterargument is that the user agreed to the Terms of Use which explicitly defined automated access as violative. In addition, these cases generally begin with a cease and desist demand letter, which explicitly informs the allegedly-infringing party that the publisher believes their rights are being violated and that they must cease and desist immediately. If the TOS argument doesn't hold, the C&D will surely qualify as a revocation of any implied license to access the content for copyright purposes. It generally also serves as explicit notice that the publisher considers the accessor to be "exceeding authorized use" of their computer systems, which is a crime under the CFAA. In Craigslist v 3Taps, the judge also commented that needing to circumvent IP bans should've made it obvious that 3Taps was "exceeding authorized access" under the CFAA.
>And copyright laws are supposed to protect only creative works, not every page on the web.
Copyright law protects all works of sufficient originality. Pretty much the only thing it doesn't protect is a plain list of facts (and in the European Union, it even protects that, known as "database rights"). The minimum standard of originality for copyright protection applies to practically every page on the web, yes.
In effect, this means that you can copy a list of names and addresses from a phone book, but you can't copy the layout. Since you can't access a web page without making a copy in RAM, if the publisher has revoked your license to access the content, even accessing and extracting the raw factual information within the body of the page is an infringement (because your RAM copy is an infringing copy).
IANAL and this is based on my layman's understanding.
I have done a lot of scraping in Python with requests and lxml and never really understood what scrapy offers beyond that. What are the main features that can't be easily implemented manually?
edit: Especially if your scraping jobs take a LONG time - days and weeks, this stuff is extra handy. Might I add a great debugging environment (scrapy shell), error handling, rate limiting, respecting robots.txt, so much more.
Nice overview! The "unfortunately-spelled threat_defence.php" just uses British spelling though.
What's wrong with British spelling? It's also the English spelling using in India, Australia, New Zealand etc. By pure numbers, more people may spell it defence than defense. Americocentrism is quite annoying from the other side :)
I use Java with simple task queue and multiple worker threads (scrapy is only singlethreaded, although uses async I/O). Failed tasks are collected into second queue and restarted when needed. Used Jsoup[1] for parsing, proxychains and HAproxy + tor [2] for distributing across multiple IPs. [1] https://jsoup.org/ [2] https://github.com/mattes/rotating-proxy
- to end the main thread only if all tasks are done
- when every running task can produce multiple new tasks
- with limiting the maximum number of running threads
- always running the maximum nubmer of threads if possible
semaphores to the rescue
Earlier quoted context omitted.
Unauthorized access, if you access the service in violation of their TOS then potentially they have a case against you. I'm not aware of it ever going to court in a case where they didn't also send a cease and desist.
I'm afraid that as long as explicit agreement is not required to make a TOS binding we'll be dealing with this crap.
We are getting small wins, but it's going to be slow going until we can get Congress to adjust both the CFAA and the Copyright Act, or until we can get SCOTUS to seriously alter the way these acts have been interpreted with reference to internet access.
Earlier quoted context omitted.
Doesn't publishing information on a public web server equals to granting authorization to download it? Why publish it otherwise? And copyright laws are supposed to protect only creative works, not every page on the web.
>Doesn't publishing information on a public web server equals to granting authorization to download it? Why publish it otherwise? This is the argument that there is an implied license. The counterargument is that the user agreed to the Terms of Use which explicitly defined automated access as violative. In addition, these cases generally begin with a cease and desist demand letter, which explicitly informs the allege…
Earlier quoted context omitted.
Anticaptcha and deatbycaptcha are some others. But it mames me feel sad to use them, as it exploits cheap labor overseas.
Most of the time they use OCR, humans are unreliable and rarely used.
Earlier quoted context omitted.
Most of the time they use OCR, humans are unreliable and rarely used.
no, at least antigate doesnt. When you hit recaptcha with known proxy urls (or generally hit it a few times per hour) the captchas get so bad that no OCR would be able to solve it, even humans struggle
I think part of it is how you crawl (phantomjs, for example, seem to hit captcha almost every time), but things like ip&proxy usage could make this trigger more often.
Earlier quoted context omitted.
right. so basically a greasemonkey script is scoped to the current page? Is there any scripting solution that is not scoped to current page? In chrome maybe?
Browser automation via (realistically Seleniun) WebDriver or a proxy that inserts scripts (like TestCafe).