21–30 of 232 posts
I use greasemonkey on firefox. Recently, I have written a crawler for a major accomondation listing website in Copenhagen. Guess what? I got a place to live right in the center in 2 weeks. I love SCRAPERS I love CRAWLERS.
so can you use greasemonkey to follow links, load new page, parse new page, just like a headless crawler?
I've used antigate for captchas and ether Tor or proxies for 403s before. Usually the browser header alone does not help for long.
Anticaptcha and deatbycaptcha are some others. But it mames me feel sad to use them, as it exploits cheap labor overseas.
Earlier quoted context omitted.
This is the first I've heard of this. Where/what laws prohibit web scraping?
It's almost always illegal in the United States. It's prohibited by a combination of the CFAA, copyright law, and contractual obligations imposed by Terms of Use, which are usually considered applicable if you load more than one page ("browsewrap"). The CFAA makes it a crime to access any computer network without authorization or in excess of granted authorization. The Terms of Use will usually prohibit "any automate…
Note that in some places this constitutes breaking the law.
How is this any different from Google doing it? It is okay for Google to crawl the Internet, but not okay to crawl Google Play? Google raising such an objection would be an ultimate irony. Edit: On second thought, I guess you are referring to overcoming 403s and Captchas?
Note that in some places this constitutes breaking the law.
How is this any different from Google doing it? It is okay for Google to crawl the Internet, but not okay to crawl Google Play? Google raising such an objection would be an ultimate irony. Edit: On second thought, I guess you are referring to overcoming 403s and Captchas?
Earlier quoted context omitted.
Anticaptcha and deatbycaptcha are some others. But it mames me feel sad to use them, as it exploits cheap labor overseas.
Most of the time they use OCR, humans are unreliable and rarely used.
I use greasemonkey on firefox. Recently, I have written a crawler for a major accomondation listing website in Copenhagen. Guess what? I got a place to live right in the center in 2 weeks. I love SCRAPERS I love CRAWLERS.
I used 1 week to selectively go through accommodations manually, then proceed to complain to a friend of mine.
She's barely human, and she found literary one-of-a-kind apartment dead center at a good price. The apartment was mine next day.
Human scrapers man.
Earlier quoted context omitted.
Anticaptcha and deatbycaptcha are some others. But it mames me feel sad to use them, as it exploits cheap labor overseas.
Most of the time they use OCR, humans are unreliable and rarely used.
Note that in some places this constitutes breaking the law.
How is this any different from Google doing it? It is okay for Google to crawl the Internet, but not okay to crawl Google Play? Google raising such an objection would be an ultimate irony. Edit: On second thought, I guess you are referring to overcoming 403s and Captchas?
Earlier quoted context omitted.
so can you use greasemonkey to follow links, load new page, parse new page, just like a headless crawler?
I think you can use a 3rd party backend service to store state of the crawler. So, when page reloads, you know which state you are in.