Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping: Bypassing “403 Forbidden,” captchas, and more

sangaline.com

Post reply on HN
Print view View original

31–40 of 232 posts

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#31

Post by cryptarch »

cmdrfred wrote: post #24
beejiu wrote: post #19

Earlier quoted context omitted.

How is this any different from Google doing it? It is okay for Google to crawl the Internet, but not okay to crawl Google Play? Google raising such an objection would be an ultimate irony. Edit: On second thought, I guess you are referring to overcoming 403s and Captchas?

Unauthorized access, if you access the service in violation of their TOS then potentially they have a case against you. I'm not aware of it ever going to court in a case where they didn't also send a cease and desist.

I'm afraid that as long as explicit agreement is not required to make a TOS binding we'll be dealing with this crap.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#32

Post by martin-adams »

bla2 wrote: post #3

Nice overview! The "unfortunately-spelled threat_defence.php" just uses British spelling though.

Well, I did not know that defense was the American spelling. I spent ages looking at that trying to find out which one was the British and which was the American spelling. You learn something every day.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#33

Post by cryptarch »

retube wrote: post #30
m00dy wrote: post #21

Earlier quoted context omitted.

I think you can use a 3rd party backend service to store state of the crawler. So, when page reloads, you know which state you are in.

right. so basically a greasemonkey script is scoped to the current page? Is there any scripting solution that is not scoped to current page? In chrome maybe?

Browser automation via (realistically Seleniun) WebDriver or a proxy that inserts scripts (like TestCafe).

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#35

Post by nip »

Scrapy is indeed excellent. One feature that I really like is Scrapy Shell [1].

It allows to run and debug the scraping code without running the spider, right from the CLI.

I use it extensively to test that my selectors (both CSS and XPATH) are returning the proper data on a test URL.

[1] https://doc.scrapy.org/en/latest/topics/shell.html

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#36

Post by Raphmedia »

ben_jones wrote: post #7
m00dy wrote: post #6

I use greasemonkey on firefox. Recently, I have written a crawler for a major accomondation listing website in Copenhagen. Guess what? I got a place to live right in the center in 2 weeks. I love SCRAPERS I love CRAWLERS.

Well the problem is when someone scrapes ALL the good listings then pre-purchases them for resale at double the cost.

How is it different than paying 50+ low-wage remote workers to "scrape" the phonebook for you and then using the information acquired for profit?

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#38

Post by codedokode »

jacquesm wrote: post #13

Note that in some places this constitutes breaking the law.

I think such laws are wrong. Scraping is what Google and Web Archive does and it serves good purposes. For example, one can make an application that compares prices for the same item at different internet shops and helps to find the cheapest offer.

I don't understand what's wrong with downloading the information that is published on a public web server. That is what that server was made for in the first place.

Of course there are people who scrape the website, add advertisement and optimize it to rank better than the original website. But this problem can solved with other measures.

And of course those who do the scraping must limit their request rate so the server doesn't get overloaded.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#39

Post by codedokode »

cookiecaper wrote: post #18
brogrammernot wrote: post #14

Earlier quoted context omitted.

This is the first I've heard of this. Where/what laws prohibit web scraping?

It's almost always illegal in the United States. It's prohibited by a combination of the CFAA, copyright law, and contractual obligations imposed by Terms of Use, which are usually considered applicable if you load more than one page ("browsewrap"). The CFAA makes it a crime to access any computer network without authorization or in excess of granted authorization. The Terms of Use will usually prohibit "any automate…

Doesn't publishing information on a public web server equals to granting authorization to download it? Why publish it otherwise?

And copyright laws are supposed to protect only creative works, not every page on the web.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#40

Post by onli »

Lxr wrote: post #37

I have done a lot of scraping in Python with requests and lxml and never really understood what scrapy offers beyond that. What are the main features that can't be easily implemented manually?

The Scrapy cloud alone is worth it. And it gives guidance, making the start very easy.
Post reply on HN