Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Web Scraping: Bypassing “403 Forbidden,” captchas, and more

sangaline.com

Post reply on HN
Print view View original

1–10 of 232 posts

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#4

Post by tehlike »

herbst wrote: post #2

I've used antigate for captchas and ether Tor or proxies for 403s before. Usually the browser header alone does not help for long.

Anticaptcha and deatbycaptcha are some others. But it mames me feel sad to use them, as it exploits cheap labor overseas.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#5

Post by herbst »

tehlike wrote: post #4
herbst wrote: post #2

I've used antigate for captchas and ether Tor or proxies for 403s before. Usually the browser header alone does not help for long.

Anticaptcha and deatbycaptcha are some others. But it mames me feel sad to use them, as it exploits cheap labor overseas.

while I agree, it may also is a option for some unskilled people to get to money they really need. IMO its better than buying cheap clothes, as captcha at least does not kill those people.

To clearify: I lived in some cheap third world cities, and i can certanly see that solving captchas could finance a rather nice life.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#7

Post by ben_jones »

m00dy wrote: post #6

I use greasemonkey on firefox. Recently, I have written a crawler for a major accomondation listing website in Copenhagen. Guess what? I got a place to live right in the center in 2 weeks. I love SCRAPERS I love CRAWLERS.

Well the problem is when someone scrapes ALL the good listings then pre-purchases them for resale at double the cost.

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#8

Post by retube »

m00dy wrote: post #6

I use greasemonkey on firefox. Recently, I have written a crawler for a major accomondation listing website in Copenhagen. Guess what? I got a place to live right in the center in 2 weeks. I love SCRAPERS I love CRAWLERS.

so can you use greasemonkey to follow links, load new page, parse new page, just like a headless crawler?

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#9

Post by richthegeek »

bla2 wrote: post #3

Nice overview! The "unfortunately-spelled threat_defence.php" just uses British spelling though.

What's wrong with British spelling? It's also the English spelling using in India, Australia, New Zealand etc. By pure numbers, more people may spell it defence than defense. Americocentrism is quite annoying from the other side :)

Re: Web Scraping: Bypassing “403 Forbidden,” captchas, and more

#10

Post by toxik »

richthegeek wrote: post #9
bla2 wrote: post #3

Nice overview! The "unfortunately-spelled threat_defence.php" just uses British spelling though.

What's wrong with British spelling? It's also the English spelling using in India, Australia, New Zealand etc. By pure numbers, more people may spell it defence than defense. Americocentrism is quite annoying from the other side :)

Sort of a tangent, but I doubt more people speak and write RE over CE.
Post reply on HN