Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Don’t Need No Stinking API: Web Scraping For Fun and Profit

blog.hartleybrody.com

Post reply on HN
Print view View original

41–50 of 176 posts

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#41

Post by goostavos »

bdcravens wrote: post #16

I've done a ton of scraping (mostly legal: on behalf of end users of an app on sites they have legit access to). This article misses something that affects several sites: JavaScript driven content. Faking headers and even setting cookies doesn't get around this. This is of course is easy to get around, using something like phantom.js or Selenium. Selenium is great because unlike all the whiz bang scraping techniques,…

In regards to the javascript problem. I'd suggest checking out the mobile versions of the sites first before you hop to a weighty solution like Selenium. Could be a very simple solution to the problem :)

I recently built a twitter bot that did some scraping and posting. I beat my head against a wall for a couple of hours trying to find a good tool to deal with all the javascript driven stuff. I happened to get an update on my phone when it dawned on me that the mobile.twitter site is, for the most part, simple html stuff. Once I realized that, I was able to programatically log into my account with no problems, and the rest of twitter was unlocked for me. I could scrape and post to (almost) my heart content.

However, there were a few very big problems. Which makes me feel that scraping is not the way to go about things. I certainly wouldn't build a service based around scraping a particular site's data.

When I had my twitter bot operational, I would get blocked from twitter for hours at a time. It seems anytime I hit their servers too hard, or crossed some threshold, I would be locked out. I'm assuming it was some kind of IP level ban, because I wasn't even able to access the site from an actual browser.

I was able to deal with the setback by setting up a script to repeatedly check its access the site, and then relaunch the scraper upon discovering access, but the solution was just a band-aid. That would translate to significant downtime if I was running a service with counted on access to their data. The ban-hammer is too easily laid down.

Finally, just as a word of caution, I'd warn prospective scrapers to be careful of just who you scrape. I've inadvertently "DDoS'd" a site when a multiprocessed script got away from me. It spawned 1000+ instances of this particular request, all of which were doing their best to beat the bejesus out of this small websites servers. The site ended up going down for a couple of hours; I assume because of a bandwidth cap or something.

So, my point being, scraping is cool, but (1) I'm unsure if I agree with relying on it over a proper API, and (2) with great power comes great responsibility! Be nice to smaller guys, and don't punish their servers to bad.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#42

Post by freshhawk »

cloverich wrote: post #34
freshhawk wrote: post #15

Earlier quoted context omitted.

So you're not so hot on the whole search engine thing? The article does slide into the sketchy side (I've always wanted an excuse to do that client side javascript trick too) but I found it more interesting because of that, these aren't secrets. Maybe if I put my "won't somebody please think of the children" hat on I agree that glorifying using trojan code to potentially ddos someones server to get around rate limits…

> So you're not so hot on the whole search engine thing? They scrape to generate links for users to go to the site . That's quite different than scraping for...any other purpose? So it seems. Would you (anyone) argue otherwise? (genuine curiosity).

Does it have to be a search based interface to the indexed data?

Does finding a link to the scrapee have to be the primary purpose of the site (and therefore google would be constantly getting "worse" by this scale)?

So how prominent does the link back have to be for it to be ok?

What about the summarized data from there that search engines are adding these days, so you don't need to leave the google results page to get your answer but the data still comes from some site that you rarely notice the name of?

edit: as to your curiosity, I honestly do not see the line that you see. Unless it's that the link back to the source is required. I don't know that I agree with that but I would understand it, although that gets harder and harder the more you massage your dataset to be useful to users.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#43

Post by codewright »

joering2 wrote: post #25
bdcravens wrote: post #16

I've done a ton of scraping (mostly legal: on behalf of end users of an app on sites they have legit access to). This article misses something that affects several sites: JavaScript driven content. Faking headers and even setting cookies doesn't get around this. This is of course is easy to get around, using something like phantom.js or Selenium. Selenium is great because unlike all the whiz bang scraping techniques,…

+1. Thanks for the hints about Selenium. My 2c about scraping - when you try to obtain data from large websites, always go for javascript content. Pages like Newegg or Amazon * may change html outline very often even without a single alteration to the front-user and even your smartest regex can have a brain fart. In contrast, even when site gets major overhaul, most likely old javascript will be left in place with al…

> even your smartest regex can have a brain fart

If you're using regex to solve this sort of problem, your code deserves to break, I'm sorry.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#44

Post by robryan »

cloverich wrote: post #34
freshhawk wrote: post #15

Earlier quoted context omitted.

So you're not so hot on the whole search engine thing? The article does slide into the sketchy side (I've always wanted an excuse to do that client side javascript trick too) but I found it more interesting because of that, these aren't secrets. Maybe if I put my "won't somebody please think of the children" hat on I agree that glorifying using trojan code to potentially ddos someones server to get around rate limits…

> So you're not so hot on the whole search engine thing? They scrape to generate links for users to go to the site . That's quite different than scraping for...any other purpose? So it seems. Would you (anyone) argue otherwise? (genuine curiosity).

They are also using title, description, some snippets from the page and taking a cached version of the site and images you can view without having to visit the site itself. They are also using this data as a product to sell advertising against.

If there wasn't so much benefit for most of all sites to be in search engine indexes you would thinking at least some would object to this scraping.

There would be lots of other scraping that websites want to prevent that takes even less data than this. It just doesn't provide much in return for the website.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#45

Post by bdcravens »

goostavos wrote: post #41
bdcravens wrote: post #16

I've done a ton of scraping (mostly legal: on behalf of end users of an app on sites they have legit access to). This article misses something that affects several sites: JavaScript driven content. Faking headers and even setting cookies doesn't get around this. This is of course is easy to get around, using something like phantom.js or Selenium. Selenium is great because unlike all the whiz bang scraping techniques,…

In regards to the javascript problem. I'd suggest checking out the mobile versions of the sites first before you hop to a weighty solution like Selenium. Could be a very simple solution to the problem :) I recently built a twitter bot that did some scraping and posting. I beat my head against a wall for a couple of hours trying to find a good tool to deal with all the javascript driven stuff. I happened to get an upd…

My scraping is part of a transactional B2B service, not a high traffic social or B2C thing, so it's a different set of problems than those who want their hands on Twitter data. These are Fortune 100's, so if I can bring down their site, they have bigger problems. :-)

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#46

Post by randomdata »

cloverich wrote: post #34
freshhawk wrote: post #15

Earlier quoted context omitted.

So you're not so hot on the whole search engine thing? The article does slide into the sketchy side (I've always wanted an excuse to do that client side javascript trick too) but I found it more interesting because of that, these aren't secrets. Maybe if I put my "won't somebody please think of the children" hat on I agree that glorifying using trojan code to potentially ddos someones server to get around rate limits…

> So you're not so hot on the whole search engine thing? They scrape to generate links for users to go to the site . That's quite different than scraping for...any other purpose? So it seems. Would you (anyone) argue otherwise? (genuine curiosity).

From what I understand, Google uses crawled data as a learning set for their translation service. There is no "this phrase was learned from: www.nytimes.com" when I do a translation, so I guess Google is still guilty?

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#47

Post by bdcravens »

codewright wrote: post #43
joering2 wrote: post #25

Earlier quoted context omitted.

+1. Thanks for the hints about Selenium. My 2c about scraping - when you try to obtain data from large websites, always go for javascript content. Pages like Newegg or Amazon * may change html outline very often even without a single alteration to the front-user and even your smartest regex can have a brain fart. In contrast, even when site gets major overhaul, most likely old javascript will be left in place with al…

> even your smartest regex can have a brain fart If you're using regex to solve this sort of problem, your code deserves to break, I'm sorry.

I've found that regex is very brittle when you don't control what comes across. DOM traversal is far more reliable.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#48

Post by [deleted] »

bdcravens wrote: post #47
codewright wrote: post #43

Earlier quoted context omitted.

> even your smartest regex can have a brain fart If you're using regex to solve this sort of problem, your code deserves to break, I'm sorry.

I've found that regex is very brittle when you don't control what comes across. DOM traversal is far more reliable.

[deleted]

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#49

Post by goostavos »

bdcravens wrote: post #40
dphase wrote: post #29

Earlier quoted context omitted.

I've done this successfully with Ruby Mechanize.

Awesome. I'd love some hints or links, as I'm always looking to refactor.

In general, if you're going the mechanize route, .retrieve() is the function your looking for.

e.g.

  br = mechanize.Browser()
  br.retrieve("https://www.google.com/images/srpr/logo3w.png, google_logo.png)[0]
Mechanize doesn't really have a proper doc, but just about everything you'd need could be figured out from the very lengthy examples page on their site.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#50

Post by derrida »

bdcravens wrote: post #47
codewright wrote: post #43

Earlier quoted context omitted.

> even your smartest regex can have a brain fart If you're using regex to solve this sort of problem, your code deserves to break, I'm sorry.

I've found that regex is very brittle when you don't control what comes across. DOM traversal is far more reliable.

Agree... basically search methods that specify a branch or leaf locally rather than the entire tree structure can more often resist layout changes.

Regex for HTML is a bad idea ... http://stackoverflow.com/questions/590747/using-regular-expr...

Post reply on HN