Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Don’t Need No Stinking API: Web Scraping For Fun and Profit

blog.hartleybrody.com

Post reply on HN
Print view View original

141–150 of 176 posts

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#141

Post by freshhawk »

frabcus wrote: post #90
freshhawk wrote: post #38

Earlier quoted context omitted.

Phantom.js and casper.js If you can't get the data from the endpoints the javascript hits then write your scraper in javascript and have it run in a headless browser, and it's the webkit engine so most sites test their site against it heavily. Either pull the data out of the javascript objects or trigger your extraction from the html by attaching to the events in the javascript.

I've started playing with zombie.js recently as well - much lighter and faster than the ones that instrument a completely full browser. But has a full Javascript engine.

That's really interesting, thanks.

I worry that it's not going to replicate a real browser accurately enough, but I'm excited to try it out a bit.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#142

Post by joesb »

chill1 wrote: post #129
kybernetyk wrote: post #81

Earlier quoted context omitted.

And your comment is another proof that people tend to assume everyone else is an idiot.

I think his comment was quite appropriate, and did not feel there was an implication that he thought anyone was dumb for not having known about the alternative approach to getting Reddit data. Often times programmers and the managers that drive them are way too quick to get going building or solving something with brute force. If they would just be patient and stop for a moment. Spending even a mere 30 minutes extra…

Saying that that was a proof of spending 15 minutes to search about Reddit API would save his time implied that kybernetyk didn't do that research.

But kybernetyk already said he did the research before and that Reddit's API is not good enough for his requirement.

So this is not the case that 15 minutes of research will save the time. And his comment meant he assumed kybernetyk didn't do research, i.e. being dumb for not doing search.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#143

Post by jonpaul »

frabcus wrote: post #90
freshhawk wrote: post #38

Earlier quoted context omitted.

Phantom.js and casper.js If you can't get the data from the endpoints the javascript hits then write your scraper in javascript and have it run in a headless browser, and it's the webkit engine so most sites test their site against it heavily. Either pull the data out of the javascript objects or trigger your extraction from the html by attaching to the events in the javascript.

I've started playing with zombie.js recently as well - much lighter and faster than the ones that instrument a completely full browser. But has a full Javascript engine.

zombie.js is not a full browser. It's a poor emulation using jsdom as its backing. http://zombie.labnotes.org/guts Beware, for some applications, jsdom is super buggy.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#144

Post by jonpaul »

freshhawk wrote: post #141
frabcus wrote: post #90

Earlier quoted context omitted.

I've started playing with zombie.js recently as well - much lighter and faster than the ones that instrument a completely full browser. But has a full Javascript engine.

That's really interesting, thanks. I worry that it's not going to replicate a real browser accurately enough, but I'm excited to try it out a bit.

Your worry is correct. http://news.ycombinator.com/item?id=4896054 I've tried scraping with it, and it failed miserably on some sites.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#145

Post by stackthatcode »

reledi wrote: post #134

Automated web testing tools, such as Watir and Selenium, are also pretty good options. I'm especially surprised Watir hasn't been mentioned yet in the comments.

Indeed - or WatiN, the .NET port of WatiR. I've done some pretty heavy duty scraping and automation with WatiN, which included building a OO framework that trivialized writing scripts. Good stuff.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#146

Post by olefoo »

bdcravens wrote: post #45
goostavos wrote: post #41

Earlier quoted context omitted.

In regards to the javascript problem. I'd suggest checking out the mobile versions of the sites first before you hop to a weighty solution like Selenium. Could be a very simple solution to the problem :) I recently built a twitter bot that did some scraping and posting. I beat my head against a wall for a couple of hours trying to find a good tool to deal with all the javascript driven stuff. I happened to get an upd…

My scraping is part of a transactional B2B service, not a high traffic social or B2C thing, so it's a different set of problems than those who want their hands on Twitter data. These are Fortune 100's, so if I can bring down their site, they have bigger problems. :-)

I wouldn't make that assumption. Do check return codes and load times, and back off if you see issues. If these sites are business partners/suppliers you have a lot to lose if things go wrong. It's worth it to develop your relationship with the business owners of the services you're touching in the correspondent organizations. And do set a User-Agent string that declares who you are and provides a link for information; if you are doing business with them, it should be on a basis of honesty.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#147

Post by SiVal »

What I wish I could do is capture Flash audio (or any audio) streams with my Mac. All I want is to listen to the audio-only content with an audio player when I'm out driving or jogging, etc. Audio-only content that has to be played off a web page usually runs into the contradiction that if I'm in a position to click buttons on my web browser (not driving, for example), I'm in a position to do my REAL work and have no time to listen to the audio. I'll go to the web page, see whatever ads they may have, but then I'd like to be able to "scrape" the audio stream into a file so I don't have to sit there staring at a static web page the whole time I'm listening.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#148

Post by codewright »

kybernetyk wrote: post #64
codewright wrote: post #43

Earlier quoted context omitted.

> even your smartest regex can have a brain fart If you're using regex to solve this sort of problem, your code deserves to break, I'm sorry.

This. You usually traverse the DOM. Either you use some XQuery /XPath magic or a library like beautiful soup.

Sizzle for life.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#149

Post by codewright »

SiVal wrote: post #147

What I wish I could do is capture Flash audio (or any audio) streams with my Mac. All I want is to listen to the audio-only content with an audio player when I'm out driving or jogging, etc. Audio-only content that has to be played off a web page usually runs into the contradiction that if I'm in a position to click buttons on my web browser (not driving, for example), I'm in a position to do my REAL work and have no…

I used to work at a company where capturing flash video and audio streams was a regular part of our work. You're not going to like the answer.

You basically have to proxy everything through a proxy that can be given a command or otherwise instructed to capture the top 3 or 4 streams from the website. From there you can either dumbly accept the largest one or start checking byte headers.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#150

Post by wslh »

SiVal wrote: post #147

What I wish I could do is capture Flash audio (or any audio) streams with my Mac. All I want is to listen to the audio-only content with an audio player when I'm out driving or jogging, etc. Audio-only content that has to be played off a web page usually runs into the contradiction that if I'm in a position to click buttons on my web browser (not driving, for example), I'm in a position to do my REAL work and have no…

My company has an API to do that but it works on Windows: http://www.nektra.com/products/deviare-api-hook-windows/
Post reply on HN