Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

I Don’t Need No Stinking API: Web Scraping For Fun and Profit

blog.hartleybrody.com

Post reply on HN
Print view View original

91–100 of 176 posts

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#91

Post by rohamg »

freshhawk wrote: post #59
randomdata wrote: post #33

Earlier quoted context omitted.

I wonder if any webapps use randomly generated IDs and class names (linked in the CSS) to prevent scraping. In my spare time, I've been playing around with "scrapers" (I like to call them web browsers, personally) that don't even look at markup. My first attempt used a short list of heuristics that proved to be eerily successful for what I was after. To the point I could throw random websites with similar content (di…

I would be really interested in knowing which heuristics or machine learning techniques produced decent results. That's if I can't convince you to open source the code. I'm working on the same problem at the moment.

What about something like http:// tubes.io

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#94

Post by RBerenguel »

mmastrac wrote: post #3

I'm surprised that no one has attempted to write a Twitter client based solely on scraping to get around the token limits.

I've written some in-browser JS to download all my tweets without needing to resort to the API and oAuth nightmare. It is indeed possible to write a client, but not recommended at all...

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#95

Post by SG- »

When scraping HTML where data gets populated with js/ajax, you can get a web inspector to look at where that data is coming from and manually GET it and it will likely be in some nice JSON.

Scraping used to be the way to get data back in the days, but websites also didn't change their layout/structure on a weekly basis too back then and were much more static when it came to the structure.

Having recently written a small app that was forced to scrape HTML and having to update it every month to make it keep working, I can't imagine doing this for a larger project and maintaining it.

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#98

Post by RaSoJo »

alexmic wrote: post #88
RaSoJo wrote: post #17

I love HTML scraping. But Javascript???...The juiciest data sets these days are increasingly in JS. For the love of me i can't get around scraping JS :( I do know that Selenium can be used for this...but am yet to see a decent example for the same. Does anyone have any good resources/examples on JS scraping that they could share?? I would be eternally grateful.

If you are using Python, you can also use pyv8 to evaluate Javascript code.

indeed i do use Python. thanks for sharing...it appears most interesting. Have started playing with it...

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#99

Post by RaSoJo »

kybernetyk wrote: post #68
RaSoJo wrote: post #17

I love HTML scraping. But Javascript???...The juiciest data sets these days are increasingly in JS. For the love of me i can't get around scraping JS :( I do know that Selenium can be used for this...but am yet to see a decent example for the same. Does anyone have any good resources/examples on JS scraping that they could share?? I would be eternally grateful.

Here's a little experiment with Reddit-automation using Selenium: https://github.com/jsz/reddit_voting

awesome...thanks :)

Re: I Don’t Need No Stinking API: Web Scraping For Fun and Profit

#100

Post by abhaga »

toyg wrote: post #4

And this is why we can't have nice things. Web scraping, as fun as it is (and btw, this title again abuses "Fun and Profit"), is not a practice we should encourage. Yes, it's the sort of dirty practice many people do, at one point or another, but it shouldn't be glorified.

There is tons of data on the govt websites in India and the only way to get to that is by scrapping the websites. Knowing that you can scrap it and be on your way feels very liberating at times. Example: Rates for Indian postal department services. Minutes of parliament houses. (great for building machine translation systems. A lot of research in MT has benefit from the availability of parallel corpus consisting of parliament proceedings in 2 or more languages. Hansard corpus from Canada. European Parliament corpus. No such luck in India.)
Post reply on HN