Yeah, that was what I did when I didn't know anything about programming, too. The next step is to learn why web scraping is discouraged, and how to use public APIs instead.
Yep, I access HN via a powerful rest API.
Tell HN: 2 weeks ago, I didnt know programming. I just wrote a web scraper.
41–43 of 43 posts
Re: Tell HN: 2 weeks ago, I didnt know programming. I just wrote a web scraper.
#42Yeah, that was what I did when I didn't know anything about programming, too. The next step is to learn why web scraping is discouraged, and how to use public APIs instead.
> The next step is to learn why web scraping is discouraged, and how to use public APIs instead. Right, just like google uses public api's to access all those sites. Web scraping is responsible for a very large portion of all the traffic on port 80 and the number of sites that have published APIs is extremely small compared to the total.
I know, it's odd how some people act like all web scraping is a terrible thing, when people have built such large businesses based on it. Companies try very hard to get scraped by Google, and don't like being scraped by other random entities so much - I suppose it depends on your intent. Poorly written spiders are a problem for system admins.
If a resource does have a public API, some very positive things about using that instead are that not only is it less resource intensive for them, it is much easier for you to create and maintain. If you're relying on scraping to get specific info from a site, it's a huge Odin when changes to their HTML break your code.
Re: Tell HN: 2 weeks ago, I didnt know programming. I just wrote a web scraper.
#43Well done! If you write a detailed account of how you did it, you are bound to inspire a lot more people to try it out.