Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What info do you web scrape for?

news.ycombinator.com

Post reply on HN
Print view View original

41–50 of 115 posts

Re: Ask HN: What info do you web scrape for?

#41

Post by [deleted] »

vijayr wrote: post #32
lumpypua wrote: post #18

I've had three primary uses of web scraping. The hard part for me has never been speed. Getting the results structured is somewhere between easy and hideously complicated. 1. Reformatting and content archival (lag times of hours to days are no prob). As an example, I put together a site to archive comments of a ridiculously prolific commenter on a site I follow. I needed the content of his comments, as well as the tr…

What about legal implications? Do you get permission from the sites you crawl?

[deleted]

Re: Ask HN: What info do you web scrape for?

#42

Post by aruggirello »

aruggirello wrote: post #37

Scraping really is a quite complex process, and not everybody does it right. Do you employ a (distributed?) crawler pool? What if a scraped page goes offline (404/410)? And, how do you handle network errors, and 403's / getting caught (and possibly blocked) - if at all? Do you conceal the scraping by employing a fake user agent? Do you (sometimes?) request permission for scraping to relevant webmasters? These are the…

BTW I write tailor made PHP+MySQL scraper scripts, targeting English or Italian language sites; contact me for more info :)

Re: Ask HN: What info do you web scrape for?

#43

Post by cblock811 »

jerhinesmith wrote: post #29

A while ago, I had the idea of creating a travel site that catered to the group of people that enjoy traveling but aren't bound by time (i.e. I want to go to X, but I don't care when -- just show me the cheapest weekend for the next 3 months). Anyway... it turns out that flight APIs are ridiculously non-existent. I ended up scraping two different airline sites, but since it was against their terms, I never took the s…

The hospitality and travel industries are very slow to update their technologies. I used to work with Ritz Carlton and St. Regis and even those brands are practically in the stone age, so I can't imagine how scraping for flight info would go.

I've thought of even building a simple event aggregator for some friends in the industry and they are blown away that it's possible. Then I remember how many venues are in cities like Charlotte and San Francisco and realize why these industries lag in technology. There just isn't a large pool of developers who want to solve their problems.

Do you have any projects you are currently working on?

Re: Ask HN: What info do you web scrape for?

#44

Post by jaequery »

I'm currently scraping data such as "tweets, comments, likes" a website gets each day so I can graph them over time.

One thing I am having a hard time scraping backlinks to websites. Currently using bing but they are paid after like 5000 queries. I really wonder how other companies like seomoz do this daily against millions of websites.

Re: Ask HN: What info do you web scrape for?

#45

Post by jpetersonmn »

I do a lot of scraping for my day job. We have a business intelligence team that will build us reports that we need from the data that we have. However I find that this process is so incredibly slow and sometimes we only need to compile the data for a one-off project. I used to use vb.net for this as that's what I started learning programming with. Now I use python/requests/bs4 for all my scraping scripts.

I've started working on a new website that will use data scraped from several vbulletin forums. I've found that even 2 vbulletin forums running the same version may have completely different html to work with. I'm assuming that it's the templates they are using that changes it so much.

I'm setting up the process so that the webscraping happens from different locations than the server were the site is hosted. The scraping scripts upload to the webserver via an api I've built for this. Mostly did this because for now I'm just using a free pythonanywhere account and their firewall would block all of this without a paid account. And then also none of these sites would see the scraping traffic coming from my website, etc...

Re: Ask HN: What info do you web scrape for?

#46

Post by frabcus »

vijayr wrote: post #32
lumpypua wrote: post #18

I've had three primary uses of web scraping. The hard part for me has never been speed. Getting the results structured is somewhere between easy and hideously complicated. 1. Reformatting and content archival (lag times of hours to days are no prob). As an example, I put together a site to archive comments of a ridiculously prolific commenter on a site I follow. I needed the content of his comments, as well as the tr…

What about legal implications? Do you get permission from the sites you crawl?

Legality of scraping is a subtle issue - I wrote up my take on it here: https://blog.scraperwiki.com/2012/04/is-scraping-legal/
Post reply on HN