Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What info do you web scrape for?

news.ycombinator.com

Post reply on HN
Print view View original

61–70 of 115 posts

Re: Ask HN: What info do you web scrape for?

#61

Post by lumpypua »

vijayr wrote: post #55
[deleted] wrote: post #41

Earlier quoted context omitted.

[deleted]

Thank you for answering. Though private and internal, you are still using the data for profit, is that correct? Does it mean that one can't sell the data, but can use it for analysis etc and still profit from it? I saw a service recently that emails app store reviews/ratings for a fee. Not sure if they are scraping or getting the reviews some other way. The same idea can be extended to lots of things like Amazon revi…

I deleted it 'cause I'm not comfortable having those details online. An extremely long story short, in our domain we're using the scraped data exactly as the owners intend, albeit via machines instead of people. Consult a lawyer.

Re: Ask HN: What info do you web scrape for?

#62

Post by lumpypua »

oz wrote: post #28
lumpypua wrote: post #18

I've had three primary uses of web scraping. The hard part for me has never been speed. Getting the results structured is somewhere between easy and hideously complicated. 1. Reformatting and content archival (lag times of hours to days are no prob). As an example, I put together a site to archive comments of a ridiculously prolific commenter on a site I follow. I needed the content of his comments, as well as the tr…

How can I contact you?

My username at google's email service.

Re: Ask HN: What info do you web scrape for?

#63

Post by dataminer »

fnbr wrote: post #19

I work as a research analyst for a Canadian provincial opposition party. Most government data is in terrible HTML tables, often dynamically generated, and almost none of it is in an easily machine readable format. I spend a lot of time downloading PDF files of data and converting them to JSON formats. I have two main recurring scrapes: - political donations. Every donation to a political party in my province above ~$…

This is very interesting work, regarding your last point, I came across this site which is searchable by employee name or government organization http://canada.landoffree.com/.

I think the salary and expense disclosure is only for Ontario, based on the sunshine list.

Re: Ask HN: What info do you web scrape for?

#65

Post by fnbr »

dataminer wrote: post #63
fnbr wrote: post #19

I work as a research analyst for a Canadian provincial opposition party. Most government data is in terrible HTML tables, often dynamically generated, and almost none of it is in an easily machine readable format. I spend a lot of time downloading PDF files of data and converting them to JSON formats. I have two main recurring scrapes: - political donations. Every donation to a political party in my province above ~$…

This is very interesting work, regarding your last point, I came across this site which is searchable by employee name or government organization http://canada.landoffree.com/ . I think the salary and expense disclosure is only for Ontario, based on the sunshine list.

I don't know much about the civil service, but I know that most provincial assemblies have expense disclosures, at least for the elected officials, e.g. [1], [2]. Federal cabinet ministers also have to disclose all of their office expenses (can't find a link, but I know it exists).

[1] BC: http://www.leg.bc.ca/Mla/remuneration/index.htm [2] Alberta: http://alberta.ca/travelandexpensedisclosure.cfm

Re: Ask HN: What info do you web scrape for?

#66

Post by Smerity »

rgrieselhuber wrote: post #60
Smerity wrote: post #54

Partial plug, but very related to the topic: if you're doing large scale analysis on the web and you don't want to have to actually run a large scale crawl, use the CommonCrawl dataset[1]! Common Crawl is a non profit organization that wants to allow anyone to use big web data. I'm one of the team behind the crawl itself. Last month (July) we downloaded 4 billion web pages. Thanks to Amazon Public Datasets, all of th…

What would it take in terms of resources, etc. to get to the point where Common Crawl was doing web scale crawls on a regular basis?

We're aiming to do monthly crawls from this point on. The main hold was automating the intensive manual steps of our crawl process. Now we have scripts that make running our 100 node EC2 cluster and processing the terabytes of web data relatively trivial.

If anyone wants to discuss sourcing well distributed crawl lists for billions of pages per month, we'd love to chat. We want to make sure we cover a diverse variety of languages and domains. Given that we're trying to get a good sample of the web, that's a difficult proposition!

Re: Ask HN: What info do you web scrape for?

#67

Post by eddywebs »

allegory wrote: post #22

I scrape Gumtree and eBay hourly using a python script for certain things I want under a certain price. The script sends me an email with the link in it and I get on top of it sharpish. Managed to bag a lot of stuff over the last couple of years for not much money. If someone bags this up as a service I'd pay for it.

@allegory could you share the the script with us ?

Re: Ask HN: What info do you web scrape for?

#68

Post by jasallen »

I am currently scraping for brand product and nutrition data. Having to build custom scrapers per brand is hell.

I have a dream to use something closer to OCR against a rendered page, rather than parsing DOM. That way it would be less custom, and I could say, for instance, "find 'protein', the thing to the right of that is the protein grams".

I, personally, don't know how to do this, but I'd be willing to pay for a more generic way to scrape nutrition data (email in profile :) )

Re: Ask HN: What info do you web scrape for?

#69

Post by samcrawford »

lumpypua wrote: post #59
samcrawford wrote: post #52

Earlier quoted context omitted.

There are services that cover at least part of what you mentioned. These effectively provide you a tool to visually build a scraper and then they automate the scraping in the background, creating an API or spreadsheet of the data. Import.io is one example, and I think there's another more recent YC-backed one. However, I tried using import.io a little while back but without much joy.

We have a non-tech intern and import.io looks like like a great tool to get him chewing up data. I'm playing with it now. Why didn't it work out for you? Beyond the wrapped browser interface being a little funky lol. (Edit: eugh, selecting data for import is really clunky.) Ask HN: Anybody got a visual scraping service they like?

It was the data extraction and selection process I couldn't get to work. I was trying to scrape a particular search on autotrader.co.uk (I wanted more up to date results than their daily emails provide, and I wanted to filter out cars that had been written off). I don't remember all the details, but I followed the tutorial video and got to the stage where you select a single item that matches your criteria and it's supposed to extrapolate from there. However I just seemed to be stuck in an infinite loop of it asking me to do this.

Re: Ask HN: What info do you web scrape for?

#70

Post by xur17 »

michaelt wrote: post #25

A lot of services with online billing refuse to send bills by e-mail, instead requiring users to log into their websites. No doubt the companies would justify this by saying e-mail isn't secure enough. The side-effect that it'll stop many users bothering to look at their bill isn't why they do it at all, no sir. I've been considering making a web scraper that goes to the phone company, electricity company, gas compan…

I started working on a tool like this over the weekend to pull down my bills, and pay the balance.

I think there is a market for something like Mint for bill paying - it's a it's a bit of a pain to remember when I have to pay all of my bills each month, and make sure to go through each one, and pay the balance on time.

Post reply on HN