Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: What info do you web scrape for?

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 115 posts

Ask HN: What info do you web scrape for?

#1

Post by cblock811 »

I have been web scraping for several months and am starting to teach it at Meetups. I'm lucky enough to work for a company that has a few pre-crawled copies of the web that I can query against and a distributed processing platform to speed up any scraping I do.

I'm running out of ideas of what to build though. I build scrapers to produce content for the company based on the data and insights I find. They are usually marketing verticals though, such as finding all websites using feedback tools (I search based on their javascript widgets), and do analysis on that info.

So if you had these resources, what would you be looking for? I love building tools that help people so any feedback/ideas would be great!

I'm also open to hearing what you would scrape for on the live web. I find that if I'm doing broad analysis then the pre-crawled copies are best, and for specific sites/information I use the live web.

Re: Ask HN: What info do you web scrape for?

#2

Post by murukesh_s »

I used to scrape web for a daily deals search engine i wrote for a client in 2010. But we scraped in realtime as the number of sites were really low (in 10s).

pre-crawled copies with distributed processing platform could be cool. you could come up with a better search engine with programmable rules that are edited collaboratively (like wikipedia)

Re: Ask HN: What info do you web scrape for?

#3

Post by chollida1 »

There is a closet industry for scraping any sort of data that can move markets. Fed, crop, weather, employment,etc.

Anything that is released at a certain time on a fixed calendar, you can bet that multiple parties are trying to scrape it as fast as possible.

If you can scrape this data( the easy part), put it in a structured format( somewhat hard) and deliver it in under a few seconds(this is where you get paid) then you can almost name your price.

It's an interesting niche that hasn't been computerized yet.

If you can't get the speed then the first 2 steps can still be useful to the large number of funds that are springing up using "deep learning" techniques to build a portfolio over timelines of weeks to months.

To answer the question of: > Wouldn't this require a huge network of various proxy IPs to constantly fetch new data from the site without being flagged and blacklisted?

This is why I gave the caveat of only looking at data that comes out at certain times. That way you only have to hit the server once, when the data comes out, or atleast a few hundred times in the seconds leading up to the data's release:)

Re: Ask HN: What info do you web scrape for?

#4

Post by finkin1 »

We do a lot of live web scraping of product information from retail sites for http://agora.sh. We basically scrape all of the essential product info and offer it to the user in an optimized view (we call it the 'product portal') that can be accessed without having to load a new page. This reduces tab sprawl and provides a general improvement to a lot of shopping workflows, such as just wanting to see larger/more images of the product (likely clothing) and being able to do so with an overlay on the existing page.

Re: Ask HN: What info do you web scrape for?

#6

Post by chrisan »

chollida1 wrote: post #3

There is a closet industry for scraping any sort of data that can move markets. Fed, crop, weather, employment,etc. Anything that is released at a certain time on a fixed calendar, you can bet that multiple parties are trying to scrape it as fast as possible. If you can scrape this data( the easy part), put it in a structured format( somewhat hard) and deliver it in under a few seconds(this is where you get paid) the…

> deliver it in under a few seconds(this is where you get paid) then you can almost name your price.

Wouldn't this require a huge network of various proxy IPs to constantly fetch new data from the site without being flagged and blacklisted?

Or are you referencing from the time you scrape data to deliver it in under 3 seconds?

Re: Ask HN: What info do you web scrape for?

#8

Post by jyu »

chollida1 wrote: post #3

There is a closet industry for scraping any sort of data that can move markets. Fed, crop, weather, employment,etc. Anything that is released at a certain time on a fixed calendar, you can bet that multiple parties are trying to scrape it as fast as possible. If you can scrape this data( the easy part), put it in a structured format( somewhat hard) and deliver it in under a few seconds(this is where you get paid) the…

That is a fairly surprising opportunity. I have experience monitoring/scraping thousands government websites for a different purpose. Considering some government sites have a round trip of well over 5 seconds, seems like it'd be a fun challenge to parse, format, and deliver it that fast.

What types of data formatting are you talking about here? Would it require a unique template for each individual site?

Re: Ask HN: What info do you web scrape for?

#9

Post by jeffclark »

kurrent wrote: post #5

Sports scores, statistics, etc are always high in demand for scraping and great to get people interested when learning scraping techniques.

Yeah, very much agreed. I have to scrape schedules together to populate games for my ticket site (http://www.boxrowseat.com).

The popular sites are wising up to scraping of information, so it's really an exercise in futility. But when it works, it's super rewarding.

Post reply on HN