Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Ask HN: Beyond Excel, how might a non-technical analyze large amounts of data?

news.ycombinator.com

Post reply on HN
Print view View original

1–10 of 11 posts

Ask HN: Beyond Excel, how might a non-technical analyze large amounts of data?

#1

Post by areohbe »

I have to do analysis on a large (in the Excel world) dataset fairly regularly. Ideally, I would like to have some type of repository that I could dump the data into on a monthly basis when I receive the reports and then be able to query and sort the data as needed. Beyond Excel, I'm not sure what the next step up in the world is, but I knew the kind folks here would not be afraid to share an opinion or two.

The data consists of multiple performance measurements for multiple unique entities on a per month basis. So currently, my Excel workbook looks something like this:

Sheet 1: Summary/Analysis Page that performances basic calculations off subsequent sheets

Sheet 2: Performance Metric 1 Column A: Unique Identifier Row A: Months B,B and beyond: Data

Sheet 3: Performance Metric 2 Column A: Unique Identifier Row A: Months B,B and beyond: Data

Sheet 4: Performance Metric 3 Column A: Unique Identifier Row A: Months B,B and beyond: Data

Re: Ask HN: Beyond Excel, how might a non-technical analyze large amounts of data?

#2

Post by Mz »

GIS software?

Most data has a physical location associated with it (street address, lat-long, something mappable). "A picture is worth a thousand words". Mapping data can help you visualize and mentally process heavy data loads. And Excel is one of the database formats typically compatible with standard software so with a smidgeon of luck you can probably kind of plug and play.

Re: Ask HN: Beyond Excel, how might a non-technical analyze large amounts of data?

#3

Post by rjempson »

It depends what you are trying to do exactly, but the excel add-in called powerpivot could be useful if you are trying to aggregate data and do adhoc analysis.

It is free :

http://www.microsoft.com/en-us/bi/powerpivot.aspx

I have analysed lots of BI tools, which could be useful in your scenario, but most require a bit of technical / database knowledge to iron out wrinkles.

Re: Ask HN: Beyond Excel, how might a non-technical analyze large amounts of data?

#8

Post by rpackard »

I worked R & D at a big consumer packaged goods company and ran into a lot of these 'excel messes' especially dealing with raw materials specifications. Here are my suggestions:

The 'R' programming language is pretty useful for statistics and could be used to aggregate a lot of files (and its free). Also if you want to generate dynamic reports using templates then you should look into sweave which is a package in 'R'. (http://www.stat.uni-muenchen.de/~leisch/Sweave/) You could build beautiful PDF reports that you simply run a script to generate if they are easily created into a template. But that would also require learning Latex for creating documents, something that is used heavily in the academic world but I have yet to see it in the business world.

Another approach which I would try is to use a programming language to dump it into a relational database and then use SQL to access the info (which is heavily used in business setting). This option also also allows for others to use it with familiar business tools. My unconventional tools of choice would be Ruby and using Rails to build an interface.

Re: Ask HN: Beyond Excel, how might a non-technical analyze large amounts of data?

#10

Post by ScottWhigham »

rjempson wrote: post #3

It depends what you are trying to do exactly, but the excel add-in called powerpivot could be useful if you are trying to aggregate data and do adhoc analysis. It is free : http://www.microsoft.com/en-us/bi/powerpivot.aspx I have analysed lots of BI tools, which could be useful in your scenario, but most require a bit of technical / database knowledge to iron out wrinkles.

+1 for PowerPivot. It kicks butt with large data sets, leverages your existing Excel skills, and extends those skills with DAX.
Post reply on HN