Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Is Microsoft Excel an Adequate Statistics Package?

practicalstats.com

Post reply on HN
Print view View original

101–108 of 108 posts

Re: Is Microsoft Excel an Adequate Statistics Package?

#101

Post by bereasonable »

newman314 wrote: post #100
bereasonable wrote: post #91

Earlier quoted context omitted.

Excel can handle millions of rows if he brings the data into PowerPivot. Note: I'm a PM on Excel

Random point but Excel sure could use a CONTAINS function for substring matches. I was looking for something like this yesterday and the solutions I found were quite ugly. Ended up having to use DSUM/DCOUNT instead which is still inelegant when one has to to multiple lookups with slightly varying parameters.

We're planning on adding support for regex. The current state of the work is shown on our User Voice site. https://excel.uservoice.com/forums/304921-excel-for-windows-...

Re: Is Microsoft Excel an Adequate Statistics Package?

#102

Post by dandermotj »

peatmoss wrote: post #74
dandermotj wrote: post #62

Earlier quoted context omitted.

So is Hadley Wickham... https://twitter.com/hadleywickham/status/748392441154248704

Well, later in that thread he did write: "I am worried by the number of people that couldn’t tell that this was sarcasm"

That was the joke I was making...

Re: Is Microsoft Excel an Adequate Statistics Package?

#103

Post by peatmoss »

dandermotj wrote: post #102
peatmoss wrote: post #74

Earlier quoted context omitted.

Well, later in that thread he did write: "I am worried by the number of people that couldn’t tell that this was sarcasm"

That was the joke I was making...

Ah dang, I got whooshed.

Re: Is Microsoft Excel an Adequate Statistics Package?

#104

Post by nightski »

fnord123 wrote: post #95
nightski wrote: post #23

Earlier quoted context omitted.

Your conclusion is that it was excel causing these errors and you are implying similar errors would not be made in Python. I think it is more because of your experience as a developer why you were able to spot and correct errors.

Sure, you can make errors in Python. But you can also write tests to validate the data in Python. And there is a culture of doing so in Python. >I think it is more because of your experience as a developer why you were able to spot and correct errors. I don't feel like I spotted errors. I wrote a script to validate the data and the script told me if there were errors (there were).

The act of writing a script to validate the data says otherwise.

Also the people who use Excel as a primary tool are not the type that write unit tests in Python generally. Or would even think to do something like that. That is my point. You, as a developer, would think of something like that. It's not that you couldn't write similar tests with excel (you could, in any .NET language). But that you thought of doing so.

Re: Is Microsoft Excel an Adequate Statistics Package?

#105

Post by nightski »

dandermotj wrote: post #27
nightski wrote: post #23

Earlier quoted context omitted.

Your conclusion is that it was excel causing these errors and you are implying similar errors would not be made in Python. I think it is more because of your experience as a developer why you were able to spot and correct errors.

It is absolutely the interface to excel that causes errors. I can take your excel spreadsheet, format cells, add things, change a reference and generally fuck it up and hand it back to you and you would never know. Whereas, if I change a text file then you can see what's changed. Similar errors are much less common in a programming language.d

You can write tests for an excel spreadsheet if that is your thing. In fact Excel can be driven and automated by any .NET language. It is quite extensible. But most people don't do that because they are not developers and the word "unit test" is not part of their routine.

Which is why they will not be switching to python or R any time soon and even if they do, it will still have similar issues as the Excel version.

Just because you know how to use git history, diffs, etc.. to spot differences in code doesn't mean that is going to help the layperson.

Re: Is Microsoft Excel an Adequate Statistics Package?

#106

Post by euyyn »

twunde wrote: post #51
dandermotj wrote: post #22

Earlier quoted context omitted.

So just to give you an opposing perspective of what you just said: "we have __one__ person that does all...". So apart from the fact he can't access a database appropriately, he's the _only_ person who does this work and none of it is documented or reviewable because that's the nature of Excel. There's some serious operational risk in that. This guy disappears tomorrow and what do you do?

It's a misconception that work in Excel can't be documented or reviewed. We have the seed data he uses and the resulting Excel files (with the formulas he used still in them). This is the equivalent to version control for programming. The results are also reviewed against 1-2 appropriate sources (invoices, historical data, multiple other reports built by IT) for accuracy. To your very legitimate point that he's the o…

I don't understand your version control equivalence. If someone introduces a non-obvious bug in his formulas, how can you discover it, and restore the file to its previous good state?

Re: Is Microsoft Excel an Adequate Statistics Package?

#107

Post by fnord123 »

nightski wrote: post #104
fnord123 wrote: post #95

Earlier quoted context omitted.

Sure, you can make errors in Python. But you can also write tests to validate the data in Python. And there is a culture of doing so in Python. >I think it is more because of your experience as a developer why you were able to spot and correct errors. I don't feel like I spotted errors. I wrote a script to validate the data and the script told me if there were errors (there were).

The act of writing a script to validate the data says otherwise. Also the people who use Excel as a primary tool are not the type that write unit tests in Python generally. Or would even think to do something like that. That is my point. You, as a developer, would think of something like that. It's not that you couldn't write similar tests with excel (you could, in any .NET language). But that you thought of doing so…

Ok then I think we're agreeing but using different bits of the same point. In Excel there is no culture of good data validation. My experience working as a developer in Python gave me that culture. Agreed.

In GGGP's case they almost certainly have no tests so I stand by the recommendation that they fight hard to get a validation system in place. Probably by moving to Python and/or a RDBMS.

Re: Is Microsoft Excel an Adequate Statistics Package?

#108

Post by PeterisP »

lordnacho wrote: post #34
ZanyProgrammer wrote: post #28

Earlier quoted context omitted.

I could've sworn I've seen people who work in the financial and maybe data science industries who've praised Excel to the heavens on HN. Like on the same level as Python, R, etc. I don't work in finance (or data science really) so I can't comment, but it seems merely a tool to me.

No way, not data science. How are you going to load in millions, possibly billions of individual numbers? With finance there's a lot of things that seem like good fits, but only if you keep them small. Something like a personal budget is fine, where everything fits on a screen.

A simple excel pivotchart on 10k-100k financial entries often works better than the best BI / MIS packages.

For most companies, all their data science fits in excel easily. Full transaction history with each individual item sold since founding the company? No problem. Every single visitor/page view on their website? I've seen such logs imported to excel to do some rough aggregation. Manufacturing data about each particular widget sub-part that ever went off your conveyor belt? Again, for many companies with hundreds of employees that would still fit in excel without issues.

There are so many companies who speak about big data while their largest datasets can fit into RAM of a cheap laptop. That doesn't mean that data science and analysis is worthless to them, quite contrary.

Post reply on HN