No, shut up. What statistical programming languages can learn from Dropbox. • Hacker News Board

Re: No, shut up. What statistical programming languages can learn from Dropbox.

#11

Post by _delirium » Tue, Feb 22, 2011, 5:37 PM UTC

kenjackson wrote: post #8

scott_s wrote: post #2

The code that makes him say "what a mess," I think is beautiful: def summary(data, key=itemgetter(0), value=itemgetter(1)): for k, group in groupby(data, key): yield (k, sum(value(row) for row in group)) Perhaps that's because I'm a programmer, and Python is a general purpose programming language. But I think that's what his complaint boils down to: the Python statistical code looks too much like Python . Which, yeah…

Perhaps that's because I'm a programmer, and Python is a general purpose programming language. Exactly. You shouldn't have to be a programmer to do statistics. Just like you shouldn't have to be a network engineer to share files. What if DropBox had stuff in there about http, ports, levels of service, bandwidth, etc... You'd probably say, "Great! I always wnated to specify that DropBox use SSL4.7 draft B over CDMA Ev…

There are GUI statistics apps for people who just want the common case, Dropbox-style: packages like Weka for data mining / predictive statistics, SPSS for descriptive statistics, and a dozen other such things.

The statisticians who choose to use a programming language like R or Python typically do it because they actually do want a programming language. I mean, that's why Bell Labs statisticians invented S (R's predecessor) to begin with.

Re: No, shut up. What statistical programming languages can learn from Dropbox.

#12

Post by tom_b » Tue, Feb 22, 2011, 5:37 PM UTC

What Dropbox does is eliminate the programmer from the equation. You don't need your IT staff to do any special setup or to follow a special process.

I think the truth is business users don't want to work with you - they just want their data relationships to be discovered in a simple and intuitive way. If your data crawling is sufficiently good, maybe you can do that.

I understand the point the article is making, but I feel rather strongly that "(y)ou’ll want a third thing – to read in and parse data" translates to the person that builds a tool that does that nicely and automatically creates pivot tables, graphs, and other dashboard-y things will probably have to hire a team to shovel the money off so he/she can breathe.

I suppose it's an ok example to talk about this re: statistical programming languages, but my own experience in the three requirements that preface the whole discussion (pivot tables, graphs, data parsing) are a big example of something just screaming for a new solution, not a new prog language . . .

Re: No, shut up. What statistical programming languages can learn from Dropbox.

#13

Post by dkarl » Tue, Feb 22, 2011, 5:39 PM UTC

If 90% of usage boils down to a small number of rigid patterns, then there is a simple solution: a handful of convenience functions. Often these functions are missing, because the demand for convenience functions is obscured by the fact that every experienced user defined them for himself years ago. That forces newbies to suffer through the unnecessary task of understanding the fully generalized API before they can accomplish simple tasks.

Languages that have good support for optional arguments, such as Python and Lisp, also make it possible to create APIs that are elegant and concise for experts but extremely intimidating for beginners. It may be more elegant to have a single function with a slew of optional arguments, and an experienced user may be able to accomplish any task quite concisely by specifying a few arguments, but a beginner would be better served by a handful of specific functions with specific names. API writers should consider providing those functions as simple wrappers to the general API, in order to provide a simpler learning curve for users who might never need more complex functionality. Examining how those wrapper functions are implemented can help intermediate users figure out the general API, too.

Re: No, shut up. What statistical programming languages can learn from Dropbox.

#14

Post by elmarks » Tue, Feb 22, 2011, 5:49 PM UTC

I can't speak for the creators of R, but I have a strong suspicion that it wasn't intended for erehweb or MBAs. It's not Microsoft Excel, its R. It's used by PhD researchers in Mathematics, Statistics, Economics and Political Science.

Re: No, shut up. What statistical programming languages can learn from Dropbox.

#15

Post by kenjackson » Tue, Feb 22, 2011, 5:50 PM UTC

_delirium wrote: post #11

kenjackson wrote: post #8

Earlier quoted context omitted.

Perhaps that's because I'm a programmer, and Python is a general purpose programming language. Exactly. You shouldn't have to be a programmer to do statistics. Just like you shouldn't have to be a network engineer to share files. What if DropBox had stuff in there about http, ports, levels of service, bandwidth, etc... You'd probably say, "Great! I always wnated to specify that DropBox use SSL4.7 draft B over CDMA Ev…

There are GUI statistics apps for people who just want the common case, Dropbox-style: packages like Weka for data mining / predictive statistics, SPSS for descriptive statistics, and a dozen other such things. The statisticians who choose to use a programming language like R or Python typically do it because they actually do want a programming language. I mean, that's why Bell Labs statisticians invented S (R's pred…

Right. I wasn't saying that there didn't exist such packages, of course there are. I was pointing out that the reason a programming language looks good to a programmer and not a statistician is due domain expertise. And of course the common trap programmers fall into is assuming the domain is programming.

And don't lump R in with Python. And good statistician would have your neck. You mention S, but again S doesn't look anything like Python either.

Re: No, shut up. What statistical programming languages can learn from Dropbox.

#16

Post by btilly » Tue, Feb 22, 2011, 5:51 PM UTC

Actually in my experience business users want one thing on that list, and if they have that thing they don't care about whether you provide the other two. They won't ask for what they want because they don't know that they can get it. But they will be happy if they get it.

They want to get at nicely organized data easily from inside of Excel. Excel is a toolbox that they already know from which they can do their own pivot tables and graphs. And they'd prefer to do that because then they can just do it instead of less efficiently having someone else do it for them.

They want it to arrive nicely aggregated and organized, since Excel is not very good at that. But they are more than happy to do the pretty reports themselves. Just get them the data.

See http://bentilly.blogspot.com/2009/12/design-of-reporting-sys... for a more detailed description of one set of experiences that taught me that.

Re: No, shut up. What statistical programming languages can learn from Dropbox.

#17

Post by icedpulleys » Tue, Feb 22, 2011, 5:53 PM UTC

peterb wrote: post #9

scott_s wrote: post #2

The code that makes him say "what a mess," I think is beautiful: def summary(data, key=itemgetter(0), value=itemgetter(1)): for k, group in groupby(data, key): yield (k, sum(value(row) for row in group)) Perhaps that's because I'm a programmer, and Python is a general purpose programming language. But I think that's what his complaint boils down to: the Python statistical code looks too much like Python . Which, yeah…

Yes, I like the Python also, but you have missed the point. For MBA-types, business types, and scientists the programming concepts are too much to learn. Why should they have to learn programming when their needs are simple? It is not just "keep it simple", it is "keep it simple" for non-programmers.

Maybe I'm missing the point too, because I don't understand why he's arguing that Python and R should cater to people that don't want to use a programming language. Isn't that akin to arguing that C is too complicated because it allows you to directly access memory rather than abstracting that away?

MBA- and business types have Excel. As a researcher, I flex both Python and R regulary -- but I want the full power of a programming language, not a couple of macros to generate a pivot table.

Re: No, shut up. What statistical programming languages can learn from Dropbox.

#18

Post by scott_s » Tue, Feb 22, 2011, 5:54 PM UTC

kenjackson wrote: post #8

scott_s wrote: post #2

The code that makes him say "what a mess," I think is beautiful: def summary(data, key=itemgetter(0), value=itemgetter(1)): for k, group in groupby(data, key): yield (k, sum(value(row) for row in group)) Perhaps that's because I'm a programmer, and Python is a general purpose programming language. But I think that's what his complaint boils down to: the Python statistical code looks too much like Python . Which, yeah…

Perhaps that's because I'm a programmer, and Python is a general purpose programming language. Exactly. You shouldn't have to be a programmer to do statistics. Just like you shouldn't have to be a network engineer to share files. What if DropBox had stuff in there about http, ports, levels of service, bandwidth, etc... You'd probably say, "Great! I always wnated to specify that DropBox use SSL4.7 draft B over CDMA Ev…

That looks close to as simple as possible, if you assume Python is to be used. My point about R was that even in a language designed for statistics, I saw dependence on common programming concepts.

Re: No, shut up. What statistical programming languages can learn from Dropbox.

#19

Post by scott_s » Tue, Feb 22, 2011, 5:57 PM UTC

peterb wrote: post #9

scott_s wrote: post #2

The code that makes him say "what a mess," I think is beautiful: def summary(data, key=itemgetter(0), value=itemgetter(1)): for k, group in groupby(data, key): yield (k, sum(value(row) for row in group)) Perhaps that's because I'm a programmer, and Python is a general purpose programming language. But I think that's what his complaint boils down to: the Python statistical code looks too much like Python . Which, yeah…

Yes, I like the Python also, but you have missed the point. For MBA-types, business types, and scientists the programming concepts are too much to learn. Why should they have to learn programming when their needs are simple? It is not just "keep it simple", it is "keep it simple" for non-programmers.

I understand that point. My point was that a Python library for statistics is not the right tool for them, but that in no way makes that statistic library or Python "bad." Python is a programming language. If you think that the users you have in mind can't handle programming, then don't give them a programming language.

Re: No, shut up. What statistical programming languages can learn from Dropbox.

#20

Post by ltjohnson » Tue, Feb 22, 2011, 5:58 PM UTC

_delirium wrote: post #11

kenjackson wrote: post #8

Earlier quoted context omitted.

Perhaps that's because I'm a programmer, and Python is a general purpose programming language. Exactly. You shouldn't have to be a programmer to do statistics. Just like you shouldn't have to be a network engineer to share files. What if DropBox had stuff in there about http, ports, levels of service, bandwidth, etc... You'd probably say, "Great! I always wnated to specify that DropBox use SSL4.7 draft B over CDMA Ev…

There are GUI statistics apps for people who just want the common case, Dropbox-style: packages like Weka for data mining / predictive statistics, SPSS for descriptive statistics, and a dozen other such things. The statisticians who choose to use a programming language like R or Python typically do it because they actually do want a programming language. I mean, that's why Bell Labs statisticians invented S (R's pred…

I am a statistician that does both research and applied work.

I use R for three reasons: (1) It's Free Software; (2) It's a programming language; (3) Other statisticians use it so it's easier for me to collaborate.

There are the usual supporting arguments for (1). (2), I've only used SAS a little bit, and it was extremely unpleasant to use it for non-built-in stuff, which makes research harder for no good reason. For (3), I have nothing against Python but most other statisticians don't use it. If I want to share my work in R, it's easy (statisticians know how to install R packages). If I want to share my work in Python, I first have to teach [most] other statisticians how to use Python. There's nothing wrong with that, but why raise the start-up cost for them?

tl;dr I conjecture that most statisticians don't want what the author is suggesting. Also, there are plenty of companies that are trying to do what the author is asking for, but most of them seem to miss the desired sweet spot, or charge lots of money, or both. I haven't taken a survey of the available software in quite some time.