Jupyter, Mathematica, and the Future of the Research Paper • Hacker News Board

Re: Jupyter, Mathematica, and the Future of the Research Paper

#131

Post by iguy » Sun, Apr 15, 2018, 5:04 PM UTC

starpilot wrote: post #40

askvictor wrote: post #27

Earlier quoted context omitted.

While I totally hear you regarding the pain of python modules (particularly on Windows), the point of python 'distributions' like anaconda and canopy is to bring the kitchen sink along, kind of like mathematica. The problem with Mathematica from a science point of view is that, being closed source, means you can't independently ensure the calculations are happening correctly. To be replicable, science involving data…

> The problem with Mathematica from a science point of view is that, being closed source, means you can't independently ensure the calculations are happening correctly. To be replicable, science involving data needs to use open source tools. Excel and MATLAB can't be used for real science?

Well since the author is an economist, questions about "real science" can be interpreted a few ways.

For a fun look into the high standards of the field, a few years ago Piketty made the mistake of sharing his Excel files, in which all sorts of crucial adjustments were hard-coded into tables of data...

https://marginalrevolution.com/marginalrevolution/2014/05/pi...

Re: Jupyter, Mathematica, and the Future of the Research Paper

#132

Post by jononor » Sun, Apr 15, 2018, 5:05 PM UTC

falkod wrote: post #43

askvictor wrote: post #30

I'm surprised there's no mention or discussion of the importance of open-source tooling for replicable science. Without seeing and reviewing the source, how can you tell that a particular calculation is right? Also, relying on costly tools such as Mathematica cuts off a sizable amount of the population from being able to replicate or play with your findings on cost grounds alone.

Long-term Mathematica user (physicist) here: I don't think the use of open source software would make most science -- maybe that does not apply to cs/datascience -- more replicable. Usually that takes an expert in the field. And usually these experts are employed at universities where Mathematica licenses are not the prime cost factor. That said, I am all for open source software. Although I would argue that probably…

Open source brings with it a strong culture of publishing code openly. It seems that the amount of public Jupyter notebooks is already higher than that of Mathematica or similar, despite those tools having a 20 year lead?

Re: Jupyter, Mathematica, and the Future of the Research Paper

#133

Post by ChrisRackauckas » Sun, Apr 15, 2018, 5:06 PM UTC

carreau wrote: post #128

ILikeConemowk wrote: post #15

Earlier quoted context omitted.

Fernando, the creator of Ipython and ultimately what morphed into Jupyter is missing from your list. That dude is an inspiration.

Sorry, Fernando as well as all the contributors are included in the "we, the Jupyter team". I did not start to mention individuals otherwise the list would be really long. I would encourage people to look at (at minimum) the steering council on Jupyter.org for a list of key people in the project. We are also striving to make contributing to Jupyter / open source recognized in Academia - for people like Fernando to ge…

>We are also striving to make contributing to Jupyter / open source recognized in Academia - for people like Fernando to get proper recognition, and future generations of scientists to have incentives to do this kind of work.

And thank you for this! Do you have a post about what efforts are going on? I myself am trying to find out how to navigate academic spaces while spending significant time doing open source development. I am curious how you're pushing for changed incentives to help people in this kind of position out.

Re: Jupyter, Mathematica, and the Future of the Research Paper

#134

Post by improbable22 » Sun, Apr 15, 2018, 5:06 PM UTC

ChrisRackauckas wrote: post #91

jhbadger wrote: post #56

Earlier quoted context omitted.

Also the idea of companies making money off of open source isn't new or nefarious and often helps people who aren't even their customer. Red Hat is the classic example, and many organizations use CentOS, a distribution based off of Red Hat's distro without paying Red Hat anything. And in scientific computing there's RStudio, which makes a great open source IDE for R besides offering prducts and services for sale.

Wow, I asked for the data that led him to his conclusion about "the monopoly" and he blocked me. So much for open science... https://twitter.com/paulmromer/status/985529525491654657

I just realised who this blog post is by, it's the Paul Romer. Apparently gravitas is important at the World Bank, because clearly he's run the tank dry.

Re: Jupyter, Mathematica, and the Future of the Research Paper

#135

Post by takluyver » Sun, Apr 15, 2018, 5:13 PM UTC

gaius wrote: post #76

nmca wrote: post #74

Earlier quoted context omitted.

Open source software, in general, tends to make things more reproducible. Sure, software licencing might not be the singular root cause, but why does that suggest we shouldn't capitalise on the improvements available?

Open source software, in general, tends to make things more reproducible Citation very much needed for that. Because you can very easily find that 6 months or a year later you update your dependencies and everything is now broken. I recently came back to a Python project after a year, updated my packages then realized: I simply cannot be bothered to unpick the mess that resulted just to add one trivial feature. Where…

Open source software is important for reproducibility for a couple of reasons. Firstly, if you record that you've done your analysis with Python 3.6.3 and Numpy 1.14.2, and it later breaks on some newer version, it's relatively easy to get the same versions you were using. Commercial software vendors are usually not keen on you downloading and running a version of their product which was superseded four years ago.

Secondly, of course, open source means that if you're not sure why two versions/functions/libraries are giving you different answers, you can go and find out. I accept that a lot of people may not have time for that, but I don't think you can fix that problem unless it's possible to dig down and follow the working.

Finally, 'reproducible by anyone with a computer' is a lot better than 'reproducible by people who buy a license for the tool I used to do it'.

Re: Jupyter, Mathematica, and the Future of the Research Paper

#136

Post by williamstein » Sun, Apr 15, 2018, 5:17 PM UTC

ChrisRackauckas wrote: post #119

williamstein wrote: post #24

Earlier quoted context omitted.

Thanks for mentioning SageMath and CoCalc (I founded both of these projects)! A minor clarification is that CoCalc is not a rebranding of SageMath, but is instead a new web application whose goal is to make it very easy to collaboratively use Sage, Jupyter, LaTeX, Julia, etc. In constract, Sage is a more traditional open source software package, which people install on their own computers. The goal of Sage is to be a…

First of all, thank you for all of your hard work. One thing I am curious about though is how you think you can match the "cohesiveness" of Mathematica. While I am not a big fan of Mathematica the language, there is something gained by a top down approach. The options all seem to have the same names, the display tools tend to compose with new functionality well, the symbolic parts integrate well with the numerical to…

> how you think you can match the "cohesiveness" of Mathematica

The basic strategy is to provide a new Python library ("the sage library" [1]), which is a layer between the user and the other dependencies. E.g., instead of just saying "use Maxima for symbolic limits", we have our own notion of symbolic functions (implemented using Ginac), and automatically convert them to and from Maxima when doing symbolic limits. The user never has to know anything about Maxima. When we first implemented this, it just used interprocess communication under the hood, but (thanks to Nils Bruin and others) we now use a direct C library interface builton top of ECL (embedded common lisp). This "new Python library", which isn't so new, since I started it in 2004, is now nearly a million lines of code.

[1] https://github.com/sagemath/sage/tree/master/src/sage

Re: Jupyter, Mathematica, and the Future of the Research Paper

#137

Post by blablabla123 » Sun, Apr 15, 2018, 5:18 PM UTC

dahart wrote: post #8

> The tie-breaker is social, not technical. The tie-breaker is financial. Jupyter is winning because it's free, not because it's social. It becomes social because of widespread adoption, and it get's widespread adoption because it's free. I love Jupyter, love love love. But there's a lot of hyperbole and opinion here. Mathematica is just a for-profit business, it's that simple. And it wouldn't be fair to deny the exa…

Also not to forget that Jupyter is just different to Mathematica. With Mathematica you can do symbolic computations, yes also Statistics and Machine Learning, but also Group Theory and what not. Jupyter does a great job as an interface for certain Statistics and Machine Learning tasks, also I'm quite sure that it needs less resources but that's all.

That said, I'm still missing a free but powerful tool for symbolic computations like Mathematica or Maple.

> The tie-breaker is financial.

Exactly, it cannot be emphasized enough. Of course as a student you get these powertools for a small price or even for free. But if you are not in University, those tools are super expensive. For a reason but there is still a need for far more open source in this area.

EDIT: I'm just realizing there is Sympy, niiceee...

Re: Jupyter, Mathematica, and the Future of the Research Paper

#138

Post by dragonwriter » Sun, Apr 15, 2018, 5:20 PM UTC

tincholio wrote: post #108

dahart wrote: post #103

Earlier quoted context omitted.

> What I wonder is why are Jupyter notebooks so popular? How easy is it to share a Jupyter notebook with someone else, versus sharing an org-mode notebook? Think about the recipients rather than how hard it is for you. Assume the person you share with needs to be able to re-run your notebook. Assume the person you share with isn't using the same operating system that you are. Be honest and think about all the steps i…

> How easy is it to share a Jupyter notebook with someone else, versus sharing an org-mode notebook? Well, now you're talking about a whole environment. Use docker, or VMs or whatever. There's plenty of options for doing this (we actually use Docker for Jupyter, to make sure we work off a consistent install, so I guess sharing can be non-trivial if you walk a bit off the beaten path, anyway) > If I choose to use a di…

> Sure, and if you need to use another language from your Jupyter notebook, you can't.

Not entirely true, BTW.

https://blog.dominodatalab.com/lesser-known-ways-of-using-no...

Re: Jupyter, Mathematica, and the Future of the Research Paper

#139

Post by gaius » Sun, Apr 15, 2018, 5:24 PM UTC

takluyver wrote: post #135

gaius wrote: post #76

Earlier quoted context omitted.

Open source software, in general, tends to make things more reproducible Citation very much needed for that. Because you can very easily find that 6 months or a year later you update your dependencies and everything is now broken. I recently came back to a Python project after a year, updated my packages then realized: I simply cannot be bothered to unpick the mess that resulted just to add one trivial feature. Where…

Open source software is important for reproducibility for a couple of reasons. Firstly, if you record that you've done your analysis with Python 3.6.3 and Numpy 1.14.2, and it later breaks on some newer version, it's relatively easy to get the same versions you were using. Commercial software vendors are usually not keen on you downloading and running a version of their product which was superseded four years ago. Se…

f you record that you've done your analysis with Python 3.6.3 and Numpy 1.14.2, and it later breaks on some newer version, it's relatively easy to get the same versions you were using

Better record which compiler you used too, and what flags, and every version of every library and everything else. It’s not as simple as you make out and it’s far from guaranteed that all those packages will still be available or compile on your OS.

Commercial software vendors are usually not keen on you downloading and running a version of their product which was superseded four years ago.

I guess you must not deal with vendors much because generally they are fine with this. It’s part of the support agreement usually, just another service. Getting an “obsolete” version for whatever reason has never been a problem for me.

By “anyone with a computer” you mean “anyone who can exactly reproduce my configuration which I don’t even know myself for certain”

Re: Jupyter, Mathematica, and the Future of the Research Paper

#140

Post by gaius » Sun, Apr 15, 2018, 5:27 PM UTC

madengr wrote: post #125

gaius wrote: post #41

Earlier quoted context omitted.

The problem with Mathematica from a science point of view is that, being closed source, means you can't independently ensure the calculations are happening correctly Have there been any high profile failures root-caused to Mathematica (or MATLAB or any similar product) getting its sums wrong? I can find any news stories etc. Plenty of serious calculations were and are done on “closed source” HP and TI calculators too…

I’ll add that many closed sourced developers will show you sections of the source code if you sign an NDA. I’ve done that on several occasions when I needed to see how a model was implemented. As long as you are not a competitor, it’s usually not an issue. I sign NDAs often to see IP.

Additionally most of Mathematica is written in Mathematica and you can just read it. Same with MATLAB.

I get the feeling that many of the Jupyter fans commenting here have only ever used a Jupyter setup and are unaware of the wider industry