Live data from Hacker News

Why Jupyter is data scientists’ computational notebook of choice

nature.com

291–300 of 308 posts

Re: Why Jupyter is data scientists’ computational notebook of choice

#291

I love notebooks as a way to present information, data, code and computations. However, I cannot stand typing any text into a web browser window. Is there any way to edit a jupyter notebook with a text editor and then run it in the browser? The native json is not really human-editable.

If you are comfortable writing python programs, you can use nbformat package ( https://github.com/jupyter/nbformat ) to generate Jupyter notebooks: http://nbviewer.jupyter.org/gist/fperez/9716279

Here is another quick example based on the gist above:

https://nbviewer.jupyter.org/gist/ontouchstart/58c62c8248540...

Re: Why Jupyter is data scientists’ computational notebook of choice

#292

Earlier quoted context omitted.

Yes. It's called Microsoft Excel. Software engineers don't like VB for the same reason they don't like Python-in-a-notebook but you cannot deny its effectiveness.

You're right about what excel is (and the whole VB ecosystem for that matter), but I think the critical difference is that the language and environment are very different. If I know the smallest amount of python (or R) I can leverage Jupyter notebooks and it is intuitive. To really get something great out of excel you have to learn excel. I think that difference is almost as important as the excel stigma.

Interestingly, excel has the advantage that the data has to always be visible. This is also a disadvantage, because it can't work with too much data.

My gut is that the amount of data it can work with more than compensates for the disadvantage.

Re: Why Jupyter is data scientists’ computational notebook of choice

#293
post #284

Earlier quoted context omitted.

Microsoft just announced an initiative like this (unfunded, community-based, likely at risk of becoming abandonware), perhaps you could combine your efforts with theirs? (The issue in their code that I'm personally most impacted by in lack of support for conda[0]) [0] https://github.com/lorenzo2897/vscode-ipe/issues/162

Fixed this in my fork :) I want to combine, but Neuron has stated they will not be accepting any PRs until December.

The Neuron extension is no longer in active development, because we are taking the functionalities further by building them directly into the Microsoft VS Code Python extension. We're still finishing up to get ready for the first official release, but if anyone is interested in giving it a try, the code is here: https://github.com/Microsoft/vscode-python. Let us know what you think :)

Re: Why Jupyter is data scientists’ computational notebook of choice

#294
post #81

Earlier quoted context omitted.

You parting "Granted..." is precisely what fills me with dread when I see notebooks. Yes, I have seen poorly done source files. I made more than a few myself. However, many of the practices we have grown into as sound programming advice seem to be largely thrown out the window for these notebooks. The irony, to me, is that I actually typically argue for the mixing of presentation and content. But to me, notebooks loo…

There is also the entire bend where these are taking off in a way that doesn't make sense. It makes perfect sense. Just not to a lot of HN readers. The average HN reader is approaching this from a perspective of "I am a professional programmer who might occasionally dabble in scientific computing, and therefore I hate this thing because it's not a professional programmer's tool designed by and for professional progra…

I'm approaching from the "I was an electric engineer and we had better tooling back wheni was in undergrad.". They just weren't free.

Re: Why Jupyter is data scientists’ computational notebook of choice

#295

Earlier quoted context omitted.

But in the case of dev and prod distinction it has nothing to do with fitting some over-constrained engineering principle, but about fitting actual science: if you cannot reproduce something, you don't have a result, you have a fluke.

I think GP here is an insightful comment. Reproducing things is indeed important, but re-running code is much too narrow a definition, and possibly distractingly narrow. Maybe your awful notebook gets the same answer you got the day before on the blackboard. Or the same answer your collaborator got independently, perhaps with different tools. Those might be great checks that you understand what you're doing. Spending…

I find that with Jupyter, re-running code does serve one useful purpose, which is to make sure that your result isn't affected by out-of-order execution or a global that you declared and forgot about. That is a real pitfall of Jupyter that has to be explained to beginners.

For my work, reproducing a result may involve collecting more data, because a notebook might be a piece of a bigger puzzle that includes hardware and physical data. This is where scripting is a two edged sword. On the one hand, it's easy to get sloppy in all of the ways that horrify real programmers. On the other hand, scripting an experiment so it runs with little manual intervention means that you can run it several times.

Re: Why Jupyter is data scientists’ computational notebook of choice

#296

Earlier quoted context omitted.

r studio supports python? does it completion and stuff like that?

You can use R Studio to execute the code, and you're preferred editor to do the editing, if you like.

try https://gryd.us

Re: Why Jupyter is data scientists’ computational notebook of choice

#297

Earlier quoted context omitted.

> When I see stuff around notebooks for "reproducibility", I'm a bit confused in that notebooks often don't specify any guidance on installation and dependencies, let alone things like arguments and options that a regular old script would. At the core of this, as some others may have already alluded to already, is that many academic scientists have not been socialized to make a distinction between development and pro…

This is kind of a broad observation, but scientists tend to borrow tools from a huge variety of fields, and use them in ways that seem un-disciplined to the practitioners of those fields. For instance, an engineer would be horrified to see me working in the machine shop without a fully dimensioned and toleranced drawing. A project manager would be disturbed to learn that I don't have a pre-written plan for my next ta…

I agree that science can't be bound by the rigid structures of most applied disciplines, and that the freedom to combine technologies in novel ways is a pre-requisite to novel findings.

What I find objectionable is the inability of scientists to explicitly delegate tasks to domain specialists in their everyday work when it makes sense. I think that it's unrealistic of you to believe that engineers always work with "a fully dimensioned and toleranced drawing" before starting work on a project and that your would work "grind to a halt". Indeed, there's a reason for the qualifier rapid in the term "rapid prototyping". If you can give an engineer general specifications for what you want and then leave him/her alone, he/she should be able to produce something that mostly fits your needs while avoiding all of the pitfalls that wouldn't have occurred to you. It would also be incorrect to assume that engineering does not involve creativity and is purely bound by rigid processes- if your requirements were strange enough, something fresh would inevitably be built.

This sort of delegation of course, is actually more efficient, since you can work on other tasks in parallel with the engineer (such as writing your next grant proposal or article or gasp teaching). Most scientists also already do this implicitly by choosing to purchase instrumentation from manufacturers like Olympus, Phillips, or Siemens rather than building it themselves.

Part of the reason for why I have such strong opinions about this matter, is that I've actually witnessed scientists waste more time messing around in fields where they were clearly out of their depth. As an example, there was a thread on a listserv in my (former) field that lasted for literally months that was solely devoted to the appearance of a website. Everyone wanted to turn the website design into an academic debate, when the website's creation (which had little to do with the substance of the scholarship itself) could have been turned over to a seasoned web developer and finished in less than a week or two.

Re: Why Jupyter is data scientists’ computational notebook of choice

#298
post #259

Earlier quoted context omitted.

They're scientists who increasingly have to do programming as part of their science. And notebooks are a godsend for them 2/3 of scientific research cannot be reproduced by other scientists. But tell us more about why scientists should ignore best practices from other fields.

Because I don't have four years to get something done, that doesn't do what I want when I finally get it, if it even works at all, and that I can't fix myself. Okay, that was extreme, and if you think I was talking about programming, it's because you have a guilty conscience. ;-) It actually applies to all interesting fields -- programming, engineering, management, classical music composition, etc. Those fields don't…

> Those fields don't even know what their best practices are

"Best practices" are a chimera. The issue at hand isn't about what is "best", but whether or not a software engineer's "good enough" practices are more likely to achieve science's goals than a graduate student's "good enough" practices.

It's also disingenuous to claim that classical music composition doesn't have "best practices" when the field of music theory exists as an explicit manifestation of "best practices" in music. Having gone to a school with a conservatory, I also believe that I know several individuals who would would disagree with your mindset regarding how the creative process can't be managed. Indeed, if creativity, as it relates to musical composition, couldn't be managed most orchestras would be brimming with anger at the number of commissions that weren't finished on time for the concert, and most Hollywood studios and Broadway shows would screech to a halt.

Re: Why Jupyter is data scientists’ computational notebook of choice

#299
post #256

Earlier quoted context omitted.

Is that atom's fault or just the Julia REPL's fault? I use the REPL directly on Windows and it seems to be really slow as it will take something like "using JuMP" and precompile the module which takes time.

I think it is Juno (the Julia package for Atom)'s fault. Atom is fine on its own, as is the Julia REPL after compilation. I just looked through Julia's settings tab in Atom, and saw the option "Fallback Renderer" with the note "Enable this if you're experiencing slowdowns in the built-in terminals." It was disabled by default, so I've just enabled it. Subjectively, I think it feels fine now. Longer use will tell, but…

I wonder if Windows is slower. It isn't terrible, but waiting 8 seconds after typing in "using JuMP" is kind of long.

Re: Why Jupyter is data scientists’ computational notebook of choice

#300

Earlier quoted context omitted.

Because I don't have four years to get something done, that doesn't do what I want when I finally get it, if it even works at all, and that I can't fix myself. Okay, that was extreme, and if you think I was talking about programming, it's because you have a guilty conscience. ;-) It actually applies to all interesting fields -- programming, engineering, management, classical music composition, etc. Those fields don't…

> Those fields don't even know what their best practices are "Best practices" are a chimera. The issue at hand isn't about what is "best", but whether or not a software engineer's "good enough" practices are more likely to achieve science's goals than a graduate student's "good enough" practices. It's also disingenuous to claim that classical music composition doesn't have "best practices" when the field of music the…

Okay, that's fair. I should not have included classical composition in that list.
Post reply on HN