Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Jupyter is data scientists’ computational notebook of choice

nature.com

Post reply on HN
Print view View original

251–260 of 308 posts

Re: Why Jupyter is data scientists’ computational notebook of choice

#251

Post by bigger_cheese »

WhompingWindows wrote: post #39
wanderfowl wrote: post #14

I like R for many things, but Python just keeps getting more compelling, particularly given the excellent machine learning packages. As these sorts of toolchain elements get better and better, and as more people realize that there's a benefit to simultaneously training researchers to run code as well as stats, I suspect we'll start to see an exodus from pure R solutions. The real question is when (and whether) new so…

The migration between languages is also industry specific. They are still teaching SAS to finance and healthcare analysts, for instance, and R and Python are still rising in healthcare specifically. Keep in mind all the legacy code and all the coders who just know SAS and don't need to change. It'll take longer for the transition than you think.

I have tried to move from SAS to R a few times at this point it's largely inertia there is so much in my org already written in SAS makes it very difficult to convince people to change.

I do like the SAS dev tools (especially Enterprise Guide). I'd really love if R had some sort of GUI front end for non technical people. Eg. I know the finance analysts in our org wouldn't have a clue how to configure their own ODBC sources -which you need to do with R Studio until it's as easy for them as SAS convincing them to switch won't get any traction.

Re: Why Jupyter is data scientists’ computational notebook of choice

#252

Post by analog31 »

lixtra wrote: post #4

What was the earliest of these tools? Mathcad? Mathematica? Maple?

  > 10 PRINT "HELLO WORLD"
  > RUN
  HELLO WORLD
  >
Just being a smart-aleck, of course, but scientists have been using interactive programming tools since they became available, and they have only grown in sophistication.

Re: Why Jupyter is data scientists’ computational notebook of choice

#253

Post by canhascodez »

kristjansson wrote: post #238
ISL wrote: post #170

Earlier quoted context omitted.

I'm not a Jupyter user, but I solve the reproducibility problem with Make. As a project moves from exploration toward production, the entire thing is wrapped into a Makefile that can flow from raw data to publication in a single call to make.

This has been my solution as well. There’s little that feels as good as running `make -B report` and watching the whole thing be rebuilt from scratch. How do you manage encapsulating each step, and passing data between them?

Would that be easier with rake than with make?

Re: Why Jupyter is data scientists’ computational notebook of choice

#254

Post by analog31 »

neuromantik8086 wrote: post #110
makmanalp wrote: post #51

The majority of the complaints I hear about notebooks I think come from a misunderstanding of what they're supposed to be. It's a mashup between a scientific paper and a repl. So it's useful for a bit of both: a) Just like with a paper, you can present scientific or mathematical ideas with accompanying visualizations or simulations. From the REPL side, as a bonus, you get interactivity, and the reader can pause and e…

> When I see stuff around notebooks for "reproducibility", I'm a bit confused in that notebooks often don't specify any guidance on installation and dependencies, let alone things like arguments and options that a regular old script would. At the core of this, as some others may have already alluded to already, is that many academic scientists have not been socialized to make a distinction between development and pro…

This is kind of a broad observation, but scientists tend to borrow tools from a huge variety of fields, and use them in ways that seem un-disciplined to the practitioners of those fields. For instance, an engineer would be horrified to see me working in the machine shop without a fully dimensioned and toleranced drawing. A project manager would be disturbed to learn that I don't have a pre-written plan for my next task. How do I even know what I'm going to do? If we adopted the most disciplined processes from every field, we'd grind to a halt.

In fact, there might be something about what attracts people to be scientists rather than engineers, that makes us bristle at doing what engineers consider to be "good" engineering.

Re: Why Jupyter is data scientists’ computational notebook of choice

#255

Post by azag0 »

bodkan wrote: post #227
azag0 wrote: post #164

I love Jupyter Notebook for experimenting and rapid creation of reports, but dislike it for not being able to use my editor and for intermingling inputs and outputs in a single file. So I'm working on an alternative frontend to Jupyter kernels, which is heavily inspired by KnitR: https://github.com/azag0/knitj It is still being developed, but it's functional and I use it every day.

This looks incredible! Does your project already support other language kernels than the Python kernel? I use R for 90% of my work, but most of it has been happening in Jupyter notebooks (which I'm not a huge fan of, despite practically living in them for the past 4 years of my life). Thanks for sharing!

I have not tested it with anything else than the Python kernel, but it uses Jupyter Client to communicate with the kernel, which is kernel agnostic. So you should be able to do just “knitj -k ...”.

Re: Why Jupyter is data scientists’ computational notebook of choice

#256

Post by celrod »

3rdAccount wrote: post #212
celrod wrote: post #132

Earlier quoted context omitted.

RStudio is also my favorite editor. All my work is data science / stats related, where I like the workflow of writing/modifying code in a .R (or .py) file, and being able to quickly experiment by running chunks in a REPL with Ctrl + Enter. R and Python are supported. No Julia, unfortunately. VS Code and Atom support similar workflows with Julia. However, the Julia Language server in VS Code is extremely unstable and…

Is that atom's fault or just the Julia REPL's fault? I use the REPL directly on Windows and it seems to be really slow as it will take something like "using JuMP" and precompile the module which takes time.

I think it is Juno (the Julia package for Atom)'s fault. Atom is fine on its own, as is the Julia REPL after compilation.

I just looked through Julia's settings tab in Atom, and saw the option "Fallback Renderer" with the note "Enable this if you're experiencing slowdowns in the built-in terminals." It was disabled by default, so I've just enabled it.

Subjectively, I think it feels fine now. Longer use will tell, but I suspect I was just running into a known issue some setups run into, and they already provided the workaround.

EDIT: Comparing running some code in Atom's terminal and a REPL running in the GNOME Terminal, the regular REPL still feels notably snappier -- even though I'm `using OhMyREPL`, which makes the REPL a bit less responsive.

I'd say Atom feels acceptable (and definitely not "mind boggling laggy" right now), and shift/ctrl + enter more convenient than switching tabs. So I will stick with it (for Julia). More time shall tell.

Re: Why Jupyter is data scientists’ computational notebook of choice

#257

Post by ubernostrum »

taeric wrote: post #81
makmanalp wrote: post #51

The majority of the complaints I hear about notebooks I think come from a misunderstanding of what they're supposed to be. It's a mashup between a scientific paper and a repl. So it's useful for a bit of both: a) Just like with a paper, you can present scientific or mathematical ideas with accompanying visualizations or simulations. From the REPL side, as a bonus, you get interactivity, and the reader can pause and e…

You parting "Granted..." is precisely what fills me with dread when I see notebooks. Yes, I have seen poorly done source files. I made more than a few myself. However, many of the practices we have grown into as sound programming advice seem to be largely thrown out the window for these notebooks. The irony, to me, is that I actually typically argue for the mixing of presentation and content. But to me, notebooks loo…

There is also the entire bend where these are taking off in a way that doesn't make sense.

It makes perfect sense. Just not to a lot of HN readers.

The average HN reader is approaching this from a perspective of "I am a professional programmer who might occasionally dabble in scientific computing, and therefore I hate this thing because it's not a professional programmer's tool designed by and for professional programmers according to the best practices of professional programmers".

The people who are actually using notebooks, meanwhile, are not professional programmers. They're scientists who increasingly have to do programming as part of their science. And notebooks are a godsend for them. We don't need to drag them all the way into our world; we need to pay attention to what they actually want, need, and find useful, and accept that it's going to differ from what we want, need, and find useful.

Re: Why Jupyter is data scientists’ computational notebook of choice

#258

Post by vhhn »

kortex wrote: post #241
jonnycomputer wrote: post #93

Earlier quoted context omitted.

If you ever have used an R Notebook written in R-Markdown, then its pretty easy to see why Jupyter Notebooks putting everything in JSON is just... infuriatingly wrong-headed. In an R Notebook, I can see my code, I can see my text, everything is exceedingly simple to understand, and I can edit it in any of the fantastic text editors out there (Jupyter's editor is not among them)

The main reason for json, I believe, is that the Jupyter client is separate from the backend. It's actually pretty trivial to run the engine on a beefy box while interacting on a light laptop (on the same subnet). With Jupyter Lab and some fiddling, you can put the server anywhere. It's also trivial to export notebooks to .py files. That said, my goodness do notebooks wreak havoc on git. I hope this in particular get…

RStudio has a server mode as well

Re: Why Jupyter is data scientists’ computational notebook of choice

#259

Post by gaius »

ubernostrum wrote: post #257
taeric wrote: post #81

Earlier quoted context omitted.

You parting "Granted..." is precisely what fills me with dread when I see notebooks. Yes, I have seen poorly done source files. I made more than a few myself. However, many of the practices we have grown into as sound programming advice seem to be largely thrown out the window for these notebooks. The irony, to me, is that I actually typically argue for the mixing of presentation and content. But to me, notebooks loo…

There is also the entire bend where these are taking off in a way that doesn't make sense. It makes perfect sense. Just not to a lot of HN readers. The average HN reader is approaching this from a perspective of "I am a professional programmer who might occasionally dabble in scientific computing, and therefore I hate this thing because it's not a professional programmer's tool designed by and for professional progra…

They're scientists who increasingly have to do programming as part of their science. And notebooks are a godsend for them

2/3 of scientific research cannot be reproduced by other scientists. But tell us more about why scientists should ignore best practices from other fields.

Re: Why Jupyter is data scientists’ computational notebook of choice

#260

Post by gaius »

makmanalp wrote: post #51

The majority of the complaints I hear about notebooks I think come from a misunderstanding of what they're supposed to be. It's a mashup between a scientific paper and a repl. So it's useful for a bit of both: a) Just like with a paper, you can present scientific or mathematical ideas with accompanying visualizations or simulations. From the REPL side, as a bonus, you get interactivity, and the reader can pause and e…

The majority of the complaints I hear about notebooks I think come from a misunderstanding of what they're supposed to be

No, the majority of complaints are that notebooks are great, but Jupyter is a bad notebook. I mean maybe it’s impressive to someone who’s never seen a notebook before but to someone used to Mathematica, MathCAD, RMarkdown, org-mode, whatever, it just seems clunky as hell. I wonder how many “data scientists” claiming it as their top choice have ever tried anything else?

Post reply on HN