Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Why Jupyter is data scientists’ computational notebook of choice

nature.com

Post reply on HN
Print view View original

241–250 of 308 posts

Re: Why Jupyter is data scientists’ computational notebook of choice

#241

Post by kortex »

jonnycomputer wrote: post #93
makmanalp wrote: post #51

The majority of the complaints I hear about notebooks I think come from a misunderstanding of what they're supposed to be. It's a mashup between a scientific paper and a repl. So it's useful for a bit of both: a) Just like with a paper, you can present scientific or mathematical ideas with accompanying visualizations or simulations. From the REPL side, as a bonus, you get interactivity, and the reader can pause and e…

If you ever have used an R Notebook written in R-Markdown, then its pretty easy to see why Jupyter Notebooks putting everything in JSON is just... infuriatingly wrong-headed. In an R Notebook, I can see my code, I can see my text, everything is exceedingly simple to understand, and I can edit it in any of the fantastic text editors out there (Jupyter's editor is not among them)

The main reason for json, I believe, is that the Jupyter client is separate from the backend. It's actually pretty trivial to run the engine on a beefy box while interacting on a light laptop (on the same subnet). With Jupyter Lab and some fiddling, you can put the server anywhere.

It's also trivial to export notebooks to .py files.

That said, my goodness do notebooks wreak havoc on git. I hope this in particular gets fixed as popularity grows.

Re: Why Jupyter is data scientists’ computational notebook of choice

#242

Post by kortex »

taeric wrote: post #220
joshuamorton wrote: post #165

Earlier quoted context omitted.

As a result of the serialize-to-json approach, jupyter supports R, python, scala, go, lua, bash, julia, and haskell, among others. Its accessible to a much wider range of programmers, at the cost of version control being a bit weirder.

That is a complete non-sequitur. Json in no way enables that. Just having a defined format enables that. Emacs org-mode is proof that a simple text format with markup rules is all you really need to support multiple languages in a single file. You lose some of the simplicity of parsing the file, but you gain a ton more.

Json enables you to chuck the notebook to any browser client anywhere on the net.

Org-mode is great, but you still have to install emacs.

Re: Why Jupyter is data scientists’ computational notebook of choice

#243

Post by rtpg »

taeric wrote: post #220
joshuamorton wrote: post #165

Earlier quoted context omitted.

As a result of the serialize-to-json approach, jupyter supports R, python, scala, go, lua, bash, julia, and haskell, among others. Its accessible to a much wider range of programmers, at the cost of version control being a bit weirder.

That is a complete non-sequitur. Json in no way enables that. Just having a defined format enables that. Emacs org-mode is proof that a simple text format with markup rules is all you really need to support multiple languages in a single file. You lose some of the simplicity of parsing the file, but you gain a ton more.

This isn't true though, right? If you were writing an org mode document about org mode, you now need an escaping mechanism to not mix your structure and text

Multi-language parsing is much harder to solve than simply enforcing some escaping mechanism in the inner protocol level and having tools do the "heavy" lifting (basically a solved problem).

Re: Why Jupyter is data scientists’ computational notebook of choice

#244

Post by taeric »

rtpg wrote: post #243
taeric wrote: post #220

Earlier quoted context omitted.

That is a complete non-sequitur. Json in no way enables that. Just having a defined format enables that. Emacs org-mode is proof that a simple text format with markup rules is all you really need to support multiple languages in a single file. You lose some of the simplicity of parsing the file, but you gain a ton more.

This isn't true though, right? If you were writing an org mode document about org mode, you now need an escaping mechanism to not mix your structure and text Multi-language parsing is much harder to solve than simply enforcing some escaping mechanism in the inner protocol level and having tools do the "heavy" lifting (basically a solved problem).

Amusingly, no. Folks have done just that.

That said, you can define away a large part of the problem.

Edit: For trivial examples of "org-mode" in an org-mode document, you need only look at the documentation of org-mode. That said, I expect there to be limitations, because they make sense. Similar to how you can pretty print json inside a jupyter notebook, but don't expect to have a notebook interpreted in the notebook. (If that makes sense.)

Re: Why Jupyter is data scientists’ computational notebook of choice

#245

Post by taeric »

kortex wrote: post #242
taeric wrote: post #220

Earlier quoted context omitted.

That is a complete non-sequitur. Json in no way enables that. Just having a defined format enables that. Emacs org-mode is proof that a simple text format with markup rules is all you really need to support multiple languages in a single file. You lose some of the simplicity of parsing the file, but you gain a ton more.

Json enables you to chuck the notebook to any browser client anywhere on the net. Org-mode is great, but you still have to install emacs.

Emacs is, amusingly, a lighter client than most browsers nowadays.

On point, a browser can not remember ifa notebook. Just parse the json. It can also parse text/plain. So, could show the org document without styling. The org document is actually readable. Json... Not so much.

To see the notebook, you have to have a Jupyter setup somewhere.

Edit: For example, see https://raw.githubusercontent.com/taeric/taeric.github.io/ma... which is the source for http://taeric.github.io/ChangeForDollar.html Not styled, and that is a short document, so probably woudn't be that tough to read in a json document, but I'm glad I don't have to.

Re: Why Jupyter is data scientists’ computational notebook of choice

#246

Post by ianamartin »

The problem with Jupyter isn't with what it does. It's the people who use it.

My experience as a data engineer/architect/application developer attached to data science teams for a while now is that most really good data scientists are very good at what they do, write somewhat competent code, and do not--in any way--care about writing good software or good application code.

Jupyter is a bane of my existence because people who use it want to use it for everything. Oh, it can have a web interface? Okay. The app is done. DEPLOY TO WEB USERS! NOW!!

It's a great tool. A lot of the people who use it are not software engineers, and they don't want to be. For a lot of people it's the straight line from point a to point b.

But in my experience, legit data scientists are pretty smart and are willing to learn a little if you're willing to give a little. This is a good exercise because they are typically skeptical about everything. So you have to be really secure about why you want certain things done certain ways, and why you definitely don't want things done other ways.

It's a good exercise for everyone involved if you have the right team dynamic and mutual, healthy respect for each other.

If you don't . . . well, then Jupyter notebooks completely suck.

Re: Why Jupyter is data scientists’ computational notebook of choice

#247

Post by huac »

jononor wrote: post #222
huac wrote: post #181

Earlier quoted context omitted.

yeah, i have a use case now where we pull data from a database, manipulate it, and then have a final table/csv/dataframe/whatever. the problem is then how to share this with non-technical users. in an ideal world, this would get inserted into a google sheet, and that sheet would just update daily after new data is loaded into the database. i'm pretty sure this is a usecase which others have and curious what people us…

Google Sheets has an API. So if putting the output there is the ideal, just use that API from Python?

yeah, issue is that if you're inputting more than ~500 rows, you'll be rate limited :|

Re: Why Jupyter is data scientists’ computational notebook of choice

#248

Post by awake »

pavanagrawal123 wrote: post #161
kylebarron wrote: post #97

Earlier quoted context omitted.

To me, my favorite part of the design of Hydrogen is that it's entirely language agnostic, and can be used with _any_ Jupyter kernel.

I'm planning on adding better language support in a couple of weeks! Don't want to limited to Python and R

One thing you may want to be aware of is that the python language server used for vscode runs pylint which performs static analysis on the code. However, Jupyter Notebook uses autocomplete by actually introspecting the variables as they are defined. This creates large differences when doing things such as selecting a column in a pandas dataframe. In jupyter if you press tab on the column name, it can autocomplete and also assumes you are getting a series, which leads to autocomplete on things like .min, .max, etc... In pylint you don't get any of this autocomplete since pylint cannot statically determine the column names so you lose the intellisense.

Re: Why Jupyter is data scientists’ computational notebook of choice

#249

Post by qwerty456127 »

makmanalp wrote: post #51

The majority of the complaints I hear about notebooks I think come from a misunderstanding of what they're supposed to be. It's a mashup between a scientific paper and a repl. So it's useful for a bit of both: a) Just like with a paper, you can present scientific or mathematical ideas with accompanying visualizations or simulations. From the REPL side, as a bonus, you get interactivity, and the reader can pause and e…

Is it actually a common skill to write meaningful non-helloworldish Python code that yields expected results without a number of iterations of debugging and correcting and without PyCharm intelligent completion, hinting and correcting features? I understand the value of Jupyter notebooks for publishing your work results but find it almost impossible to use it to actually do the work - it feels million times more convenient to code in PyCharm then copy-paste the code to Jupyter once it's ready.

Re: Why Jupyter is data scientists’ computational notebook of choice

#250

Post by throwaway487548 »

Jupyter could be viewed as a modern reminiscence of Lisp Machine UI, without the elegance of homoiconicity, of course.

Python is a good "glue" for optimized C++ or fortran libraries which the core of things like tensorflow or numpy. Everything fits together nicely.

Post reply on HN