Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nbdev: A literate programming environment that democratizes best practices

github.blog

Post reply on HN
Print view View original

51–60 of 88 posts

Re: Nbdev: A literate programming environment that democratizes best practices

#51

Post by contravariant »

konjin wrote: post #46
musingsole wrote: post #44

Earlier quoted context omitted.

No true scotsman. It's more literate programming than not and chasing the promise of the ideal literate programming environment is what created the novel notebook environment in the first place.

1). Notebooks were invented by Mathematica in the 80s. 2). Words have meanings and literate programming is defined extremely well by Knuth in his 1983 paper. This is not what was described there any more than the WWW is Xanadu. 3). That is not what No True Scotsman means.

> That is not what No True Scotsman means.

Setting aside whether you are or are not right, can we just appreciate the irony in that statement for a moment?

Re: Nbdev: A literate programming environment that democratizes best practices

#52

Post by justnotworthit »

jph00 wrote: post #23
ipsum2 wrote: post #14

Earlier quoted context omitted.

For some strange reason, everyone in the ML community refers to 'increasing adoption' as 'democratizing'. It's my pet peeve.

There are two lead definitions for "democratize" in the Oxford English Dictionary. One of them is: "make (something) accessible to everyone" So the usage here is entirely consistent with standard English usage. It is also consistent with the French etymology (démocratiser), which has as a dictionary definition "Rendre démocratique, populaire" (i.e. to make popular).

If I let everyone on my street borrow my bike, can I say "I've democratized my bike"?

Re: Nbdev: A literate programming environment that democratizes best practices

#53

Post by justnotworthit »

pmdulaney wrote: post #5

I love democracy, but somehow there is something cloying about the use of "democratize" in this context.

It suggests you've stolen fire from the gods (or the arcane halls of wizards/academics) and fed the starving masses.

Pretty good for an ide.

Re: Nbdev: A literate programming environment that democratizes best practices

#54

Post by nestorD »

I have used Nbdev and I am not a fan. It creates friction when one wants to contribute (in my case, to fast.ai) and forces you to write your code with notebooks which is the point but also not great when you are writing code rather than producing a display mixing text and pictures. Plus, while notebooks should favour documentation in theory, you can also end up with notebooks full of blob of code with transitions text that does not help you undertsanding what's going on.

Case in point, here is a random notebook from the fastai repository, a python file would be simpler to read and shorter: https://github.com/fastai/fastai/blob/master/nbs/09b_vision....

Re: Nbdev: A literate programming environment that democratizes best practices

#55

Post by tlarkworthy »

kowlo wrote: post #31
jph00 wrote: post #27

Earlier quoted context omitted.

I coined the term - and it turns out someone else did too, for something else. So be it. If someone else can think of a better term that's never been used before, then I'll happily use that instead. The earlier usage mentioned in Wikipedia is entirely uncited there however, and seems to have only been used in one academic project AFAICT.

How are you sure you didn't just read it somewhere and then forgot? It has been mentioned many times in related literature (even in publication titles) https://scholar.google.com/scholar?q=%22exploratory+programm... Here's one from 1988 https://dl.acm.org/doi/abs/10.1145/51607.51614 Are these unrelated? Is Nbdev not only a "new programming environment", but also a new concept that needs a new name?

"In some cases the estimates may be obvious. Perhaps the story is similar to others that have already been completed. In other cases the story may be very difficult to estimate and may require exploratory programming."

Kent Beck and Martin Fowler

http://index-of.es/Java/Planning%20Extreme%20Programming.pdf

Its a commonly understood term AFAIK

Re: Nbdev: A literate programming environment that democratizes best practices

#56

Post by zomglings »

nbdev looks promising. I'm wondering if it solves what I see as the biggest pain of using notebooks when I'm doing data science work.

When a notebook gets large, it can be difficult to keep track of dependencies between cells. For workflows in which you have to run cells n_1, n_2, ..., n_k before running cell n.

I try to organize my cells so that if I run them from first to last, all dependencies are covered (e.g. "Restart kernel and run all cells).

Unfortunately, this doesn't help when I discover a bug in cell n_2 and don't want to run ALL cells n_2 + 1, ..., n-1, n because some of them carry out expensive operations.

When working in my editor, the way I resolve this is to make a light CLI wrapper around my program (if __name__ == "__main__": import argparse; ...) and my CLI commands encode all this dependency information.

Is it possible to get this kind of experience in a Jupyter notebook without building a custom plugin (I think a frontend plugin would suffice)?

Re: Nbdev: A literate programming environment that democratizes best practices

#57

Post by hamelsmu »

erikgaas wrote: post #48

So I use this in production at my company. It's an awesome tool. Personally when I'm coding in python I like to prototype in jupyter, copy code over, and then reimport anyway. Nbdev streamlines everything so I can write docs, tests, and code all in one place. And since the docs are just a jekyll site I can copy it to our documentation aws bucket in continuous integration. And with one command I can run all the notebo…

Really interesting! Do you mind sharing what your company is? (I am the author of the blog post)

Re: Nbdev: A literate programming environment that democratizes best practices

#59

Post by kowlo »

tlarkworthy wrote: post #55
kowlo wrote: post #31

Earlier quoted context omitted.

How are you sure you didn't just read it somewhere and then forgot? It has been mentioned many times in related literature (even in publication titles) https://scholar.google.com/scholar?q=%22exploratory+programm... Here's one from 1988 https://dl.acm.org/doi/abs/10.1145/51607.51614 Are these unrelated? Is Nbdev not only a "new programming environment", but also a new concept that needs a new name?

"In some cases the estimates may be obvious. Perhaps the story is similar to others that have already been completed. In other cases the story may be very difficult to estimate and may require exploratory programming." Kent Beck and Martin Fowler http://index-of.es/Java/Planning%20Extreme%20Programming.pdf Its a commonly understood term AFAIK

> Its a commonly understood term AFAIK

I agree - it's what made me raise an eyebrow... However, based on their comment above, the lead author of Nbdev believes to have coined the term.

My opinion is that due diligence and attribution are important. If I believed I'd coined a new term, I'd check first. Mistakes are easy to make, but when highlighted, perhaps corrections are more appropriate than negotiating with the person highlighting them:

From the lead author (jph00): ... If someone else can think of a better term that's never been used before, then I'll happily use that instead.

Re: Nbdev: A literate programming environment that democratizes best practices

#60

Post by Jugurtha »

zomglings wrote: post #56

nbdev looks promising. I'm wondering if it solves what I see as the biggest pain of using notebooks when I'm doing data science work. When a notebook gets large, it can be difficult to keep track of dependencies between cells. For workflows in which you have to run cells n_1, n_2, ..., n_k before running cell n. I try to organize my cells so that if I run them from first to last, all dependencies are covered (e.g. "R…

>Unfortunately, this doesn't help when I discover a bug in cell n_2 and don't want to run ALL cells n_2 + 1, ..., n-1, n because some of them carry out expensive operations.

Running all cells has its disadvantages, but what I found out was that often, there are bugs elsewhere than in cell n_2. I treat that as unit tests: run them all because even though I think only this is breaking, fixing it could have broken some other part.

Many other "Jupyter killers", as sensationalist blog post titles call them, claim they have done away with Jupyter's dirty "hidden state", but reading further about how the "Jupyter killer" avoids re-doing heavy computation by "caching" compute-intensive results and you want to tell them "get your mind right, which is it?".

The way we do it is schedule the notebook[0] to make sure everything works.

- [0]: https://iko.ai/docs/notebook/#long-running-notebooks

Post reply on HN