Live data from Hacker News

Noweb – A Simple, Extensible Tool for Literate Programming

cs.tufts.edu

11–20 of 54 posts

Re: Noweb – A Simple, Extensible Tool for Literate Programming

#11
post #6

Hijacking this topic to talk about something I've been thinking about lately: literate diffs. I find that the order of diffs given by git is not optimized for helping a reviewer understand the change. Sometimes the order of files will not be in the most logical way; sometimes unrelated changes (e.g., a text editor removing blanks at the end of lines) create noise; etc. I've been thinking that it would be interesting…

Take a look at the term "Semantic Source Diff", eg https://martinfowler.com/bliki/SemanticDiff.html Tools in this space date back to the 1990s. There is a recent upsurge of interest, a number of capable tools for different languages are currently available.

Pijul gives me hope semantic diffs may become common - see the "Dependencies" paragraph of https://pijul.org/posts/2020-11-07-towards-1.0/. The HN comments on that - https://news.ycombinator.com/item?id=25032956 - are a nice read as well.

Re: Noweb – A Simple, Extensible Tool for Literate Programming

#12
post #5

I am working on a tool, which can take a collection of MarkDown files with fragments of C code, and can combine these into a single C file, where all fragments are placed in an order, such that they can be compiled. Because defines can change the meaning of code depending where you place them, there are some restrictions on the input files. An example of the type of input file I have in mind, is given at https://gith…

Check out lit.sh, it's a super simple shell script that does what you're after: https://github.com/vijithassar/lit It's basically a one way markdown to commented source converter. One really slick thing is that since it just comments out the markdown you lose no information about line numbers and can interpret errors from your compiler, etc. with ease (something which is almost impossible with more complex literate programming systems like noweb).

Re: Noweb – A Simple, Extensible Tool for Literate Programming

#13
post #6

Hijacking this topic to talk about something I've been thinking about lately: literate diffs. I find that the order of diffs given by git is not optimized for helping a reviewer understand the change. Sometimes the order of files will not be in the most logical way; sometimes unrelated changes (e.g., a text editor removing blanks at the end of lines) create noise; etc. I've been thinking that it would be interesting…

What you’re describing is already possible with Git: rebase and committing chunk/lines allows you to organize your changes coherently. The trick is finding ways to get into the habit of doing it that way and staying consistent with the whole team.

Edit: i’m not saying that this is a solved problem. I think the parent’s point is valid. I am just saying that there are some tools that make this possible and I agree that there is a definite need for improvements in this area.

Re: Noweb – A Simple, Extensible Tool for Literate Programming

#14
I used to be a serious advocate of LP. I spent a lot of time in Pascal Web and later CWeb. I really appreciated how the latter emitted #line directives in its output so that compiler error messages (all too common in those pre/proto-IDE days) and debugger traces would refer to the original CWeb source code. I also tended to have a single cweb file generate both the .c and .h files for a program which unfortunately meant that I would end up recompiling code that depended on the .h even if there was no change to the .h file. Something similar could be useful now, but a lot of the tooling around LP was geared towards printing out source code to refer to rather than working with it on-screen (which made sense in those days of 80x24 text-only displays). The Pascal Web Changefile mechanism was pure brilliance. It was by far superior to the standard C practice of using preprocessor directives to manage compilation for different targets.

Re: Noweb – A Simple, Extensible Tool for Literate Programming

#16
post #6

Hijacking this topic to talk about something I've been thinking about lately: literate diffs. I find that the order of diffs given by git is not optimized for helping a reviewer understand the change. Sometimes the order of files will not be in the most logical way; sometimes unrelated changes (e.g., a text editor removing blanks at the end of lines) create noise; etc. I've been thinking that it would be interesting…

I would love if my VCS tool could also keep track of things like "This diff is the result of running sed -i s/this/that/g .py". I usually split out such mechanical changes anyway into a separate commit, but it would be clearer for reviewers to see that (most review tools show you the overall diff of the entire branch you want to merge by default, making you click further to see patch-by-patch changes), and it would also be easier for me* if the VCS could re-run the sed command when I rebased.

(An obvious next step is Coccinelle-style semantic patches, but let's start with sed!)

Re: Noweb – A Simple, Extensible Tool for Literate Programming

#17
post #6

Hijacking this topic to talk about something I've been thinking about lately: literate diffs. I find that the order of diffs given by git is not optimized for helping a reviewer understand the change. Sometimes the order of files will not be in the most logical way; sometimes unrelated changes (e.g., a text editor removing blanks at the end of lines) create noise; etc. I've been thinking that it would be interesting…

I believe most literate programming tools are language-agnostic, so you could probably do that with this tool!

Re: Noweb – A Simple, Extensible Tool for Literate Programming

#18

Literate Programming is one of those ideas I keep coming to. There is an idea there that touches on something I find to be true about software development, namely that the communication of an idea to other humans is the most critical piece. There is a similar idea in Naur's paper "Programming as Theory Building." That said, I've never loved the LaTeX-centric nature of most tools. I don't like heavier markup systems w…

I am less convinced now after an initial period of enthusiasm. I can get around things locally fairly well with or without assistance. What takes effort to comprehend a code base is its overarching organization and internal interfaces that often only exist in the heads of the creators.

But if one is into literate programming it is definitely a must to check out the Leo Editor http://leoeditor.com

Re: Noweb – A Simple, Extensible Tool for Literate Programming

#19
(BTW, Norman's server seems to be suffering under the load; https://web.archive.org/web/20210223015500/https://www.cs.tu... has your (Way)Back if you're suffering problems in accessing it.)

I've been interested in literate programming for a long time; for my self-bootstrapping PEG parser https://github.com/kragen/peg-bootstrap/blob/master/peg.md I wrote my own noweb-like system called HandAxeWeb in Lua (5.x) https://github.com/kragen/peg-bootstrap/blob/master/handaxew.... It accepts input in Markdown, HTML, ReStructuredText, etc., and it's only a couple hundred lines of Lua.

For HandAxeWeb (named following the convention of StoneKnifeForth—the intent was to make it simple enough that even fairly early stages of a bootstrap could be literate programs), I wanted to be able to include multiple versions of a program in the same document, because I think it's often helpful to see the temporal development of a program from a simpler version to a more complex version. The simpler version is easier to understand and helps you focus on the most fundamental aspects of the program. https://www.youtube.com/watch?v=KoWqdEACyLI is a 5'30" screencast (not by me) of explaining the development of a Pong program in this fashion. I think it's usually easier to understand a program in this fashion than by trying to understand the final complete version bit by bit, the way something like "TeX: The Program" forces you to do.

Still, generally speaking, soi-disant literate programming tools—including my own—generally fail to take advantage of the most compelling aspect of the computer as a communication medium: its ability to simulate. When I dive into a new code base, it's never entirely by reading it—whether top-down, bottom-up, or in any other order. The cross-reference links added by things like CWEB (or, you know, ctags) are helpful, of course, but invariably I want to see the output of the program, which CWEB doesn't support at all! (Although Knuth's TeX: The Program does manage to include TeX output despite being written in CWEB, that's in a sense sort of a coincidence; this is not a feature CWEB can provide for any programs other than TeX and METAFONT.)

Books like Algorithms, by Knuth's student Sedgewick, are full of graphical representations of the outputs of the algorithms being discussed, and this is enormously helpful—perhaps even more so than the source code; see https://algs4.cs.princeton.edu/22mergesort/ for some examples from the current version of the book, which is lamentably in Java. It's better still, though, when you can edit the code and see the results—when diving into a new code base, I tend to execute modified versions of the code a lot, whether in the debugger or with extra logging or what. Paper books can't do this, but that's no excuse for not doing it when we're writing for readers who have computers.

Philip Guo's Python Tutor http://pythontutor.com/ provides dynamic visualization of the memory contents of, in theory, arbitrary code (in the supported languages, including of course Python, but also C, C++, Java, JS, and Ruby). There are things you can display with animation that you can't display in a static page, but Algorithms gets quite far with static printed images, and I think static visualization is better when you can make it work, for reasons explored at length in Bret Victor's http://worrydream.com/MagicInk/.

Python Tutor doesn't scale to large programs, but Dorothea Lütkehaus and Andreas Zeller's DDD can control GDB to create such visualizations for anything you can run under GDB (or JDB, Ladebug, pydb, or perl -d). Unfortunately there's no way to share the output of either DDD or Python Tutor, except maybe a screencast, and despite having been around since 01995, DDD has never been popular, I suspect because its Motif UI is clumsy to use. https://edoras.sdsu.edu/doc/ddd/article20.html shows what it looked like in 02002 and https://youtu.be/cKQ1qdo79As?t=106 shows what it looked like in 02015.

Of course, spreadsheets are by far the most popular programming environment, and they have always displayed the program's output when you open it—even to the exclusion of the code, mostly. I've experimented with this sort of thing in the past, with things like http://canonical.org/~kragen/sw/bwt an interactive visualization of the Burrows–Wheeler transform, and so it's been heartening to see modern software development moving in this direction.

The simplest version of this is things like Python's doctest, where you manually paste textual snippets of output in the code itself, and a testing tool automatically verifies that they're still up-to-date; Darius Bacon's Halp https://github.com/darius/halp is a more advanced version of this, where the example output updates automatically, so you can make changes to the program and see how they affect the results.

The most polished versions of this approach seem to have adopted the name "explorable explanations", and many of the best examples are Amit Patel's, which he has at different times termed "interactive illustrations" https://simblob.blogspot.com/2007/07/interactive-illustratio..., "active essays" (I think? Maybe I'm misremembering and that term was current in Squeak around 02003: http://wiki.squeak.org/squeak/1125), and "interactive visual explanations". I wrote a previous comment about this on here in 02019: https://news.ycombinator.com/item?id=20954056.

However, Amit's explorables, like many other versions of the genre, de-emphasize the underlying code to the point where they both don't display the actual code and don't let you edit it. They're intended to visualize an algorithm, not a codebase.

Mike Bostock, d3.js's original author, has created https://bl.ocks.org/ for sharing explorable explanations made with d3, and is doing a startup called ObservableHQ which makes things like this a lot easier to build: https://beta.observablehq.com/d/e639659056145e88 but at the expense of a certain amount of polish and presentational freedom. Also, unfortunately, ObservableHQ programs seem to be tied to the company's website—you can download their output, but very much unlike TeX, the programs will only be runnable until the company goes out of business. So if you aspire to make a lasting contribution to human intellectual heritage, like TeX, GCC, or d3.js itself, ObservableHQ is not for you.

R Markdown (by JJ Allaire—yes, the Cold Fusion dude—and Yihui Xie, among others) is one of the more interesting developments here; as with noweb or HandAxeWeb, you edit something very close to the "woven" version of the source code (in a dialect of Markdown); but, in a separate file alongside, RStudio maintains the results of executing the code, which are included in the "woven" output, and may be textual or graphical. Moreover, as with Halp or ObservableHQ, these results are displayed in a notebook-style interface as you're editing the code. https://bookdown.org/yihui/rmarkdown/notebook.html has a variety of examples, and Xie is rightly focused on reproducibility, which is very challenging to achieve with the existing tooling. https://bookdown.org/ lists a number of books that have been written with R Markdown, and https://github.com/rstudio/rmarkdown explains the overall project.

Of course the much more common notebook-style interface, and the one that popularized the interaction style, is Jupyter (influenced by SageMath), which mixes input and output indiscriminately in the same file and peremptorily makes backward-incompatible changes in file formats; the result is a lot of friction with version-control systems. Nevertheless, it supports inline LaTeX, it's easy to use and compatible with a huge variety of existing software, and it can include publication-quality visualizations, so there's a lot of code out there in Jupyter notebooks now, far more than in any system that purports to be a "literate programming" system. Notable examples include Peter Norvig's œuvre (there's a list at https://github.com/norvig/pytudes#pytudes-index-of-jupyter-i...). I find this a very comfortable and powerful medium for this form of literate programming; recent examples include https://nbviewer.jupyter.org/url/canonical.org/~kragen/sw/de..., https://nbviewer.jupyter.org/url/canonical.org/~kragen/sw/de..., and https://nbviewer.jupyter.org/url/canonical.org/~kragen/sw/de..., which are maybe sort of embarrassingly bad but I think demonstrate the potential of the medium for vernacular expression of vulgar software, as well as lofty Norvig-type things.

Konrad Hinsen has written about the reproducibility and lock-in problems introduced by Jupyter, for example in https://khinsen.wordpress.com/2015/09/03/beyond-jupyter-what..., and has been using Tudor Gîrba's Glamorous Toolkit https://gtoolkit.com/ to explore what comes next. He's been hitting the reproducibility problem pretty hard in http://www.activepapers.org/ but the primary intent there is, as with the explorable-explanations stuff, code as a means to producing research ("How should we package and publish the outcomes of computer-aided research"), rather than maintainability and understandability of code itself. I think this is a promising direction for literate programming as such, too.

Post reply on HN