Live data from Hacker News

The one about Lisp interactivity

blog.fogus.me

21–30 of 51 posts

Re: The one about Lisp interactivity

#21
post #4

The thing I still don't understand about REPL driven development is how you manage state, and how you manage threads. If I have a task running and it depends on a collection of global variables, those global variables now need some machinery around them so that they can be edited from the REPL thread. If you replace a function, there now needs to be decisions made about when you now begin usage of the new function. T…

> The state of the in-memory image had gotten totally out of sync with the codebase.

I have not used it, but I heard that Clojure Clerk is supposed to help with this.

https://github.com/nextjournal/clerk

Re: The one about Lisp interactivity

#22

Smalltalk only got a mention in the footnotes but I think it deserves a bigger entry when talking about ways to interact with programs. If a REPL is talking to and conversing with a program then environments like Pharo take it a step further by letting you interactively and graphically look under the hood as well. It's an amazing way to interact with software once you get used to it and I think well worth checking ou…

And we can even talk about back in time debugging such as https://github.com/hpi-swa-lab/squeak-tracedebugger

Re: The one about Lisp interactivity

#23

In the opening paragraphs, this post indicates that this other post of David Vujuc [1] falls victim to common misconceptions about what a REPL is. I'm not sure what misconceptions this post is referring to; perhaps I also have these misconceptions. But I also don't think this post clarifies that. Could anyone here make it more explicit? [1]: https://davidvujic.blogspot.com/2022/08/joyful-python-with-r...

The Vujic post describes repl-driven development as "[evaluating] variables, code-blocks, functions, or an entire module [to] get instant feedback, just by hitting a key combination in your favorite code editor."

Speaking from the perspective of long experience with Lisp and Smalltalk environments, I agree with fogus here: this is a misconcpetion--or at least an impoverished version of repl-driven development. The existence of a repl does not constitute the repl-driven programming that fogus is talking about, nor that I was talking about in the blog post he references.

In its full form, repl-driven development means communicating directly with the live dynamic environment of your running program, which contains, in addition to the code you're developing, systematic support for inspecting, controlling, and modifying all of its code by directly interacting with it, and without needing to stop and restart the program in order to do that.

Clojure can do some of that, but not all of it. For example, unlike any implementation of Clojure I'm aware of, both Smalltalk and Common Lisp implementations support handling an error or other exception by starting up a nested repl within the dynamic context of the error so that you can inspect the live stack frames that are pending in the context of the error, modify any variables or functions or methods that are pending, and restart the computation at the frame of your choice.

Both Smalltalk and Common Lisp implementations support handling an undefined type or method by defining it interactively while the program waits, suspended in the error that alerted you to the missing definition, and then resuming execution after you've supplied the missing definition.

Both Smalltalk and Common Lisp implementations support redefining classes that have live instances while the program runs, they automatically catch references to those instances when they're referenced, and they automatically update them to reflect the new definitions (dropping you into a nested repl to specify how to do that, if that's needed).

No Clojure implementation I know of provides these features. Moreover, it's not just these specific features that are missing from Clojure and implementations of the other languages that have been mentioned here; also missing is the fundamental design orientation reflected in Common Lisp and its ancestral Lisps, and in Smalltalk: they were designed with the tacit assumption that the normal way to write a program was to start the runtime going and then change it bit-by-bit into the program you want by telling it interactively, feature-by-feature, how to be that program.

I and other people have made this point over and over for the past couple of years--and that's fine. I think the fact that it needs to be said over and over simply illustrates the misconception that fogus refers to: folks who have worked with repls have the notion that having a repl means that you're doing repl-driven programming. It doesn't--at least not in the sense that fogus is talking about, or that I'm talking about.

The unfortunate thing is that if you think that's all there is to repl-driven programming, there's a whole other layer of affordances that you're missing.

I'll briefly address two auxiliary points, because they always seem to come up.

First, I do not claim that repl-driven programming is objectively better than any other kind. If the affordances I'm talking about don't interest you, if you're happy without them, more power to you. All I care about is that I personally prefer them, and I want them to continue to exist and be further developed so that I and others who prefer them will continue to have them available. I think that making more people aware of those affordances increases the chances of that happening.

Second, someone will think that "repl-drive programming" means doing all your coding at a repl prompt. It doesn't mean that. It means writing your program by communicating with a read-eval-print loop--a repl--to tell the runtime how to become the program you want. The repl prompt isn't the repl; it's just one particular UI for the repl.

I work mainly in Common Lisp, and rarely type expressions at a repl prompt. Some influential repl-driven systems, such as Smalltalk-80 and Interlisp-D, may not even show you a prompt unless you specifically ask for it.

Repl-driven programming means talking to your running program while it runs, telling it how to change itself into the program you want. How you talk to the repl is a separate matter.

In a Smalltalk image, it usually means using the System Browser and related tools to find the classes and methods you want to modify and telling them to change. The Smalltalk image automatically saves those changes in the image itself, in the Changes file, and in the Sources file. Nowadays it probably also saves them to a git or other VCS repo.

In a Common Lisp environment, it usually means writing expressions in a source file and tapping a keystroke to send the the change expression, or its whole context, or the whole file, or all changed files, to the Lisp for compilation and loading into the running program. I and everyone I've worked with for years has kept those source files in a version-control system, just like any other code.

With respect to Clojure specifically, the subset of repl-driven programming features that it provides are good as far as they go. They aren't the whole enchilada, though, and when I work with Clojure I always miss the Common Lisp and Smalltalk features that are missing.

Re: The one about Lisp interactivity

#24

In the opening paragraphs, this post indicates that this other post of David Vujuc [1] falls victim to common misconceptions about what a REPL is. I'm not sure what misconceptions this post is referring to; perhaps I also have these misconceptions. But I also don't think this post clarifies that. Could anyone here make it more explicit? [1]: https://davidvujic.blogspot.com/2022/08/joyful-python-with-r...

The Vujic post describes repl-driven development as "[evaluating] variables, code-blocks, functions, or an entire module [to] get instant feedback, just by hitting a key combination in your favorite code editor." Speaking from the perspective of long experience with Lisp and Smalltalk environments, I agree with fogus here: this is a misconcpetion--or at least an impoverished version of repl-driven development. The ex…

Typically an advanced UI where one types into a REPL is called a 'Listener' in Lisp. Examples for Listeners: the MCL Listener, Genera's Listener, LispWorks Listener, the SLIME Listener and others.

For an impression of a Genera Listener I would recommend to see Kalman Reti's Youtube video: https://www.youtube.com/watch?v=o4-YnLpLgtk He shows there the Lisp Machine Listener debugging/interacting with mixed Lisp and C code.

MCL and LispWorks IDEs Listeners are running in an integrated editor. The running programming runs inside the development environment.

SLIME's Listener uses an external editor (GNU Emacs) for the Listener.

Genera has an integrated application as a Listener and that one is not based on an editor.

That's also a significant difference if the Listener is an internal tool, compared to an externally attached tool. External: from a user point of view, I use an IDE and connect to a running Lisp. Internal: I use the IDE and spawn a new Listener window (which could be on another X11 screen in case of an X11-based GUI). Usually the integration with internal Listeners is higher, but they may be more fragile, since they share the process & UI with the running program.

Using an editor as a base substrate has some advantages: one has usually better editing support in the Listener. But as Genera shows, a Listener does not need to run on top of an editor to be powerful. The Genera listener has for example full output recording, each listener is also a drawing plane and remembers all output and associates it with the displayed Lisp objects. That makes the interaction with code and data extremely convenient, a feature which is not provided by evaluating code from an editor buffer. SLIME provides a similar feature, but in a very limited way. The richer the Listener UI, the more of the interaction of the user will be in the Listener. Thus often an exclusive use of the editor to evaluate code is either a sign of a powerful editor integration or a weak Listener implementation. In Genera the Lisp listener is also not only a powerful data explorer, but also a shell with a lot of commands for exploring the Lisp system. A portable and in some ways slightly less polished / extensive version is the McCLIM listener. Example: https://mcclim.common-lisp.dev/static/media/screenshots/bund...

Also a Lisp might provide Listeners as panes of application frames. Thus an application window (either a tool of the IDE or any application GUI window) includes a corresponding Listener as a pane. As a simple example I can open a LispWorks Inspector and add a Listener pane. Any result from evaluation in the Listener will be displayed in a Inspector, with history.

Re: The one about Lisp interactivity

#26

Smalltalk only got a mention in the footnotes but I think it deserves a bigger entry when talking about ways to interact with programs. If a REPL is talking to and conversing with a program then environments like Pharo take it a step further by letting you interactively and graphically look under the hood as well. It's an amazing way to interact with software once you get used to it and I think well worth checking ou…

Similar to a Lisp REPL / Listener is in Smalltalk the Workspace, in Pharo the Playgound.

Re: The one about Lisp interactivity

#27

Earlier quoted context omitted.

> Only fools would write that in the REPL, never commit it to a source file, and be surprised when they couldn't reproduce the system state later on. This happened a few times while I was developing https://laarc.io . It’s so addictive to just paste new functions into a repl and see the changes instantly that it’s easy to go a few hours without committing and realize your changes rely on a now-deleted function. :) Wo…

> It’s so addictive to just paste new functions into a repl and see the changes instantly that it’s easy to go a few hours without committing and realize your changes rely on a now-deleted function. :) I think Slime is a pretty happy middle ground. Instead of just pasting, the function gets written in a source file and then C-c C-c to evaluate it in the repl. I did once or twice run into similar issues though from de…

Doing a full reload of the system once in a while (especially after large changes) also helps.

Just because you can keep a REPL open for days doesn't mean that's always the best idea.

Re: The one about Lisp interactivity

#28
post #27

Earlier quoted context omitted.

> It’s so addictive to just paste new functions into a repl and see the changes instantly that it’s easy to go a few hours without committing and realize your changes rely on a now-deleted function. :) I think Slime is a pretty happy middle ground. Instead of just pasting, the function gets written in a source file and then C-c C-c to evaluate it in the repl. I did once or twice run into similar issues though from de…

Doing a full reload of the system once in a while (especially after large changes) also helps. Just because you can keep a REPL open for days doesn't mean that's always the best idea.

Another way to deal with that is patching the running Lisp and "undefine" no longer needed functions, variables, classes, ...

Re: The one about Lisp interactivity

#29
post #4

The thing I still don't understand about REPL driven development is how you manage state, and how you manage threads. If I have a task running and it depends on a collection of global variables, those global variables now need some machinery around them so that they can be edited from the REPL thread. If you replace a function, there now needs to be decisions made about when you now begin usage of the new function. T…

You’re right. I think the other commenters aren’t being straightforward with you. It is a race condition to load a file one function at a time, because any other thread can preempt you. Arc has a particularly elegant solution to this. Any code you want to happen atomically, you wrap in (atomic …) So (atomic x y z) Will do x, y, then z. During this time, no other threads are allowed to run. Therefore your load-file fu…

But if your tail recursion gets compiled down into the standard "label whatever, code blablabla, set address register to label whatever, jump" type-code, how do you change the address that the register gets set to?

Unless you do a lookup in the environment every time you recur to find where you have to jump to, but that would kill your performance stone dead.

Re: The one about Lisp interactivity

#30
post #29

Earlier quoted context omitted.

You’re right. I think the other commenters aren’t being straightforward with you. It is a race condition to load a file one function at a time, because any other thread can preempt you. Arc has a particularly elegant solution to this. Any code you want to happen atomically, you wrap in (atomic …) So (atomic x y z) Will do x, y, then z. During this time, no other threads are allowed to run. Therefore your load-file fu…

But if your tail recursion gets compiled down into the standard "label whatever, code blablabla, set address register to label whatever, jump" type-code, how do you change the address that the register gets set to? Unless you do a lookup in the environment every time you recur to find where you have to jump to, but that would kill your performance stone dead.

I suspect (but cannot prove) that environment lookups are cached, so that when the new definition is loaded with the same name as the previous jump target, the compiled code is invalidated and switches back to interpreted mode. But now that I talk through it, that sounds pretty hard.

Occasionally I dive into racket’s good old C code for masochistic pleasure (it’s good code, just … very C, and very old). I always come away with a feeling of “well, that’s an interesting puzzle… I wonder what will happen when X happens” for every X that catches my attention. (Yesterday X was “How do racket’s thread-local parameters propagate their current values to new threads spawned by a subthread?)

Still, I’m ~50% confident that my original explanation is right, even if the details are wrong. I would be shocked if a compiled tail-recursive function didn’t jump into its own new definition when a new definition is loaded for the previous jump target (the global function name). Test it out and check what happens empirically. :)

Who knows. I’ve switched to Python long ago for daily tasks, since occasionally I enjoy “actually getting things done quickly, rather than endlessly researching interesting theoretical programming questions,” and Python is 60x slower than JavaScript (which doesn’t have tail recursion either), so #shruggyface. The world has apparently decided that tail recursion was an idea best left to the 90’s.

EDIT: After thinking it over, I bet you’re right and my original point about tail recursion is mistaken. And indeed, I remember now that in Arc, all the threads tend to be while-loops that just call another global function to do its work. Probably for this exact reason.

Very impressive callout. And a nice reminder to limit myself to talking about the things I actually do, not what I might theoretically do. At least not without qualifications and disclaimers. Thanks!

Post reply on HN