Forth: The programming language that writes itself

It's a real cool idea to compile everything down to lambda calculus and then you solve all semantics issues. (If something fits) you can convert 1:1, use general libraries in one language in others without loss etc. Ah, what a beautiful world it could be!

Thanks! I'll take a look.

Thanks! You might want to git clone http://canonical.org/~kragen/sw/pavnotes2.git/.

Like many here, the annual "language" issue of Byte Magazine in 1980 introduced me to Forth. Although I was enrolled as an engineer in college, I was a frustrated (mediocre) programmer but my upstate NY institute did not offer a Comp Sci degree at that time. Forth was a gateway drug for me and demystified the concept of compiling for me. Prior to Forth, when a thousand Freshman engineers were writing their Fortran IV projects for an 8am class the next morning on mainframe 3270 terminals, we imagined that the operators must have to continuously pour water on the compiler to keep it cool. Yeah, it was 1980 and computers were still a little magic.

But Forth and threaded code was a life changer; it explained so much! My 3rd year, I partially implemented a 32-bit Forth in IBM S360 Assembler but getting I/O to work was my downfall (mostly due to my poor skills and lack of experience.) But the threaded interpreter and the basic stack ops all worked. But then I was introduced to Lisp...

But my love of Forth never left me. I make my living with C in the early days, but predominantly SQL and Bash these days. When I had my second (third?) midlife crisis, I got two tattoos on my arm: one is the Y Combinator in Lisp (I 'lost' a trailing parenthesis due to a cut-n-paste error in the template given to the tattoo artist, so I had to go back and get another tattoo with an error message pointing out the missing parenthesis.), the second tattoo is the implementation of an ANSI Forth word:

    : ? @ . ;

The fact that I could write an entire function with only punctuation characters was mind-blowing and reminds me to approach problems in unique ways. The tattoos are also great ways to start up conversations in bars...

TL;DR: I'm trying to Forth a Lisp.

This is long winded, but maybe you have some thoughts here.

I've been building a DOM-builder API recently in Rust. The existing options (there are many) tend to use textual templating, which can't reason well about anything, or basic macros, which never support indentation. I wanted something that was just code, where I'd be in full control on the indentation (or lack thereof) programmatically. The closest equivalent is Python's Dominate [1], though its indentation mechanisms are buggy.

So I built a system using the traditional tree where Nodes own other Nodes at random addresses, and I built a renderer that renders those nodes and concatenates their strings recursively. It ended up working but it was hacky and very slow for the large inputs. In release mode, it was taking almost a minute to render 70 files, and I want about two orders of magnitude lower.

I ran it through profilers and optimized it a bit, but wanted to see if I could simplify the architecture and reduce the amount of work the computer could do. I read about flattening ASTs [2] and how through optimizing that format, you can end up with a sort of bytecode [3]. I also looked into Data-Oriented Design, watching Mike Acton's famous talk [4], Andrew Kelley's talk about DoD in Zig [5], and reading through the DoD book by Richard Fabian [6].

I ended up with something that works quite well for traversing and rendering, which is a stack that can be traversed and rendered in O(n), but I lost my nice Dominate-like API. As in, I can build these beautiful, flat trees, but to embed those trees in my code, I need to either materialize a tree in the traditional style first and then push it onto these stacks, or do some sort of macro magic to make these stack pushes.

I wonder if this is a common issue with stack-based programming. It is, in my case, quite simple for the computer, but hard to fit into an API without building up the stack manually!

---

1. https://pypi.org/project/dominate/

2. https://www.cs.cornell.edu/~asampson/blog/flattening.html

3. https://old.reddit.com/r/ProgrammingLanguages/comments/mrifd...

4. [Mike Acton] https://www.youtube.com/watch?v=rX0ItVEVjHc

5. [Zig] https://www.youtube.com/watch?v=IroPQ150F6c

6. https://www.dataorienteddesign.com/dodbook.pdf