Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The Problem with Macros

ianthehenry.com

Post reply on HN
Print view View original

31–40 of 70 posts

Re: The Problem with Macros

#31

Post by reikonomusha »

ianthehenry wrote: post #22
reikonomusha wrote: post #10

As unsatisfying as it may sound, Common Lisp taught me that, truly, none of this function binding capture stuff really matters in practice. Millions of lines of Common Lisp code have been running for decades without running into problems with capture. So I purport solving this problem is akin to solving a 0.00000001% issue if we measure the frequency of encountering this error writing thousands of lines of Lisp per d…

> I purport solving this problem is akin to solving a 0.00000001% issue It seems like Common Lisp has done quite a lot to solve this problem already, which is why it is a non-issue in Common Lisp. > non-externalizable, non-PRINTable function objects Yes, this would be a very annoying technique in a language with non-externalizable, non-PRINTable functions. > This is contrary to Lisp’s DNA of interactive development a…

It’s ambiguous to me whether you’re suggesting there are compiled languages with printable and externalizable functions.

Given that many useful functions are actually closures over lexical environments possibly shared by other functions, and compiled to native code, I’d be impressed to find a language where such objects are printable and externalizable. I don’t think they are in Janet.

Re: The Problem with Macros

#32

Post by reikonomusha »

waterhouse wrote: post #18
reikonomusha wrote: post #10

As unsatisfying as it may sound, Common Lisp taught me that, truly, none of this function binding capture stuff really matters in practice. Millions of lines of Common Lisp code have been running for decades without running into problems with capture. So I purport solving this problem is akin to solving a 0.00000001% issue if we measure the frequency of encountering this error writing thousands of lines of Lisp per d…

> redefining that function won’t take effect in previously expanded code Yeah, that is an important issue. What do you think of the following solution? 1. The semantics of the language are that, every time a function is called, any macros in its body get reexpanded. 2. The language implementation is expected to notice that, in normal cases, neither the macro nor the functions it calls have been redefined, and therefo…

By “print”, I mean it in the Lisp sense of “print readably”. In Lisp, PRINT really means “serialize as text that can be deserialized with READ.” In Common Lisp (and Janet), functions print fine in the colloquial sense:

    > (print #'print)
    #
The character sequence ‘#<’ means it’s “unreadable” though.

Re: The Problem with Macros

#33

Post by waterhouse »

reikonomusha wrote: post #31
ianthehenry wrote: post #22

Earlier quoted context omitted.

> I purport solving this problem is akin to solving a 0.00000001% issue It seems like Common Lisp has done quite a lot to solve this problem already, which is why it is a non-issue in Common Lisp. > non-externalizable, non-PRINTable function objects Yes, this would be a very annoying technique in a language with non-externalizable, non-PRINTable functions. > This is contrary to Lisp’s DNA of interactive development a…

It’s ambiguous to me whether you’re suggesting there are compiled languages with printable and externalizable functions. Given that many useful functions are actually closures over lexical environments possibly shared by other functions, and compiled to native code, I’d be impressed to find a language where such objects are printable and externalizable. I don’t think they are in Janet.

> closures over lexical environments possibly shared by other functions

The same problem exists with lists (or any other mutable object). If x and y point to the same list, and you print x and read it back in, and then modify the list x points to, then (naively) it won't modify y. If you did want to preserve such structure sharing, one approach would be to print the entire environment and make liberal use of #n= notation. (I've occasionally done things like this.)

> compiled to native code

Assuming compilation is a deterministic function of the source code and the environment, and assuming the user hasn't changed the environment, it seems printing the source code should suffice, and the runtime can redo the compilation when necessary.

Re: The Problem with Macros

#34

Post by reikonomusha »

waterhouse wrote: post #28
fiddlerwoaroof wrote: post #23

Earlier quoted context omitted.

I think there might be another issue here: function objects aren't "externalizable" and so you can run into weird errors if you try to compile a form with literal functions.

Why aren't they externalizable? You could print a lambda as # . The arglist and probably the body would be straightforward (unless the body contains more literal functions, but then you recurse). Printing the env might involve printing complex or even circularly linked structures, but it's no more difficult than pretty-printing objects in general. (Some things, like a pointer to an FFI object corresponding to an open…

If both F and G are closures over x, and you externalize F, and later externalize G, how do you make sure their environments are enjoined when F and G are reified in a different Lisp process?

In Common Lisp, what about LOAD-TIME-VALUE?

Recursive printing of the environment just to serialize a single function seems atrocious and hazardous.

Re: The Problem with Macros

#35

Post by reikonomusha »

waterhouse wrote: post #33
reikonomusha wrote: post #31

Earlier quoted context omitted.

It’s ambiguous to me whether you’re suggesting there are compiled languages with printable and externalizable functions. Given that many useful functions are actually closures over lexical environments possibly shared by other functions, and compiled to native code, I’d be impressed to find a language where such objects are printable and externalizable. I don’t think they are in Janet.

> closures over lexical environments possibly shared by other functions The same problem exists with lists (or any other mutable object). If x and y point to the same list, and you print x and read it back in, and then modify the list x points to, then (naively) it won't modify y. If you did want to preserve such structure sharing, one approach would be to print the entire environment and make liberal use of #n= nota…

Functions with compiled code referring to address offsets in the closure environment are not printable and I’d hazard to say they can’t be without compromising something else. The function being purportedly serialized is already in a representation very far removed from its source code, and has mutable state that isn’t just from its closure environment. LOAD-TIME-VALUE (in Common Lisp) is another problem that allocates something akin to a C ‘static’ variable. Compiler macros (in Common Lisp) also make things hairier. Lack of run-time representation of reader macros (in Common Lisp) are the cherry on top—though this is only an issue if you want your serialized representation to mimic the source code closely.

Lists do have issues of sharing data with other lists, but most times the lists that are sharing data are somehow adjacent to one another allowing the printer to collect shared references.

If we allow ourselves the freedom to ignore shared closure environments, L-T-V and other Common Lisp complications, then serializing a function would still be this very expensive recursive procedure to serialize all transitive callees to capture any possible shared state, and I think that is too much to ask for, especially in the context we are speaking (making functions appropriate objects to be printed as a part of an S-expression).

It’s all doable if everything is slow, interpreted, and first-class though. Or maybe you’re serializing FORTH words. :)

Re: The Problem with Macros

#36

Post by kazinator »

fiddlerwoaroof wrote: post #24
reikonomusha wrote: post #10

As unsatisfying as it may sound, Common Lisp taught me that, truly, none of this function binding capture stuff really matters in practice. Millions of lines of Common Lisp code have been running for decades without running into problems with capture. So I purport solving this problem is akin to solving a 0.00000001% issue if we measure the frequency of encountering this error writing thousands of lines of Lisp per d…

I strongly agree here: my experience is that a Lisp-2 with namespaced symbols has very few issues with function-name capture. While, at one point, Lisp-2s may have been for performance or other implementation details, I find that Lisp-2s are much nicer to code in because you just don't worry about name collisions (as long as you know a handful of relatively simple rules about macros).

My experience that a certain systems programming language of Unix origins featuring 1 namespace and an unhygienic token-based macro system also has few issues with function-name capture, even in code bases with seven digit LOCs.

That makes me severely disinterested and skeptical about hygienic macros.

They are too weird for the little benefit they provide. You can't look at a piece of code and know what it expands to.

Hygienic macros break lexical scope, because code produced by a macro being invoked in some file "A.lisp" is seeing lexical variables defined in some different file "B.lisp". Lexical scope must be physically enclosed and contained. If you expand it here, it sees the variables that are here, and not some other ones elsewhere that you don't see here.

Given a nesting like (x (y (z a))), if y binds a, that must be like a brick wall; there is no way that the a reference in z can get around it, to connect with a binding of a produced by x. Either that a refers to z's own binding, or else to y. Yet hygienic macros can perpetrate a lexical-scope-destroying wormhole which can do that: the inner a reference can bypass a definition set up by y, and go to the one in x.

It's a downright security issue, like going around your company's firewall, or escaping a container or chroot jail or whatever.

Re: The Problem with Macros

#37

Post by fiddlerwoaroof »

kazinator wrote: post #36
fiddlerwoaroof wrote: post #24

Earlier quoted context omitted.

I strongly agree here: my experience is that a Lisp-2 with namespaced symbols has very few issues with function-name capture. While, at one point, Lisp-2s may have been for performance or other implementation details, I find that Lisp-2s are much nicer to code in because you just don't worry about name collisions (as long as you know a handful of relatively simple rules about macros).

My experience that a certain systems programming language of Unix origins featuring 1 namespace and an unhygienic token-based macro system also has few issues with function-name capture, even in code bases with seven digit LOCs. That makes me severely disinterested and skeptical about hygienic macros. They are too weird for the little benefit they provide. You can't look at a piece of code and know what it expands to…

All I know is that in Python, JS and Clojure is I’ve accidentally written code like:

    def foo(str):
      v = str(1)
      . . .
Which resulted in head-scratching errors like “str is not a function”.

EDIT: I agree that hygienic macros aren't the right way to solve this issue.

Re: The Problem with Macros

#38

Post by nathell »

Clojure alleviates this to a great extent by automatically namespace-qualifying symbols within quasiquoted forms, which prevents clashes between let-bound symbols (which are never qualified) and ones from the global scope.

It’s still possible to shoot yourself in the foot (see https://blog.danieljanus.pl/2020/01/21/middleware/ for my personal story), but for that you need `binding`, `with-redefs` or `alter-var-root`, which are all relatively rarely seen.

Re: The Problem with Macros

#39

Post by dunefox »

jsnell wrote: post #19
komon wrote: post #15

Earlier quoted context omitted.

"no notes" is an idiom meaning "I have no suggestions for improvement" The author was being sincere, rather than sarcastic as your initial parse suggests

I was not suggesting they were being sarcastic. I was suggesting they had not actually noticed that the error message actually answered their question: "Why can’t I shadow this function?". This is the section of the standard the error message points at: http://www.lispworks.com/documentation/lw71/CLHS/Body/11_aba... And this is the reasoning for why that section exists, linked to from that page: http://www.lispworks.…

I read it the same way. I haven't heard this idiom either.

Re: The Problem with Macros

#40

Post by TeMPOraL »

waterhouse wrote: post #18
reikonomusha wrote: post #10

As unsatisfying as it may sound, Common Lisp taught me that, truly, none of this function binding capture stuff really matters in practice. Millions of lines of Common Lisp code have been running for decades without running into problems with capture. So I purport solving this problem is akin to solving a 0.00000001% issue if we measure the frequency of encountering this error writing thousands of lines of Lisp per d…

> redefining that function won’t take effect in previously expanded code Yeah, that is an important issue. What do you think of the following solution? 1. The semantics of the language are that, every time a function is called, any macros in its body get reexpanded. 2. The language implementation is expected to notice that, in normal cases, neither the macro nor the functions it calls have been redefined, and therefo…

1. might fly in some purely-functional Lisp, but most (all?) real-world Lisps can have side effects in macro expansion. And by "side effects", I don't mean altering some global variable or adding a function to the symbol table - Lisp macros have access to the full language runtime, so a macroexpansion may just as well send some HTTP requests or hit a database.

Ironically, the author of the article was correct in their view that he initially described as wrong. Macros are just functions, typically taking code and outputting code. They just didn't understand the evaluation rules surrounding macro invocation.

Post reply on HN