If I remember correctly, Postgres was initially developed in Lisp, but then rewritten. Was that a mistake, or evidence against the thesis of this article?
It is not that simple
Uniform Structured Syntax, Metaprogramming and Run-Time Compilation
21–30 of 45 posts
Re: Uniform Structured Syntax, Metaprogramming and Run-Time Compilation
#22If I remember correctly, Postgres was initially developed in Lisp, but then rewritten. Was that a mistake, or evidence against the thesis of this article?
Not really, the original Postgres was developed in a mix of 17000 lines of Lisp and 63000 lines of C. This was difficult to develop/debug at that time. Probably still would be. It had a 'gigantic' memory footprint of 4 MB - the all-in-C version only used 1 MB. The Lisp version was also slower and they didn't use features like GC... http://db.cs.berkeley.edu/papers/ERL-M90-34.pdf
> By a process of elimination, we decided to try writing POSTGRES in LISP. We expected that it would be especially easy to write the optimizer and inference engine in LISP, since both are mostly tree processing modules. Moreover, we were seduced by AI claims of high programmer productivity for applications written in LISP.
Yes, that's what I remembered, they started out using LISP.
> Our feeling is that the use of LISP has been a terrible mistake for several reasons.
"Terrible mistake" is pretty unambiguous language.
Re: Uniform Structured Syntax, Metaprogramming and Run-Time Compilation
#23Earlier quoted context omitted.
Not really, the original Postgres was developed in a mix of 17000 lines of Lisp and 63000 lines of C. This was difficult to develop/debug at that time. Probably still would be. It had a 'gigantic' memory footprint of 4 MB - the all-in-C version only used 1 MB. The Lisp version was also slower and they didn't use features like GC... http://db.cs.berkeley.edu/papers/ERL-M90-34.pdf
> By a process of elimination, we decided to try writing POSTGRES in LISP. We expected that it would be especially easy to write the optimizer and inference engine in LISP, since both are mostly tree processing modules. Moreover, we were seduced by AI claims of high programmer productivity for applications written in LISP. Yes, that's what I remembered, they started out using LISP. > Our feeling is that the use of LI…
> Yes, that's what I remembered, they started out using LISP.
Only for the optimizer and the inference engine.
The authors also had no prior experience developing an application in a hybrid of C and Lisp.
> "Terrible mistake" is pretty unambiguous language.
4MB memory footprint was a terrible mistake at that time.
Re: Uniform Structured Syntax, Metaprogramming and Run-Time Compilation
#24Earlier quoted context omitted.
I think more analogous to the examples Kay was giving would be for a single program to quietly change the rules for function application, or to quietly change the behavior of CLOS. I think of an embedded DSL, based on a macro like it is here, as a bit different than that, because it's not doing anything quietly -- it applies only to the parenthesized extent of the code that begins with your macro name: (my-dsl-macro…
You still have the problem that the `my-dsl-macro` is a black box. That macro can do anything it wants to the sub-AST. The problem is not about hygiene re: variables but about the meaning of symbols. A function application within it that doesn't obviously have anything to do with the DSL is also subject to the macro's will to have its meaning changed. This can make composition difficult. Who guarantees that if you co…
> Who guarantees that if you combine `my-dsl-1-macro` with `my-dsl-2-macro` or simply with general purpose code, that they don't interfere in odd ways?
Nobody does. But that's where the power lies. You can't have unrestricted power with restricted features. Lisp gives you unrestricted power. It's up to you to use it correctly, for w/e definition of correct your context makes sense.
Re: Uniform Structured Syntax, Metaprogramming and Run-Time Compilation
#25This is talking about evaluating dynamic expressions over (semi-?)-structured row-oriented data, for the purpose of filtering. It contrasts a tree interpreter in C++ with a JITted dynamically generated Lisp expression, with some hand-waving away of what the equivalent JIT in C++ would be, seemingly dismissing it as taking too long (is that what "unpause cosmic time" alludes to? I'm not sure). The tree interpreter is…
Unless you provide one of
- a simple VM implementation - code using a JIT framework - using Apache Impala on this particular dataset
which performs on par and didn't take astonishingly long to write then these are just vague claims and hypotheses.
Re: Uniform Structured Syntax, Metaprogramming and Run-Time Compilation
#26Earlier quoted context omitted.
I think more analogous to the examples Kay was giving would be for a single program to quietly change the rules for function application, or to quietly change the behavior of CLOS. I think of an embedded DSL, based on a macro like it is here, as a bit different than that, because it's not doing anything quietly -- it applies only to the parenthesized extent of the code that begins with your macro name: (my-dsl-macro…
You still have the problem that the `my-dsl-macro` is a black box. That macro can do anything it wants to the sub-AST. The problem is not about hygiene re: variables but about the meaning of symbols. A function application within it that doesn't obviously have anything to do with the DSL is also subject to the macro's will to have its meaning changed. This can make composition difficult. Who guarantees that if you co…
True, `my-dsl-1-macro` potentially does anything inside its syntactic parentheses extent, including possibility manipulating syntax within it that someone just glancing at the code might assume is a code snippet that's pasted in verbatim, including what looks like a use of `my-dsl-2-macro`.
The verbatim paste seems like the usual case for macros.
This kind of potential expectation violation isn't specific to macros. For example, some functions could potentially mutate arguments when, at a casual glance, the expectation would be that that it doesn't mutate. (Even with an explicitly const argument, the language usually permits mutating something referenced by that immediate const, so we're back to expectations again.)
I think this goes back to conventions and judiciousness. And perhaps also IDE-supported easily-accessible documentation, and good naming.
Re: Uniform Structured Syntax, Metaprogramming and Run-Time Compilation
#27Earlier quoted context omitted.
> By a process of elimination, we decided to try writing POSTGRES in LISP. We expected that it would be especially easy to write the optimizer and inference engine in LISP, since both are mostly tree processing modules. Moreover, we were seduced by AI claims of high programmer productivity for applications written in LISP. Yes, that's what I remembered, they started out using LISP. > Our feeling is that the use of LI…
> Yes, that's what I remembered, they started out using LISP. Only for the optimizer and the inference engine. The authors also had no prior experience developing an application in a hybrid of C and Lisp. > "Terrible mistake" is pretty unambiguous language. 4MB memory footprint was a terrible mistake at that time.
They obviously started writing Postgres in LISP, because "we soon realized that parts of the system were more easily coded in C" wouldn't make sense if writing a hybrid had been the initial plan.
They tried going all LISP at first, and failed. Was it them, or was it LISP?
Re: Uniform Structured Syntax, Metaprogramming and Run-Time Compilation
#28Does it matter that McCarthy won the Turing Award before Alan Kay? What a silly point.
Is it awful that functions are black boxes? That they are bundled in libraries?
Does it matter that for loops don't leave a stack trace? How can poor programmers debug them w/o a stack trace?
Can you imagine that overriding a method completely changes what a bundle of methods does in a class?
It is so sad that every time a new idea comes out, the "resistance" (programmers who often lack experience with it or experienced an inferior implementation of the idea) are the loudest to complain and hold sw dev back.
Re: Uniform Structured Syntax, Metaprogramming and Run-Time Compilation
#29This is talking about evaluating dynamic expressions over (semi-?)-structured row-oriented data, for the purpose of filtering. It contrasts a tree interpreter in C++ with a JITted dynamically generated Lisp expression, with some hand-waving away of what the equivalent JIT in C++ would be, seemingly dismissing it as taking too long (is that what "unpause cosmic time" alludes to? I'm not sure). The tree interpreter is…
Unless you provide one of - a simple VM implementation - code using a JIT framework - using Apache Impala on this particular dataset which performs on par and didn't take astonishingly long to write then these are just vague claims and hypotheses.
A lot of people have experience with simple VMs. There's nothing super unique about this problem that requires every claim about it to be accompanied by a proof by construction.
Re: Uniform Structured Syntax, Metaprogramming and Run-Time Compilation
#30Earlier quoted context omitted.
> Yes, that's what I remembered, they started out using LISP. Only for the optimizer and the inference engine. The authors also had no prior experience developing an application in a hybrid of C and Lisp. > "Terrible mistake" is pretty unambiguous language. 4MB memory footprint was a terrible mistake at that time.
They obviously started writing Postgres in LISP, because "we soon realized that parts of the system were more easily coded in C" wouldn't make sense if writing a hybrid had been the initial plan. They tried going all LISP at first, and failed. Was it them, or was it LISP?
Well they were using LISP without GC because they couldn't afford stop-the-world pauses and concurrent GC didn't exist yet. If it was LISP, the several aspects of LISP they describe having trouble with are all obsolete considerations.