Just what does “code as data” mean anyway? (2014) • Hacker News Board

Re: Just what does “code as data” mean anyway? (2014)

#21

Post by PaulHoule » Fri, Feb 16, 2018, 3:12 PM UTC

pjc50 wrote: post #9

PaulHoule wrote: post #5

One of the failures of conventional programming libraries is that parsing libraries such as yacc work in only one direction. In 2018 we could easily have libraries that work both ways by default, but we don't. Bidirectional parsing is great for code generation and opens up a lot of things you could do easily, but because common parsing libraries are unidirectional, people aren't aware of what you can do and don't cla…

What do you actually mean by "bidirectional parsing" - re-generating the original input, or something equivalent to it, from the AST?

Yep.

"Something equivalent" is pretty easy.

To be able to regenerate the original input (spacing and other foibles) is a bit harder but also possible. Commercial vendors developed tools that can do this in the 00's to support UML and UML-like tooling.

Related to this is getting sensible output structures. For instance, many parsing libraries based on parsing combinators output a "lispy" combination of lists/dicts/scalars which is dependent on the details of the grammar, for ex.

https://github.com/neogeny/tatsu

it is particularly obnoxious when operator precedence is implemented by creating extra nodes in the parse tree, and even more obnoxious when you have to copy those already over-complicated structures in your grammar to deal with complex context.

Even in 2018 people are still using some compiler compilers that use a "callback hell" interface that dates back to yacc. That might have been OK in 1978, but it is one of the many unergonomic things that put compiler compilers out of reach for many application programmers.

Re: Just what does “code as data” mean anyway? (2014)

#22

Post by dmitriid » Fri, Feb 16, 2018, 3:13 PM UTC

peatmoss wrote: post #7

And, as I wrote in another thread yesterday, it allows you to work with your text editor at a higher level of abstraction than non-lisp-coding people may have seen before ( https://news.ycombinator.com/item?id=16386380 ). Rather than thinking in letters, words, lines, or paragraphs, editor modes like paredit let you edit in terms of the structure of your code. I find this really hard to give up after lispy sessions.

> Rather than thinking in letters, words, lines, or paragraphs, editor modes like paredit let you edit in terms of the structure of your code.

Is there a single programmer on this planet who thinks about code in terms of letters, words, lines, or paragraphs?

Newsflash: it's 2018, not 1960, and code editors are capable of things most lisp "editor modes" can only dream about.

Re: Just what does “code as data” mean anyway? (2014)

#23

Post by kazinator » Fri, Feb 16, 2018, 3:15 PM UTC

dmitriid wrote: post #19

One of the greatest lies that Lispers have is: "Lisp has no syntax". Syntax is defined as "the structure of statements in a computer language." What Lisp has, and is, is a syntax to describe an AST. If you get the syntax wrong, your program won't run. And even that syntax isn't uniform across the various Lisps (some will throw in weird chars and constructs here and there to make dealing with common structures easier…

That particular select instance could indeed just be an ordinary function, whereby the (where ...) and (order ...) are just evaluated argument expressions, also calling ordinary constructors for objects that influence the query. It doesn't really demonstrate the ability to manipulate syntax.

Things start to get more interesting when some of the inputs need to be lambdas. Still, the sugaring of those can be in the arguments, and select can remain a function. Now suppose that a clause can, say, bind a lexical variable that a later clause can somehow usefully refer to; things like that.

Re: Just what does “code as data” mean anyway? (2014)

#24

Post by PaulHoule » Fri, Feb 16, 2018, 3:16 PM UTC

icebraining wrote: post #15

PaulHoule wrote: post #5

One of the failures of conventional programming libraries is that parsing libraries such as yacc work in only one direction. In 2018 we could easily have libraries that work both ways by default, but we don't. Bidirectional parsing is great for code generation and opens up a lot of things you could do easily, but because common parsing libraries are unidirectional, people aren't aware of what you can do and don't cla…

If by code you mean actual text (as opposed to macropy, which generates ASTs that get directly compiled), what does that open up? I've found code generation to be mostly annoying, unless it's just fixing up code written by humans. What's the point of generating a bunch of code, so that it can be re-parsed to be executed? Might as well skip the intermediate step. To me it seems like having the computer physically pres…

For many of the databases I use I've written libraries that can parse the query language (SQL, SPARQL, Arangodb) do analysis and transformations on the queries and convert them back to text to send on to the server.

Sometimes you are working with objects that are intrinsic to your runtime and there is no need to generate text, but sometimes you are modifying somebody else's configuration file and you need to. Or maybe you are coding in Python and you want to generate C, etc. Or maybe you are busting HTML down to a DOM tree so you can template with a set of modification operators like you do in JavaScript as opposed to various forms of text templating that have the same problems as C macros.

In Java you can load classes from bytecode (even written in Clojure!) which opens up a huge opportunity for runtime code generation. The commonly used Spring framework uses it heavily.

Other times though you want to package some objects in a JAR to give to some system that doesn't have your code generator. The generated Java files work great with an IDE in terms of cross-referencing, autocompletion, documentation lookup, etc.

Re: Just what does “code as data” mean anyway? (2014)

#25

Post by 3pt14159 » Fri, Feb 16, 2018, 3:17 PM UTC

quadcore wrote: post #12

See what I did there? Despite the good intention, no, because you're trying to explain lisp using lisp. If I knew lisp, I wouldnt need you to explain what code-as-data means. Maybe it's time for a different strategy. As a matter of fact, I've tried to explain code-as-data using OOP just this morning: https://news.ycombinator.com/item?id=16389514

Your linked comment is great but it sweeps one thing under the rug.

There is a reason lisp didn't win. We put up with it (I use emacs, so I have to write it at times) but in general it's harder to reason about transformable lists of code than it is about plain old objects and their methods. Almost every problem elegantly solved in lisp has a parallel elegant solution in Ruby (using blocks, dynamic method definition, or otherwise) and I generally find the resulting code both more maintainable as well as more readable. Sometimes with lisp things get "spooky" in a way that just doesn't happen in practice with Ruby, even if it theoretically possible to say redefine String#inspect. Though I completely agree that inheritance gets clunky and I strongly favour composition.

Re: Just what does “code as data” mean anyway? (2014)

#26

Post by scottmsul » Fri, Feb 16, 2018, 3:25 PM UTC

Would it be possible to do the same thing in Haskell using partial function application, where the first argument is the name of the tag and the second argument makes the "" tags and applies string concatenation?

Re: Just what does “code as data” mean anyway? (2014)

#27

Post by Steltek » Fri, Feb 16, 2018, 3:25 PM UTC

PaulHoule wrote: post #21

pjc50 wrote: post #9

Earlier quoted context omitted.

What do you actually mean by "bidirectional parsing" - re-generating the original input, or something equivalent to it, from the AST?

Yep. "Something equivalent" is pretty easy. To be able to regenerate the original input (spacing and other foibles) is a bit harder but also possible. Commercial vendors developed tools that can do this in the 00's to support UML and UML-like tooling. Related to this is getting sensible output structures. For instance, many parsing libraries based on parsing combinators output a "lispy" combination of lists/dicts/sca…

You want a decompiler that produces the original language, not just marked up assembly. For pure VM or interpreted languages, you're right this doesn't seem impossible but has unacceptable tradeoffs for most people. For starters, most companies like that compilation is a one-way action.

Whitespace is easy, btw. Just enforce code style in the language like gofmt. You don't need to preserve what you can derive programmatically. As a bonus, gofmt is awesome.

Re: Just what does “code as data” mean anyway? (2014)

#28

Post by junke » Fri, Feb 16, 2018, 3:34 PM UTC

Why call `(map eval inner)`?

Re: Just what does “code as data” mean anyway? (2014)

#29

Post by quadcore » Fri, Feb 16, 2018, 3:36 PM UTC

3pt14159 wrote: post #25

quadcore wrote: post #12

See what I did there? Despite the good intention, no, because you're trying to explain lisp using lisp. If I knew lisp, I wouldnt need you to explain what code-as-data means. Maybe it's time for a different strategy. As a matter of fact, I've tried to explain code-as-data using OOP just this morning: https://news.ycombinator.com/item?id=16389514

Your linked comment is great but it sweeps one thing under the rug. There is a reason lisp didn't win. We put up with it (I use emacs, so I have to write it at times) but in general it's harder to reason about transformable lists of code than it is about plain old objects and their methods. Almost every problem elegantly solved in lisp has a parallel elegant solution in Ruby (using blocks, dynamic method definition,…

I agree, OO is ultimately good enough. Problem is, coding using OOP is like printing ideas on paper: you dont wanna do that because it's hard to discard ideas that's been printed on paper. You wanna use post-its instead because post-its are easy to discard. Refactoring is hard using OO and refactoring is the key difference between waterfall and good software development. Refactoring is easier, even fun, using functional programming because the medium is more malleable.

Re: Just what does “code as data” mean anyway? (2014)

#30

Post by kevas » Fri, Feb 16, 2018, 3:44 PM UTC

icebraining wrote: post #15

PaulHoule wrote: post #5

One of the failures of conventional programming libraries is that parsing libraries such as yacc work in only one direction. In 2018 we could easily have libraries that work both ways by default, but we don't. Bidirectional parsing is great for code generation and opens up a lot of things you could do easily, but because common parsing libraries are unidirectional, people aren't aware of what you can do and don't cla…

If by code you mean actual text (as opposed to macropy, which generates ASTs that get directly compiled), what does that open up? I've found code generation to be mostly annoying, unless it's just fixing up code written by humans. What's the point of generating a bunch of code, so that it can be re-parsed to be executed? Might as well skip the intermediate step. To me it seems like having the computer physically pres…

Can you please elaborate a bit more on what was such a nuisance when working with this method. I'm currently building something out that is using this technique to (hopefully) save lines in code, dynamic user form generation based on with permissions, etc...