Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Julia Macros for Beginners

jkrumbiegel.com

Post reply on HN
Print view View original

31–36 of 36 posts

Re: Julia Macros for Beginners

#31

Post by StefanKarpinski »

dan-robertson wrote: post #24

The frustration of Julia macros for me was never knowing what AST would be produced for a given expression. This is a bit more manageable if you have a typed AST (e.g. OCaml but ppxes have other issues) or an obvious one (e.g. lisp). I like the way rust handles it where the macros operate on a tree of non-delimiter leaves and [delimiter, subtree list, delimiter] nodes which can allow for figuring out what the input t…

Tip: the first move when trying to write a macro is doing `Meta.@dump` on examples of argument expressions you want your macro to consume and produce. Then write code that transforms the inputs to the outputs.

Re: Julia Macros for Beginners

#32

Post by ryanmonroe »

krumbie wrote: post #28
ryanmonroe wrote: post #26

Earlier quoted context omitted.

To reference variables in the outer scope, you would do mutate(df, b = .env$a + 1) And if you have a string (contained in a_var) which identifies a variable you can do mutate(df, b = .data[[a_var]] + 1) You could argue these feel clumsy, but I wouldn’t say it’s “hard” to do either of these things with dplyr.

I don't think it's just about whether it's hard to do, your syntax example looks short enough and one can memorize these two patterns relatively quickly. However, both patterns are another special case how identifiers are resolved in the expression. Aren't `.env` and `.data` both valid variable and column names? So what happens if I have a column named `.data`? Another example, which is the reason why we chose the `:…

I hadn't thought of that tradeoff. After testing just now, if you have a column named `.data` or `.env` those constructs work as if there was no such column, and actually in that case `mutate(df, b = .data + 1)` is an error.

Personally I'll happily take not being able to use those as column names if it means I can avoid always typing : before every in-data variable, but your comment gave me a better understanding of why it would be bad for some other person or scenario, perhaps where short term ease-of-use is lower on the list of priorities.

For your second example, it doesn't come up in R because a data frame column cannot be a function. Columns must be vectors (including lists) and you could have a vector where one or all elements are functions, but the column itself cannot not be a function (functions are not vectors), so there's no ambiguity there. To call a function stored in your data frame you'd have to access an element of the column, and any access method, e.g. `[[` or `$` would make the resulting set of characters invalid as the name of an object (without backticks, which would then disambiguate the intent)

    df % 
      mutate(y = x[[1]](3))
Separate from dplyr, in R when you use `(` to call a function it searches only for functions by that name.

    log

Re: Julia Macros for Beginners

#33

Post by dan-robertson »

StefanKarpinski wrote: post #31
dan-robertson wrote: post #24

The frustration of Julia macros for me was never knowing what AST would be produced for a given expression. This is a bit more manageable if you have a typed AST (e.g. OCaml but ppxes have other issues) or an obvious one (e.g. lisp). I like the way rust handles it where the macros operate on a tree of non-delimiter leaves and [delimiter, subtree list, delimiter] nodes which can allow for figuring out what the input t…

Tip: the first move when trying to write a macro is doing `Meta.@dump` on examples of argument expressions you want your macro to consume and produce. Then write code that transforms the inputs to the outputs.

Right but it wasn’t obvious to me (at the time) what a change to the source code would do to the ast, so it wasn’t easy to know all the cases to handle, especially with quasiquotation (I think double-backtick style programming was basically impossible).

For example, maybe you want to handle something that looks like:

  foo ~~> bar
In lisp syntax (and recall that is what the Julia ast is: everything is a head and then arguments) it might look like:

  (~~> foo bar)
  ; or
  (op ~~> foo bar)
But if you change to e.g.

  foo ~~> bar + 5
You might get

  (+ (~~> foo bar) 5)
Or

  (~~> foo (progn (+ bar 5)))
I don’t remember what you got or which cases were tricky, only that I could never guess what the output of dump would be.

Re: Julia Macros for Beginners

#34

Post by eigenspace »

medo-bear wrote: post #23
cbkeller wrote: post #21

Earlier quoted context omitted.

As it happens, in addition to the femptolisp in the Julia parser, there is actually a secret s-expression syntax for Julia itself. There's no built in REPL mode for it, but you can hack one in about a dozen lines: https://gist.github.com/brenhinkeller/44051118c2f9d18b26dc76...

nice! one annoying nit pick for me though is using commas as data separators. when you need to input data by hand into a multi dim array this can get annoying very quickly

There's https://github.com/swadey/LispSyntax.jl for that.

Re: Julia Macros for Beginners

#35

Post by cbkeller »

medo-bear wrote: post #23
cbkeller wrote: post #21

Earlier quoted context omitted.

As it happens, in addition to the femptolisp in the Julia parser, there is actually a secret s-expression syntax for Julia itself. There's no built in REPL mode for it, but you can hack one in about a dozen lines: https://gist.github.com/brenhinkeller/44051118c2f9d18b26dc76...

nice! one annoying nit pick for me though is using commas as data separators. when you need to input data by hand into a multi dim array this can get annoying very quickly

It's true; I think this syntax was probably made more for reading than writing since the main place it appears in the base language is just `Meta.show_sexpr`, but it's still interesting to play around with, and parsing it has some fun properties like that you can use Julia's standard syntax as effectively a preprocessor syntax for the s-expression syntax.

Re: Julia Macros for Beginners

#36

Post by pdeffebach »

ryanmonroe wrote: post #32
krumbie wrote: post #28

Earlier quoted context omitted.

I don't think it's just about whether it's hard to do, your syntax example looks short enough and one can memorize these two patterns relatively quickly. However, both patterns are another special case how identifiers are resolved in the expression. Aren't `.env` and `.data` both valid variable and column names? So what happens if I have a column named `.data`? Another example, which is the reason why we chose the `:…

I hadn't thought of that tradeoff. After testing just now, if you have a column named `.data` or `.env` those constructs work as if there was no such column, and actually in that case `mutate(df, b = .data + 1)` is an error. Personally I'll happily take not being able to use those as column names if it means I can avoid always typing : before every in-data variable, but your comment gave me a better understanding of…

In Julia you could have an `AbstractVector` type also be callable, or more likely a vector of callable objects (and the operation is performed row-wise).

I agree it's unlikely that a user will name their column `.data`. But it certainly saves developer effort from thinking about these issues.

The larger concern, really, is that Julia needs to know which things are columns and which things are variables in an expression at parse time in order to generate fast code for a DataFrame. It needs to do this without inspecting the data frame, since the data frame's contents aren't known at parse time.

One option would be to make all literals columns. But then you run into issues with things like `missing`, which would have to be escaped or not recognized as a column. Its hard to predict all the problems there, and any escaping rules would definitely have to be more complicated than R's. So we require `:` and take the easy way out, which has the added benefit for new users who might get confused about the variable-column distinction.

Post reply on HN