Live data from Hacker News

TXR – A Programming Language for Convenient Data Munging

nongnu.org

61–70 of 74 posts

Re: TXR – A Programming Language for Convenient Data Munging

#61
post #39
post #9

It would be interesting to have a DSL for data munging, but I am afraid TXR is not it. My requirements would be that the language should be functional and total. Most transformations that we do on data do not require Turing completeness or recursion. I think it would be useful to write these down in a language with semantics that is easy to analyze.

Then I think you will like https://tkatchev.bitbucket.io/tab/index.html "It's statically-typed and type-infered. It also infers memory consumption and guarantees O(n) memory use. It is designed for concise one-liner computations right in the shell prompt. It features both a mathematics library and a set of data slicing and aggregation primitives. It is faster than all other interpreted languages with a similar scope.…

Looks very interesting, but I am not the author.

Re: TXR – A Programming Language for Convenient Data Munging

#62

Earlier quoted context omitted.

Lambdas are limited to a single expression. Threads don't work the way anyone would ever want threads to work because o the GIL. That said, homoiconicity (s-expressions) is not a feature I want. It makes code like reading a wall of text compared to a nicely laid out magazine.

Since Python lambdas also are expressions, they cannot contain statements, due to Python adhering to the Algol-like statement/expression syntactic paradigm. If a lambda expression could contain statements, that would mean that almost any other kind of Python expression could also contain statements by containing a lambda expression. But, here above, I'm writing more from the angle of supporting multi-line lambdas tha…

>Strictly speaking, you are only bemoaning the lack of multiple expression support, not multi-line lambdas.

Oh, I'm not bemoaning. I'm just answering GP about the likely limitations that the previous poster was complaining about.

>Maybe this is a common trick? I don't use Python; I hardly know anything about it.

I work around lambdas by naming internal functions. It's easier to read intent if I tell you what I'm trying to do.

Re: TXR – A Programming Language for Convenient Data Munging

#63
post #43

Earlier quoted context omitted.

Gotcha. I do agree about homoiconicity. It's great for writing macros, and terrible for everything else. Of course, go too far in the other direction and you get perl, so... yeah.

I don't want to reopen this 50+ years old can of worms, but I have 2 questions: 1. Is it about homoiconicity in general or specifically s-expressions [EDIT: I see GP writes about s-exps specifically, missed it at first]? Prolog, Erlang, TCL, and Rebol (just some examples) are homoiconic, but not s-exps based. What do you think about them? 2. Do you often read code without syntax highlighting and proper indentation? A…

>I see GP writes about s-exps specifically, missed it at first

My experience is with s-expressions being a wall of text. I haven't used prolog (end erlang uses the same syntax) enough to have a beef with them. Perhaps I was being over general.

Re: TXR – A Programming Language for Convenient Data Munging

#64

Earlier quoted context omitted.

Yep. Seriously. R w/tidyverse is a ridiculously powerful data wrangling tool especially when dealing with text files. I tend use Notepad++ when starting out on a data-wrangling adventure. It has an uncanny ability, unlike any other editor, to open hundreds of files at the same time and to perform regex operations on all of them without dropping dead. I uses Notepad++ for initial manual exploration to get the lay of t…

>I tend use Notepad++ I assume, then, that your file sizes are not so big. N++ is not good with big (>25% of your ram) file sizes, refusing to open them. Is R/tidyverse also limited on the size of the file it can handle? In my job i routinely work with up to 100GB files.

I guess it depends on what your definition of "big" is, I've never had to deal with 100GB files!

Re: TXR – A Programming Language for Convenient Data Munging

#65

Earlier quoted context omitted.

Since Python lambdas also are expressions, they cannot contain statements, due to Python adhering to the Algol-like statement/expression syntactic paradigm. If a lambda expression could contain statements, that would mean that almost any other kind of Python expression could also contain statements by containing a lambda expression. But, here above, I'm writing more from the angle of supporting multi-line lambdas tha…

> Maybe this is a common trick? No. It's obvious and trivial, but you'd be on a verge of being called names if you tried to use it in a Python codebase. Lambdas in Python are limited to a single expression by convention - which in Python-land is scarily rigid and specific - rather than just by the language spec. Before `... if ... else ...` was added to the language as an expression (I think around 2.5), people had t…

My experience with Go is that the syntax is more rigid as compared to Python but the community is much less idealistic.

As long as your code is linted with gofmt and it compiles (and doesn’t abuse reflection), the community tolerates more creative uses of the syntax to get around some of the pitfalls of the language – but there is of course less opportunity for creative syntax than in a dynamic language like Python.

Re: TXR – A Programming Language for Convenient Data Munging

#66

Earlier quoted context omitted.

I don't want to reopen this 50+ years old can of worms, but I have 2 questions: 1. Is it about homoiconicity in general or specifically s-expressions [EDIT: I see GP writes about s-exps specifically, missed it at first]? Prolog, Erlang, TCL, and Rebol (just some examples) are homoiconic, but not s-exps based. What do you think about them? 2. Do you often read code without syntax highlighting and proper indentation? A…

>I see GP writes about s-exps specifically, missed it at first My experience is with s-expressions being a wall of text. I haven't used prolog (end erlang uses the same syntax) enough to have a beef with them. Perhaps I was being over general.

> My experience is with s-expressions being a wall of text.

That's sad, but unfortunately not that rare. Making a Lisp readable is not as easy as doing so in Python.

In my experience, Lisp code written by an experienced lisper who cares about readability can often be much more readable than well-written Python (assuming equal levels of proficiency in their respective languages). On the other hand, the number of ways you could destroy the readability of Lisp source is endless, they are always close by, and are even context-dependent. Beginners or people without the focus on readability use those ways rather liberally.

The situation is the exact opposite: Python defines the lower bound on the code readability ("Readability counts" -> PEP-8 -> linters -> (lately) `black`) while Lisps, in general, don't even have an authoritative PEP-8 equivalent.

On the other hand, Python also has an upper bound on how readable it can be: its syntax is rich, but if you find yourself in a place where it's not rich enough, you're on your own. You could just use the expressive power of the language to hijack the syntax and beat it into shape better suited for your problem, but it will be almost certainly seen as un-Pythonic.

On this point, Lisps have a huge advantage, because you can change the language parser on the fly easily, and you can add whatever syntax sugar you need in a couple of lines of a macro. In other words, Lisps give programmers tools for making their code as readable as they want (and are able to) while at the same time allowing them to write "walls of text" (and honestly, that'd be a very polite way of describing some of the Lisp code I've seen) which - in readability - could be one of the worst among many languages I've seen.

So, what I want to say here is that it's possible - and not that hard - to write readable Lisp code. Unfortunately, a programmer has to both think of readability when writing and have skills to make their ideas on readability into reality.

In effect, yes, there's a lot of Lisp code which is hard to ingest. Some style guides are there or are being created, paredit helps a lot, I'm not aware of any linters yet, but they should start appearing at some point. On the other hand, Lisp code skillfully crafted for readability is rivals (and sometimes surpasses) Python at its best.

I'm not sure what Lisp code you've seen, but I assume it was all of the former kind. This is unfortunate. Without knowing what was it you were reading/working with it's hard to recommend anything, but I found examples in "How to Design Programs" quite readable: https://htdp.org/2019-02-24/part_five.html and there are also other books and Open Source projects with code worth reading, but I'd have to dig through my bookmarks, which I don't have the time for right now, sorry :(

TLDR: Lisps - Schemes, Racket, CL, PicoLisp, Emacs and TXR Lisps to name a few - can be used to write astonishingly readable and to-the-point code, but the languages do absolutely nothing to discourage using them to write the most unreadable mess under the heavens. As for the reasons for this - I've honestly no idea at all.

Re: TXR – A Programming Language for Convenient Data Munging

#67

Earlier quoted context omitted.

>I see GP writes about s-exps specifically, missed it at first My experience is with s-expressions being a wall of text. I haven't used prolog (end erlang uses the same syntax) enough to have a beef with them. Perhaps I was being over general.

> My experience is with s-expressions being a wall of text. That's sad, but unfortunately not that rare. Making a Lisp readable is not as easy as doing so in Python. In my experience, Lisp code written by an experienced lisper who cares about readability can often be much more readable than well-written Python (assuming equal levels of proficiency in their respective languages). On the other hand, the number of ways…

> Python defines the lower bound on the code readability

Ignoring ;... comments for a moment, if we squash a Lisp program into one line and remove all non-essential whitespace, it's possible to recover it into nicely formatted code by machine, more or less.

> TLDR: Lisps - Schemes, Racket, CL, PicoLisp, Emacs and TXR Lisps to name a few - can be used to write astonishingly readable and to-the-point code, but the languages do absolutely nothing to discourage using them to write the most unreadable mess under the heavens. As for the reasons for this - I've honestly no idea at all.

This view is unbalanced without noting that Javascript, Rust, C, Perl, Java, Scala, Go, Kotlin, ... and a large number of other languages, have the flexible formatting that allows for unreadable code. Ruby, anyone? https://github.com/mame/quine-relay/blob/master/QR.rb

> As for the reasons for this - I've honestly no idea at all.

Bad formatting is a bug that is fixable in the actual code (greatly assisted by automation) and a minor social problem in programming that is treatable with education and experience. Therefore, it is nearly a non-issue.

Re: TXR – A Programming Language for Convenient Data Munging

#68

Earlier quoted context omitted.

> My experience is with s-expressions being a wall of text. That's sad, but unfortunately not that rare. Making a Lisp readable is not as easy as doing so in Python. In my experience, Lisp code written by an experienced lisper who cares about readability can often be much more readable than well-written Python (assuming equal levels of proficiency in their respective languages). On the other hand, the number of ways…

> Python defines the lower bound on the code readability Ignoring ;... comments for a moment, if we squash a Lisp program into one line and remove all non-essential whitespace, it's possible to recover it into nicely formatted code by machine, more or less. > TLDR: Lisps - Schemes, Racket, CL, PicoLisp, Emacs and TXR Lisps to name a few - can be used to write astonishingly readable and to-the-point code, but the lang…

You're right, of course, on all points (BTW, WTF is with the downvotes??), but they are technicalities of interest to lispers. I omitted these because I wanted to present a convincing argument that fnord123 simply had bad luck and encountered bad Lisp code. And that there's a lot of good, readable code written in s-exps out there.

The resistance to s-exps in the general population of programmers is bad "news" (if something 50 years old can be called that...) for Lisps. It's hard to fight it in general terms. Pointing out that C, JS, PERL, etc. are often much worse in terms of readability - while obviously true - doesn't really help in convincing someone to look at s-exps differently. This is why I chose Python for comparison and tried to present a positive argument, saying that you can write code "even more readable than Python at its best" in Lisp.

I ignored automatic formatting because it's not part of the language, but of tooling. The problem with tooling is that not everyone uses it. I've had a "pleasure" of working with a 50+ kloc Clojure code base written mostly by C programmers who didn't know or care about formatting tools - honestly, it was a nightmare. Of course, each file could be automatically reformatted into something sensible, but the fact that it was written the way it was and the language did nothing to prevent that still stands. In Python, you at least would get the indentation right.

Readability is a hard problem in general. You're right that it's also a matter of education in the community. You're right that it's almost negligible a problem for lispers themselves, as they know how to reformat the code automatically with a single key press. It is a problem, though, for people who come into contact with Lisp code for the first time. I wanted to convince fnord123 that it's not the syntax itself, but rather how it is used that's a problem - like with every other kind of syntax out there, by the way. I'd be extremely happy if he reconsidered and tried to read some of the better-written s-exps based code.

Re: TXR – A Programming Language for Convenient Data Munging

#69

This seemed interesting, but when I went through the "Accepted Stack Overflow" links on the main page, I thought "how would I do this in an R tidyverse stack?" and set the goal that my responses should be shorter, clearer, or ideally both, and that I would favour clearer answers to code golf, except that when posting to HN I collapse the code into a single line while in R there would be linebreaks at each semicolon o…

> when posting to HN I collapse the code into a single line

If you just put two spaces of indentationo on every line, you get a verbatim block in typewriter font,

  like
  this.

Re: TXR – A Programming Language for Convenient Data Munging

#70

Earlier quoted context omitted.

> Python defines the lower bound on the code readability Ignoring ;... comments for a moment, if we squash a Lisp program into one line and remove all non-essential whitespace, it's possible to recover it into nicely formatted code by machine, more or less. > TLDR: Lisps - Schemes, Racket, CL, PicoLisp, Emacs and TXR Lisps to name a few - can be used to write astonishingly readable and to-the-point code, but the lang…

You're right, of course, on all points (BTW, WTF is with the downvotes??), but they are technicalities of interest to lispers. I omitted these because I wanted to present a convincing argument that fnord123 simply had bad luck and encountered bad Lisp code. And that there's a lot of good, readable code written in s-exps out there. The resistance to s-exps in the general population of programmers is bad "news" (if som…

In Lisp languages, we don't use pure S-exps for everything; we have notations. We have 'X instead of (quote X), `(,A ,@B) instead of (list* 'A B), and numerous # notations.

In the area of arithmetic, although the basic operators are functions invoked using (f arg ...), we give them short names like +, -, * and /. Why? The obvious reason is that we would find it irksome to be writing (add ...) and (mul ...).

Lisp can have notations, and they can be had without disturbing the Lisp syntax. Notations that are related to major program organization have payoff.

In TXR Lisp there is relatively small set of new notations, which all have correspondence to S-exp forms, the same way that 'X corresponds to (quote x).

  ;; slot access
  obj.x.y.z  -->  (qref x y z)
Of course, people are going to prefer this to something like:

  (slot-value (slot-value x 'y) 'z)
Then:

  ;; unbound slot access
  .x.y.z --> (uref x y z)

  ;; method call
  obj.x.(f a b)  --> (qref x (f a b))

  ;; x.f(blah).g(foo).xyzzy(x, y) pattern:

  x.(f blah).(g foo).(xyzzy x y)  ;; looks like this

  ;; sequence indexing, function calls (the "DWIM" operator)
  [array i]  --> (dwim array i)
  [f x y]    --> (dwim f x y)

  ;; ranges:
  a..b  --> (rcons a b)

  ;; slice
  [str 0..3] --> [dwim str (rcons 0 3)]

  ;; Python-like negative indexing:

  [str -4..:] --> [dwim str (rcons -4 :)] ;; : means "default value: one index past end sequence (its length)".

  ;; quasistrings -- recently appeared JavaScript in strikingly similar form!
  `@a @b ...` --> (sys:quasi @a " " @b) --> (sys:quasi (sys:var a) " " (sys:var b))

  ;; word list literals
  #"a b c"  --> ("a" "b" "c")

  ;; quasi word list literals
  #`a @b c`  --> (sys:quasilist `a` `@b` `c`)

Some Lisp syntax is streamlined:

   (lambda (a b c : x y  . r) ...)  ;; a b c required, x y optional, r rest
Dot notation allowed without preceding atom:

   (. x)  ->  x
Improper lists can be function calls:

   ;; TXR: verbosity-free wrapping of a function.
   (defun wrapper (. args)
     (wrapped . args))

   ;; CL:
   (defun wrapper (&rest args)
     (apply #'wrapped args))
This works even if the thing in the dot position is a symbol macro expanding to a compound form. The reason is that the code walker/expander will recognize and transform (func ... . rest) into (sys:apply (fun func) ... rest) first, and then expand macros. (I.e. we can't work this into existing Lisps like CL implementations without going down to that level.)

   ;; the : symbol -- symbol named "" in keyword package:
   ;; used as a "third boolean" in various places

   (func 1 2 : 4)  ;; use default value for optional arg, pass 4 for the next one
                   ;; diminishes need for keyword args


   ;; built-in regex syntax
   #/a.*b/

   ;; C-like character escapes
   "\t blah \x1F3 ... \e[32a"

   ;; multi-line strings with leading whitespace control:

   "Four sc \
    ore
   \ and seven years ago"  -> "Four score and seven years ago"


   ;; Simple commenting-out of object with #;

   #; (this is 
         commented out)
Also, there is no programmable reader in TXR Lisp; no reader macros. I'm not a big fan of reader macros. They are only useful for winning "I can have any damn syntax in my language" arguments. Problem is, the whole territory of "any damn syntax" is a wasteland of bad syntax, nt to mention mutually incompatible syntax.
Post reply on HN