Critique of Lazy Sequences in Clojure

> Is it for built-in map or it would work in a general way for, say, `myMap f (myMap g xs)` ?

Just the stdlib `map`, but it's all ordinary library code. You can easily add your own rewrite rules.

> I've been circling around lisp for a couple of years. I'm starting in a month, I'll spend several hours a day.

Is this just a personal goal you’re setting? I’m curious because I’m in a similar position, so I’d love to hear your plan!

It can be legitimate to have side effects in lazy processing and in particular to rely that a lazy sequence is not accessed beyond the visible access that is coded in the program.

Suppose we make a sequence of numbers which grows very rapidly, so that by the time we hit the 17th one, we have a bignum that is gigabytes wide.

You probably don't want this to be chunked in batches of 32.

Another situation might be if we have some side effect: the lazy sequence is connected to some external API somehow or foreign code. You might want it so that the observable behaviors happen only to the extent that the sequence is materialized.

The advice to be careful with side effects is good in general; not sure why you're downvoted.

TXR Lisp also fails this test:

  1> (len
       (with-stream (s (open-file "/usr/share/dict/words"))
         (get-lines s)))
  ** error reading #: file closed
  ** during evaluation of form (len (let ((s (open-file "/usr/share/dict/words")))
                                      (unwind-protect
                                        (get-lines s)
                                        (close-stream s))))
  ** ... an expansion of (len (with-stream
                                (s (open-file "/usr/share/dict/words"))
                                (get-lines s)))
  ** which is located at expr-1:1

The built-in solution is that when you create a lazy list which reads lines from a stream, that lazy list takes care of closing the stream when it is done.

If the lazy list isn't processed to the end, then the stream semantically leaks; it has to be cleaned up by the garbage collector when the lazy list becomes unreachable.

We can see with strace that the stream is closed:

  $ strace txr -p '(flow "/usr/share/dict/words" open-file get-lines len)'
  [...]read(3, "d\nwrapper\nwrapper's\nwrappers\nwra"..., 4096) = 4096
  read(3, "zigzags\nzilch\nzilch's\nzillion\nzi"..., 4096) = 826
  read(3, "", 4096)                       = 0
  close(3)                                = 0
  fstat64(1, {st_mode=S_IFCHR|0620, st_rdev=makedev(136, 0), ...}) = 0
  write(1, "102305\n", 7102305
  )                 = 7
  exit_group(0)                           = ?
  +++ exited with 0 +++

It is possible to address the error issue with reference counting. Suppose that we define a stream with a reference count, such that it has to be closed that many times before the underlying file descriptor is closed.

I programmed a proof of concept of this today. (I ran into a small issue in the language run-time that I fixed; the close-stream function calls the underlying method and then caches the result, preventing the solution from working.)

  (defstruct refcount-close stream-wrap
    stream
    (count 1)

    (:method close (me throw-on-error-p)
      (put-line `close called on @me`)
      (when (plusp me.count)
        (if (zerop (dec me.count))
          (close-stream me.stream throw-on-error-p)))))

  (flow
    (with-stream (s (make-struct-delegate-stream
                      (new refcount-close
                           count 2
                           stream (open-file "/usr/share/dict/words"))))
      (get-lines s))
    len
    prinl)

With my small fix in stream.c (already merged, going into Version 292), the output is:

  $ ./txr lazy2.tl
  close called on #S(refcount-close stream # count 2)
  close called on #S(refcount-close stream # count 1)
  102305

One close comes from the with-stream macro, the other from the lazy list hitting EOF when its length is being calculated.

Without the fix, I don't get the second call; the code works, but the descriptor isn't closed:

  $ txr lazy2.tl
  close called on #S(refcount-close stream # count 2)
  102305

In the former we see the call to close in strace; in the latter we don't.

I have not read yet, but how much of this critique is relevant to Python and overuse of generators?