Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

ClojureScript Builds, Rebooted

adzerk.com

Post reply on HN
Print view View original

21–29 of 29 posts

Re: ClojureScript Builds, Rebooted

#21

Post by weavejester »

brandonbloom wrote: post #17
weavejester wrote: post #15

Earlier quoted context omitted.

There's certainly a lot you can do to constrain I/O, but removing side-effects when possible will always be better than merely restricting them. I do like the idea of immutable files and repeatable builds, but I don't think this negates the benefits of maximising the time you spend working with data. For instance, there might be a task that automatically cleans up some files, but you want to keep those files around.…

> removing side-effects when possible will always be better than merely restricting them I disagree. Effects are a very natural mental model for a great deal of problems and constraining yourself to purity is both impractical and quickly experiences diminishing returns. Furthermore, if you can intercept effects, you can impose purity upon them. For an extreme example, consider application virtualization and container…

I realised I might not have been very clear in my previous comment. Let me see if I can improve it with an example.

Let's forget about all other considerations and instead consider the simplest possible build system we can conceive. This build system should take a directory structure of source files, and produce a directory structure of output files.

If our sole consideration is simplicity, we might construct a build system like:

    (defn -main [task & args]
      (-> (read-dir (cwd))
          (run-task task args)
          (write-dir (cwd))))
So we take every file in the current working directory, read everything into memory, perform some functional transformation that produces a data structure of output files, then write that to disk. This minimises I/O, and gives us a functional data structure to play around with.

It's a naive approach, and one made without regard for memory or efficiency, but given that the amount of memory on a modern machine is far larger than the source directory is likely to be, it actually seems feasible.

However, we can also consider optimisations that don't alter the behaviour. For instance, we could only read in files when their contents are accessed. In order to protect against changes, we could check the modification date, and abort if it changes. It's a compromise, but a small one.

We might also conceive of a system where the contents of the file are memory mapped, or held in some temporary file, or any number of clever ways to avoid keeping the file in memory while not breaking the integrity of the data structure.

This is just a toy example, and lacking in many areas like network I/O, but it's easier to start simple and add complexity when necessary, than it is to start from an assumption of complexity and try to work backward to simplicity. This is why I think it's incorrect to start with side-effectful functions, because that means starting from complexity.

Re: ClojureScript Builds, Rebooted

#22

Post by brandonbloom »

michaniskin wrote: post #19
brandonbloom wrote: post #18

Earlier quoted context omitted.

I still don't understand what the "task abstraction" is or what it provides. It seems to me that it's simply a Clojure function with a corresponding command line interface. Is that fair? Does it do something else too? If it's just a function, I don't know why command-line support is valuable. For interactive use, the Clojure REPL is just fine. For automated use, you only need a single shell utility like perl or awk t…

Yes, in the post I didn't get too technical with the treatment of tasks; I'll elaborate a little here. First, the command line thing. You're right that the command line isn't strictly required to use boot, you can do everything at the REPL or perl/awk etc., as you pointed out. But for me it's really useful just ergonomically to be able to use command line arguments to configure ad-hoc builds because they can be very…

> it's really useful just ergonomically to be able to use command line arguments to configure ad-hoc builds

It's largely also what contributes to "works for me" build environments... It's better to have a just one way to do it interface and discourage excessive tinkering with parameters. The more parameters, the more likely for your dev env to be unstable across individual checkouts or developers. I know it's idealistic, but I think we should strive for zero-arg builds, which oddly means not making it easier to configure them.

I'll have to think on all the other stuff you wrote, since it's not totally clear to me yet. I may ping you again after I noodle a bit.

Re: ClojureScript Builds, Rebooted

#23

Post by michaniskin »

weavejester wrote: post #20
brandonbloom wrote: post #17

Earlier quoted context omitted.

> removing side-effects when possible will always be better than merely restricting them I disagree. Effects are a very natural mental model for a great deal of problems and constraining yourself to purity is both impractical and quickly experiences diminishing returns. Furthermore, if you can intercept effects, you can impose purity upon them. For an extreme example, consider application virtualization and container…

> Effects are a very natural mental model for a great deal of problems and constraining yourself to purity is both impractical and quickly experiences diminishing returns. I can't personally recall a problem where purity was feasible but impractical, though I can think of a few examples of the opposite. > If you intercept all file IO, you can recover the same data. Yes, if you record all I/O, you could restore files…

The problem with the read-everything-into-memory approach is that this is not how the JVM ecosystem works. Things written for the JVM use the classpath primarily, and things in memory secondarily if at all.

We can't control the fact that the CLJS compiler, for example, is looking for source files on the classpath instead of in some FileSet object proxy. If we admit the use of tools written by the Java community at large we suffer by adding another leaking half-abstraction to the mix.

We actually did some experiments with fuse filesystems but the performance is just not there yet. When fuse performance becomes comparable to Java NIO it may become a viable option, and would solve all of these problems. You could then have a "membrane" approach, where the JVM is only manipulating a filesystem proxy, and you have complete control over when and how to reify that and write to the filesystem.

Re: ClojureScript Builds, Rebooted

#24

Post by michaniskin »

brandonbloom wrote: post #22
michaniskin wrote: post #19

Earlier quoted context omitted.

Yes, in the post I didn't get too technical with the treatment of tasks; I'll elaborate a little here. First, the command line thing. You're right that the command line isn't strictly required to use boot, you can do everything at the REPL or perl/awk etc., as you pointed out. But for me it's really useful just ergonomically to be able to use command line arguments to configure ad-hoc builds because they can be very…

> it's really useful just ergonomically to be able to use command line arguments to configure ad-hoc builds It's largely also what contributes to "works for me" build environments... It's better to have a just one way to do it interface and discourage excessive tinkering with parameters. The more parameters, the more likely for your dev env to be unstable across individual checkouts or developers. I know it's idealis…

Your point about repeatability is valid. As a policy matter we would never advocate building a project without codifying the process as zero-arg tasks in the build.boot file. You can see this in our own projects (eg. https://github.com/tailrecursion/boot-useful/blob/master/bui...). This project has one way to build the project jar file:

    (deftask build-jar
      "Build project jar file."
      []
      (comp (pom) (add-src) (jar) (install)))
When you build the jar file you just do

    boot build-jar
This ensures repeatability, etc.

But we don't only use boot for repeatable builds! Boot is in a unique position in that it's on the intersection of application and environment. That is to say, boot can be used to "bootstrap" the application. With boot we can create sort of self-configuring applications, where the entry point of the application is the build.boot file. This is a very clear win, for example, when running Clojure applications in docker on Elastic Beanstalk, etc. (We'll write that up, too.)

Re: ClojureScript Builds, Rebooted

#25

Post by weavejester »

michaniskin wrote: post #23
weavejester wrote: post #20

Earlier quoted context omitted.

> Effects are a very natural mental model for a great deal of problems and constraining yourself to purity is both impractical and quickly experiences diminishing returns. I can't personally recall a problem where purity was feasible but impractical, though I can think of a few examples of the opposite. > If you intercept all file IO, you can recover the same data. Yes, if you record all I/O, you could restore files…

The problem with the read-everything-into-memory approach is that this is not how the JVM ecosystem works. Things written for the JVM use the classpath primarily, and things in memory secondarily if at all. We can't control the fact that the CLJS compiler, for example, is looking for source files on the classpath instead of in some FileSet object proxy. If we admit the use of tools written by the Java community at la…

Reading everything into memory wasn't supposed to be a complete solution. Not everything can be that simple. However, it seems to me to be better to start from a simple base and add complexity in as necessary, then start from a complex base and try to achieve simplicity.

But let's run with the idea of loading everything into some immutable in-memory data structure, just to see where it goes. So long as we write everything in Clojure we're fine, but the moment we start hitting things adapted for the JVM, such as the CLJS compiler, we run into problems as you point out.

However, it's not too hard to conceive of possible solutions. Let's start with a simple, but naive way around it. We'll take the files in memory, write them to a temporary directory, and then generate a CLJS compiler with a classpath pointing to that directory. When the compiler is done, we take the result and load it into memory again.

Again, this is solution that aims for simplicity rather than performance, but optimisations immediately suggest themselves. If the files exist on disk, we symlink them or point the classpath directly at them. If we don't need the CLJS output file's content, we can defer loading it into memory.

Re: ClojureScript Builds, Rebooted

#26

Post by bsima »

michaniskin wrote: post #24
brandonbloom wrote: post #22

Earlier quoted context omitted.

> it's really useful just ergonomically to be able to use command line arguments to configure ad-hoc builds It's largely also what contributes to "works for me" build environments... It's better to have a just one way to do it interface and discourage excessive tinkering with parameters. The more parameters, the more likely for your dev env to be unstable across individual checkouts or developers. I know it's idealis…

Your point about repeatability is valid. As a policy matter we would never advocate building a project without codifying the process as zero-arg tasks in the build.boot file. You can see this in our own projects (eg. https://github.com/tailrecursion/boot-useful/blob/master/bui... ). This project has one way to build the project jar file: (deftask build-jar "Build project jar file." [] (comp (pom) (add-src) (jar) (ins…

> running Clojure applications in docker on Elastic Beanstalk, etc. (We'll write that up, too.)

Yes. I would love to see a writeup on that.

Re: ClojureScript Builds, Rebooted

#27

Post by michaniskin »

weavejester wrote: post #25
michaniskin wrote: post #23

Earlier quoted context omitted.

The problem with the read-everything-into-memory approach is that this is not how the JVM ecosystem works. Things written for the JVM use the classpath primarily, and things in memory secondarily if at all. We can't control the fact that the CLJS compiler, for example, is looking for source files on the classpath instead of in some FileSet object proxy. If we admit the use of tools written by the Java community at la…

Reading everything into memory wasn't supposed to be a complete solution. Not everything can be that simple. However, it seems to me to be better to start from a simple base and add complexity in as necessary, then start from a complex base and try to achieve simplicity. But let's run with the idea of loading everything into some immutable in-memory data structure, just to see where it goes. So long as we write every…

Haha, yes! Now we're cookin'! The "simple, but naive way" you describe above is pretty much the way boot does things. I'd say you could look at the boot cljs task to see this but setting up the environment for the CLJS compiler is pretty tricky so the code there isn't as clear and elegant as I'd like.

In boot tasks don't fish around in the filesystem to find things. Not for input nor output. Tasks obtain the list of files they can access via functions: boot.core/src-files, boot.core/tgt-files, et al. These functions return immutable sets of java.io.File objects. However, these Files are usually temp files managed by boot.

Boot does things like symlinking (actually we use hard links to get structural sharing, and the files are owned by boot so we don't have cross-filesystem issues to worry about), and we shuffle around with classpath directories and pods.

So stay tuned for the write-up of the filesystem stuff, I think it might be right up your alley!

Re: ClojureScript Builds, Rebooted

#28

Post by weavejester »

michaniskin wrote: post #27
weavejester wrote: post #25

Earlier quoted context omitted.

Reading everything into memory wasn't supposed to be a complete solution. Not everything can be that simple. However, it seems to me to be better to start from a simple base and add complexity in as necessary, then start from a complex base and try to achieve simplicity. But let's run with the idea of loading everything into some immutable in-memory data structure, just to see where it goes. So long as we write every…

Haha, yes! Now we're cookin'! The "simple, but naive way" you describe above is pretty much the way boot does things. I'd say you could look at the boot cljs task to see this but setting up the environment for the CLJS compiler is pretty tricky so the code there isn't as clear and elegant as I'd like. In boot tasks don't fish around in the filesystem to find things. Not for input nor output. Tasks obtain the list of…

It sounds like there's a lot in Boot I'd like, particularly in how it deals with the filesystem. I'm still not convinced about the design, but it's clear I don't know enough about it to make a decision on it.

If nothing else, I'm sure there will be parts in it I'll want to steal ;)

Re: ClojureScript Builds, Rebooted

#29

Post by brandonbloom »

weavejester wrote: post #20
brandonbloom wrote: post #17

Earlier quoted context omitted.

> removing side-effects when possible will always be better than merely restricting them I disagree. Effects are a very natural mental model for a great deal of problems and constraining yourself to purity is both impractical and quickly experiences diminishing returns. Furthermore, if you can intercept effects, you can impose purity upon them. For an extreme example, consider application virtualization and container…

> Effects are a very natural mental model for a great deal of problems and constraining yourself to purity is both impractical and quickly experiences diminishing returns. I can't personally recall a problem where purity was feasible but impractical, though I can think of a few examples of the opposite. > If you intercept all file IO, you can recover the same data. Yes, if you record all I/O, you could restore files…

> Yes, if you record all I/O, you could restore files that have been deleted by a previous task.

I guess I wasn't very clear either. I didn't mean deleted files. I meant the metadata you'd get from a "declarative" build.

Post reply on HN