Clojure 1.7 is now available

Transducers create functions that are meant to be passed to reduce (i.e. take accumulator and element and return a new accumulator).

Its possible to implement most other functions (map, filter, take etc.) using reduce. Transducers take advantage of that.

Unfortunately, if you try to do that, you'll notice that the implementation is sometimes tied with the way you build the entity: e.g. for vectors, map implemented in terms of reduce would start with an empty vector then append to it.

That sucks. We want operation chains that are independent of the data structure they operate on. We want them to work on vectors, lists, channels, whatever - anything that can have a reduce-like operation (anything reducible)

However, it turns out you don't necessarily have to recreate the entity (e.g. list) when chaining. All you need to know how to do is invoke the next operation in the chain, i.e. the next reduction function. For example:

* map can apply the function on the element to get a new result, then call the next reduction function with the accumulator and that new result.

* filter can check the element and either return the old accumulator or apply the next reduction function with the element, etc.

Therefore, transducers simply take an additional argument: the next reduction function that should be applied. This lets you build up a chain of operations (e.g. map filter etc) that doesn't care about the kind of entity they're operating on. Only the last "step" must be a reduction function that needs to know how to build the result.

Language design requires tradeoffs: the power of a feature, vs. how much more complex it makes the language to read/learn/etc. Making something easy to use doesn't commit one to anti-intellectualism; it just shifts the field of intellectual challenge elsewhere.

See Yegge's Perl essays[1] for examples of core-language features that are 'powerful' but problematic; hopefully that convinces you that at least the form of argument is legitimate. It could be that those Perl features are bad whereas transducers are great. I happen to share some skepticism of transducers because they overly-resemble existing features (partial application) and can be implemented pretty easy with existing Clojure tools. Also the marquee use case (sharing transformations between sequences and channels) so far seems exceedingly rare to me, though maybe things are moving in a direction where that's more important.

> And like you said, if you don't understand it you don't have to use it.

True to an extent, but (a) the high prominence given to the feature may well lead to a situation where you can't read other peoples' code without learning the concept, and (b) it's easy to create transducers by accident.

[1] e.g. ".." in https://sites.google.com/site/steveyegge2/ancient-languages-...

Much Clojure code relies on applying nested transformations to sequences:

    (reduce + 0 (filter odd? (map inc (range 1000))))

Conceptually, this takes an input sequence of 1000 elements (range 100), then creates an intermediate sequence of (map inc), then creates an intermediate sequence of (filter odd?), then reduces that with +. (I am glossing over a number of internal optimizations - chunking and transients, but the idea holds.)

Transducers let you represent the transformation parts as an independent (reusable) thing:

    (def xf (comp (map inc) (filter odd?)))

You then apply that composite transformation in a single pass over the input:

    (transduce xf + 0 (range 1000))

transduce combines (a) input iteration (b) transformation application and (c) what to do with the results - in this case apply + as a reduce. Other functions also exist that make different choices about how to combine these parts: into collects results into a collection, sequence can incrementally produce a result, eduction can delay execution, core.async chans can apply them in a queue-like way.

There are a number of benefits here:

1. Composability - transformations are isolated from their input and output contexts and thus reusable in many contexts.

2. Performance - the sequence example above allocates two large intermediate sequences. The transduce version does no intermediate allocation. Additionally, some collection inputs are "internally reducible" (including range in 1.7) which allows them to be traversed more quickly via reduce/transduce contexts. The performance benefits of these changes can be dramatic as you increase the size of input data or number of transformations. If you happen to have your data in an input collection (like a vector), into with transducers can be used to move data directly from one vector to another without ever producing a sequence.

3. Eagerness - if you know you will process the entire input into the entire output, you can do so eagerly and avoid overhead. Or you can get laziness via sequence (although there are some differences in what that means with transducers). The upshot is you now have more choices.

4. Resource management - because you have the ability to eagerly process an external input source, you have more knowledge about when it's "done" so you can release the resource.

One advantage to language specific packages is that IDE's have a uniform target across platforms (PyCharm has pip support for example).

That's a win if you switch around a lot.

You may feel less intimidated by "Clojure for the Brave and True." The landing page features a rodent with an eye patch and a sword.

http://www.braveclojure.com/