Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Rich Hickey – Inside Transducers [video]

youtube.com

Post reply on HN
Print view View original

21–30 of 50 posts

Re: Rich Hickey – Inside Transducers [video]

#21

Post by kbeaty »

moomin wrote: post #14
nickik wrote: post #2

Again a more technical talk. I really liked it. Specially the end bit about pipelines, that seams like it would be really usful. Transducers overall are quite intressting, I am exited to see what other transducer context people come up with.

It's not that clear there will be one. Transducers have three steps: begin, during and end. In practice, start is only used for reduce operations and outside of reduce operations end can only be used for its side effects. There's also definitional issues e.g. can you have an async transducer?

You can indeed use transducers in asynchronous contexts. In fact, that is what intrigues me most about the abstraction.

Transducers are defined such that you can abstract the context of input, output and iteration and focus on the transformation of each element from an independent source individually. The implementation of each transducer accepts another transformation in a "pipeline" (eventually the output sink) and accepts an input during an external iteration process. The implementation decides what to do with it (map changes with a function, filter ignores certain values with a predicate, etc.) You can also define transducers that send multiple values for each input (think string.split) or some that do not send any until completion (think buffered results).

Since the iteration source and sink are abstracted from the transformation, you can use the same transformation in other contexts (Promises, event streams, IO streams, CSP, etc.)

I've been experimenting with transducers in asynchronous contexts in JavaScript if anyone is interested, for example:

[1]: http://simplectic.com/projects/underscore-transducer/ [2]: http://simplectic.com/projects/underarm/ [3]: https://github.com/transduce/transduce-async [4]: https://github.com/kevinbeaty/transduce-stream [5]: https://github.com/transduce/transduce-push [6]: https://github.com/transduce/transduce-then

Re: Rich Hickey – Inside Transducers [video]

#25

Post by nilliams »

ultimape wrote: post #3

For those not familiar with Clojure, here's a great demonstration of the concept done up in JavaScript: http://jlongster.com/Transducers.js--A-JavaScript-Library-fo...

For JavaScripters, there's also the 'Like Underscore, but lazier' Lazy.js http://danieltao.com/lazy.js/

Re: Rich Hickey – Inside Transducers [video]

#26

Post by drostie »

charlysisto wrote: post #11

So Transducers generalize the usage of Enumerable functions such as map, reduce, filter, flatMap (...). Please could someone tell me if this is conceptually different from ruby's Enumerable module which only needs the class its included in to implement `each` so anything can be enumerable ? Or is it a similar but just in translated to the FP world ?

Transducers are a different kind of thing, not just a generalized usage of enumerable functions.

Clojure transducers are isomorphic to functions of type `a -> [b]`, that is, functions which take some thing and return lists of other things. You can implement map and filter in this language:

    tmap f a = [f a]
    tfilter p a = if p a then [a] else []
You can also compose two such functions:

    concat :: [[x]] -> [x]
    concat [] = []
    concat (x : xs) = x ++ concat xs
    
    concatMap f list = concat (map f list)
    
    compose lb_a lc_b = \a -> concatMap lc_b (lb_a a)
One reduction function (`foldr`) turns out to just be the identity function [a] -> [a], as I recall.

The interesting thing about Clojure transformers is that they have a strange continuation-passing form, such that this function `compose` that I wrote above for the type `a -> [b]` is actually in Clojure just a function composition. That is, instead of `a -> [b]` we see `forall r. (b -> r -> r) -> (a -> r -> r)`, which composes with normal function composition.

Re: Rich Hickey – Inside Transducers [video]

#27

Post by dkersten »

ajuc wrote: post #7
jonahx wrote: post #4

Earlier quoted context omitted.

From that link: "The reduce function is the base transformation; any other transformation can be expressed in terms of it (map, filter, etc)." This seems so obvious in retrospect -- I can't believe I had never made that connection before.

For me the "ephiphany" was when I realized reduce don't need to be (T,T)->T, but can be (T1, T2) -> T1.

They can also be (T1,T2)->T3

Re: Rich Hickey – Inside Transducers [video]

#29

Post by Skinney »

seanmcdirmid wrote: post #17
cfeduke wrote: post #15

Earlier quoted context omitted.

Transducers _can be_ parallelized (TODO) but some transducer implementations do contain state, like `take` and therefore cannot be parallelized. The big thing is no intermediate results.

So transducers are like .Net/LINQ enumerables?

Nah, Clojure seqs are more like .NET/LINQ enumerables.

Functions that work over seqs/enumerables return a new seq/enumerable. From what I understand, and I'm sure I am wrong, a transducer receives and transforms a value, and may call the next transducer with the transformed value.

The nice thing is that a transducer does not create intermediate results (a seq/enumerable), and that it doesn't make any assumptions on the underlying datastructure. So you can, for instance, send a value through a transducer and directly place it in a list, instead of creating a set of enumerables and then call .ToList on that. Since transducers can work with any value, you also aren't tied to enumerables. So transducers should be a little more universal and performant than enumerables.

Re: Rich Hickey – Inside Transducers [video]

#30

Post by nickik »

cfeduke wrote: post #15
YuriNiyazov wrote: post #13

Earlier quoted context omitted.

Yes; There are differences. 1. Transducers are parallel under the covers. Since the expectations is that the code that the predicate or mapping functions that you pass into map, filter and reduce are pure (no variables are changed, no state is modified, just a calculation that is only dependent on arguments) the parallelism is hidden away from you, but it's there. Ruby's Enumerable can't do parallelism 2. When you co…

Transducers _can be_ parallelized (TODO) but some transducer implementations do contain state, like `take` and therefore cannot be parallelized. The big thing is no intermediate results.

The big thing is not even that, the big thing is that you can seperate your logic from the data that you want to process. So instead of (map inc [1 2 3 4]) you can do (def inc-transducer (map inc)) and then use that for any datastructures, channels or whatever other transducer context people come up with. To be sure, you would not do that with (map inc) but if you have more logic, you could.
Post reply on HN