Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Simple apply/filter/reduce package in Go

github.com

Post reply on HN
Print view View original

21–30 of 68 posts

Re: Simple apply/filter/reduce package in Go

#21

Post by seanmcdirmid »

eropple wrote: post #17
seanmcdirmid wrote: post #9

Earlier quoted context omitted.

Debugging. Try debugging a Select/Where stream in C#; it is a PITA. I often find myself unwinding my list comprehensions into for loops because I need to debug. There is a reason Haskell's type system has to be so strong :)

I was pretty confused by this post for a minute until I remembered I've been on VS2015 since the first preview release. This problem is greatly reduced there.

I saw that in June, and I'm interested in how effective it is, and what the usability characteristics are (it looked complex at the time, but it was an early demo). But going from 2010 to 2013 was a painful, and I don't want to throw this in a VM, so I'll just wait for RTM.

Re: Simple apply/filter/reduce package in Go

#22

Post by adrusi »

drcode wrote: post #8
bunderbunder wrote: post #6

Earlier quoted context omitted.

In most languages, the for loop ends up being faster. Sometimes noticeably so. Me, I generally start with declarative syntax, but the profiler frequently tells me to go back and change it. Being a systems programmer, wonder if it's easier for him to just use the for loop as a default. Performance demands are always high in systems programming (because there'll be a whole stack of additional software standing on top o…

That's the essence of Clojure's transducers http://clojure.org/transducers ...they give you the declarative syntax of filter/reduce/etc but can have the same evaluation strategy as for loops, with comparable performance.

That's not why clojure has transducers. You can use a mapping transducer to express map, but it's actually going to have slightly worse performance than regular map.

There's nothing inherently slow about higher order collection functions, rust has them and they compile to the same machine code as the equivalent loop construct. Its just that most languages implement them on the wrong data structures. Functional languages implement them on lazy lists, which is good, but has overhead. A lot of languages implement them on arrays, which is bad because then it needs to process the whole array at once, and allocate all the memory, even though the next transformation in the pipeline doesn't need all that memory. Rust implements them on iterators, which have all the sane benefits as lazy lists, but fit better into a performance-focused imperative language.

Re: Simple apply/filter/reduce package in Go

#23

Post by ryanthejuggler »

pmahoney wrote: post #13

This isn't "reduce" as I know it. It requires the user function return the same data type as contained by the slice. Furthermore, for a slice of size 1, it simply returns that single element. case 1: return in.Index(0) ... if !goodFunc(fn, elemType, elemType, elemType) { ... panic } So I could not, for example, reduce a slice of numbers into a struct of (min,max,mean).

The "official" way to do this is map, then reduce. The way reduce is meant to be used exactly matches this implementation.

Consider that you have a large quantity of numbers that you want to get the min, max, mean for. If you write something like the following:

    function minMaxMean(list) {
      return list.reduce(function(lastState, n) {
        var min = lastState.min,
            max = lastState.max,
            sum = lastState.sum,
            count = lastState.count;
        if(n  max) max = n;
        sum += n;
        count += 1;
        return {
          min: min,
          max: max,
          sum: sum,
          count: count
        }
      }, {min: Infinity, max: -Infinity, sum: 0, count: 0});
    }
...then you're assuming that the reduce function will run once, over a single list of numbers, in order from left to right. However, if you implement it as the following:

    function minMaxMean(list) {
      return list.map(function(n){
        return {
          min: n,
          max: n,
          sum: n,
          count: 1
        }
      }).reduce(function(a, b) {
        return {
          min: (a.min  b.max ? a.max : b.max),
          sum: a.sum + b.sum,
          count: a.count + b.count
        }
      });
    }
...then you can distribute this out across multiple threads/machines/etc, update it when new data comes in, reduce in any order.

Re: Simple apply/filter/reduce package in Go

#24

Post by fosforsvenne »

mhd wrote: post #18
fosforsvenne wrote: post #15

Earlier quoted context omitted.

The letter W isn't used in Swedish except in proper names and the occasional loanword.

And just when I felt so smart about using "ö" and not "ø" for fake furniture…

Actually, I both noticed and appreciated that :). And a lot of IKEA product names are just first names anyway, so it _was_ overly pedantic.

Re: Simple apply/filter/reduce package in Go

#25

Post by epidemian »

hyperpape wrote: post #19
falcolas wrote: post #2

But you can't write generic functions in Go! Oh, wait. He just did. As a caveat to my exasperated sarcasm, I do realize he's using reflection to identify and type the data at runtime, as opposed to compile time as with C++ templating, but this is kind of generalization is still quite useful when writing general purpose library code. Personally, I'd not be inclined to use this either, the number of times I've actually…

This only handles functions of type a -> a -> a ( https://github.com/robpike/filter/blob/master/reduce.go ), whereas a generic reduce takes functions of type a -> a -> b. So this is certainly not proof that you can write generics in go. See also pmahoney's comment in this thread: https://news.ycombinator.com/item?id=9315721 .

Minor nitpick: a generic reduce should take functions of type a -> b -> a, where a is the type of the reduced values, and b is the type of the elements in the sequence to be reduced.

Re: Simple apply/filter/reduce package in Go

#26

Post by espadrine »

ryanthejuggler wrote: post #23
pmahoney wrote: post #13

This isn't "reduce" as I know it. It requires the user function return the same data type as contained by the slice. Furthermore, for a slice of size 1, it simply returns that single element. case 1: return in.Index(0) ... if !goodFunc(fn, elemType, elemType, elemType) { ... panic } So I could not, for example, reduce a slice of numbers into a struct of (min,max,mean).

The "official" way to do this is map, then reduce. The way reduce is meant to be used exactly matches this implementation. Consider that you have a large quantity of numbers that you want to get the min, max, mean for. If you write something like the following: function minMaxMean(list) { return list.reduce(function(lastState, n) { var min = lastState.min, max = lastState.max, sum = lastState.sum, count = lastState.c…

There are many cases of elegantly using reduce() with a different return type than the list's item type.

Here is a JS function which computes the combined length of all strings in a list:

    stringsLen = (strings) => strings.reduce((acc, item) => acc += item.length, 0);
    
    stringsLen(['hello', 'world'])  //> 10
Sure, you can argue that this works, but it misses the point.

    stringsLen = (strings) => strings.map(s => s.length).reduce((acc, n) => acc += n, 0);
    
     stringsLen(['hello', 'world'])

Re: Simple apply/filter/reduce package in Go

#27

Post by [deleted] »

bunderbunder wrote: post #6
inglor wrote: post #4

I don't get why someone would rather write a for loop than use declarative data syntax. If I want to get the names of all administrators doing `users.Where(user => user.isAdmin()).Select(user => user.Name)` is so much nicer than using a for loop - or maybe he's suggesting we start writing FOR loops instead of SQL for our databases too?

In most languages, the for loop ends up being faster. Sometimes noticeably so. Me, I generally start with declarative syntax, but the profiler frequently tells me to go back and change it. Being a systems programmer, wonder if it's easier for him to just use the for loop as a default. Performance demands are always high in systems programming (because there'll be a whole stack of additional software standing on top o…

[deleted]

Re: Simple apply/filter/reduce package in Go

#28

Post by amscanne »

hyperpape wrote: post #19
falcolas wrote: post #2

But you can't write generic functions in Go! Oh, wait. He just did. As a caveat to my exasperated sarcasm, I do realize he's using reflection to identify and type the data at runtime, as opposed to compile time as with C++ templating, but this is kind of generalization is still quite useful when writing general purpose library code. Personally, I'd not be inclined to use this either, the number of times I've actually…

This only handles functions of type a -> a -> a ( https://github.com/robpike/filter/blob/master/reduce.go ), whereas a generic reduce takes functions of type a -> a -> b. So this is certainly not proof that you can write generics in go. See also pmahoney's comment in this thread: https://news.ycombinator.com/item?id=9315721 .

You could very easily relax that constraint by changing a couple of lines. Rob Pike has simply chosen the classic MapReduce form.

I'm not sure why you would need "proof" that generics are possible in Go. You have reflection and type assertions, and their capabilities are well documented. Does it give you generics? All depends on your definition of generics.

Re: Simple apply/filter/reduce package in Go

#29

Post by TheCoelacanth »

falcolas wrote: post #2

But you can't write generic functions in Go! Oh, wait. He just did. As a caveat to my exasperated sarcasm, I do realize he's using reflection to identify and type the data at runtime, as opposed to compile time as with C++ templating, but this is kind of generalization is still quite useful when writing general purpose library code. Personally, I'd not be inclined to use this either, the number of times I've actually…

It's 30 lines long (the equivalent in Haskell would be 1 fairly simple line of code). Type errors are detected at runtime instead of at compile type, and it's not as generic as actual reduce.

Re: Simple apply/filter/reduce package in Go

#30

Post by topkekz »

metaphorm wrote: post #10
svisser wrote: post #7

Earlier quoted context omitted.

It's easy to dismiss an opinion but that doesn't mean it's wrong. For example, if you like filter/reduce you may criticize languages that don't encourage it (Go, Python, etc). But often this leads to copying ideas from one language resulting in code that's hard to maintain in another language which encourages different ways of expressing the same logic.

filter and reduce are both builtins in Python and I see nothing discouraging anyone from taking full advantage of them.

http://www.artima.com/weblogs/viewpost.jsp?thread=98196

http://python-history.blogspot.com/2009/04/origins-of-python...

Post reply on HN