Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Origins of Python's “Functional” Features (2009)

python-history.blogspot.com

Post reply on HN
Print view View original

11–20 of 37 posts

Re: Origins of Python's “Functional” Features (2009)

#11

Post by clusmore »

m_mueller wrote: post #8
aftbit wrote: post #7

Earlier quoted context omitted.

You could also use the list comprehension form of that: [x for x in mixed_widgets if 'widgets' in x][0]['widgets'] More readable? I dunno. More "Pythonic"? Definitely. Related: I wish the list type in Python included an analogue to dict's ".get(key, default)" operation.

definitely more readable - it describes in basic english words what it's doing and it's basically the same as mathematical set notation. The main question to me is the performance implication though. Are generator expressions doing the iteration at C level like map and does that give performance parity then? What about branching - am I correctly assuming that filter is always faster than comprehensions or generator e…

Must watch: https://www.youtube.com/watch?v=OSGv2VnC0go

I believe that the list comprehension will be faster than filter, but as always, any time you replace readable code with unreadable code for performance reasons, you damn well better time it.

Re: Origins of Python's “Functional” Features (2009)

#12

Post by m_mueller »

clusmore wrote: post #11
m_mueller wrote: post #8

Earlier quoted context omitted.

definitely more readable - it describes in basic english words what it's doing and it's basically the same as mathematical set notation. The main question to me is the performance implication though. Are generator expressions doing the iteration at C level like map and does that give performance parity then? What about branching - am I correctly assuming that filter is always faster than comprehensions or generator e…

Must watch: https://www.youtube.com/watch?v=OSGv2VnC0go I believe that the list comprehension will be faster than filter, but as always, any time you replace readable code with unreadable code for performance reasons, you damn well better time it.

That's actually the talk I was thinking about, but I guess I forgot how exactly Raymond described generator expressions there. Thanks for linking it again!

Re: Origins of Python's “Functional” Features (2009)

#13

Post by viraptor »

clusmore wrote: post #11
m_mueller wrote: post #8

Earlier quoted context omitted.

definitely more readable - it describes in basic english words what it's doing and it's basically the same as mathematical set notation. The main question to me is the performance implication though. Are generator expressions doing the iteration at C level like map and does that give performance parity then? What about branching - am I correctly assuming that filter is always faster than comprehensions or generator e…

Must watch: https://www.youtube.com/watch?v=OSGv2VnC0go I believe that the list comprehension will be faster than filter, but as always, any time you replace readable code with unreadable code for performance reasons, you damn well better time it.

~4.5 times faster with comprehension.

    In [11]: timeit.timeit('''list(filter(lambda x : ('widgets' in x), mixed_widgets))[0]['widgets']''', '''nw={'abc': 'def'};w={'widgets':'s'};mixed_widgets=[nw]*100+[w]+[nw]*100''', number=100000)
    Out[11]: 2.6956532129988773

    In [12]: timeit.timeit('''[x for x in mixed_widgets if 'widgets' in x][0]['widgets']''', '''nw={'abc': 'def'};w={'widgets':'s'};mixed_widgets=[nw]*100+[w]+[nw]*100''', number=100000)
    Out[12]: 0.5911771030077944
But not generating the list at all is still going to be faster (with bigger gains for bigger data)

    In [13]: timeit.timeit('''next(x for x in mixed_widgets if 'widgets' in x)['widgets']''', '''nw={'abc': 'def'};w={'widgets':'s'};mixed_widgets=[nw]*100+[w]+[nw]*100''', number=100000)
    Out[13]: 0.3324074839911191

Re: Origins of Python's “Functional” Features (2009)

#14

Post by m_mueller »

viraptor wrote: post #13
clusmore wrote: post #11

Earlier quoted context omitted.

Must watch: https://www.youtube.com/watch?v=OSGv2VnC0go I believe that the list comprehension will be faster than filter, but as always, any time you replace readable code with unreadable code for performance reasons, you damn well better time it.

~4.5 times faster with comprehension. In [11]: timeit.timeit('''list(filter(lambda x : ('widgets' in x), mixed_widgets))[0]['widgets']''', '''nw={'abc': 'def'};w={'widgets':'s'};mixed_widgets=[nw]*100+[w]+[nw]*100''', number=100000) Out[11]: 2.6956532129988773 In [12]: timeit.timeit('''[x for x in mixed_widgets if 'widgets' in x][0]['widgets']''', '''nw={'abc': 'def'};w={'widgets':'s'};mixed_widgets=[nw]*100+[w]+[nw]…

I guess my next question is then - why is anyone using filter/map instead of comprehensions or even generator expressions? Familiarity when coming from FP?

Re: Origins of Python's “Functional” Features (2009)

#15

Post by viraptor »

m_mueller wrote: post #14
viraptor wrote: post #13

Earlier quoted context omitted.

~4.5 times faster with comprehension. In [11]: timeit.timeit('''list(filter(lambda x : ('widgets' in x), mixed_widgets))[0]['widgets']''', '''nw={'abc': 'def'};w={'widgets':'s'};mixed_widgets=[nw]*100+[w]+[nw]*100''', number=100000) Out[11]: 2.6956532129988773 In [12]: timeit.timeit('''[x for x in mixed_widgets if 'widgets' in x][0]['widgets']''', '''nw={'abc': 'def'};w={'widgets':'s'};mixed_widgets=[nw]*100+[w]+[nw]…

I guess my next question is then - why is anyone using filter/map instead of comprehensions or even generator expressions? Familiarity when coming from FP?

In my experience filter/map are used by people who just don't know about comprehensions, or are not used to having them available. It takes some time to start using them where properly.

Re: Origins of Python's “Functional” Features (2009)

#16

Post by tommikaikkonen »

While it's far from a functional language, I find that it's possible to build functional style abstractions without too much headache. The iterator protocol is so well supported by the language and the standard libraries that using map, filter, itertools, generator expressions & comprehensions gets you a long way. You get the rest of the way by writing any specialized combinators you need and then imperative code to tie things together. You can build up a lazy pipeline of data transformations that works pretty well.

There are some gotchas where you need to explicitly copy (itertools.tee) iterators when you have multiple consumers, and be careful with mutable values, but it's manageable.

It would be nice to use curried functions as functools.partial gets verbose, but at that point you're far from Pythonic code.

Re: Origins of Python's “Functional” Features (2009)

#17

Post by clusmore »

m_mueller wrote: post #14
viraptor wrote: post #13

Earlier quoted context omitted.

~4.5 times faster with comprehension. In [11]: timeit.timeit('''list(filter(lambda x : ('widgets' in x), mixed_widgets))[0]['widgets']''', '''nw={'abc': 'def'};w={'widgets':'s'};mixed_widgets=[nw]*100+[w]+[nw]*100''', number=100000) Out[11]: 2.6956532129988773 In [12]: timeit.timeit('''[x for x in mixed_widgets if 'widgets' in x][0]['widgets']''', '''nw={'abc': 'def'};w={'widgets':'s'};mixed_widgets=[nw]*100+[w]+[nw]…

I guess my next question is then - why is anyone using filter/map instead of comprehensions or even generator expressions? Familiarity when coming from FP?

Well, for one I believe they pre-date comprehensions. Having them as functions is also occasionally useful for partial application, e.g.

  from functools import partial
  to_strings = partial(map, str)
  # vs
  def to_strings(seq):
      return (str(elem) for elem in seq)

Re: Origins of Python's “Functional” Features (2009)

#18

Post by tommikaikkonen »

Alex3917 wrote: post #5
foota wrote: post #4

Earlier quoted context omitted.

Which is supposed to be functional in that example?

The one line would be functional. Here is a good example: list(filter(lambda x : ('widgets' in x), mixed_widgets))[0]['widgets'] I almost always go for more lines of readable code rather than less lines of unreadable code. But in cases like this, you can either write that as one line of garbage unreadable code or six lines of garbage unreadable code. So I'd rather just leave the overall codebase more dense to make it…

Implementing a find function makes this read a lot nicer. You could also call it find_first or find_next.

    def find(predicate, iterable, default=None):
        """Returns the first value that matches predicate, otherwise default=None"""
        return next(
            (x for x in iterable if predicate(x)),
            default
        )
Which turns the expression to

    find(lambda x: 'widgets' in x, mixed_widgets)['widgets']

Re: Origins of Python's “Functional” Features (2009)

#19

Post by m_mueller »

clusmore wrote: post #17
m_mueller wrote: post #14

Earlier quoted context omitted.

I guess my next question is then - why is anyone using filter/map instead of comprehensions or even generator expressions? Familiarity when coming from FP?

Well, for one I believe they pre-date comprehensions. Having them as functions is also occasionally useful for partial application, e.g. from functools import partial to_strings = partial(map, str) # vs def to_strings(seq): return (str(elem) for elem in seq)

I can see it together with partial, yes, that's when it can become a bit cleaner. Another reason why I use map is when I want to use multiprocessing or multithreading (with IO heavy functions). But on a fine grained level of code I find it really hurts readability compared to comprehensions.
Post reply on HN