Brilliant or insane code?

There's also flexibility improvements to do with lazy evaluation, and it makes the Twisted guys not hate you.

it depends on the kind of code you write. i guess if you're writing web server stuff, documenting this makes sense. but in maths-related code, it's pretty standard. you use something very similar to transpose matrices, for example.

(and the original article is dealing with coords in graphics, which is "maths-related code" in my book, but perhaps not in everyone's)

I made a few more interesting (to me) measurements. As always, you have to measure your performance with your actual input data to see what's "best".

Test 1: Boring, small array of integers

    In [28]: arr = range(0, 300)

    In [29]: %timeit [(arr[3*x], arr[3*x+1], arr[3*x+2]) for x in range(len(arr)/3)]
    10000 loops, best of 3: 27.2 us per loop

    In [30]: %timeit numpy.reshape(arr, (-1, 3))
    10000 loops, best of 3: 45.2 us per loop

    In [31]: %timeit zip(*([iter(arr)]*3))
    100000 loops, best of 3: 6.25 us per loop

This roughly matches the article's timing ratios, so far so good.

Test 2: Use numpy's random number generation to get a small array of floats

    In [32]: arr = numpy.random.ranf(300)

    In [33]: %timeit [(arr[3*x], arr[3*x+1], arr[3*x+2]) for x in range(len(arr)/3)]
    10000 loops, best of 3: 54 us per loop

    In [34]: %timeit numpy.reshape(arr, (-1, 3))
    1000000 loops, best of 3: 1.06 us per loop

    In [35]: %timeit zip(*([iter(arr)]*3))
    10000 loops, best of 3: 39.7 us per loop

numpy is two orders of magnitude faster here; it's evidently using a highly optimized internal codepath for random sequence generation, which I'd guess is a common thing to do in numeric analysis. I assume it's using a generator, so there's no actual array being created, blowing up the CPU cache lines etc.

Test 3: Verify that analysis by interfering with numpy

    In [36]: arr = [x for x in numpy.random.ranf(300)]

    In [37]: %timeit [(arr[3*x], arr[3*x+1], arr[3*x+2]) for x in range(len(arr)/3)]
    10000 loops, best of 3: 26.2 us per loop

    In [38]: %timeit numpy.reshape(arr, (-1, 3))
    10000 loops, best of 3: 48.5 us per loop

    In [39]: %timeit zip(*([iter(arr)]*3))
    100000 loops, best of 3: 6.55 us per loop

Yep.

Test 4: Larger data set, no interference

    In [40]: arr = numpy.random.ranf(3000000)

    In [41]: %timeit [(arr[3*x], arr[3*x+1], arr[3*x+2]) for x in range(len(arr)/3)]
    1 loops, best of 3: 624 ms per loop

    In [42]: %timeit numpy.reshape(arr, (-1, 3))
    1000000 loops, best of 3: 1.06 us per loop

    In [43]: %timeit zip(*([iter(arr)]*3))
    1 loops, best of 3: 335 ms per loop

The numpy time doesn't change at all from test 2 despite the larger size, but the others suffer. Again, I suspect numpy is being intelligent here; my guess is that it doesn't actually apply the function and generate the real output, it just wraps the random generator in another one.

Test 5: Larger data set, interfering with numpy

    In [44]: arr = [x for x in numpy.random.ranf(3000000)]

    In [45]: %timeit [(arr[3*x], arr[3*x+1], arr[3*x+2]) for x in range(len(arr)/3)]
    1 loops, best of 3: 321 ms per loop

    In [46]: %timeit numpy.reshape(arr, (-1, 3))
    1 loops, best of 3: 354 ms per loop

    In [47]: %timeit zip(*([iter(arr)]*3))
    10 loops, best of 3: 83.6 ms per loop

There we go; we're back to roughly the original timing ratios.

So, surprise! You always have to measure. Measure, measure measure. My bias is to write code first for legibility and modifiability, and then optimize hot spots if needed (and add comments, please, when you do so).

Without doing deeper analysis I'd say one moral of the Python story is, this shows the potential power of generators. But in real-world data sets this isn't always ideal -- is it faster to load up the whole data set in memory and blast through it, or load it from disk on demand with a generator? In really high performance scenarios, is it faster to preprocess the data to fit into the CPU's cache lines? You can't tell without measuring, and you have to measure in the environment you're deploying to, since the answer may be different on a machine with 1GB RAM vs. one with 128GB RAM, or 32KB L1 cache vs. 8KB.

Recently I was downvoted 2-3 times on S.O. for an answer that was claimed to be non-idiomatic. So, I cleaned it up, but it really irked me. What I had written was totally fine. It shouldn't have hurt anyone's eyes. It was direct. It was in-your-face. It was not magic.

Reading this post brought back that feeling.

If people don't understand a completely valid and terse way of coding something, sometimes instead of bothering to understand it, they will bash it. Sometimes, this is a totally valid way to vent frustration, and then they learn something new, and all is good. But sometimes, it just gets left as "this is wrong" and then someone else thinks it is wrong, and so on. That is wrong, and tech leads or architects that enforce such crap will bug the living shit out of good developers and lose them.

I appreciate clarity. But, terse one-liners can be just as clear if not clearer than code that unnecessarily adds more methods/functions/names/local vars and claims to be "more testable", etc.

You shouldn't have to sacrifice the ability to be terse and clear at the same time. Testing is no excuse for code bloat. You can likely write a test that executes the behavior without having to atomize it. Assess the amount of production and test code you are writing. How much more code are you actually having to write in order to test, both in the tests themselves and in the code which you are having to test?

It happened here with Python and it happens in many languages. It even happens with laws and regulations in government. If someone gets the same thing done just as ethically but without the bureaucracy, just appreciate it as another perhaps better way of doing something. Don't bash it publically because you don't understand it.

    The left-to-right evaluation order of the
    iterables is guaranteed.

http://docs.python.org/2/library/functions.html#zip