Live data from Hacker News

Fast Python loops

python.org

31–40 of 54 posts

Re: Fast Python loops

#31
post #6

The author states: "There's a general technique to avoid quadratic behavior in algorithms like this. I coded it as follows for strings of exactly 256 items:" def f5(list): string = "" for i in range(0, 256, 16): # 0, 16, 32, 48, 64, ... s = "" for character in map(chr, list[i:i+16]): s = s + character string = string + s return string I am not understanding what the technique is or why using a step size of 16 in the…

Found a long but thorough answer here - doesn't become linear but does go down to n log n. The step size of 16 is chosen as it is √256

http://cs.stackexchange.com/questions/52360/from-guidos-essa...

Re: Fast Python loops

#34
post #6

The author states: "There's a general technique to avoid quadratic behavior in algorithms like this. I coded it as follows for strings of exactly 256 items:" def f5(list): string = "" for i in range(0, 256, 16): # 0, 16, 32, 48, 64, ... s = "" for character in map(chr, list[i:i+16]): s = s + character string = string + s return string I am not understanding what the technique is or why using a step size of 16 in the…

The idea is to reduce the amount of redundant copying of characters: you end up doing a few more concatenations in the outer loop, but the concatenations in the inner loop are of short strings.

Importantly, if you remove the restriction of the input list being "exactly 256 items", then the method is still quadratic.

A linear-time algorithm for this would copy each input character exactly once, which is effectively what the method based on array.tostring() does.

The chunk size of 16 is not as significant as the technique of constructing+concatenating chunks, although it is optimal for input length 256. In general I think you'd want a chunk size about the square-root of the expected input length, to minimise the number of copied characters.

EDIT: maths

Concatenating strings of length M and N is linear in O(M+N), because that's how many characters you're copying.

Number of characters copied if you construct a string of length N by concatenating one character at a time

  = (0+1) + (1+1) + (2+1) + (3+1) + ... + ((N-1)+1) + (N+1)
  = (N +1)*N / 2
Number of characters copied if you construct a string of length N by concatenating a chunk of length 16 each time

  = (0+16) + (16+16) + (32+16) + ... + ((N-16)+16)
  = 16*(0+1) + 16*(1+1) + 16*(2+1) + ...
  = 16 * (N/16 +1)*(N/16) / 2
         ^^^^^^^^^^^^^^^
This is where the "technique" comes in: although the algorithm is still quadratic you're effectively moving a constant factor out the front.

Note that you also have the cost of constructing the chunks each time, which becomes the dominant cost if you have too many chunks.

In general, if you have a length kM string which you construct from k chunks of length M, the number of characters copied

  = M * (k+1)*k / 2  +  k * (M+1)*M / 2
... which (rounding and integer constraints aside) is minimised for M = k, i.e. when the chunk size is the square root of the input length. Hence, for input length 256 we take chunk size 16.

Re: Fast Python loops

#35
post #14

Earlier quoted context omitted.

It would fail when you pass it 128. He's passing in values from 0-255. "Ascii" is sort of a misnomer in his post.

It's better to fail than to convert something that probably isn't latin1!

The latin1 codec doesn't fail for any values in the 0-255 range. It probably should, since there are values that don't map to a valid character. I don't know if this is deliberate and guaranteed or just an artifact of the current implementation. There should have been a 'byte' encoding that was explicitly made for 1:1 conversions.

Re: Fast Python loops

#37
I didn't know about the array library, but

    "".join(map(chr, list))
would have been my first choice, partly for style and partly for avoiding string concatenation (recent speedups aside).

Re: Fast Python loops

#38

I didn't know about the array library, but "".join(map(chr, list)) would have been my first choice, partly for style and partly for avoiding string concatenation (recent speedups aside).

or `"".join([chr(ch) for ch in s])`. It's only 2-3x slower than the array solution, but a lot more rubust.

The main issue IIUC is the original suggestion was O(N^2).

Re: Fast Python loops

#39
I think this is the most idiomatic way of doing it in modern python:

> "".join(chr(x) for x in list_of_ints)

This article is really really really out of date

Re: Fast Python loops

#40

doesn't 'b' in array.array('b'... and 'B' denote unsigned and signed conversion, thus the 0-256 as desired? or am i missing something?

> Note: since this essay was written, the 'B' typecode was added to the array module, which stores unsigned bytes, so there's no reason to prefer g1() any more.
Post reply on HN