Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Show HN: ZFS Implementation in Python

github.com

Post reply on HN
Print view View original

91–97 of 97 posts

Re: Show HN: ZFS Implementation in Python

#91

Post by int_19h »

twa927 wrote: post #84
weberc2 wrote: post #64

Earlier quoted context omitted.

Python is also a GC’d language...

CPython is mostly reference-counted.

With a synchronous garbage collector for cycles. Which is like the worst of both worlds, since you get the constant overhead of refcounting, plus unpredictable interruptions of unspecified duration that can happen every time a new object that might contain references to other objects is created.

To be fair, the GC can be disabled. But it's only safe to do so when you know there are no cycles, and even when such guarantee can be had for your own code, I've never seen a library guarantee that to API clients.

Re: Show HN: ZFS Implementation in Python

#92

Post by int_19h »

twa927 wrote: post #41
spullara wrote: post #13

Earlier quoted context omitted.

Faster that CPython doesn't mean it is fast.

I was trying to speed up a log processing service running on PyPy by rewriting it in Java. I was surprised that the result was about twice slower (I know Java quite well and I didn't see obvious optimizations; most of the time was spent in GC). So it can be quite fast even in more absolute terms (VM languages), at least for some types of code.

I had a binary parser written in Python that took around 30 seconds on typical input on CPython. PyPy took that down to about 10 seconds. Rewriting it in C# took it down to 200 ms.

Re: Show HN: ZFS Implementation in Python

#94

Post by Annatar »

lunixbochs wrote: post #2

Note: this was implemented without referencing any ZFS source code and should not be subject to the CDDL.

CDDL isn't a fascist license like GPL, so there is no penalty for looking at the source code licensed under CDDL. Let's leave the GPL ideology out of this.

Re: Show HN: ZFS Implementation in Python

#95

Post by twa927 »

int_19h wrote: post #92
twa927 wrote: post #41

Earlier quoted context omitted.

I was trying to speed up a log processing service running on PyPy by rewriting it in Java. I was surprised that the result was about twice slower (I know Java quite well and I didn't see obvious optimizations; most of the time was spent in GC). So it can be quite fast even in more absolute terms (VM languages), at least for some types of code.

I had a binary parser written in Python that took around 30 seconds on typical input on CPython. PyPy took that down to about 10 seconds. Rewriting it in C# took it down to 200 ms.

If this was using a loop processing a single byte in an iteration I would expect a greater speedup on PyPy. I've seen 100x speedup in such cases.

Re: Show HN: ZFS Implementation in Python

#96

Post by int_19h »

twa927 wrote: post #95
int_19h wrote: post #92

Earlier quoted context omitted.

I had a binary parser written in Python that took around 30 seconds on typical input on CPython. PyPy took that down to about 10 seconds. Rewriting it in C# took it down to 200 ms.

If this was using a loop processing a single byte in an iteration I would expect a greater speedup on PyPy. I've seen 100x speedup in such cases.

Not single byte, but individual fields (float32/int32/string etc). Yes, I expected a much more significant speed-up as well. It's probably because a lot of that code was driven by reflection-type techniques.

Curiously, IronPython did better than anything (but still slow). Haven't tried Jython.

Compiling the whole thing with Cython was less effective than PyPy.

Re: Show HN: ZFS Implementation in Python

#97

Post by twa927 »

int_19h wrote: post #96
twa927 wrote: post #95

Earlier quoted context omitted.

If this was using a loop processing a single byte in an iteration I would expect a greater speedup on PyPy. I've seen 100x speedup in such cases.

Not single byte, but individual fields (float32/int32/string etc). Yes, I expected a much more significant speed-up as well. It's probably because a lot of that code was driven by reflection-type techniques. Curiously, IronPython did better than anything (but still slow). Haven't tried Jython. Compiling the whole thing with Cython was less effective than PyPy.

Yes, reflection voids many JIT paths in PyPy, AFAIK. Maybe it was worth rewriting the Python code to get rid of the reflection?
Post reply on HN