Show HN: ZFS Implementation in Python • Hacker News Board

Re: Show HN: ZFS Implementation in Python

#51

Post by tigershark » Sat, May 04, 2019, 9:55 PM UTC

twa927 wrote: post #48

tigershark wrote: post #46

Earlier quoted context omitted.

I know that it’s asking a lot, but any chance that you can post a minimum reproducible sample? From what I know it is quite smelly...

I don't have access to this codebase now but I'll try to write some benchmark.

If you manage to do it would be awesome, otherwise thanks a lot anyway for the effort :) I’m just curious to understand why it happens because it’s exactly the opposite of what I would expect. The only explanation that comes to my mind is excessive gc as someone else already mentioned, but it would be interesting to see the original code.

Re: Show HN: ZFS Implementation in Python

#52

Post by klyrs » Sat, May 04, 2019, 10:01 PM UTC

cure wrote: post #16

j88439h84 wrote: post #6

Earlier quoted context omitted.

Dont forget, pypy is fast.

PyPy is faster than Python, yes. But Go, C and many other (compiled) languages are way faster than PyPy. Plus, if you use a language like Go or Rust then you avoid Python's GIL and you'll have much more reasonable memory usage. Best of all, deploying is a matter of copying a binary, rather than having to deal with the absolute disaster that is Python packaging.

Filesystem with a GIL, what could possibly go wrong? /s

Re: Show HN: ZFS Implementation in Python

#53

Post by moonbug » Sat, May 04, 2019, 10:11 PM UTC

fragmede wrote: post #45

j88439h84 wrote: post #6

Earlier quoted context omitted.

Dont forget, pypy is fast.

pypy is fast, but even were this written in C, there's still the kernel-userland boundary to contend with.

Fuse

Re: Show HN: ZFS Implementation in Python

#54

Post by michaelmrose » Sat, May 04, 2019, 10:20 PM UTC

twa927 wrote: post #41

spullara wrote: post #13

Earlier quoted context omitted.

Faster that CPython doesn't mean it is fast.

I was trying to speed up a log processing service running on PyPy by rewriting it in Java. I was surprised that the result was about twice slower (I know Java quite well and I didn't see obvious optimizations; most of the time was spent in GC). So it can be quite fast even in more absolute terms (VM languages), at least for some types of code.

The fact that a singular implementation was better than java says less about the languages and more about the particular software.

Re: Show HN: ZFS Implementation in Python

#55

Post by twa927 » Sat, May 04, 2019, 10:24 PM UTC

tigershark wrote: post #51

twa927 wrote: post #48

Earlier quoted context omitted.

I don't have access to this codebase now but I'll try to write some benchmark.

If you manage to do it would be awesome, otherwise thanks a lot anyway for the effort :) I’m just curious to understand why it happens because it’s exactly the opposite of what I would expect. The only explanation that comes to my mind is excessive gc as someone else already mentioned, but it would be interesting to see the original code.

I started doing it but yes, it's too much effort to get two full benchmarks, I'm sorry :). But I think it went down to the inefficiency of String.split(): https://stackoverflow.com/questions/37007189/string-split-te... and generally the Java's String built-in methods not being GC-friendly: https://stackoverflow.com/questions/20336459/garbage-friendl.... I'm guessing that when such parts can be coded in a non-VM environment (CPython/PyPy runtime) they can be made much faster, and Java to these days has the standard library coded in pure Java?

Re: Show HN: ZFS Implementation in Python

#56

Post by GuB-42 » Sat, May 04, 2019, 10:50 PM UTC

j88439h84 wrote: post #6

josteink wrote: post #4

Earlier quoted context omitted.

So we port this python to something not slow, and all the kernel-people can shut up about ZFS being terrible ;)

Dont forget, pypy is fast.

Maybe plenty fast for most applications but a filesystem is not one of these IMHO, especially for something as naturally resource hungry as ZFS.

A good filesystem implementation requires tight memory management and good control of what happens at the OS level. I am not saying it can't be be done in python, but it clearly isn't the right tool for the job.

I meant that for a production implementation. Python is perfectly fine for a proof of concept, in fact, it may be better than jumping straight down to C. But keeping it for production is foolish IMHO.

Re: Show HN: ZFS Implementation in Python

#57

Post by nine_k » Sat, May 04, 2019, 10:52 PM UTC

cure wrote: post #16

j88439h84 wrote: post #6

Earlier quoted context omitted.

Dont forget, pypy is fast.

PyPy is faster than Python, yes. But Go, C and many other (compiled) languages are way faster than PyPy. Plus, if you use a language like Go or Rust then you avoid Python's GIL and you'll have much more reasonable memory usage. Best of all, deploying is a matter of copying a binary, rather than having to deal with the absolute disaster that is Python packaging.

Go? A GC'd language in kernel? (Well, yes, this has been done, from Lua to Haskell, but only experimentally.)

Re: Show HN: ZFS Implementation in Python

#58

Post by ummonk » Sat, May 04, 2019, 11:32 PM UTC

mfsch wrote: post #49

Does someone know whether it would be legal for someone to go through the ZFS code and write a specification of the features this author hasn’t figured out yet? I.e. could someone write a detailed description of the missing functionality that doesn’t include any details about the implementation so other people can implement it in non-CDDL code?

Edit: yeah that is how you avoid copyright infringement https://en.wikipedia.org/wiki/Clean_room_design

Original comment: I could swear this was actually the standard practice for writing an implementation of an unknown file format or interface without infringing on copyright. But I don't remember the term for it.

Re: Show HN: ZFS Implementation in Python

#59

Post by hu3 » Sat, May 04, 2019, 11:34 PM UTC

nine_k wrote: post #57

cure wrote: post #16

Earlier quoted context omitted.

PyPy is faster than Python, yes. But Go, C and many other (compiled) languages are way faster than PyPy. Plus, if you use a language like Go or Rust then you avoid Python's GIL and you'll have much more reasonable memory usage. Best of all, deploying is a matter of copying a binary, rather than having to deal with the absolute disaster that is Python packaging.

Go? A GC'd language in kernel? (Well, yes, this has been done, from Lua to Haskell, but only experimentally.)

I wouldn't advise writing low level stuff in Go but people do enjoy a challenge from time to time: https://news.ycombinator.com/item?id=18399389

Re: Show HN: ZFS Implementation in Python

#60

Post by atomicwrites » Sat, May 04, 2019, 11:38 PM UTC

mfsch wrote: post #49

Does someone know whether it would be legal for someone to go through the ZFS code and write a specification of the features this author hasn’t figured out yet? I.e. could someone write a detailed description of the missing functionality that doesn’t include any details about the implementation so other people can implement it in non-CDDL code?

That's called a clean room implementation and was the standard way to make x-compatible products (like for example, the bios on an IBM PC clone). Not sure what the current legal standing of that method is.

EDIT: Ninjad because I left the reply in a tab without posting.