Show HN: ZFS Implementation in Python • Hacker News Board

Re: Show HN: ZFS Implementation in Python

#71

Post by [deleted] » Sun, May 05, 2019, 3:40 AM UTC

twa927 wrote: post #55

tigershark wrote: post #51

Earlier quoted context omitted.

If you manage to do it would be awesome, otherwise thanks a lot anyway for the effort :) I’m just curious to understand why it happens because it’s exactly the opposite of what I would expect. The only explanation that comes to my mind is excessive gc as someone else already mentioned, but it would be interesting to see the original code.

I started doing it but yes, it's too much effort to get two full benchmarks, I'm sorry :). But I think it went down to the inefficiency of String.split(): https://stackoverflow.com/questions/37007189/string-split-te... and generally the Java's String built-in methods not being GC-friendly: https://stackoverflow.com/questions/20336459/garbage-friendl... . I'm guessing that when such parts can be coded in a non-VM envi…

[deleted]

Re: Show HN: ZFS Implementation in Python

#72

Post by burmecia » Sun, May 05, 2019, 3:59 AM UTC

PaulHoule wrote: post #33

i love userspace implementations of filesystems. note that the issues are entirely different from those with a kernel implementation since you aren't having to think about page cache et all.

There is an userspace implementations of filesystems ZboxFS: https://github.com/zboxfs/zbox.

Re: Show HN: ZFS Implementation in Python

#73

Post by Conan_Kudo » Sun, May 05, 2019, 5:03 AM UTC

yjftsjthsd-h wrote: post #66

moonbug wrote: post #53

Earlier quoted context omitted.

Fuse

If you're going to run the file system in user space, there's no reason not to just use normal ZFS. The problem with ZFS licensing is only in combining CDDL+GPL in one unit. If you're working across the kernel/userspace boundary, there's already no problem. ZoL even already ships a fuse version that works fine.

> ZoL even already ships a fuse version that works fine.

This is not true. A FUSE implementation is wanted though: https://github.com/zfsonlinux/zfs/issues/8

Re: Show HN: ZFS Implementation in Python

#74

Post by d0mine » Sun, May 05, 2019, 5:36 AM UTC

twa927 wrote: post #41

spullara wrote: post #13

Earlier quoted context omitted.

Faster that CPython doesn't mean it is fast.

I was trying to speed up a log processing service running on PyPy by rewriting it in Java. I was surprised that the result was about twice slower (I know Java quite well and I didn't see obvious optimizations; most of the time was spent in GC). So it can be quite fast even in more absolute terms (VM languages), at least for some types of code.

I would have expected it: https://stackoverflow.com/questions/9371238/why-is-reading-l...

Re: Show HN: ZFS Implementation in Python

#75

Post by yjftsjthsd-h » Sun, May 05, 2019, 5:45 AM UTC

Conan_Kudo wrote: post #73

yjftsjthsd-h wrote: post #66

Earlier quoted context omitted.

If you're going to run the file system in user space, there's no reason not to just use normal ZFS. The problem with ZFS licensing is only in combining CDDL+GPL in one unit. If you're working across the kernel/userspace boundary, there's already no problem. ZoL even already ships a fuse version that works fine.

> ZoL even already ships a fuse version that works fine. This is not true. A FUSE implementation is wanted though: https://github.com/zfsonlinux/zfs/issues/8

Oops; I didn't realize the existing fuse version wasn't built from ZoL. Thanks for pointing that out.

Re: Show HN: ZFS Implementation in Python

#76

Post by riffraff » Sun, May 05, 2019, 6:06 AM UTC

yjftsjthsd-h wrote: post #67

dr0verride wrote: post #8

Earlier quoted context omitted.

It's clear to me that the solution is to integrate python into the kernel.

I assume you mean this as a joke, but I would point out that at least one of the BSD family has gone and baked lua into their kernel. Granted, lua is rather meant for that kind of thing and python isn't, but it is entertaining to point out an interpreted language that has been stuck into a unix kernel:)

there was lua in linux too, with lunatik https://github.com/lunatik-ng/lunatik-ng

Re: Show HN: ZFS Implementation in Python

#77

Post by iiv » Sun, May 05, 2019, 7:08 AM UTC

hueving wrote: post #68

aerovistae wrote: post #38

Earlier quoted context omitted.

One of my favorite short stories, and nobody else has ever read it. So glad to see someone else reference it.

Link?

The original title is "Pierre Menard, Author of the Quixote": http://www.coldbacon.com/writing/borges-quixote.html

Re: Show HN: ZFS Implementation in Python

#78

Post by twa927 » Sun, May 05, 2019, 7:38 AM UTC

auscompgeek wrote: post #70

twa927 wrote: post #55

Earlier quoted context omitted.

I started doing it but yes, it's too much effort to get two full benchmarks, I'm sorry :). But I think it went down to the inefficiency of String.split(): https://stackoverflow.com/questions/37007189/string-split-te... and generally the Java's String built-in methods not being GC-friendly: https://stackoverflow.com/questions/20336459/garbage-friendl... . I'm guessing that when such parts can be coded in a non-VM envi…

CPython and PyPy are both VMs, similarly to the JVM.

I meant the inside of a VM (its implementation), which is coded in C/RPython. Java's VM is coded in C++, but I don't think C++ is used for any regular library functions, while C is used heavily for Python's stdlib.

Re: Show HN: ZFS Implementation in Python

#79

Post by tigershark » Sun, May 05, 2019, 7:40 AM UTC

twa927 wrote: post #55

tigershark wrote: post #51

Earlier quoted context omitted.

If you manage to do it would be awesome, otherwise thanks a lot anyway for the effort :) I’m just curious to understand why it happens because it’s exactly the opposite of what I would expect. The only explanation that comes to my mind is excessive gc as someone else already mentioned, but it would be interesting to see the original code.

I started doing it but yes, it's too much effort to get two full benchmarks, I'm sorry :). But I think it went down to the inefficiency of String.split(): https://stackoverflow.com/questions/37007189/string-split-te... and generally the Java's String built-in methods not being GC-friendly: https://stackoverflow.com/questions/20336459/garbage-friendl... . I'm guessing that when such parts can be coded in a non-VM envi…

Yes, Java has nothing like C# Span to avoid these kind of problems, but I thought that also python would be affected in a similar way... Anyway, thanks for sharing.

Re: Show HN: ZFS Implementation in Python

#80

Post by y4mi » Sun, May 05, 2019, 7:48 AM UTC

emmelaich wrote: post #69

klyrs wrote: post #52

Earlier quoted context omitted.

Filesystem with a GIL, what could possibly go wrong? /s

A lot less will go wrong than a filesystem without a GIL. GIL is for safety and correctness, not speed.

Uh, no?

Python's global interpreter lock was added for single threaded speed and c library integrations, which often can't be used multithreaded

There was some talk about removing it recentlish to improve pythons multithreaded performance and Guido said something along the lines of

> "I'll remove it as long as single threaded performance doesn't suffer"

Which nobody succeeded in