PlentyFS: PoC read-only filesystem with random contents generated on demand

The inspiration for this was wanting to develop and test file access and backup tools on many large files, without actually requiring the physical storage for origin data, and minimal memory or system overheaad.

Files are regular files (not FIFO's, named pipes, sockets, or symlinks), and data are random (not simply ASCII null as with /dev/zero).

Access is read only.

To be clear, this is a test rig, not for actual data storage.

Re: PlentyFS: PoC read-only filesystem with random contents generated on demand

#5

Post by rwmj » Mon, Oct 19, 2020, 7:18 AM UTC

For something similar at the block level try:

  nbdkit random 1G
  nbd-client localhost /dev/nbd0

Any size can be used and there is a seed= parameter to get the same data between runs. You might need "modprobe nbd".

As in this article, we also use this for testing.

Re: PlentyFS: PoC read-only filesystem with random contents generated on demand

#6

Post by rwmj » Mon, Oct 19, 2020, 7:27 AM UTC

dredmorbius wrote: post #4

The inspiration for this was wanting to develop and test file access and backup tools on many large files, without actually requiring the physical storage for origin data, and minimal memory or system overheaad. Files are regular files (not FIFO's, named pipes, sockets, or symlinks), and data are random (not simply ASCII null as with /dev/zero). Access is read only. To be clear, this is a test rig, not for actual dat…

Some comments: The seed should be an optional parameter (I see there's a comment in the code mentioning this). This is very important for CI testing where a bug only happens for specific file content and you need to reproduce the bug locally.

Why use a cryptographic hash? For nbdkit-random-plugin we use a fast PRNG (xoshiro256-starstar 1.0 as it happens, but the specific one is not important). I doubt a cryptographically secure hash is worth the extra computational cost to users.

Re: PlentyFS: PoC read-only filesystem with random contents generated on demand

#7

Post by baq » Mon, Oct 19, 2020, 11:28 AM UTC

rwmj wrote: post #5

For something similar at the block level try: nbdkit random 1G nbd-client localhost /dev/nbd0 Any size can be used and there is a seed= parameter to get the same data between runs. You might need "modprobe nbd". As in this article, we also use this for testing.

ndb is also quite fun in that you can rate-limit easily, so you can check what happens with your app on slow storage. don't make it too slow, the Linux kernel doesn't like that and can hang completely.

Re: PlentyFS: PoC read-only filesystem with random contents generated on demand

#8

Post by dredmorbius » Mon, Oct 19, 2020, 6:12 PM UTC

rwmj wrote: post #5

For something similar at the block level try: nbdkit random 1G nbd-client localhost /dev/nbd0 Any size can be used and there is a seed= parameter to get the same data between runs. You might need "modprobe nbd". As in this article, we also use this for testing.

https://en.wikipedia.org/wiki/Network_block_device

If I'm understanding correctly, this still requires one instance with real (vs. virtual) storage on it? The advantage of PlentyFS is that it effectively requires no storage anywhere.

I didn't write PlentyFS, but was participating (mostly not helpfully) in Lars Wirzenius's thread on Mastodon.

My idea was to look at an HTTP or WebDAV solution using FUSE. The server would "create' the content (thinking along the line of HTTP honeypots), the FUSE module would present the filesystem-based access. There's an existing WebFS fuse modele, also a WebDAV one IIRC.

Unfortunately, HHTP doesn't offer directory presentation, making full filesystem semantics difficult.

I'm looking at other possible options as well, the concept is interesting.

Re: PlentyFS: PoC read-only filesystem with random contents generated on demand

#9

Post by dredmorbius » Mon, Oct 19, 2020, 6:13 PM UTC

rwmj wrote: post #6

dredmorbius wrote: post #4

The inspiration for this was wanting to develop and test file access and backup tools on many large files, without actually requiring the physical storage for origin data, and minimal memory or system overheaad. Files are regular files (not FIFO's, named pipes, sockets, or symlinks), and data are random (not simply ASCII null as with /dev/zero). Access is read only. To be clear, this is a test rig, not for actual dat…

Some comments: The seed should be an optional parameter (I see there's a comment in the code mentioning this). This is very important for CI testing where a bug only happens for specific file content and you need to reproduce the bug locally. Why use a cryptographic hash? For nbdkit-random-plugin we use a fast PRNG (xoshiro256-starstar 1.0 as it happens, but the specific one is not important). I doubt a cryptographic…

I'm really just a bystanderhere. File an issue w/ the project.

Re: PlentyFS: PoC read-only filesystem with random contents generated on demand

#10

Post by minoru » Mon, Oct 19, 2020, 7:46 PM UTC

rwmj wrote: post #6

dredmorbius wrote: post #4

The inspiration for this was wanting to develop and test file access and backup tools on many large files, without actually requiring the physical storage for origin data, and minimal memory or system overheaad. Files are regular files (not FIFO's, named pipes, sockets, or symlinks), and data are random (not simply ASCII null as with /dev/zero). Access is read only. To be clear, this is a test rig, not for actual dat…

Some comments: The seed should be an optional parameter (I see there's a comment in the code mentioning this). This is very important for CI testing where a bug only happens for specific file content and you need to reproduce the bug locally. Why use a cryptographic hash? For nbdkit-random-plugin we use a fast PRNG (xoshiro256-starstar 1.0 as it happens, but the specific one is not important). I doubt a cryptographic…

Hi, the author here. Thanks for taking a look at the code!

I picked SHA-1 because it was the simplest way to generate pseudorandom data: I knew there's a Rust crate for it which I used before. Works fine for a proof-of-concept :) I'm now reading up on other hashes and ciphers. My tentative plan was to replace SHA-1 with AES-NI in counter mode, but if I can find something that works on all CPUs while giving me a comparable performance, I'll go for that. (Should probably look at PRNGs as well; if I concatenate the file seed with the counter and use that as a seed for a PRNG, then ask it for a value -- it should be good enough too.)

I doubt Lars has requirements for the quality of randomness, but we'll see.