Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

A distributed Posix file system built on top of Redis and S3

github.com

Post reply on HN
Print view View original

41–50 of 120 posts

Re: A distributed Posix file system built on top of Redis and S3

#41

Post by Maakuth »

keyle wrote: post #39

So... a file system on top of a virtualized file system hosted on someone else's computer across the land, with "Outstanding Performance: The latency can be as low as a few milliseconds" ... Milliseconds disk access is outstanding? I mean, amazing, and, maybe you know, use a file system.

Remember that a network filesystem, can't really be faster than the network latency. If you are not in the same rack with your filesystem server, you can expect a few milliseconds of network latency. So a network filesystem that has a scalable design and is POSIX compatible, being in the same latency ballpark with the network sounds quite nice actually.

Re: A distributed Posix file system built on top of Redis and S3

#42

Post by hilbertseries »

QuinnyPig wrote: post #30
14u2c wrote: post #12

Earlier quoted context omitted.

Out of curiosity how do you use a DNS service as a database?

Excellently. I use it as a database excellently: https://www.lastweekinaws.com/blog/route-53-amazons-premier-...

OOC, what are the downsides of storing service discovery information like this in TXT records? As opposed to using ZK, consul, etcd or a more standard solution?

Re: A distributed Posix file system built on top of Redis and S3

#44

Post by QuinnyPig »

hilbertseries wrote: post #42
QuinnyPig wrote: post #30

Earlier quoted context omitted.

Excellently. I use it as a database excellently: https://www.lastweekinaws.com/blog/route-53-amazons-premier-...

OOC, what are the downsides of storing service discovery information like this in TXT records? As opposed to using ZK, consul, etcd or a more standard solution?

There are better tools for the job now. This is basically a misuse of a service instead of using things that are purpose built.

I mean, your inventory is either a grep or a zone transfer this way, for one.

Re: A distributed Posix file system built on top of Redis and S3

#45

Post by tutfbhuf »

I think the major problem is latency. Try sshfs (fs over ssh) and see what I mean. Don't get me wrong, I use and like sshfs for a quick data transfer, but it's just not good enough to run your application with.

For a stable POSIX filesystem in production latency is key. Often times in a datacenter 10GE is recommended for network storage solutions, not because of bandwidth (which is also important), but for the 10x reduced latency of a 10GE NIC. Most applications simply expect response times of microseconds or a few milliseconds at most from a POSIX Filesystem, they simply cannot run (without modifying the codebase) on something much slower.

But if you had to rewrite your application anyways, then you might want to use plain S3 without the FS, that is easier to operate in the long run.

Re: A distributed Posix file system built on top of Redis and S3

#47

Post by fock »

toomuchtodo wrote: post #11
cmwelsh wrote: post #10

Would not an NFS solution have this kind of caching and durability built-in? Without doing actual “Jepsen tests” (they are almost a generic term at this point due to their name) how would this improve my life versus buying an NFS vendor solution, or rolling my own?

NFS doesn’t scale to effectively infinity with an underlying object store. This is to give you a ton of storage without using a traditional volume target with your app that, for whatever reason, requires a posix filesystem. I’m sure someone from AWS can’t comment, but I imagine this is how AWS’ EFS service is built (NFS wire protocol to clients, but using S3 and metadata caching under the hood). Blobs or blocks doesn…

well, NFS may not scale to infinity, but easily beats this thing I guess... And for scaling to infinity: how about benchmarking vs. GPFS, BeeGFS or Gluster?

Re: A distributed Posix file system built on top of Redis and S3

#49

Post by casselc »

QuinnyPig wrote: post #44
hilbertseries wrote: post #42

Earlier quoted context omitted.

OOC, what are the downsides of storing service discovery information like this in TXT records? As opposed to using ZK, consul, etcd or a more standard solution?

There are better tools for the job now. This is basically a misuse of a service instead of using things that are purpose built. I mean, your inventory is either a grep or a zone transfer this way, for one.

If only AWS Cloud Map wasn't $0.10/resource outside of ECS you could reasonably have your API cake and query it using DNS too.

Re: A distributed Posix file system built on top of Redis and S3

#50

Post by daviesliu »

tutfbhuf wrote: post #45

I think the major problem is latency. Try sshfs (fs over ssh) and see what I mean. Don't get me wrong, I use and like sshfs for a quick data transfer, but it's just not good enough to run your application with. For a stable POSIX filesystem in production latency is key. Often times in a datacenter 10GE is recommended for network storage solutions, not because of bandwidth (which is also important), but for the 10x re…

Good, that's why we choose Redis for the metadata. When Redis is deployed in same DC or VPC, the latency could be about 0.2ms - 0.5ms.

Later on, We will try the client cache of Redis, which could also reduce the latency for some metadata operations down to a few microseconds.

Post reply on HN