Live data from Hacker News

Introducing the Infinit file system

blog.infinit.one

51–60 of 63 posts

Re: Introducing the Infinit file system

#51
post #9

This looks really great and I love the direction. Hopefully they're working on making it more consumer-friendly. ;) Questions: 1. They say that "While most solutions (Dropbox, GlusterFS, OwnCloud etc.) store your files unprotected in the cloud or on a specific server, we took a diametrically opposed direction by relying heavily on encryption. Whenever a file is stored in Infinit, it is cut into chunks, every chunk is…

Sorry that I'm late to this thread!

The first question appears to have been answered by others but I will add that even though Infinit is currently closed source, you can build your infrastructure without using the Hub. The Hub is just in place to make it easy to fetch user public keys, network and volume descriptors and endpoints at runtime. You can do this manually using the CLI (--export/--import instead of --push/--fetch).

We're using the 3.x branch of FUSE for OS X and haven't had an issue with it yet.

Re: Introducing the Infinit file system

#52

Really cool, but I use NixOS for everything. You mentioned it's open source so--want to help me package it? It's probably not very hard.

It's not open source just yet. We want to open it in a responsible way (i.e.: not a massive code dump) so it will take a bit of time.

You can post a request for packages here in the meantime if you would like: http://infinit-sh.uservoice.com

Re: Introducing the Infinit file system

#53

Earlier quoted context omitted.

The part about GlusterFS storing files unencrypted is untrue, BTW. We've had full at-rest encryption using client-only keys for years before they came along. Makes me wonder how many of their other "original ideas" are just failure to do basic research before jumping in to make a buck.

I was going to agree but decided to load up web site to be sure: https://www.gluster.org/ I don't see a list of features at all here or an obvious link to them. Most sites have something like this (see "Why Sector/Sphere?"): http://sector.sourceforge.net/ Or this: http://lustre.org/about/ Wait, that one almost sucks as much as GlusterFS site. It could also use a page detailing exactly what it can and can't do with ob…

Let me Google "GlusterFS encryption" for you.

http://www.gluster.org/community/documentation/index.php/Fea...

https://www.gluster.org/community/documentation/images/e/e2/...

...and more. I think that still falls under "casual research" ... besides which, implementers and promoters should be doing more than casual research anyway. The fact that the Gluster documentation sucks doesn't change that. Plenty of others have implemented such end-to-end encryption as well, and the trend is already increasing. (I find that quite gratifying BTW since I felt like a lone voice in the wilderness when I was writing about the importance of keeping keys on the clients five-plus years ago.) What they present as a differentiator is very much a me-too nowadays so, again, it calls their other claims of originality or uniqueness into question.

Re: Introducing the Infinit file system

#54
post #50
post #47

Earlier quoted context omitted.

Infinit's goal is not to compete against IPFS which is focusing on providing a protocol for distributing content. IPFS does not focus on providing redundancy, fault tolerance, rebalancing or file-level functionalities such as access control, versioning etc. This is what Infinit is doing. Two very different solutions even though they may share some technical similarities.

I'm not sure what IPFS you're reading about.... redundancy - Any node in the IPFS network can provide the requested data. That's because the identity of the data is with the name, and not the server you got the data from. fault tolerance - Absolutely does provide fault tolerance. The filesystem is a SHA256 hash in what they call a multihash. Because everything has a hash-name, the file system is a self-certifying fil…

I'm reading what I can find :) Sorry if I missed information, it was not my intention to say that IPFS was not good, just to say that the purpose was different from Infinit's.

Redundancy: https://github.com/ipfs/ipfs/issues/50. One of IPFS' core developers clearly states that there is no redundancy in IPFS. I took my information from there because it was the only one I could find with Google. It is not because one node can retrieve the data that the data is actually stored multiple times and that the storage servers coordinate to maintain consistency. From what I understand, IPFS content-hashes information (as Infinit does) but does not replicate it. As such, if a server goes down, its hosted content becomes unavailable; very much like the Web. This is not the case of Infinit (if you defined a replication factor above 1). Again, different purposes, my goal is not to say one is better; redundancy obviously has a cost.

Fault tolerance: Yes nodes can go down in IPFS but the system will not function as before as some data may be unavailable, in the worst-case scenario, permanently, should the failure be fatal. Sorry, I should have made myself clear. IPFS is fault-tolerant but does not ensure availability. Bittorrent as well. Some files may become so rare that you cannot access it anymore. Infinit ensures that all the files are available at any time. Self-certifying is another concept altogether.

Rebalancing: I don't understand exactly what you mean but if you want to provide a POSIX-compliant reliable (available/durable) file system, you need fault tolerance and rebalancing, which is to recreate missing replicas and possibly move data around as servers are added or fail. If you can't do that, then again you have the problem of potentially having unavailable pieces of information. It is fine for some systems such as the Web (not critical if you can't access some rare pages), but it is super critical for an enterprise file system in production. Again, different purposes I believe as IPFS (from what I understand, sorry if I missed something) is trying to provide a HTTP replacement: a new peer-to-peer hypermedia protocol (from http://ipfs.io).

Access control: File-level encryption is one thing but access control something else entirely: how to allow other users to read/write files and how to manage the keys. What about removing files? What about groups? Subgroups? And what about integrating into an enterprise directory (LDAP)? These compose a complete set of access control capabilities from my point of view. File-level encryption is obviously the basis for access control but is not enough. As you stated, it is planned in the protocol.

Versioning: Good to know that IPFS will be providing versioning, I didn't know that. Thanks for pointing that out.

To conclude, both projects seem to rely on some similar tech such as content hashing. IPFS seems to be going the way of a protocol for accessing data in a decentralized way (like the Web in a more modern way) but without providing redundancy, consistency and availabilities garantees. Storj is going this way for instance, providing a peer-to-peer object store (S3-like). Infinit however seems to be more focused on providing a POSIX-compliant peer-to-peer, reliable, secure and fault-tolerant file system for Ops and DevOps.

Re: Introducing the Infinit file system

#55

Earlier quoted context omitted.

Dropbox deduplicate content ... so a file that several people have uploaded is only actually stored once. They also allow you to access your files over a web interface, and no decryption happens "in browser". So yes, DropBox most certainly have the decryption keys. Your data is protected at rest, and on the wire using TLS, but it's not end-to-end encryption.

Do you have any reference to their de-duplication system?

There is a blog post[0] which references deduplication. But apparently this may have been disabled[1].

[0] https://blogs.dropbox.com/dropbox/2011/07/changes-to-our-pol...

[1] http://webapps.stackexchange.com/questions/54633/does-dropbo...

Re: Introducing the Infinit file system

#56

Earlier quoted context omitted.

I was going to agree but decided to load up web site to be sure: https://www.gluster.org/ I don't see a list of features at all here or an obvious link to them. Most sites have something like this (see "Why Sector/Sphere?"): http://sector.sourceforge.net/ Or this: http://lustre.org/about/ Wait, that one almost sucks as much as GlusterFS site. It could also use a page detailing exactly what it can and can't do with ob…

Let me Google "GlusterFS encryption" for you. http://www.gluster.org/community/documentation/index.php/Fea... https://www.gluster.org/community/documentation/images/e/e2/... ...and more. I think that still falls under "casual research" ... besides which, implementers and promoters should be doing more than casual research anyway. The fact that the Gluster documentation sucks doesn't change that. Plenty of others have…

Why would I Google GlusterFS encryption? You think a casual researcher should type every feature of importance into Google with an apps name to confirm if it has that feature? That's ridiculous. It's features or benefits should be easy to find on the homepage. Documentation, esp "What This Does," is critical to OSS success.

Here's what Gluster doc menu said: one that goes to instsllation page; more detailed installation; admin guide; developer guide; upgrade guide. Geez, I don't need any of that. Just want to know what the hell it does with what features.

They need to fix it.

Re: Introducing the Infinit file system

#57

Earlier quoted context omitted.

Let me Google "GlusterFS encryption" for you. http://www.gluster.org/community/documentation/index.php/Fea... https://www.gluster.org/community/documentation/images/e/e2/... ...and more. I think that still falls under "casual research" ... besides which, implementers and promoters should be doing more than casual research anyway. The fact that the Gluster documentation sucks doesn't change that. Plenty of others have…

Why would I Google GlusterFS encryption? You think a casual researcher should type every feature of importance into Google with an apps name to confirm if it has that feature? That's ridiculous. It's features or benefits should be easy to find on the homepage. Documentation, esp "What This Does," is critical to OSS success. Here's what Gluster doc menu said: one that goes to instsllation page; more detailed installat…

If somebody is going to make a specific claim, in a published document, about another project having or not having a feature, then damn right they should Google for the combination. A lot of OSS projects end up having pieces of information scattered all over presentations and blog posts and who the hell knows what else. All of your goalpost-moving about what should be on the Gluster website doesn't change the fact that Infinit's characterization was inaccurate and trivially revealed as such by a single obvious Google search.

Re: Introducing the Infinit file system

#58
post #9

This looks really great and I love the direction. Hopefully they're working on making it more consumer-friendly. ;) Questions: 1. They say that "While most solutions (Dropbox, GlusterFS, OwnCloud etc.) store your files unprotected in the cloud or on a specific server, we took a diametrically opposed direction by relying heavily on encryption. Whenever a file is stored in Infinit, it is cut into chunks, every chunk is…

LTFS is based on FUSE and has support from all the major tape vendors. It's been pretty stable so far but does have some speed limitations.

Re: Introducing the Infinit file system

#59

Earlier quoted context omitted.

Why would I Google GlusterFS encryption? You think a casual researcher should type every feature of importance into Google with an apps name to confirm if it has that feature? That's ridiculous. It's features or benefits should be easy to find on the homepage. Documentation, esp "What This Does," is critical to OSS success. Here's what Gluster doc menu said: one that goes to instsllation page; more detailed installat…

If somebody is going to make a specific claim, in a published document, about another project having or not having a feature, then damn right they should Google for the combination. A lot of OSS projects end up having pieces of information scattered all over presentations and blog posts and who the hell knows what else. All of your goalpost-moving about what should be on the Gluster website doesn't change the fact th…

You're semi-right here. People saying why their offering is better than competitors better know what the competition offers. Marketing 101 says dig deep to find that plus differentiators. That much I agree with.

That said, your excuse for GlusterFS site being screwed up is that other projects are screwups, too. Makes no sense. Let's put it into perspective: Gluster people could spend under 5 minutes typing up and posting that page. Instead, they expect all potential users or contributors to spend 10m-1hr digginh through docs for same information. Meanwhile, many FOSS pages state clearly what their software does.

So, no excuses. It's just laziness and foolish on top of that given they want more adoption of a tool they won't describe haha. The responsibility is on them to present their work in a clear way given that goal. They're failing on that right now.

Re: Introducing the Infinit file system

#60
This seems like an interesting project to watch and I'd be interested more in it as A) Windows support gets better, and B) source is opened.

I'm definitely comparing to my usage of BitTorrent Sync today. I see in the FAQ a comparison for BT Sync a few things of interest.

BT Sync does have a more filesystem-like mode. (It's a part of BT Sync "Pro", if that makes a differences.) It's also scalable to available resources, although some of that through manual management of which devices are connected to a BT Sync share and which have which copies of which files.

Also, BT Sync Pro does support At-Rest security with some management. A UX for encrypted shares was added in recent versions of the software and it supported through some command line effort in previous versions. I've seen tutorials for setting up BT Sync "know nothing peers" on, for instance, EC2 storing encrypted blocks to S3.

Post reply on HN