111–120 of 125 posts
Hopefully some day an implementation in something other than Go will emerge. The current Go implementation is mind-numbingly slow and uses huge amounts of memory.
Have a look at librevault [1] instead, it is written in C++ [2].
Earlier quoted context omitted.
Just a heads up: that almost certainly means SyncThing is failing to recognize some classes of conflicts and concurrency races, and is destroying some changes. Doing this right is a hard problem. I know they can be a pain to deal with, but we don't spin out conflicts for no reason! Disclosure: I work on the Dropbox sync engine.
Do you have any specific bug or design reference to the Syncthing codebase that illustrates this, or do you simply believe that it's impossible to do it faster than Dropbox does it? Fwiw, i frequently get notifications from Syncthing that it's found a conflict it can't resolve and asks me to resolve manually.
We've likely put 25-100x more resources into solving this problem over the last 10 years, and we just have a lot more data b/c we have 100s of millions of users on lots of platforms using Dropbox with every application you can imagine. So we're able to tease out the "long tail" of weird file system and application behavior in a way that's very difficult for smaller projects. Truly durable conflict management in the face of arbitrary mutations by user applications on the filesystem ends up being a really, really hard problem to cover exhaustively. The Dropbox client handles literally hundreds of special cases.
So, yeah, I believe it is (in general) safe to assume that Dropbox is probably doing A-more-correct-thing for a complicated (and admittedly confusing) reason when it comes to sync behavior. But we're not perfect--we do still find surprises from time to time, so feel free to contact support if you see something that looks wrong!
Earlier quoted context omitted.
This is the blocker for me too. Support for a zero knowledge server where even if someone gets access to the filesystem on the server, they can only see it encrypted. Other than that, Syncthing is absolutely amazing!
I would like to mix it. Then my desktop could be a mirror for my friend without knowing his files and vice versa.
Is there something like Syncthing that works like a RAID system? I'd like to use multiple computers in different locations and use them to backup and expand my storage size. Basically redundant but distributed storage with N node fail tolerance. Preferably interfaceable via a "/mnt/cloud" directory so I can use it from my laptop and easily write backup scripts and store data into it. If this existed I'd have somethin…
You set up one introducer, where storage nodes and clients meet and exchange each other's address. After that your clients can use the storage as one big single, end-to-end encrypted, resilient space where N nodes out of M can fail without any effect on your data.
Is there something like Syncthing that works like a RAID system? I'd like to use multiple computers in different locations and use them to backup and expand my storage size. Basically redundant but distributed storage with N node fail tolerance. Preferably interfaceable via a "/mnt/cloud" directory so I can use it from my laptop and easily write backup scripts and store data into it. If this existed I'd have somethin…
Yup, there is Tahoe-LAFS ( https://tahoe-lafs.org/trac/tahoe-lafs ) and it doesn't receive the amount of love it deserves. You set up one introducer, where storage nodes and clients meet and exchange each other's address. After that your clients can use the storage as one big single, end-to-end encrypted, resilient space where N nodes out of M can fail without any effect on your data.
Edit: Also, can you say "this group of nodes count's as a single location" for failure protection? So if I have two locations where I store servers and then 10 other one off data collection locations can I say "I want you to treat these datacenters as one node since they are very likely to fail together if they fail".
Earlier quoted context omitted.
Yup, there is Tahoe-LAFS ( https://tahoe-lafs.org/trac/tahoe-lafs ) and it doesn't receive the amount of love it deserves. You set up one introducer, where storage nodes and clients meet and exchange each other's address. After that your clients can use the storage as one big single, end-to-end encrypted, resilient space where N nodes out of M can fail without any effect on your data.
Is it possible to mount Tahoe-LAFS as a file system? If it is, then it's exactly what I'm looking for. Edit: Also, can you say "this group of nodes count's as a single location" for failure protection? So if I have two locations where I store servers and then 10 other one off data collection locations can I say "I want you to treat these datacenters as one node since they are very likely to fail together if they fail…
Here it is directly from the Q&A...
" Q12: If I had 3 locations each with 5 storage nodes, could I configure the grid to ensure a file is written to each location so that I could handle all servers at a particular location going down? "
" A: Not directly. We have a wiki page and some tickets (linked from the wiki page) about this but it's deeper than it looks and we haven't come to a conclusion on how to build it.
The current system will try to distribute the shares as widely as possible, using a different pseudo-random permutation for each file, but it is completely unaware of server properties like "location". If you have more free servers than shares, it will only put one share on any given server, but you might wind up with more shares in one location than the others.
For example, if you have 15 servers in three locations A:1/2/3/4/5, B:6/7/8/9/10, C:11/12/13/14/15, and use the default 3-of-10 encoding, your worst case is winding up with shares on 1/2/3/4/5/6/7/8/9/10, and not use location C at all. The most likely case is that you'll wind up with 3 or 4 shares in each location, but there's nothing in the system to enforce that: it's just shuffling all the servers into a ring, starting at 0, and assigning shares to servers around and around the ring until all the shares have a home.
The possible distributions of shares into locations (A, B, C) are:
(3, 3, 4) 1500 (2, 4, 4) 750 (2, 3, 5) 600 (1, 4, 5) 150 (0, 5, 5) 3 sum = 3003
So you've got a 50% chance of the ideal distribution, and a 1/1000 chance of the worst-case distribution. "
Earlier quoted context omitted.
This is the price you have to pay for a (secure) decentralised system, but syncthing tries to help where it can. In general, Resilio Sync's 'key per folder' model seems to work better. I can give a key to other people and they can join the swarm without extra work on either sides. Also, it has a logical extension to encrypted-only peers: they get a derived key that can be used to sync in the swarm, but cannot decrypt…
Yeah, the only problem is you can never "un-give" a key, which means you either have to rebuild your cluster under a different key (having to re-give key to everyone else apart from that one guy you want to kick out), or deal with the person having access to the data forever.
Earlier quoted context omitted.
Do you have a reference to this 32 node cap for Resilio? I just tried doing a quick search and did not see any specific results showing this limit. Thanks.
I found the original thread, turns out the peer count is 50 (not 32 like I originally stated). Here's the thread in question: https://forum.resilio.com/topic/30525-please-please-please-p...
https://forum.resilio.com/topic/41520-max-number-of-users-fo...