Really great code - clean and extensible! I also really like inotify for this sort of code and actually use it myself to kick off my build scripts automatically when changes are detected. BTW, have you run into too many files/dirs issues with inotifywait?
Show HN: Pullbox – A dead-simple dropbox alternative using Git
21–30 of 32 posts
Re: Show HN: Pullbox – A dead-simple dropbox alternative using Git
#22Earlier quoted context omitted.
It is based on Git so uses the same mechanism. When a merge cannot be done because of conflict, you can resolve in the usual git workflow.
It's also possible to specify other conflict resolution options, such as `git merge --strategy=recursive --strategy-option=theirs` [0], which avoids the need for manual conflict resolution (and it's "worked for me"). Discarded hunks are still available in the repo history, in case you needed to restore a file/hunk that was discarded in this manner. [0]: http://git-scm.com/docs/git-merge (be sure to look for the "ours…
Thanks for the tip. I wanted to keep my implementation extremely bare-bones and simple so I chose to leave conflict resolution (which itself is a rare event in my a use-case as I usually only make modifications on one synced machine at a time).
Re: Show HN: Pullbox – A dead-simple dropbox alternative using Git
#23Despite all the competitors everyone is linking to here, I bet you learned a ton while building this. What was most surprising to you? What was the trickiest part?
It was a ton of fun on a saturday afternoon when I wrote this code. I was trying to avoid rolling my own syncing solution but I guess my Google-fu failed me :)
My goal was to write a very thin wrapper around the workflow I would follow if I had to sync the changes myself manually. The tricky part was in figuring out how to inform multiple client machines when the backup server noticed a change in the file system. I wanted to avoid writing a server-side component that I had to install on the server and maintain.
When I found that I could use a combination of "ssh" and "inotifywait" (run inotifywait on the server using ssh from the client and listen for changes), I was pleasantly surprised that this even worked! I see my implementation in this aspect as the equivalent of AJAX long-polling that used to be applied for chat like communication in the browser in some implementations. i.e. When some modification happens on the server filesystem, the "inotifywait" command quits thereby unblocking the "ssh" command upon which I do a "git pull".
Because of the above, I was able to keep my implementation really simple - The whole functionality was achieved in under 300 lines of code.
Re: Show HN: Pullbox – A dead-simple dropbox alternative using Git
#24Earlier quoted context omitted.
Linking to alternatives is not exactly constructive criticism for someone looking for feedback from a Show HN. It feels dismissive and out of spirit of what's trying to be accomplished here. Without any commentary or questions, it's almost like you're saying, "Why did you bother?" Now, I very much doubt that was your intention, but please have some empathy for the person taking a chance when sharing something on Show…
Point taken, and you're right. I'm extremely sensitive about my work too. My deepest apologies to the OP!
No apologies required! Thanks for taking the time to post the links. I discovered "syncthing" via this thread and while it sucks that I did not come across it before, I am excited to try it out.
Re: Show HN: Pullbox – A dead-simple dropbox alternative using Git
#25Really great code - clean and extensible! I also really like inotify for this sort of code and actually use it myself to kick off my build scripts automatically when changes are detected. BTW, have you run into too many files/dirs issues with inotifywait?
Thank you! I tried hard to keep the code very simple sacrificing some 20%-scenario requirements (such as automatic conflict resolution).
I am using this to synchronize my Markdown based notes files across machines. There are under a hundred files now so I haven't hit any issues in that department yet.
However, there is a gradual memory leak which persists even when I kill and restart my process. I observed this only on a KVM based Linux guest - not sure if this is because of inotify based listening but I'm going to have to dig deeper to find out.
Re: Show HN: Pullbox – A dead-simple dropbox alternative using Git
#26Earlier quoted context omitted.
Point taken, and you're right. I'm extremely sensitive about my work too. My deepest apologies to the OP!
No apologies required! Thanks for taking the time to post the links. I discovered "syncthing" via this thread and while it sucks that I did not come across it before, I am excited to try it out.
You're a champ mate. Try out git annex too. The presentation might put you off a bit but it's really powerful.
Re: Show HN: Pullbox – A dead-simple dropbox alternative using Git
#27Really great code - clean and extensible! I also really like inotify for this sort of code and actually use it myself to kick off my build scripts automatically when changes are detected. BTW, have you run into too many files/dirs issues with inotifywait?
Thank you! I tried hard to keep the code very simple sacrificing some 20%-scenario requirements (such as automatic conflict resolution). I am using this to synchronize my Markdown based notes files across machines. There are under a hundred files now so I haven't hit any issues in that department yet. However, there is a gradual memory leak which persists even when I kill and restart my process. I observed this only…
Nice clean code. A couple of places to start looking are probably in keeprunning(). Any file handle issues on repeated runs?
The threads in lines 245-247 feel like they could deadlock or have a race, esp if changes happen nearly simultaneously on both ends, or if a change on one end is received and sync'ed to the other. I'm not sure how git handles multiple writes? Almost wonder if forking might cure the memory leak too or at least make it easier to see (see my other comment of some shell code).
Re: Show HN: Pullbox – A dead-simple dropbox alternative using Git
#28Really great code - clean and extensible! I also really like inotify for this sort of code and actually use it myself to kick off my build scripts automatically when changes are detected. BTW, have you run into too many files/dirs issues with inotifywait?
Thank you! I tried hard to keep the code very simple sacrificing some 20%-scenario requirements (such as automatic conflict resolution). I am using this to synchronize my Markdown based notes files across machines. There are under a hundred files now so I haven't hit any issues in that department yet. However, there is a gradual memory leak which persists even when I kill and restart my process. I observed this only…
Forking might be easier to debug.. but I think I might have missed some of the functionality in your script.. something like (totally untested):
(while true
do
inotifywait \
--exclude '.*\.sw.' \
--recursive \
--event close_write \
--event delete \
--event moved_to \
--event moved_from \
dir1/
git push
sleep 0.1
done
) 2>&1 >/var/log/push_log.log &
(while true
do
cat # (second EOF cannot be indented)
done
) 2>&1 >/var/log/pull_log.log &
echo "Kicked off pull and push processes."
waitRe: Show HN: Pullbox – A dead-simple dropbox alternative using Git
#29Earlier quoted context omitted.
Thank you! I tried hard to keep the code very simple sacrificing some 20%-scenario requirements (such as automatic conflict resolution). I am using this to synchronize my Markdown based notes files across machines. There are under a hundred files now so I haven't hit any issues in that department yet. However, there is a gradual memory leak which persists even when I kill and restart my process. I observed this only…
Nice clean code. A couple of places to start looking are probably in keeprunning(). Any file handle issues on repeated runs? The threads in lines 245-247 feel like they could deadlock or have a race, esp if changes happen nearly simultaneously on both ends, or if a change on one end is received and sync'ed to the other. I'm not sure how git handles multiple writes? Almost wonder if forking might cure the memory leak…
The memory "leak" persists across restarts of my process. That is what beats me. If memory were leaking while the process was alive and was reclaimed by the OS upon killing the process, then I could debug the process itself.
I read that running multiple git commands at the same time on the same repo should be fine (I can't remember where I read this). It is probably better to synchronize regardless!
Re: Show HN: Pullbox – A dead-simple dropbox alternative using Git
#30Earlier quoted context omitted.
Thank you! I tried hard to keep the code very simple sacrificing some 20%-scenario requirements (such as automatic conflict resolution). I am using this to synchronize my Markdown based notes files across machines. There are under a hundred files now so I haven't hit any issues in that department yet. However, there is a gradual memory leak which persists even when I kill and restart my process. I observed this only…
Forking might be easier to debug.. but I think I might have missed some of the functionality in your script.. something like (totally untested): (while true do inotifywait \ --exclude '.*\.sw.' \ --recursive \ --event close_write \ --event delete \ --event moved_to \ --event moved_from \ dir1/ git push sleep 0.1 done ) 2>&1 >/var/log/push_log.log & (while true do cat EOF # (second EOF cannot be indented) done ) 2>&1…
That is Gold! Implementing the functionality in shell itself was what I considered first however I needed two things
1. Make it work on OSX 2. Use a lock file to prevent multiple instances from running when I put this in crontab
I found watchman[0]. Got to see if I can use that in place of "inotifywait".