41–50 of 91 posts
I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME. ----- I would like to use something like git to store the source code to every piece of software I use, and the binaries. That…
But, as in all things software, "it depends."
It depends on what the tools are, and what we are writing.
In my own case, I have the luxury of writing fully native Swift code for Apple devices. I don't need to work with anything off the device, except for fairly direct interfaces, like USB, TCP/IP or Bluetooth.
Usually.
I have written "full stack" systems that included self-authored SDKs in Swift, and self-authored servers in PHP/JS.
I avoid dependencies like the plague. Some of them are excellent, and well worth the effort, but I have encountered very few that really make my life as an Apple device programmer that much easier. The rare ones I do use (like SOAPEngine, or ffmpeg, for instance), are local to the development environment, and usually quite well-written and supported.
If I were writing an app that reflected server-provided utility on a local device, then there's a really good chance that I'd use an SDK/dependency with network connectivity, like GraphQL, or MapBox. These are great services, but ones that I don't use (at the moment).
I'm skeptical of a lot of "Big Social Media" SDKs. I believe that we just had an issue with the FB SDK.
That said, if I were writing an app that leveraged FB services, I don't see how I could avoid their SDK.
So I write fully native software with Swift, and avoid dependencies. That seems to make my life easier.
But Xcode is still a really crashy toolbox.
I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME. ----- I would like to use something like git to store the source code to every piece of software I use, and the binaries. That…
I agree that git is a great model, however it's often hard to explain to new users why merge conflict require time to resolve, and nothing saves you from the added work. Life just isn't completely decentralizable... sorry.
Decentralization is a spectrum, not the opposite of centralization.
----
One way to partially address to merge conflict problem is to explode application files into directory hierarchies.
For example, Word .doc files and Photoshop .PSD files are basically huge hierchical data structures inside a single file. I believe video formats also have significant hierarchical structure.
A lot of them even have immutable portions and mutable portions -- e.g. for storing an entire version history.
So if those were exploded into something that the OS (or tools like git/rsync) could understand, like a tree of files, then you would have a lot fewer merge conflicts.
That's how people tend structure their git repos too. If you have a frequently edited file that's hard to merge, then that's a smell, and you can fix it.
This won't solve every problem but again there needs to be something in between "rewrite Photoshop as Figma" and "email around a bunch of PSD files" (which is a very flaky distributed system on top of e-mail.)
Earlier quoted context omitted.
The worlds most trafficked database is probably Google search* index and its not sqlite and it's certainly distributed. * Maybe Facebook, I dunno, the point stands.
Modern browsers all use sqllite, so more than 90% of user accessing that index are doing through an app using sqllite. Lot’s of things come with sqllite without you knowing about it
Admittedly this is a fairly pedantic point.
I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME. ----- I would like to use something like git to store the source code to every piece of software I use, and the binaries. That…
It's NixOS.
I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME. ----- I would like to use something like git to store the source code to every piece of software I use, and the binaries. That…
So, as a consumer of the repo, I would clone the repo, and then have a post-checkout hook check the blacklisted file extensions and grab their contents: the magnet link, all that's left is to find peers, download the file and replace the text file with a hardlink to the binary. Maybe this is all too laggy, asynchronous, and mismatched, but I think git+magnet could be a cool combination.
I found this repo [1] that generates the magnet link for you, I would just need to find away to use the repo to make all the contributors peers to each other, so we can download the binaries from whoever is nearest / highest bandwidth from us.
Earlier quoted context omitted.
Actually git “suffers” from two of the problems he listed: cache coherency and, as with all filesystem based approaches, serialization. These don’t matter for git which manages to push all the coherency issues onto the user and which can afford to operate (in computational terms, not human terms) very slowly on small amounts of data. I’m not saying git is slow (it’s gratifyingly fast) but it has a remarkably smaller…
I don't agree with that framing... When you say "cache coherency" you're implicitly assuming some authoritative state. The point of git is that there isn't a single authoritative state. Not all apps will work with that model, but more apps than you think would. I guess the difference is how fine-grained you want the updates to be. For example something like Figma in the browser (a collaborative photoshop) implements…
I think you’re missing the point of the post you refer to.
For git, there _technically_ isn’t a single authoritave state, but for many, if not most, git use cases, there _sociologically_ is. Projects typically have one repo that is _the_repo_: the repo most merges are done to, that releases get built from, whose url you give when you tell people where the project lives, etc.
All other clones of that repo, sociologically, are just caching the main repo and changes you make.
Then, humans have to decide when to flush what part(s) of the write cache to the main server, and when that results in conflicts, humans have to resolve them.
That’s, I think, what “which manages to push all the coherency issues onto the user” means.
Earlier quoted context omitted.
I don't agree with that framing... When you say "cache coherency" you're implicitly assuming some authoritative state. The point of git is that there isn't a single authoritative state. Not all apps will work with that model, but more apps than you think would. I guess the difference is how fine-grained you want the updates to be. For example something like Figma in the browser (a collaborative photoshop) implements…
“The point of git is that there isn't a single authoritative state.“ I think you’re missing the point of the post you refer to. For git, there _technically_ isn’t a single authoritave state, but for many, if not most, git use cases, there _sociologically_ is. Projects typically have one repo that is _the_repo_: the repo most merges are done to, that releases get built from, whose url you give when you tell people whe…
I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME. ----- I would like to use something like git to store the source code to every piece of software I use, and the binaries. That…
I've been bouncing an idea around for a while, on how could I use git as a back-end for a filesharing/chat/collaboration suite -- I think it would work to have a git hook, pre-commit, that replaces all binary files / blacklisted file extensions, with a text file whose contents are the magnet address to download the large binary file over torrent - so the file name, permissions etc don't change, but a small text file…
I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME. ----- I would like to use something like git to store the source code to every piece of software I use, and the binaries. That…
Well, you may already be aware, but Emacs is actually 'network-enabled' in this way through TRAMP, and to a lesser extent, the emacs client/server protocol.
There are also many issues other than file syncing that networking has to solve. And also, even for regular files, there are many ways to interact with them, and many protocols that existing systems can already speak.