Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fragile narrow laggy asynchronous mismatched pipes kill productivity

thume.ca

Post reply on HN
Print view View original

21–30 of 91 posts

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#21

Post by dimatura »

chubot wrote: post #12

I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME. ----- I would like to use something like git to store the source code to every piece of software I use, and the binaries. That…

I have been interested in "git for binary data" for a while, mostly for ML/computer vision purposes.

I've tried quite a few systems. Of course, there's git-lfs (which keeps "pointer" files and blobs in a cache), which I do use sometimes - but it has a quite few things I don't like. It doesn't give you a lot of control on where the files are stored and how the storage is managed on the remote side. The way it works means there'll be two copies of your data, which is not great for huge datasets.

Git-annex (https://git-annex.branchable.com/) is pretty great, and ticks almost every checkbox I want. Unlike git-lfs, it uses symlinks instead of pointer files (by default) and gives you a lot of control in managing multiple remote repositories. On the other hand, using it outside of Linux (e.g., MacOS) has always been a bit painful, specially when trying to collaborate with less technical users. I also get the impression that the main developer doesn't have much time for it (understandably - I don't think he makes any money off it, even if there were some early attempts).

My current solution is DVC (https://dvc.org/). It's explicitly made with ML in mind, and implements a bunch of stuff beyond binary versioning. It does lack a few of the features of git-annex, but has the ones I do care about most - namely, a fair amount of flexibility on how the remote storage is implemented. And the one thing I like the most is that it can work either like git-lfs (with pointer files), like git-annex (with soft- or hard-links), or -- my favorite -- using reflinks, when running on filesystems that support it (e.g. APFS, btrfs). It also is being actively developed by a team at a company, though so far there doesn't seem to be any paid features or services around it.

Pachyderm (https://www.pachyderm.com) also seems quite interesting, and pretty ideal for some workflows. Unfortunately it's also more opinionated, in that it requires using docker for the filesystem, as far as I can tell.

Edit: a rather different alternative I've resorted to in the past -- which of course lacks a lot of the features of "git for binary data" -- is simply to do regular backups of data to either borg or restic, which are pretty good deduplicating backup systems. Both allow you to mount past snapshots with FUSE, which is a nice way of accessing earlier versions of your data (read-only, of course). These days, this kind of thing can also be done with ZFS or btrfs as well, though.

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#22

Post by jodrellblank »

lostcolony wrote: post #5

Failure is such a fun thing to think about, and it gets handwaved away so often. So many devs, architects, product owners, etc, just focus on happy path, and leave failure unspecced, unhandled, and just hope it never happens. And then boast about 99% uptime, but once you start questioning them you find out they get weekly pages they have to go investigate (and really the system is behaving weirdly a solid 10% of the…

On the subject of failures, you might like this blog post https://danluu.com/postmortem-lessons/ if you haven't seen it before.

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#24

Post by trishume »

tlarkworthy wrote: post #20

I love the catagorization. But decent software should be distributed. I dislike single teams that begat 10s of miscroservices, but it should be buy, not build, features from specialized 3rd parties. Thus a decent modern installation should be leaning on a ton of 3rd party services (e.g. Identity providers, databases, caches) because they all do a better job than the hand rolled local one. It's how you outsource exper…

I think it should be possible to buy or use software from third parties. One thing I'm disappointed about and think we need better tools to avoid is the fact that third parties provide their tools as services rather than libraries. There's reasons they do that, deploying a library that all your customers can easily use is hard right now, but there's no reason it has to be that way.

Some things can't easily be libraries like databases, but other things like some caches, miscellaneous operations like image resizing (depending on an architecture that can handle the load on those servers), and a bunch of other things could just be libraries.

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#25

Post by at_a_remove »

Yes, I have a little Python library for managing network shares in Windows.

It has things like automatic retries that "back off" slowly, switching to cached IPs in case DNS is down, and checking to see if all of the drive-letters are full and either re-using a letter or creating a "letter-less" share. I had to develop it during a period of great instability within our network. It's ... large and over-engineered, but it just keeps on truckin'.

On the other hand, it has been quite useful going forward, so that's a plus.

I tend to program fairly defensively, in layers, right down to the much maligned Pokemon exception handling. The results don't have the, ah, velocity that is so often praised but they'll be there ticking along years later.

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#26

Post by nemetroid »

trishume wrote: post #24
tlarkworthy wrote: post #20

I love the catagorization. But decent software should be distributed. I dislike single teams that begat 10s of miscroservices, but it should be buy, not build, features from specialized 3rd parties. Thus a decent modern installation should be leaning on a ton of 3rd party services (e.g. Identity providers, databases, caches) because they all do a better job than the hand rolled local one. It's how you outsource exper…

I think it should be possible to buy or use software from third parties. One thing I'm disappointed about and think we need better tools to avoid is the fact that third parties provide their tools as services rather than libraries. There's reasons they do that, deploying a library that all your customers can easily use is hard right now, but there's no reason it has to be that way. Some things can't easily be librari…

> Some things can't easily be libraries like databases

The world's most widely deployed database engine (SQLite) is only available as a library.

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#27

Post by chubot »

trishume wrote: post #16
chubot wrote: post #12

I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME. ----- I would like to use something like git to store the source code to every piece of software I use, and the binaries. That…

This is also an interesting system in that it's an example of how you can get away with a non-distributed system if your problem is small enough but eventually that falls over. Once you get to large corporation monorepos git operations start to get real slow and use too much hard disk, so you end up with them either creating/using a new VCS or doing some complicated undertaking like https://devblogs.microsoft.com/bha…

Yeah I agree, it's sort of an open problem, but I guess a bunch of arrows are pointing toward FUSE.

Distri uses FUSE and it appears Microsoft's GVFS uses FUSE or whatever Windows technology is the equivalent. (My teammates developed Google's equivalent about 14 years ago, using FUSE, so it's something I've used / seen several times.)

FUSE requires some kernel support (a module), which git of course doesn't require. That is a barrier, but perhaps not an insurmountable one. Basically I would like to offload all the "network" work to the OS, so applications are free of that logic.

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#28

Post by chubot »

gumby wrote: post #19
chubot wrote: post #12

I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME. ----- I would like to use something like git to store the source code to every piece of software I use, and the binaries. That…

Actually git “suffers” from two of the problems he listed: cache coherency and, as with all filesystem based approaches, serialization. These don’t matter for git which manages to push all the coherency issues onto the user and which can afford to operate (in computational terms, not human terms) very slowly on small amounts of data. I’m not saying git is slow (it’s gratifyingly fast) but it has a remarkably smaller…

I don't agree with that framing... When you say "cache coherency" you're implicitly assuming some authoritative state. The point of git is that there isn't a single authoritative state.

Not all apps will work with that model, but more apps than you think would.

I guess the difference is how fine-grained you want the updates to be. For example something like Figma in the browser (a collaborative photoshop) implements a lot of custom application-specific sync with CRDTs and so forth.

Maybe you need the really fine-grained updates, or maybe you just need some Github-like site which allows coarse-grained collaboration.

In other words, there can be a network model between "e-mail a .PSD file" and Figma. I think this "in between" would scale to more applications than reinventing sync inside every app. Imagine audio editors, video editors, 3D modellers, etc. Rewriting all those in the style of Figma is prohibtive.

I would rather have open file formats and application-agnostic sync, like git. And I think it's a lot cheaper to develop, although current software business models don't really support its development.

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#29

Post by chubot »

dimatura wrote: post #21
chubot wrote: post #12

I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME. ----- I would like to use something like git to store the source code to every piece of software I use, and the binaries. That…

I have been interested in "git for binary data" for a while, mostly for ML/computer vision purposes. I've tried quite a few systems. Of course, there's git-lfs (which keeps "pointer" files and blobs in a cache), which I do use sometimes - but it has a quite few things I don't like. It doesn't give you a lot of control on where the files are stored and how the storage is managed on the remote side. The way it works me…

Thanks for the response! I have heard of dvc and git annex, and it's probably time to give them another try :)

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#30

Post by tlarkworthy »

trishume wrote: post #24
tlarkworthy wrote: post #20

I love the catagorization. But decent software should be distributed. I dislike single teams that begat 10s of miscroservices, but it should be buy, not build, features from specialized 3rd parties. Thus a decent modern installation should be leaning on a ton of 3rd party services (e.g. Identity providers, databases, caches) because they all do a better job than the hand rolled local one. It's how you outsource exper…

I think it should be possible to buy or use software from third parties. One thing I'm disappointed about and think we need better tools to avoid is the fact that third parties provide their tools as services rather than libraries. There's reasons they do that, deploying a library that all your customers can easily use is hard right now, but there's no reason it has to be that way. Some things can't easily be librari…

The world is distributed. Money is distributed. You need services to interact with the world. The large portion of useful business services cannot be encapsulated into a hermetic library.

If you think that distributed service clients as libraries is enough to solve the issue of distributed computing, that is incorrect as most of the same distributed systems crap will still happen.

Post reply on HN