Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fragile narrow laggy asynchronous mismatched pipes kill productivity

thume.ca

Post reply on HN
Print view View original

51–60 of 91 posts

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#51

Post by nitwit005 »

I've found people have these problems inside of their datacenter, where there is reliable low latency bandwidth, but where things might rebooted due to upgrades or maintenance.

Common example is data being pushed between systems with HTTP. Take the simplest case of propagating a boolean value. You toggle some setting in the UI, and it sends an update to another system with an HTTP request, retrying on a delay if it can't connect. This has two problems. The first is that if the user toggles a setting on and then off, you can have two sets of retries going, producing a random result when the far end can be connected to again. The second, is that the machine doing the retries might get rebooted, and people often fail to persist the fact that a change needs to be pushed to the other system.

I've seen this issue between two processes on the same machine, so technically you don't even need a network.

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#53

Post by jancsika »

> Untrusted: If you don’t want everything to be taken down by one malfunction you need to defend against invalid inputs and being overwhelmed. Sometimes you also need to defend against actual attackers.

Sorry, but unless your centralized alternative is only used internally by troglodytes you have to at least defend against invalid inputs.

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#55

Post by kortex »

dimatura wrote: post #21
chubot wrote: post #12

I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME. ----- I would like to use something like git to store the source code to every piece of software I use, and the binaries. That…

I have been interested in "git for binary data" for a while, mostly for ML/computer vision purposes. I've tried quite a few systems. Of course, there's git-lfs (which keeps "pointer" files and blobs in a cache), which I do use sometimes - but it has a quite few things I don't like. It doesn't give you a lot of control on where the files are stored and how the storage is managed on the remote side. The way it works me…

+1 for DVC. Setting up the backing store can be some extra work if you are doing that yourself, but after that it's a breeze.

What do you use for the backing store?

Git-lfs has been a pain in my seat since my first use of it. Most of the issues stem from the pointer files that have to be filtered/smudged pre/post commit.

Haven't used git-annex myself, but I have heard from coworkers that cross-OS is a pain.

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#56

Post by shijie »

evadne wrote: post #11

Do you have a moment to talk about our Lord and Saviour, Erlang/OTP?

My thoughts exactly! As the Elixir code at my company continues to grow at a rapid rate, writing OTP services for everything has allowed me to never have to think about entire classes of bugs and edge cases simply by virtue of the patterns inherent in OTP and Elixir/Erlang.

Right tool for the right job.

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#57

Post by kortex »

tlarkworthy wrote: post #30
trishume wrote: post #24

Earlier quoted context omitted.

I think it should be possible to buy or use software from third parties. One thing I'm disappointed about and think we need better tools to avoid is the fact that third parties provide their tools as services rather than libraries. There's reasons they do that, deploying a library that all your customers can easily use is hard right now, but there's no reason it has to be that way. Some things can't easily be librari…

The world is distributed. Money is distributed. You need services to interact with the world. The large portion of useful business services cannot be encapsulated into a hermetic library. If you think that distributed service clients as libraries is enough to solve the issue of distributed computing, that is incorrect as most of the same distributed systems crap will still happen.

Totally agree. I have found that all of the pain points OP article mentions, are things that still crop up in monoliths, and by hardening against them, the app is more antifragile. Ex:

Fragile: writing fault tolerant (because the endpoint is unreliable) hedges against the case when unforseen conditions throw a spanner into the works.

Async/Laggy: some operations just take longer. Being able to gracefully route control flow around blockages improves performance.

Pipes: I've found it's helpful to have ser/de interfaces throughout the program, as it forces you to use immutable data structures - aka functional programming. This helps reduce state space.

Put together, I find having various "shear lines" in a monolith codebase, where I could split into a service if I wanted, greatly improves robustness.

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#60

Post by nybble41 »

jazzyjackson wrote: post #46
chubot wrote: post #12

I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME. ----- I would like to use something like git to store the source code to every piece of software I use, and the binaries. That…

I've been bouncing an idea around for a while, on how could I use git as a back-end for a filesharing/chat/collaboration suite -- I think it would work to have a git hook, pre-commit, that replaces all binary files / blacklisted file extensions, with a text file whose contents are the magnet address to download the large binary file over torrent - so the file name, permissions etc don't change, but a small text file…

What you are describing sounds a lot like git-annex[1] with the IPFS remote[2]. There is also a bittorrent remote[3] but it doesn't handle uploading content; you have to create the .torrent file or magnet link yourself and add it to the repo, at which point git-annex will download the content automatically on checkout.

[1] https://git-annex.branchable.com/ [2] https://git-annex.branchable.com/special_remotes/ipfs/ [3] https://git-annex.branchable.com/special_remotes/bittorrent/

Post reply on HN