Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Fragile narrow laggy asynchronous mismatched pipes kill productivity

thume.ca

Post reply on HN
Print view View original

81–90 of 91 posts

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#81

Post by dimatura »

kortex wrote: post #55
dimatura wrote: post #21

Earlier quoted context omitted.

I have been interested in "git for binary data" for a while, mostly for ML/computer vision purposes. I've tried quite a few systems. Of course, there's git-lfs (which keeps "pointer" files and blobs in a cache), which I do use sometimes - but it has a quite few things I don't like. It doesn't give you a lot of control on where the files are stored and how the storage is managed on the remote side. The way it works me…

+1 for DVC. Setting up the backing store can be some extra work if you are doing that yourself, but after that it's a breeze. What do you use for the backing store? Git-lfs has been a pain in my seat since my first use of it. Most of the issues stem from the pointer files that have to be filtered/smudged pre/post commit. Haven't used git-annex myself, but I have heard from coworkers that cross-OS is a pain.

Mostly S3. I used to do SSH, but these days I can afford to keep the data in the cloud. I do appreciate the possibility of migrating to other stores if needed in the future, though - might have to soon, for $reasons.

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#82

Post by jupp0r »

hinkley wrote: post #72
m463 wrote: post #65

Earlier quoted context omitted.

Software engineering evolves to get the most out of your hardware. If you sort of turn that idea around in your head, software will always end up trying to barely work on the shittiest hardware. Think about it. Chips work on 30 nm. What's our next step, to make 30 nm more solid and reliable? No, it's to go to 20 nm or 10 nm and get the same software to work. The same goes with systems stuff. Do we work on having much…

I've heard it said that the reason bits of the Roman aqueducts are still standing is that they were overbuilt by at least a factor of 2. The craftsman often wants to overbuild something. It's part of what gets them out of bed in the morning, or lets them sleep at night. What if you - or someone else - got your margin of error wrong, like the Citibank building? A little extra to deal with unexpected situations isn't s…

If you compare the aqueducts to software, you might draw the completely wrong conclusions. Software that was extremely well built in the 70s is mostly gone (or if it's still there, it's not exactly admired by todays software craftsmen (looking at you, airline ticket booking systems)).

What makes software endure over time is its ability to react to changes of wildly different varieties (market shifts, changing hardware, new technology, ...). The Linux kernel might still be popular in 30 years, but it will likely look very different from today. If it looks the same in the way the aqueducts do, it won't be relevant anymore and will most likely be forgotten eventually.

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#83

Post by zrm »

colechristensen wrote: post #63
btreecat wrote: post #38

So I have been thinking about software projects lately, and I have come to the conclusion that a lot of these tools/solutions exist to "build houses" when most of us are just throwing together lean-to sheds and dog houses. Software projects today are naturally more complex and have more complex tooling the same way building a house today requires more knowledge and skill than it did 50 years ago. Then there are some…

French cooking's mise en place or first order retrievability by Adam Savage or Lean Engineering or whatever. There's a common thread among them, having what you need, when you need it, where you need it, and understanding how to use it. This is what is lacking. The original Unix philosophy build by greybeards and university linguistics and language professors had this at it's core. The "do one thing well" combined wi…

> Not using knowledge and tools to amplify what I do, but spending 60% of my time confused as hell about something which should be obvious and is only obvious afterwards, 20% trying to convince people of things they're often reluctant to believe, and 20% actually using built up knowledge and tools to do many many things very quickly. It's frustrating.

It also creates some nasty perverse incentives. Because if you spend the hours to find the best way to do something that should be simple, it often turns out to actually be simple. API function #487 does just exactly what you need, if only it hadn't taken seven hours to find it. So then you spend a day writing five lines of code that should have taken half an hour, but in the end you have a good solution.

But there is an alternative to that. You can reinvent the wheel. Instead of spending your time understanding the unnecessarily complicated thing with poor documentation, write some new code to do just your thing.

And who looks more productive to the boss? The one who spent all day to write five lines of code, or the one who added a complex new feature that clearly took a lot of time and effort (but will now take ten times more effort to maintain)?

This is especially nasty because it's not always an obvious answer. Sometimes the existing wheel is made of glass and is shaped like a triangle and reinventing it is totally worth it, and then you spend all day discovering that and still have to spend tomorrow doing what a lazier person did yesterday.

Whereas without the unnecessary complexity the right answer would have been obvious much sooner.

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#84

Post by zrm »

breeny592 wrote: post #74
colechristensen wrote: post #63

Earlier quoted context omitted.

French cooking's mise en place or first order retrievability by Adam Savage or Lean Engineering or whatever. There's a common thread among them, having what you need, when you need it, where you need it, and understanding how to use it. This is what is lacking. The original Unix philosophy build by greybeards and university linguistics and language professors had this at it's core. The "do one thing well" combined wi…

> Complexity. This is the enemy, the second enemy is bad attempts to reduce complexity which often end up adding more complexity than they take away, just harder to find. This is true at every level of the systems design process - often by trying to make a system "simpler" i.e. less complex for the end user, the complexity is shifted further down the stack into application code, or even to the underlying infrastructu…

There is a difference between inherent complexity and manufactured complexity.

Inherent complexity can only be moved around, but even then you still want to move it to a place that can handle it properly. If the best you can do is a complex decision tree then it should at least be well-audited and well-documented instead of having a dozen separate buggy ones that all give different results for unknown reasons.

But much of today's complexity isn't inherent. It's manufactured. It's just not required to be there at all and there is much to be gained by taking it out.

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#85

Post by hoseja »

hinkley wrote: post #72
m463 wrote: post #65

Earlier quoted context omitted.

Software engineering evolves to get the most out of your hardware. If you sort of turn that idea around in your head, software will always end up trying to barely work on the shittiest hardware. Think about it. Chips work on 30 nm. What's our next step, to make 30 nm more solid and reliable? No, it's to go to 20 nm or 10 nm and get the same software to work. The same goes with systems stuff. Do we work on having much…

I've heard it said that the reason bits of the Roman aqueducts are still standing is that they were overbuilt by at least a factor of 2. The craftsman often wants to overbuild something. It's part of what gets them out of bed in the morning, or lets them sleep at night. What if you - or someone else - got your margin of error wrong, like the Citibank building? A little extra to deal with unexpected situations isn't s…

But it won't look as good on the quarterly report. What profit has the managerial class extracted from the aqueduct?

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#86

Post by ChrisMarshallNY »

MauranKilom wrote: post #10
dylan604 wrote: post #7

Earlier quoted context omitted.

every. damn. time. i'm a freelancer, so there's not a dev team behind me. the client likes what i made, and wants to start using it. realizing this, i've started spending much more time on the UI/UX (even though i'm not one of those guys) to at least make the tool useable in a dogfooding way.

Maybe it's not 100% relevant to your case, and maybe you've already seen it, but just in case: https://www.joelonsoftware.com/2002/02/13/the-iceberg-secret...

Joel Spolsky is my hero. There are so many nails he’s been hitting on the head for decades.

Engineering (to me) is a craft; not just a vocation.

I also think that the software industry’s Ponce de Leon obsession might be part of the problem.

Younger folks tend to “dream big,” which I think may be a factor behind the ageism.

“They haven’t had some grizzled old fart tell them that it can’t be done!” is the argument that I’ve heard.

Tru dat, but one of the things that older folks have, is experience getting things done, often, highly optimized, polished, and of excellent quality. That tends to require a lot of tenacity and patience.

Shipping is boring.

Ever watch a building go up? A good prefab looks complete after three months, but doesn’t open for another nine months. It looks awesome and shiny, but is still behind a rent-a-fence. What gives?

That’s because all that interior work; the finish carpentry, the drywall, the painting, etc., take forever, and these are the parts of the building that see everyday use, so they need to be absolutely spot-on. The outside is mostly a pigeon toilet. It doesn’t need to be as complete; a solid frame and watertight is sufficient. They just needed it to keep the rain out, while the really skilled craftspeople got their jobs done.

I like to make stuff polished, tested and complete. I don’t like making pigeon toilets.

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#87

Post by rkangel »

zrm wrote: post #83
colechristensen wrote: post #63

Earlier quoted context omitted.

French cooking's mise en place or first order retrievability by Adam Savage or Lean Engineering or whatever. There's a common thread among them, having what you need, when you need it, where you need it, and understanding how to use it. This is what is lacking. The original Unix philosophy build by greybeards and university linguistics and language professors had this at it's core. The "do one thing well" combined wi…

> Not using knowledge and tools to amplify what I do, but spending 60% of my time confused as hell about something which should be obvious and is only obvious afterwards, 20% trying to convince people of things they're often reluctant to believe, and 20% actually using built up knowledge and tools to do many many things very quickly. It's frustrating. It also creates some nasty perverse incentives. Because if you spe…

This is why good documentation is such a wonderful feature. I like Elixir for lots of reasons but the documentation for the core language and particularly for Phoenix (the primary web framework) is the best technical documentation I have seen in any context, ever.

It makes such a difference to your ability to get things done with unfamiliar libraries etc.

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#88

Post by steve76 »

What's the opposite?

"Sturdy broad quick harmonious complimentary pipes boost productivity"

Details are inescapable, whether you are painting or composing or writing software. Ideas have holes in them and designs fall apart in your hands during implementation.

The best approach I found is to just write down mistakes. I'm building software that reverses the freelance market. Instead of an endless list of buyers requests, there are dedicated stages buyers and programmers select, and everyone works together. Instead of tracking project goals, mistakes are track and there's specialized production stages to software design.

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#89

Post by microtherion »

chubot wrote: post #12

I think git is a good model for what would otherwise be "laggy async and mismatched" distributed systems. It has a fast sync algorithm, and after you sync, everything works locally on a fast file system. You explicitly know when you're hitting the network, rather than hitting it ALL THE TIME. ----- I would like to use something like git to store the source code to every piece of software I use, and the binaries. That…

git has some robustness problem over slow, fragile connections, though. It will not corrupt data, but if your connection cannot stay up long enough to clone an entire repo, you need to resort to ugly workarounds.

Re: Fragile narrow laggy asynchronous mismatched pipes kill productivity

#90

Post by RandyRanderson »

TLDR he means microservices. We now have a generation that doesn't even recall a non-MS world.

Tristan, the generation between us has created the IT world you describe. You'll probably spend the next 20 years of your career dealing with that mess. Sorry about that.

Post reply on HN