I’ve reached an interesting point in my career. I started out truly in love with “sitting around and dreaming up new architectural diagrams that can comfortably sit inside [my head].” However, I don’t derive any pleasure from that anymore. Possibly because I’ve had to face the hard reality of what happens when the architecture “outgrow[s] the size of any one person’s head.” I’ve come to be very interested in two things: figuring out how to delay that as long as possible and what to do when you can’t delay anymore. The first seems to be bringing in as little software (in any form) as possible. Dig and dig to find the actual problem and then solve that using as few tools and abstractions that the team can possibly tolerate. The second seems to be about process and Conway’s law. Figure out communication structures to distribute the management of complexity.
The collapse of complex software
261–270 of 304 posts
Re: The collapse of complex software
#262One great tool for fighting complexity is to Always Consider Option Zero: whenever comparing possible solutions to a problem, consider the option (Option Zero) where you simply don’t solve the problem. It’s amazing how much it changes the conversation when you stack up the pros and cons of the status quo, against the pros and cons of each solution. Quite often you realize the best path is much simpler than you though…
I’m definitely gonna try this out. Thanks for bringing it up!
Re: The collapse of complex software
#263Earlier quoted context omitted.
> tackling more complex problems This is the 'induced demand' argument. As perfection seems impossible, and mistakes inevitable, this seems likely. However, the more I look into issues, the more I realize that a very small number of errors introduced early on is what ultimately causes a plethora of them. Just fixing a very small amount of these mistakes would have untold effects on computing over the long-term. That…
This seems like a fantasy considering how big the development teams for these projects are. Multics did not have thousands of developers. Linus was just more practical about software development, was maybe better at building a community, and more importantly focused on x86 commodity HW which was a bet that only Microsoft also happened to make.
The cumulative system design contributing to MULTICS, however, probably was a order of magnitude greater work than the implementation effort. On the shoulders of Giants may be clichèd , but it also seems increasingly as if the contemporary world has decided to deprecate inherited wisdom. Blame awful policy decisions starting under Regan..
Re: The collapse of complex software
#264Earlier quoted context omitted.
> tackling more complex problems This is the 'induced demand' argument. As perfection seems impossible, and mistakes inevitable, this seems likely. However, the more I look into issues, the more I realize that a very small number of errors introduced early on is what ultimately causes a plethora of them. Just fixing a very small amount of these mistakes would have untold effects on computing over the long-term. That…
> As teams get bigger, communication sales factorially, That sounds a bit drastic. Surely at worst the scaling is quadratic?
Correct, sales does have to scale factorially to the development man years. That's the problem with the prevailing economic contradiction. See Jamie Dimon, JPM, three days ago.. I've never witnessed a bulge bracket chief concluding the most public statements with, roughly, "doh, I dunno, it's crazy" before. My father cut his banking teeth in the Great Depression. That was actually much more simple. (Whether because the tools of the trade were more simple or just thankfully..) "Hurricane" is the search term for Dimon's delivery...
Re: The collapse of complex software
#265Earlier quoted context omitted.
> Include an Hardware Asset FK in your Link M2M table. Model each binary link explicitly, so a Y cable = 2-3 different Links that point to the same cable Asset. Or you can have single Links with M2M inputs/outputs. But definitely don't model Y cables explicitly. Our cables have stickers with cable IDs, which are stored in the CMDB (and can be used to trace the endpoints through patch fields, for example). So if you m…
Can't you just have 2 tables? One for 2 end and one for 3 ends?
Or just have a cable type column (which could be a pointer/foreign key back to a table of cable types, so you can add the cable with 4 ends that comes out next year).
Re: The collapse of complex software
#266Earlier quoted context omitted.
The problem is "pulling in a tried and tested solution" means pulling in a slice of the whole stack: CSS, Javascript, client/server communication, API endpoints, data-flow logic, and database schema. People are very very bad at this because they believe that software engineering principles apply only to little tiny portions of this stack. If you actually apply your software engineering principles to the whole stack,…
Kinda reminds me of what they do in some areas of the finance world with Kx System's Kdb+. That entire install is tiny tiny and you get a high performance array language for querying your SSD database with either the main language "k" or a syntactical sugar dialect sitting on top called "q" that is more like SQL. I think of that as optimizing the entire stack for what the customer wants of rapid/high-speed analytics.
KDB is unique for translating "in house" tech to marketable CV entry status. Having Haskell and CS(FB) experience requires educating the recruitment agencies, for comparable recognition. Personally I consider every ill in computing as symptoms of the intellectual property system, from making reading potential prior art (aka learning) verboten to non disclosures making the Peter Principle look like the common cold to covid.
Re: The collapse of complex software
#267Earlier quoted context omitted.
Can't you just have 2 tables? One for 2 end and one for 3 ends?
Or just have a cable type column (which could be a pointer/foreign key back to a table of cable types, so you can add the cable with 4 ends that comes out next year).
Yeah that sounds even better and what I typically see in production databases
Re: The collapse of complex software
#268Earlier quoted context omitted.
I want to believe you, but there are just so many complicated problems IN REALITY that we have to model in software that I don't really see complexity going down. Just an example: around here, most people have a first (given) and a last (family) name. If I don't model that as separate, I have trouble interfacing with other software. If I do, I have trouble with people from other cultures that don't follow that conven…
> We haven't really found a way to reuse these among different technologies. Like, if you once figured out the perfect password reset workflow with Ruby on Rails, and your next job uses Python + Django, you're back to square one. C and C++ libraries with light language-specific wrappers largely serve this purpose, in practice. It's plausible that, say, a PHP postgres client lib and a Node postgres client lib will sha…
>> We haven't really found a way to reuse these among different technologies.
The Common Language Environment on VMS, or its lesser(but infinitely more available hence virtuous by effect) imitation, dot net CLR . FFIs, if you bought a good environment. Sounds like if open source implementation of good foreign interfaces existed, the OP's problem wouldn't exist, let alone system level interoperability.
Re: The collapse of complex software
#269Earlier quoted context omitted.
> tackling more complex problems This is the 'induced demand' argument. As perfection seems impossible, and mistakes inevitable, this seems likely. However, the more I look into issues, the more I realize that a very small number of errors introduced early on is what ultimately causes a plethora of them. Just fixing a very small amount of these mistakes would have untold effects on computing over the long-term. That…
> As teams get bigger, communication sales factorially, That sounds a bit drastic. Surely at worst the scaling is quadratic?
The number of channels is exponential, that's just Metcalfe's law.
The ability to suss-out meaning behind those channels and come to a shared understanding pushes it to factorial. If A is speaking to B and C, A needs to also think about what communication is happening between B and C, and this is different to what B knows about C and C knows about B.
In a carefully balanced classroom you might manage to get everyone knowing the same things, and this will allow knowledge to scale. Think about how much better TV got once they knew that viewers would watch every episode. What happens in software is that specialization quickly comes into play such that this is impossible in any organization that doesn't use mob programming. Other people might as well be speaking a different language.
Next time you are in a Sprint meeting, think about how much you don't understand. Even as the team lead or architect who designs the entire system and whose job it is to understand everything it will be a shocking amount - it's black boxes all the way down. You'll claim that you can't know everything and that anyone who tries will fail.
And this is made worse as the only true way to learn something is to do it, and if you aren't actually challenging yourself on something, anything you learn without doing will just fade away - as Spaced Repetition shows.
The bigger the team, the bigger the software, the more inevitable the collapse.
Re: The collapse of complex software
#270The article refers to this kind of complexity:
[0] https://twitter.com/internetofshit/status/986006653605687296...>