Nobody knows how the whole system works
surfingcomplexity.blog
Nobody knows how the whole system works
1–10 of 226 posts
Re: Nobody knows how the whole system works
#2> Nobody knows how the whole system works
True.
But in all systems up to now, for each part of the system, somebody knew how it worked.
That paradigm is slowly eroding. Maybe that's ok, maybe not, hard to say.
Re: Nobody knows how the whole system works
#3Strange article. The problem isn’t that everyone doesn’t know how everything works, it’s that AI coding could mean there is no one who knows how a system works.
Re: Nobody knows how the whole system works
#4> Nobody knows how the whole system works True. But in all systems up to now, for each part of the system, somebody knew how it worked. That paradigm is slowly eroding. Maybe that's ok, maybe not, hard to say.
> But in all systems up to now, for each part of the system, somebody knew how it worked.
If the project is legacy or the people just left the company that’s just not true.
Re: Nobody knows how the whole system works
#5That's not how things work in practice.
I think the concern is not that "people don't know how everything works" - people never needed to know how to "make their own food" by understanding all the cellular mechanisms and all the intricacies of the chemistry & physics involved in cooking. BUT, when you stop understanding the basics - when you no longer know how to fry an egg because you just get it already prepared from the shop/ from delivery - that's a whole different level of ignorance, that's much more dangerous.
Yes, it may be fine & completely non-concerning if agricultural corporations produce your wheat and your meat; but if the corporation starts producing standardized cooked food for everyone, is it really the same - is it a good evolution, or not? That's the debate here.
Re: Nobody knows how the whole system works
#6But people are expected to understand the part of the system they are responsible for at the level of abstraction they are being paid to operate.
This new arrangement would be perfectly fine if they aren't responsible when/if it breaks.
Re: Nobody knows how the whole system works
#7Strange article. The problem isn’t that everyone doesn’t know how everything works, it’s that AI coding could mean there is no one who knows how a system works.
Including the AI, which generated it once and forgot.
This is going to be a big problem. How do people using Claude-like code generation systems do this? What artifacts other than the generated code are left behind for reuse when modifications are needed? Comments in the code? The entire history of the inputs and outputs to the LLM? Is there any record of the design?
Re: Nobody knows how the whole system works
#8That's not how things work in practice. I think the concern is not that "people don't know how everything works" - people never needed to know how to "make their own food" by understanding all the cellular mechanisms and all the intricacies of the chemistry & physics involved in cooking. BUT, when you stop understanding the basics - when you no longer know how to fry an egg because you just get it already prepared fr…
Most people have no idea how to hunt, make a fire, or grow food. If all grocery stores and restaurants run out of food for a long enough time people will starve. This isn't a problem in practice though, because there are so many grocery stores and restaurants and supply chains source from multiple areas that the redundant and decentralized nature makes it not a problem. Thus it is the same with making your own food. Eventually if you have enough robots or food replicators around knowing how to make food becomes irrelevant, because you always will be able to find one even if yours is broken. (Note: we are not there yet)
Re: Nobody knows how the whole system works
#9Strange article. The problem isn’t that everyone doesn’t know how everything works, it’s that AI coding could mean there is no one who knows how a system works.
This happens even today. If a knowledgeable person leaves a company and no KT (or more likely, poor KT) takes place, then there will be no one left to understand how certain systems work. This means the company will have to have a new developer go in and study the code and then deduce how it works. In our new LLM world, the developer could even have an LLM construct an overview for him/her to come up to speed more quickly.
Re: Nobody knows how the whole system works
#10> Nobody knows how the whole system works True. But in all systems up to now, for each part of the system, somebody knew how it worked. That paradigm is slowly eroding. Maybe that's ok, maybe not, hard to say.
> But in all systems up to now, for each part of the system, somebody knew how it worked. If the project is legacy or the people just left the company that’s just not true.
> If the project is legacy or the people just left the company that’s just not true.
Yeah, that's why I said "knew" instead of "knows".