Live data from Hacker News

Nobody knows how the whole system works

surfingcomplexity.blog

201–210 of 226 posts

Re: Nobody knows how the whole system works

#201

There are many layers to this. But there is one style of programming that concerns me. Where you neither understand the layer above you (why the product exists and what the goal of the system is) nor the layer below (how to actually implement the behavior). In the past, many developers barely understood the business case, but at least they understood how to translate into code, and could put backpressure on the busin…

You're simply describing the end state of a hyper capitalist system, as outlined by classic Marxist theory.

The core operating principle of which says capitalism requires and promotes systems that enforce the separation of labor from the product they produce. This precludes fellow laborers from meaningfully communicating with each other; knowledge sharing could expose more of how the product "works" after all! Only in final combination, following an undisclosed (to the worker) larger plan, does the product become whole and provide utility.

So not knowing "what happens" in layers "above" and "below" you for your specific work unit is key. This is the "de-skilling" tenet of capitalism and is required for exploitation, conformity, at scale. As labor units become smaller, they require less skill and time to produce, rendering laborers "conditioned to a machine." In other words, workers must acquiesce their skills in the name of "progress" of the system itself. This can easily be sold to the laborers, couched by real world data highlighting the obvious efficiency gains, along with a heavy bonus of having to do less work yourself.

Only by making ever smaller parts of a whole, awhile hiding the utility of those parts produced, can capital rob labor of their value (their skill, their products, their output.)

This very same system lends itself to outcompeting private labor by way of parallelization: as it just so happens that smaller slices of work tend to parallelize better than larger ones. If you can operate at a scale that bespoke creators have no chance of replicating on their own, you "win!" The beautiful moat, the envy of all.

In other words, you're just describing being a worker in a highly efficient capitalist machine! Look! We're almost there! I can just about smell all the "winning" from here...

Re: Nobody knows how the whole system works

#202
Good design allows systems to work without anyone knowing how the whole thing works.

AI and humans are labor that can be put to work designing and vetting such systems. The problem with AI isn’t that it builds things we don’t understand. It’s that we do not have much experience with its failure modes, limitations and risks. There are many unknown unknowns.

It’s directly analogous to the problem of hiring, management, outsourcing and contacting. Sure, we know that labor can produce massive, highly reliable systems nobody fully understands. But how do we coordinate labor, AI and human, to successfully produce the systems we actually need? What failure modes and advantages does AI introduce into the mix for specific projects?

That’s where the uncertainty comes from, not the lack of comprehensive knowledge of the systems themselves.

Re: Nobody knows how the whole system works

#203
post #54
post #39

Earlier quoted context omitted.

> that's a whole different level of ignorance, that's much more dangerous. Why? Is it more dangerous to not know how to fry an egg in a teflon pan, or on a stone over a wood fire? Is it acceptable to know the former but not the latter? Do I need to understand materials science so I can understand how to make something nonstick so I’m not dependant on teflon vendors?

It's relative, not absolute. It's definitely more dangerous to not know how to make your own food than to know something about it - you _need_ food, so lacking that skill is more dangerous than having it. That was my point, really - that you probably don't need to know "materials science" to declare yourself competent enough in cooking so that you can make your own food. Even if you only cooked eggs in teflon pans, y…

When it comes to food prep, I'd agree with you that the more time of your life passes, the more irresponsible is the risk of not knowing how to fry an egg, for example.

At the same time, you only need to learn how to fry an egg once, and you won't forget it. You can go your entire life without ever having to fry an egg yourself - but if you ever had to, you could.

When it comes to coding, the analogy breaks down, I think. Aside from the obviously different stakes (survival versus control of your device), coding also requires keeping up with a lot of changing domain knowledge. It'd be as if an egg is one week savoury, another week sweet, and another a poisonous mushroom. It's also less of a single skill like writing a for loop, and more of a combination of skills and experiments, like organizing a banquet.

Coding today suffers from having too many types of eggs, many of which exist because some communities prefer them. I also don't like the solution "let the LLM do it", but it's much easier. Still, if we manage to stabilize patterns for the majority of use cases, frying the proverbial egg will no longer be as much of domain knowledge, choice or elitism as it is today.

Re: Nobody knows how the whole system works

#204

There are many layers to this. But there is one style of programming that concerns me. Where you neither understand the layer above you (why the product exists and what the goal of the system is) nor the layer below (how to actually implement the behavior). In the past, many developers barely understood the business case, but at least they understood how to translate into code, and could put backpressure on the busin…

Vaguely reminds me of a phenomenon in farming an acquaintance recently told me. Apparently automation is now so great (if you can afford it), that operators only need to sit in their tractors and let the machine do the rest. You're effectively a weight for the seat contact switch.

Re: Nobody knows how the whole system works

#205
post #90

This article is about people using abstractions without knowing how they work. This is fine. This is how progress is made. But someone designed the abstraction (e.g. the Wifi driver, the processor, the transistor), and they made sure it works and provides an interface to the layers above. Now you could say a piece of software completely written by a coding agent is just another abstraction, but the article does not r…

> Now you could say a piece of software completely written by a coding agent is just another abstraction,

Abstractions come with both syntactic and semantic behaviour specifications. In other words their implementation can have bugs. An LLM never has a bug, it always produces "something", whether this is what you wanted is on you to verify.

Re: Nobody knows how the whole system works

#206
post #8
post #5

That's not how things work in practice. I think the concern is not that "people don't know how everything works" - people never needed to know how to "make their own food" by understanding all the cellular mechanisms and all the intricacies of the chemistry & physics involved in cooking. BUT, when you stop understanding the basics - when you no longer know how to fry an egg because you just get it already prepared fr…

Most people have no idea how to hunt, make a fire, or grow food. If all grocery stores and restaurants run out of food for a long enough time people will starve. This isn't a problem in practice though, because there are so many grocery stores and restaurants and supply chains source from multiple areas that the redundant and decentralized nature makes it not a problem. Thus it is the same with making your own food.…

If I could hunt, it wouldn't actually matter, because nearly all the animals I would want are in stables. So all I would need to do is find a large enough rock and throw it at them, until they die. The much larger problem would be to keep all the other humans from doing that before me.

Re: Nobody knows how the whole system works

#207
post #72
post #3

Strange article. The problem isn’t that everyone doesn’t know how everything works, it’s that AI coding could mean there is no one who knows how a system works.

Just because there is someone who could understand a given system, that doesn’t mean there is anyone who actually does. I take the point to be that existing software systems are not understood by anyone most of the time.

I do not know about you all, but I need to understand the system before I can change anything, otherwise I would introduce tons of bugs. Heck, without knowing the system I do not even know what I *want* to change.

Re: Nobody knows how the whole system works

#208
post #4

> Nobody knows how the whole system works True. But in all systems up to now, for each part of the system, somebody knew how it worked. That paradigm is slowly eroding. Maybe that's ok, maybe not, hard to say.

> But in all systems up to now, for each part of the system, somebody knew how it worked. If the project is legacy or the people just left the company that’s just not true.

Which is why such code is routinely thrown away and rewritten from scratch or just not really touched at all.

Re: Nobody knows how the whole system works

#209

Earlier quoted context omitted.

> Granted one person can't know/do everything, watch me try, at least. > but large companies in particular seem allergic to granting you any visibility whatsoever. It's particularly annoying If the blind spot is directly causing customer pain, find metrics that demonstrate the impact. If it ends up driving away your customers, then your company is securing itself to death.

> customer pain > driving away your customers > company death You are implying efficient market theory, which is bunk. Example: Our banks have endless painful papercuts yet most of us don't change banks just because of one pain. We each respond to our own complex of costs and benefits (or risks versus rewards). Second example: I use an iPhone because I judge it to be more secure yet I'm constantly fighting the same b…

> Our banks have endless painful papercuts yet most of us don't change banks just because of one pain.

Only because they're all painful. If there was a bank that was recognized as perfect, people would switch in short order. Switching to another bank that is also painful is not worth the effort.

> I use an iPhone because I judge it to be more secure yet I'm constantly fighting the same bugs and misfeatures that seem to never get fixed/improved.

Only because nobody else sells an iPhone. People would start switching over to other, less buggy iPhone on the market if there was such a thing.

> You are implying efficient market theory, which is bunk.

The efficient market theory says that, in an active market, prices rapidly reflect all publicly available information. How does that apply here, bunk or not?

Post reply on HN