Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Nobody knows how the whole system works

surfingcomplexity.blog

Post reply on HN
Print view View original

41–50 of 226 posts

Re: Nobody knows how the whole system works

#41

Post by stoneforger »

lijok wrote: post #39
virgilp wrote: post #5

That's not how things work in practice. I think the concern is not that "people don't know how everything works" - people never needed to know how to "make their own food" by understanding all the cellular mechanisms and all the intricacies of the chemistry & physics involved in cooking. BUT, when you stop understanding the basics - when you no longer know how to fry an egg because you just get it already prepared fr…

> that's a whole different level of ignorance, that's much more dangerous. Why? Is it more dangerous to not know how to fry an egg in a teflon pan, or on a stone over a wood fire? Is it acceptable to know the former but not the latter? Do I need to understand materials science so I can understand how to make something nonstick so I’m not dependant on teflon vendors?

You do need to be able to understand nonstick coating is unhealthy and not magic. You do need to understand your options for pan frying for not sticking are a film of water or an ice cube if you don't want to add an oil into the mix. Then it really depends what you are cooking on how sticky it will be and what the end product will look like. That's why there are people that can't fry an egg, people that cook, chefs, and Michelin chefs. Because nuance matters, it's just that the domain where each person wants to apply it is different. I dont care about nuance in hockey picks but probably some people do. But some domains should concern everyone.

Re: Nobody knows how the whole system works

#42

Post by mojuba »

> AI will make this situation worse.

Being an AI skeptic more than not, I don't think the article's conclusion is true.

What LLM's can potentially do for us is exactly the opposite: because they are trained on pretty much everything there is, if you ask the AI how the telephone works, or what happens when you enter a URL in the browser, they can actually answer and break it down for you nicely (and that would be a dissertation-sized text). Accuracy and hallucinations aside, it's already better than a human who has no clue about how the telephone works or where to even begin if the said human wanted to understand it.

Human brains have a pretty serious gap in the "I don't know what I don't know" area, whereas language models have such a vast scope of knowledge that makes them somewhat superior, albeit at a price of, well, being literally quite expensive and power hungry. But that's technical details.

LLMs are knowledge machines that are good at precisely that: knowing everything about everything on all levels as long as it is described in human language somewhere on the Internet.

LLMs consolidate our knowledge in ways that were impossible before. They are pretty bad at reasoning or e.g. generating code, but where they excel so far is answering arbitrary questions about pretty much anything.

Re: Nobody knows how the whole system works

#43

Post by stoneforger »

lynguist wrote: post #29
mamp wrote: post #3

Strange article. The problem isn’t that everyone doesn’t know how everything works, it’s that AI coding could mean there is no one who knows how a system works.

No I think the problem is AI coding removes intentionality . And that introduces artifacts and connections and dependencies that shouldn’t be there if one had designed the system with intent. And that makes it eventually harder to reason about. There is a difference in qualia in it happens to work and it was made for a purpose . Business logic will strive more for it happens to work as a good enough .

Excellent point. The intention of business is profit, how it arrives there is considered incidental. Any product no matter what, as long as it sells. Compounding effects in computing, the internet and miniaturisation, have enabled large profit margins that further compound these effects. They think of this as a machine that can keep on printing more money and subsuming more and more as software and computers are pervasive.

Re: Nobody knows how the whole system works

#44

Post by stoneforger »

ahnick wrote: post #9
mamp wrote: post #3

Strange article. The problem isn’t that everyone doesn’t know how everything works, it’s that AI coding could mean there is no one who knows how a system works.

This happens even today. If a knowledgeable person leaves a company and no KT (or more likely, poor KT) takes place, then there will be no one left to understand how certain systems work. This means the company will have to have a new developer go in and study the code and then deduce how it works. In our new LLM world, the developer could even have an LLM construct an overview for him/her to come up to speed more qu…

Yes but every time the "why" is obscured perhaps not completely because there's no finished overview or because the original reason cannot be derived any longer from the current state of affairs. Its like the movie memento: you're trying to piece together a story from fragments that seem incoherent.

Re: Nobody knows how the whole system works

#47

Post by supriyo-biswas »

PandaStyle wrote: post #13

Perhaps a dose of pragmatism is needed here? I am no CS major, nor do I fully understand the inner workings of a computer beyond "we tricked a rock into thinking by shocking it." I'd love to better understand it, and I hope that through my journey of working with computers, i'll better learn about these underlying concepts registers, bus's, memory, assembly etc Practically however, I write scripts that solve real wor…

There are so many resources, for example, https://cpu.land.

Re: Nobody knows how the whole system works

#48

Post by cess11 »

Yeah, it's not a problem that a particular person does not know it all, but if no one knows any of it except as a black box kind of thing, that is a rather large risk unless the system is a toy.

Edit: In a sense "AI" software development is postmodern, it is a move away from reasoned software development in which known axioms and rules are applied, to software being arbitrary and 'given'.

The future 'code ninja' might be a deconstructionist, a spectre of Derrida.

Re: Nobody knows how the whole system works

#49

Post by conorcleary »

mrkeen wrote: post #16

Adam Jacob It’s not slop. It’s not forgetting first principles. It’s a shift in how the craft work, and it’s already happened. This post just doubled down without presenting any kind of argument. Bruce Perens Do not underestimate the degree to which mostly-competent programmers are unaware of what goes on inside the compiler and the hardware. Now take the median dev, compress his lack of knowledge into a lossy model,…

"I don't need to know about hardware, I'm writing software."

"I don't need to know about software engineering, I'm writing code."

"I don't need to know how to design tests, ____ vibe-coded it for me."

Post reply on HN