Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We automated everything except knowing what's going on

eversole.dev

Post reply on HN
Print view View original

51–60 of 73 posts

Re: We automated everything except knowing what's going on

#51

Post by aetherson »

kennethops wrote: post #45
bluGill wrote: post #40

You cannot understand everything. That has been the case since long before AI. I have a vague idea how the linux kernel works, and I could figure it out (I once found and fixed a bug in FreeBSD device drivers) - but I don't, I just trust it works. I've never looked at sqlite to understand how it works - I know enough SQL to be dangerous and trust it works. I know very in depth how the logging framework of my project…

>You cannot understand everything. I 100% agree with this in a individual person sense, but in a humanity sense someone does understand linux very deeply and is very intentional on how they change it which to me is how I gain trust in it. does trust change when the entire SLDC is AI?

Genuine question: is there a big inherent difference between "I don't understand this thing but I think this other human does," and "I don't understand this but I think this other AI does"?

If your answer is "yes," do you think that's inherent to the (metaphysical?) fact of it being AI or to specific limitations to current AI? If the latter, what changes to AI would let you trust it?

Re: We automated everything except knowing what's going on

#52

Post by jollyllama »

andai wrote: post #21

There's a funny angle to all this. There was an article last year where the author asked AI for a web app. It installed a gigabyte of node modules and crashed on startup. He told it to calm down and just use php, it gave him 100 lines with no dependencies that worked the first time. The Pieter Levels stack :) Of course, this is ideal for solo entrepreneur. If you are employed, then you cannot finish it in 100 lines.…

You don't have to go whole-hog and go back to PHP, a reasonable Django application would suffice :)

But the point remains: the NPM monoculture is indefensible.

Re: We automated everything except knowing what's going on

#53

Post by FromTheFirstIn »

seethishat wrote: post #47

Abstractions have been happening since the 1970s when ASM was replaced by the C Programming Language. From there we got C++ (look, it actually has a string type that most humans understand!) then we got memory safe managed languages like Go that is almost human readable, runs almost everywhere and doesn't have buffer overflows. ASM was machine specific. C was portable but required expert programmers. C++ was even mor…

Abstractions truncate the decision space of the layer above them by making understandable trade offs. LLMs don’t abstract anything, your code is still in python or php or Go. It just feels like they abstract if you don’t understand the output since not understanding the layer down is what we associate with non-leaky reliable abstractions. LLMs are abstractions the same way that your code editor is an abstraction- it’s not a layer, it’s an interface.

Re: We automated everything except knowing what's going on

#54

Post by philipstorry »

Not a bad article - thanks!

Others are pointing out that you cannot understand everything - and that's true enough.

But you only need to understand what's important. The experience of a good expert helps you to find that out.

As a systems administrator the recent AWS outage in the Middle East is the best recent example. There will be roughly three types of companies, separated by their understanding:

- Don't Understand - these companies thought that the cloud would handle this kind of thing for them, and are probably going to be doing a lot of finger-pointing in the near future.

- Do Understand, Don't Care - these companies did understand that high availability meant going multi-region, but decided against it for whatever reason. Probably cost vs perceived likelihood. These companies know that they've made a mistake. Short term they're wondering how to survive it, long term they'll be re-assessing their risk acceptance. Many may decide to stay single-region, but at least understand why.

- Do Understand, Do Care - these companies will simply be checking that their procedures worked for any manual parts of their failover, plus possibly looking at any improvements they can make given the real-life experience they've gained.

An LLM is just going to tell you how to implement it. It's not going to be thinking "what sort of availability do we require?", it may never start that conversation unless explicitly prompted. And even then it's going to return consensus opinions, which may not be what you want when evaluating risk.

I'd love to think a lot of companies will be looking at this event and updating their own risk register or justifying their existing risk decisions for hosting. But let's be honest - most won't even have thought about it, and won't until it goes wrong.

Re: We automated everything except knowing what's going on

#55

Post by afry1 »

iammjm wrote: post #41
afry1 wrote: post #2

"The future belongs to whoever understands what they just shipped." Perfect summary. It's like we invented a world where you can finally, _finally_ speedrun an enormous legacy codebase and all patted ourselves on the back like that was a good thing.

We are speedrunning legacy "codebases" all the time. Or do you conjure up your own pickaxe, mine your own minerals, produce your own electricity, and construct your own computers and networks first before you go off to develop an application? Would you even know how to do those things? That is all enormous legacy codebase that we speedrun all the time. Just add one more to it.

I sure don't.

But when I'm using all of those things (pickaxe, mineral mine, power station, internet network hub), I know that there was a thinking human being that took some measure of human care and consideration when creating them. And that there are people on the other side of the economic transaction to talk to or hold accountable when something goes wrong.

Re: We automated everything except knowing what's going on

#56

Post by valdork59 »

bluGill wrote: post #40

You cannot understand everything. That has been the case since long before AI. I have a vague idea how the linux kernel works, and I could figure it out (I once found and fixed a bug in FreeBSD device drivers) - but I don't, I just trust it works. I've never looked at sqlite to understand how it works - I know enough SQL to be dangerous and trust it works. I know very in depth how the logging framework of my project…

"In short, I suggest that the programmer should continue to understand what he is doing, that his growing product remains firmly within his intellectual grip. It is my sad experience that this suggestion is repulsive to the average experienced programmer, who clearly derives a major part of his professional excitement from not quite understanding what he is doing. In this streamlined age, one of our most undernourished psychological needs is the craving for Black Magic and apparently the automatic computer can satisfy this need for the professional software engineer, who is secretly enthralled by the gigantic risks he takes in his daring irresponsibility. For his frustrations I have no remedy......"

Re: We automated everything except knowing what's going on

#57

Post by bluGill »

aetherson wrote: post #51
kennethops wrote: post #45

Earlier quoted context omitted.

>You cannot understand everything. I 100% agree with this in a individual person sense, but in a humanity sense someone does understand linux very deeply and is very intentional on how they change it which to me is how I gain trust in it. does trust change when the entire SLDC is AI?

Genuine question: is there a big inherent difference between "I don't understand this thing but I think this other human does," and "I don't understand this but I think this other AI does"? If your answer is "yes," do you think that's inherent to the (metaphysical?) fact of it being AI or to specific limitations to current AI? If the latter, what changes to AI would let you trust it?

I don't know. AI has an understand of some really complex things, but it also does some really stupid things. Depending on which it did most recently for me I change my answer.

The question is does AI understand well enough to maintain that thing for whatever maintenance I need to do in the future?

Re: We automated everything except knowing what's going on

#58

Post by raesene9 »

caseyohara wrote: post #48
raesene9 wrote: post #33

Earlier quoted context omitted.

There's a massive difference between launching a piece of software and launching a successful business. Over the last couple of months I've seen a load of new "product launches" in my niche but when you look at them they're largely vibecoded and don't show deep understanding and sustainability, so it's pretty likely you'll never see them as successful businesses. Looking at some of the related places like /r/sideproj…

Then, respectfully, what is the point? Does the trillions-of-dollars AI industry exist to support a few hobbyists building niche products to scratch their own itch? I thought the promise here is increased productivity, presumably in the economic sense. There seems to be a lot of hype, and has been for years, but I’m not seeing it materialize as actual economic output. Surely by now there should be lots of businesses…

Whilst I have no special knowledge, my expectation is it'll do both. If you reduce the barriers to coding you'll get more code, both at the hobbyist/one-person level and also at the large corp level.

Whether that translates into more value for those larger corps is the trillion dollar question :) Writing code is a small part of the process of finding and shipping features that customers want, so it remains to be seen how much LLM tools translate it.

I think it's fairly widely accepted that from a financial standpoint we're in an AI/LLM bubble. There has been more investment than we're likely to see financial benefits, but it's impossible to predict to what degree (if you can predict that and the timing you can make a lot of money!!)

Re: We automated everything except knowing what's going on

#59

Post by whynotmaybe »

bluefirebrand wrote: post #46
whynotmaybe wrote: post #44

I'm still balancing on whether we "need" to know what's happening. Very few understand deeply what's happening within the computer between the cou and the bridges and the rest. The fdiv bug in 1994 took us all by surprise because we were in a situation where bug couldn't exist in hardware, because it either works or it doesn't. When I'm using firebase or aws, I don't know the underlying system, I don't know why some…

> I'm still balancing on whether we "need" to know what's happening. Of course we do. Otherwise we start trying to water crops with Brawndo > Very few understand deeply what's happening within the computer between the cou and the bridges and the rest But it's very very important that those people deeply understand it. We cannot replace their actual knowledge with LLM approximations of their knowledge

>Otherwise we start trying to water crops with Brawndo

But it got electrolytes !

For now it seems we're veering towards even more specialization in roles.

The concept of "Full Stack" might disappear because, well any LLM can center a div now and create the sql query that fills it with data.

When asking a LLM about "any sre interview question that LLM got wrong ?", it identifies the questions and the corrects answers.

I should take some time to discuss with an LLM to see their answer for one of my last assignment where the underlying issue revealed by bad performance was simply that the client shouldn't have used relational db for almost immutable documents.

And that they should have bought a math library instead of writing their own.

Re: We automated everything except knowing what's going on

#60

Post by irickt »

kennethops wrote: post #45
bluGill wrote: post #40

You cannot understand everything. That has been the case since long before AI. I have a vague idea how the linux kernel works, and I could figure it out (I once found and fixed a bug in FreeBSD device drivers) - but I don't, I just trust it works. I've never looked at sqlite to understand how it works - I know enough SQL to be dangerous and trust it works. I know very in depth how the logging framework of my project…

>You cannot understand everything. I 100% agree with this in a individual person sense, but in a humanity sense someone does understand linux very deeply and is very intentional on how they change it which to me is how I gain trust in it. does trust change when the entire SLDC is AI?

SDLC ? software development life cycle ?
Post reply on HN