Oracle engineers caused five days software outage at U.S. hospitals

Title should be "Oracle execs caused five days software outage at US hospitals". If the systems helped save lots of life, you bet the engineers wouldn't be thanked for that and it probably wouldn't even be a news article

I'm not so sure. I did an experiment last year. I was writing a prototype in a language I'm not well versed in, using unfamiliar tools and only somewhat familiar libraries. I made heavy use of ChatGPT and other tools to get the job done. (Well, actually, I wrote several pieces of software this way, but only one major for-work prototype).

Some observations:

Initial velocity was high. It felt like I was getting things done. This was exciting and satisfying.

The code I wrote was structurally unsound as it ended up having no overall plan. This became more and more evident as the prototype was completed. Yes, it was a prototype, but usually I use prototypes to think about structure. (Choosing the right way to structure a problem is a key part of solving the problem)

My retention of new knowledge was terrible. Ditto for developing any deep understanding of the tools and libraries I was using. I had skipped the entire process by which I usually learn and just had the LLMs tell me solutions. This provided a poor basis for turning the prototype into production code because I had to catch up on all the thinking that should have been part of producing the prototype. And I had to sit down and read the documentation.

LLMs are only as good as your prompts. You actually need to know quite a bit in order to formulate good prompts. Being a fairly experienced programmer (and also having a some knowledge of LLMs and a former career in web search) I had significant advantages novices do not have.

Now imagine a novice who lacks the experience to see the shortcomings of their code (hey, it works doesn't it!?) and the ability to introspect and think about why they failed.

Half of the time the LLM would lead me astray and pursue paths that resulted in poor solutions when better solutions would have been more obvious if I'd just read documentation. It is easy to become focused on exhausting paths that the LLM sent you down in the first place and poke it to give you something that works.

I have nearly 40 years of programming experience. It scared me how stupid relying too much on a LLM made me. Yes, it got me started quickly and it feels faster. However the inflection point that comes some time into the learning process, where things start to click, didn't materialize. Mostly I was just poking the LLM in various ways to make it spit out solutions to tiny parts of the problem at a time.

I still use LLMs. Every day. But I use it more as a fancy search engine. Sometimes I use it to generate ideas - to try detect blind spots (solutions I don't know about, alternative approaches etc). Having been burnt by LLMs hallucinating I consistently ask LLMs to list their sources and then go and look at those.

LLMs are *NOT* mentors. A mentor isn't someone who does the thinking and the work for you. A mentor is also not an alternative to reference material or the means by which you find information. You're expected to do that yourself. A mentor is not someone who eliminates the need to grind through things. You have to grind through things to learn. There is no alternative if you are going to learn.

A mentor is someone who uses their experience and insight to help your personal growth.

> engineers conducting maintenance work mistakenly deleted critical storage connected to a key database

I'd say poor process management. Why is an engineer even deleting critical storage (I take that to mean that they are deleting something of a file-system). Perhaps they where dropping a database, but you wouldn't do it like that in a critical environment. You'd disable the database access first, and then after some time, weeks, you'd drop the database, after doing a final backup.

It could also be disconnecting a SAN, deleting a storage pool, something like that, but your process should say: Check of read activity, off-line the storage, anything non-destructive, and the only later, once you've verified that everything runs without this resource, do you delete.

At previous jobs I've worked with healthcare system. You have processes, you follow those proesses to the letter and you never delete anything as your first step. Deleting is almost always done by going into read-only mode and ageing out the data.

The fact that recovery time is four days tells me that no one followed a single procedure. Because there should be a written step by step plan, including recovery and risk assessment and when the change manager sees: "Recovery time five days" they will ask questions and stop you.

This is probably the only answer less sexy than technical debt.

I made a similar experience. The database just stopped working one day, triggered by a normal restart without any updates or patches. Fixed the configuration, tested it, restarted multiple times, everything works. After a week it was not working again, without me being able to fix it except doing a fresh reinstall on a clean VM.

Luckily more and more customers switched to Postgres and I no longer have to deal with it.

It’s called vibe coding now.