I've just started my career - I wonder what my first fuck-up will be :)
The day I locked everyone out of the company intranet
11–20 of 130 posts
Re: The day I locked everyone out of the company intranet
#12I was working on an old old old "ERP" system written in D3 PICK. It's a database, programming language and OS all in one with roots in tracking military helicopter parts in the 1960's. I was working on it in the mid-2000s.
It had SQL like syntax for manipulating data, but it was interactive. So you would SELECT the rows from the table that you wanted, then those rows would be part of your state. You would then do UPDATE or DELETE without any kind of WHERE, because the state had your filter from the previous SELECT.
It has a fun quirk though - if your SELECT matched no rows, the state would be empty. So SELECT foo WHERE 1=2 would select nothing.
UPDATE and DELETE are perfectly valid actions even without a state...
Working late one night, I ran a SELECT STKM WHERE something that matched nothing, then before I realised I realised my state had no rows matched, I followed up with DELETE STKM.
Yep, the entire Stock Movements table four the last 20+ years of business were gone.
The nightly backup had not run, and I didn't want to lose an entire day of processing to roll back to the previous night.
I spent the entire night writing a program to recreate that data based on invoices, purchase orders, stocktake data, etc. I was able to recreate every record and got home about 9am. Lots of lessons learnt that night.
Re: The day I locked everyone out of the company intranet
#13Always do a select with your criteria before doing a Delete or update. Don’t ask me how I learned this.
That, and some DB tools (line JetBrains DataGrip) block UPDATEs and DELETEs without a WHERE condition.
Re: The day I locked everyone out of the company intranet
#14Always do a select with your criteria before doing a Delete or update. Don’t ask me how I learned this.
Then start a transaction, and only commit when the row numbers match.
Re: The day I locked everyone out of the company intranet
#15I was working on an old old old "ERP" system written in D3 PICK. It's a database, programming language and OS all in one with roots in tracking military helicopter parts in the 1960's. I was working on it in the mid-2000s. It had SQL like syntax for manipulating data, but it was interactive. So you would SELECT the rows from the table that you wanted, then those rows would be part of your state. You would then do UPD…
You recreated a production database of 20 years data by hand overnight?
You deserve an award !!!
Re: The day I locked everyone out of the company intranet
#16Use chatGPT to double check too.
"Yes, you can delete those items via that query."
"But now all has been deleted??!?! WTF HEEELP"
"I apologize for my previous answer, you are correct, you now deleted everything!"
Re: The day I locked everyone out of the company intranet
#17My manager had got me to look at backups.. but for cheap.
I decided on bacula - I had the clients installed on all the computers in the office, and it worked for some small tests.
My manager decided we would try this with a USB drive attached to one of the servers (somehow this didn't seem like a bad idea).
In the morning, very uncaffinated he sent me to the other site - an unmanned basement office with the servers.
Being uncaffinated I forgot the door password and set off the alarm.
I had to go into the office and phone him with the alarm going to get the code to turn the alarm off.
OK, that was stressful but sorted out at least.
I plugged in the hard drive to the selected server and headed back.
Once I got back it turned out all the websites on that server had gone down - trying to send all the backups to this poor USB harddrive had overwhelmed the IO on that-era Linux server and the poor thing just froze.
Fairly soon after I was let go, and joined my friends at a much more fun company making mobile games.
Re: The day I locked everyone out of the company intranet
#18I was working on an old old old "ERP" system written in D3 PICK. It's a database, programming language and OS all in one with roots in tracking military helicopter parts in the 1960's. I was working on it in the mid-2000s. It had SQL like syntax for manipulating data, but it was interactive. So you would SELECT the rows from the table that you wanted, then those rows would be part of your state. You would then do UPD…
You recreated a production database of 20 years data by hand overnight? You deserve an award !!!
Only that day's data was added on top of backup from last night...
Re: The day I locked everyone out of the company intranet
#19My manager had got me to look at backups.. but for cheap. I decided on bacula - I had the clients installed on all the computers in the office, and it worked for some small tests. My manager decided we would try this with a USB drive attached to one of the servers (somehow this didn't seem like a bad idea). In the morning, very uncaffinated he sent me to the other site - an unmanned basement office with the servers.…
It's the responsibility of the technical person to uphold engineering ethics, especially in the face of potentially inadequate recovery and security solutions.
I was once let go from a big name university for refusing to weaken and rush changes to a payment processing network (PCI-DSS) when there was "no time" to review them in detail. That's a future FBI press conference sort of thing when it all comes crashing down. Not long after, all SS#s, DOBs, and deets for every employee was stolen from a "rogue" laptop taken by a consultant, likely to be sold on carder and identity theft forums because of an utter failure at data protection processes. That place was a shitshow because they didn't have the professional ethics or leadership backbone to do what was prudent and necessary.
Re: The day I locked everyone out of the company intranet
#20I was working on an old old old "ERP" system written in D3 PICK. It's a database, programming language and OS all in one with roots in tracking military helicopter parts in the 1960's. I was working on it in the mid-2000s. It had SQL like syntax for manipulating data, but it was interactive. So you would SELECT the rows from the table that you wanted, then those rows would be part of your state. You would then do UPD…
You recreated a production database of 20 years data by hand overnight? You deserve an award !!!
he only recreated the ones not saved by the nightly backup, meaning he used the previous nightly backup and recreated from invoices the last ~23 hours.