51–60 of 456 posts
[deleted]
Things like this would have been much less likely to happen in the past -- you know, when the developers only had access to the development database.
But then someone had an idea... think of how great it would be if we got rid of the operations folks and gave responsiblity for prod to the developers, too!
If you are using postgres, configure it to keep the WAL logs for at least 24 hours. They could have used point-in-time recovery to not lose any data from this at all.
> Computers are just too complex and there are days when the complexity gremlins win. Wow. But then again it's not like programmers handle dangerous infrastructure like trucks, military rockets or nuclear power plants. Those are reserved for adults
These is pretty common, as devs tool belts have grown longer over time.
I think at some point we will stop automating or reverse some of the automation.
> Computers are just too complex and there are days when the complexity gremlins win. Wow. But then again it's not like programmers handle dangerous infrastructure like trucks, military rockets or nuclear power plants. Those are reserved for adults
I feel that computers make it easier for this danger to be more indirect, however. The examples you give are physical, and even the youngest of child would likely recognise they are not regular items. A production database, meanwhile, is visually identical to a test database, if measures are not made to make it distinct. Adults though we may be, we're human, and humans can make really daft mistakes without the right…
Can you imagine if instead of a physical lock it just said “are you sure you wish to turn on this machine”. “Of course I want to turn it on, that’s why I pressed the button”
Some software makes it a lot harder for the user to mess up now. When deleting a repo on GitLab you have to type the name of the repo before pressing delete and then it puts it in a pending deletion state for a month before it’s actually deleted. Unfortunately for developers we typically get minimal cli tools which will instantly cause a lot of damage without any way to undo.
If I screw up one parameter, instead of deleting only unconfirmed users, I could delete all users. I have two redundant checks, first when the query is run to get the unconfirmed users, and then again checking the user's confirmed status before deleting them. And then I check one more time further down in the code for good measure. Not because I think the result will be different, but just in case one of the lines of code is altered somehow.
I put BIG LOUD comments everywhere of course. But it still terrifies me.
> Computers are just too complex and there are days when the complexity gremlins win. > However, we will figure out what went wrong and ensure that that particular error doesn’t happen again. How can you say statement 2 just after statement 1 ? Isn't statement 1 just plain acceptance of defeat ? And looking at all the replies here, is this a feel good thread for the mistakes you made ?
> Computers are just too complex and there are days when the complexity gremlins win. > However, we will figure out what went wrong and ensure that that particular error doesn’t happen again. How can you say statement 2 just after statement 1 ? Isn't statement 1 just plain acceptance of defeat ? And looking at all the replies here, is this a feel good thread for the mistakes you made ?
I like to think that, by addressing the known bugs as they pop up, over time you can box the complexity gremlins into tighter and more predictable spaces. Though as long as humans are building these systems, that box will always be there, and the predictability of those gremlins' behavior will only go so far.
Earlier quoted context omitted.
...and same credentials apparently also - there are lots of things that could have prevented something like this.
Keep going, I'm writing this down!
As well as pretty much every other privilege they don't legitimately need to use on a daily basis -- which, for prod, should be most of them (quite possibly including "delete").
If or when they really need to delete a ton of rows all in one go, they can be given a (temporary) set of credentials that they can use to do that, once, and which are then revoked immediately afterwards -- after another set of eyes reviews the script they've written to actually perform the operation, of course.
Basic best practices aren't difficult or some secret thing that only the experts know about. It is necessary to actually implement and follow them, however!
Sure, it can be a pain in the ass sometimes. Will it be worth it when it eventually saves your ass one day, "after a couple of glasses of red wine"? Absolutely.
> Computers are just too complex and there are days when the complexity gremlins win. Wow. But then again it's not like programmers handle dangerous infrastructure like trucks, military rockets or nuclear power plants. Those are reserved for adults
One explanation for the author feeling that way is that the system is has too much automation. Being in a situation where you take on more responsibilities of the system at a shallower level leads to less industry expertise. This, as it turns out, places the security of the system in a precarious position. These is pretty common, as devs tool belts have grown longer over time. I think at some point we will stop autom…
Literally just the automation of the test suite. That's 1 automation.
> These is pretty common
? Waiting for FB to delete their db
SSH tunnel from localhost to prod on database port?
Another is to change my terminal theme to a red background before connecting to anything in production...never want to click the ‘psql ...’ tab, run “truncate table app_user cascade” and realize afterwards it was a lingering connection to production...