Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We deleted the production database by accident

keepthescore.co

Post reply on HN
Print view View original

311–320 of 456 posts

Re: We deleted the production database by accident

#311

Post by corobo »

qz2 wrote: post #302
corobo wrote: post #301

Earlier quoted context omitted.

Most will hide it away because being truthful will hurt current business or future career prospects because people like yourself exist who want everyone shitting themselves at the prospect of being honest. In a blame free environment you find the underlying issue and fix it. In a blame full environment you cover up the mistake to avoid being fired and some other person does it again later down the line

No. There’s a third option where people accept responsibility and are rewarded for that rather than hide from it one way or another. I have a modicum of respect for people who do that. I don’t for people who shovel it under a rug or point a finger which are the points you are referring to. I’ve been in both environments and neither end up with a positive outcome. If I fuck up I’m the first person to put my hand up. C…

See that leading with a "No." there

That's what we're talking about. I hope you don't have direct reports.

Next time be honest "Just shut the conversation down, everyone's a dumbass, I'm right, you're dumb" it'll be quicker than all this back and forth actually trying to get to a stable middle ground :)

Re: We deleted the production database by accident

#312

Post by qz2 »

corobo wrote: post #311
qz2 wrote: post #302

Earlier quoted context omitted.

No. There’s a third option where people accept responsibility and are rewarded for that rather than hide from it one way or another. I have a modicum of respect for people who do that. I don’t for people who shovel it under a rug or point a finger which are the points you are referring to. I’ve been in both environments and neither end up with a positive outcome. If I fuck up I’m the first person to put my hand up. C…

See that leading with a "No." there That's what we're talking about. I hope you don't have direct reports. Next time be honest "Just shut the conversation down, everyone's a dumbass, I'm right, you're dumb" it'll be quicker than all this back and forth actually trying to get to a stable middle ground :)

That's a tad ironic is it not?

All I am calling for is people to take responsibility.

Re: We deleted the production database by accident

#313

Post by [deleted] »

qz2 wrote: post #312
corobo wrote: post #311

Earlier quoted context omitted.

See that leading with a "No." there That's what we're talking about. I hope you don't have direct reports. Next time be honest "Just shut the conversation down, everyone's a dumbass, I'm right, you're dumb" it'll be quicker than all this back and forth actually trying to get to a stable middle ground :)

That's a tad ironic is it not? All I am calling for is people to take responsibility.

[deleted]

Re: We deleted the production database by accident

#314

Post by inglor_cz »

jurre wrote: post #267
qz2 wrote: post #259

Earlier quoted context omitted.

I disagree. Culturally speaking we like to pat people on their back when they do something stupid and comfort them. But most of the time this isn’t productive because it doesn’t instil the requisite fear required when working out what decision to make. What happens is we have growing complacency and disassociation from consequences. Do you press the button on something potentially destructive because your are confide…

You should have zero fear instilled when pressing any button. The system or process has failed if a single person pressing a button can bring something down unintended. Fix the system/process, don’t “instill fear” onto the person, it’s toxic, plus now you have to make sure any new person on boarded has “fear instilled”, and that’s just hugely unproductive

I recalled Akimov pressing the AZ-5 button in Chernobyl...

Re: We deleted the production database by accident

#317

Post by nucleardog »

stepbeek wrote: post #214
emodendroket wrote: post #122

Earlier quoted context omitted.

There is sense in not building more services than you need. But many folks end up finding it hard to break away from their monolith and then it becomes an albatross. Not sure how to account for that.

If a team doesn’t have the engineering know how to split a monolith into distinct services in a straightforward fashion, then I’m not sure that team will have the chops to start with microservices.

Dealing with existing code and moving it forward in significant ways without taking down production is always much more challenging than writing new code, whatever form those new ways take.

You can get by with one strong lead defining services and interfaces for a bunch of code monkeys that write what goes behind them.

Given an existing monolithic codebase, you can’t specify what high level services should exist and expect juniors to not only make the correct decisions on where the code should land up but also develop a migration plan such that you can move forward with existing functionality and customer data rather than starting from zero.

Re: We deleted the production database by accident

#318

Post by robjan »

fauigerzigerk wrote: post #236
john_moscow wrote: post #139

Earlier quoted context omitted.

It really depends on your database size. This works just fine for ~300MB databases. Git gc takes pretty good care of the fluff and once every couple of years I reset the repository to prune the old snapshots. The big plus is that you can reuse your existing git infrastructure, so the marginal setup costs are minimal. You can always switch to a more specialized solution if the repository size starts bugging you, but d…

>It really depends on your database size. This isn't just about size though. You're storing all customer data on all developer machines. You're just one stolen laptop away from your very own "we take the security of our customers' data very seriously" moment.

Nobody said the devs have access to those repos

Re: We deleted the production database by accident

#319

Post by ClumsyPilot »

qz2 wrote: post #259
skrebbel wrote: post #240

I'm appalled at the way some people here receive an honest postmortem of a human fuck-up. The top 3 comments, as I write this, can be summarized as "no, it's your fault and you're stupid for making the fault". This is not good! We don't want to scare people into writing less of these. We want to encourage people to write more of them. An MBA style "due to a human error, we lost a day of your data, we're tremendously…

I disagree. Culturally speaking we like to pat people on their back when they do something stupid and comfort them. But most of the time this isn’t productive because it doesn’t instil the requisite fear required when working out what decision to make. What happens is we have growing complacency and disassociation from consequences. Do you press the button on something potentially destructive because your are confide…

"fear required when working out what decision to make"

People like you keep making the same mistake, creating companies/organisations/industries/societies that run on fear of failure. We've tried it a thousand times, and it never works.

You can't solve crime by making all punishments hearsh death, we've tried that in 1700 in Britain and crimerate was sky high.

This culture gave us disasters in USSR and famine in China.

The only thing that can solve this problem is structural change.

Re: We deleted the production database by accident

#320

Post by csomar »

Nextgrid wrote: post #89
sushshshsh wrote: post #84

Earlier quoted context omitted.

As a mid level developer contributing to various large corporate stacks, I would say the systems are too complex and it's too easy to break things in non obvious ways. Gone are the days of me just being able to run a simple script that accesses data read only an exports the result elsewhere as an output.

This is why I am against the current trend of over-complicating stacks for political or marketing reasons. Every startup nowadays wants microservices and/or serverless and a mashup of dozens of different SaaS (some that can't easily be simulated locally) from day 1 while a "boring" monolithic app will get them running just fine.

For the most part, we are not complicating stuff. Today's requirements are complicated. We used to operate from the commandline on a single processor. Now things are complicated: People expect a Web UI, High availability, integration with their phone, Email notification, 2FA authentication, and then you have things like SSL/HTTPS, Compliance, and you need to log the whole thing for errors or compliance or whatever.

Sometimes it's simpler to go back to a commandline utility, sometimes it's not.

Post reply on HN