Live data from Hacker News

We deleted the production database by accident

keepthescore.co

201–210 of 456 posts

Re: We deleted the production database by accident

#201

Quote: "Note that host is hardcoded to localhost. This means it should never connect to any machine other than the developer machine. Also: of course we use different passwords and users for development and production. We’re too tired to figure it out right now. The gremlins won this time." No they didn't. Instead one of your gremlins ran this function directly on the production machine. This isn't rocket science, ju…

2 cents his hosts file points localhost to the prod db IP

Re: We deleted the production database by accident

#202

Quote: "Note that host is hardcoded to localhost. This means it should never connect to any machine other than the developer machine. Also: of course we use different passwords and users for development and production. We’re too tired to figure it out right now. The gremlins won this time." No they didn't. Instead one of your gremlins ran this function directly on the production machine. This isn't rocket science, ju…

If managed DBs on DigitalOcean are anything like those on AWS, you can not run commands directly on them since SSH is prohibited. EDIT: there's also the deal with different credentials for dev and prod envs.

Re: We deleted the production database by accident

#203

> Computers are just too complex and there are days when the complexity gremlins win. Wow. But then again it's not like programmers handle dangerous infrastructure like trucks, military rockets or nuclear power plants. Those are reserved for adults

Yep, I hate when I'm dealing with a system literally comprised of logic and a magic gremlin shows up to ruin my day. Seems like they though a casual "everyman" type of explanation would suffice, but really who would trust them after this?

I understand that web interfaces are trivial, unimportant work (it s what i do), but how can one sleep with such an unresolved mystery?

Re: We deleted the production database by accident

#204
post #187
post #137

Earlier quoted context omitted.

Not original poster, but up to 2010, default MySQL table type was MyISAM, which does not support transactions.

When a server runs out of memory a lot of strange things can happen. It can even fail while in the middle of a transaction commit. So transactions won't fix this.

No. That is exactly what a transactional DB is designed to prevent. The journal gets appended with both the old and the new data and physically written to disk, and only then the primary data representation (data and B-tree blocks) gets updated in memory, then eventually that changed data is written to DB files on disk. If the app or DB crashes during any stage, it will reconstruct primary data based on journalled, comitted changes. DBs shouldn't attempt to allocate memory during the critical phase, and should be able to recover even on failed allocations at any time by just crashing and let regular start-up recovery clean up. Though a problem on Linux might be memory overcomitting.

Edit: and another problem is disk drives/controller caches lying and reporting write completion when not all data has actually reached stable storage

Re: We deleted the production database by accident

#206

Earlier quoted context omitted.

Or a blog on being unable to drive your kid to an emergency room because you just finished a glass of wine over dinner. A problem with devices of that type is that they only test for a potential source of inability to drive safely. What we want is to test for an inability to drive safely. And while one is easy and might give some quick wins, the drawbacks scare me too much.

Being unable to take your car when your child needs to go to the ER would be terrible. Actually getting in the car while under the influence such of stress and alcohol sounds worse. I know someone who had a glass of wine just before her daughter needed to be brought to the hospital. This was just two days ago. She simply concluded she could not drive. Luckily, she was able to get a taxi.

Maybe we should, as a society, invest in taxis equipped with medical facilities and trained personnel so that they can provide first response medical treatment while on the way to the ER.

I'm sure that would save a lot of lives. An ambulatory medical service, if you will.

Re: We deleted the production database by accident

#207
I did something similar once - I had fiddled with my /etc/hosts and subsequently connected to the production database without realizing. I dropped a table but thankfully it wasn't much of a deal - the monitoring rang the bell and I recreated it a few seconds later. All that happened was that I had logged out several hundred users.

Re: We deleted the production database by accident

#208
post #187
post #137

Earlier quoted context omitted.

Not original poster, but up to 2010, default MySQL table type was MyISAM, which does not support transactions.

When a server runs out of memory a lot of strange things can happen. It can even fail while in the middle of a transaction commit. So transactions won't fix this.

Transactions should fix this. That's what the Write Ahead Log and similar techniques are for.

Re: We deleted the production database by accident

#209

Quote: "Note that host is hardcoded to localhost. This means it should never connect to any machine other than the developer machine. Also: of course we use different passwords and users for development and production. We’re too tired to figure it out right now. The gremlins won this time." No they didn't. Instead one of your gremlins ran this function directly on the production machine. This isn't rocket science, ju…

2 cents his hosts file points localhost to the prod db IP

Yep. Nowadays a kubectl port-forward makes something like this all too easy. They accidentally had the kubecontext point at the production cluster instead of dev, set up the port-forward to the database, and whoops! At least that's how this could happen to me, even with my years of experience in doing unexpected things to production databases.

Re: We deleted the production database by accident

#210

Earlier quoted context omitted.

This is why I am against the current trend of over-complicating stacks for political or marketing reasons. Every startup nowadays wants microservices and/or serverless and a mashup of dozens of different SaaS (some that can't easily be simulated locally) from day 1 while a "boring" monolithic app will get them running just fine.

Microservices are easy to build and throw away these days. In startups time to market is more important than future investment in devops. In terms of speed of delivery they are not worse than monolithic architecture (also not better). For similar reasons SaaS is and must be the default way to build IT infrastructure, because it has much lower costs and time to deploy, compared to custom software development.

If you're talking about a web application or API back end with a smallish startup team, time to market is definitely going to be much longer for a microservices architecture compared to developing a "monolith" in a batteries included framework like eg rails, using a single database.
Post reply on HN