Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

We deleted the production database by accident

keepthescore.co

Post reply on HN
Print view View original

221–230 of 456 posts

Re: We deleted the production database by accident

#221

Post by vorticalbox »

duskwuff wrote: post #157
ethbr0 wrote: post #115

Earlier quoted context omitted.

> Please don't use csv Could you elaborate? I'm interested in the specific reasons.

High-quality software which interacts with real transactional SQL databases is readily available, free, and easy to use. Building an inferior solution yourself, like a CSV-based database, doesn't make your product any better. (If anything, it will probably make it worse .)

It was never suggested to use a csv database, it was suggested that for small amounts of read only data a csv or other format file is a better option and I agree.

Re: We deleted the production database by accident

#222

Post by auroranil »

michelpp wrote: post #83

> Computers are just too complex and there are days when the complexity gremlins win. I'm sorry for your data loss, but this is a false and dangerous conclusion to make. You can avoid this problem. There are good suggestions in this thread, but I suggest you use Postgres's permission system to REVOKE DROP action on production except for a very special user that can only be logged in by a human, never a script. And NE…

Tom Scott made a mistake with a similar outcome as this article, but with an SQL query that is much more subtle than DROP.

https://www.youtube.com/watch?v=X6NJkWbM1xk

By all means, find ways to fool-proof the architecture. But be prepared for scenarios where some destructive action happens to a production database.

Re: We deleted the production database by accident

#223

Post by daniellarusso »

theamk wrote: post #133
dnautics wrote: post #107

Earlier quoted context omitted.

Please don't use csv. At the very least use SQLite. But hosted sqls are probably the smart thing to do.

Do use CSV (and other similar formats) for read-only data which fits entirely in the memory. It is great for data safety -- chown/chmod the file, and you can be sure your scripts won't touch this. And if you are accessing live instance, you can be pretty sure that you won't accidentally break it by obtaining a database lock. Now "csv" in particular is kinda bad because it never got standardized, so if you you have co…

I agree with the JSON suggestion, but what advantage is there to TSV versus CSV?

I have experienced pain with both characters (tab and comma), particularly when I am not the one creating the output file.

Re: We deleted the production database by accident

#224

Post by nullsense »

amingilani wrote: post #99
tempestn wrote: post #88

Earlier quoted context omitted.

Seems unwise to have an employee doing anything with production servers on their first day, let alone while everyone else is asleep.

It does but that was an exceptional role. The company needed emergency patches to a running product while they hired a whole engineering team. As such, I was the only one around doing things, and there wasn't any documentation for me to work off of. I actually waited until nightfall just incase I bumped the server offline because we had low traffic during those hours.

What's the story behind this company/job? Was it some sort of total dumpster fire?

Re: We deleted the production database by accident

#225

Post by tamrix »

Nextgrid wrote: post #89
sushshshsh wrote: post #84

Earlier quoted context omitted.

As a mid level developer contributing to various large corporate stacks, I would say the systems are too complex and it's too easy to break things in non obvious ways. Gone are the days of me just being able to run a simple script that accesses data read only an exports the result elsewhere as an output.

This is why I am against the current trend of over-complicating stacks for political or marketing reasons. Every startup nowadays wants microservices and/or serverless and a mashup of dozens of different SaaS (some that can't easily be simulated locally) from day 1 while a "boring" monolithic app will get them running just fine.

I think we're hitting peak tech. All this "technical" knowledge just dates itself in a year's time anyway.

Eventually, you come to realise that the more tech you've got, the more problems you have. .

Now developers spend more time googling errors and plugging in libraries and webservices together than writing any actual code.

Sometimes I wish for a techless cloudless revolution when we just go back to the foundations of computers and is use plain text wherever possible.

Re: We deleted the production database by accident

#226

Post by akerro »

sushshshsh wrote: post #84
michelpp wrote: post #83

> Computers are just too complex and there are days when the complexity gremlins win. I'm sorry for your data loss, but this is a false and dangerous conclusion to make. You can avoid this problem. There are good suggestions in this thread, but I suggest you use Postgres's permission system to REVOKE DROP action on production except for a very special user that can only be logged in by a human, never a script. And NE…

As a mid level developer contributing to various large corporate stacks, I would say the systems are too complex and it's too easy to break things in non obvious ways. Gone are the days of me just being able to run a simple script that accesses data read only an exports the result elsewhere as an output.

I'm on similar position as you, but I say systems are as complex as their designed made them and it's on you to change it.

Re: We deleted the production database by accident

#227

Post by moksly »

sushshshsh wrote: post #84
michelpp wrote: post #83

> Computers are just too complex and there are days when the complexity gremlins win. I'm sorry for your data loss, but this is a false and dangerous conclusion to make. You can avoid this problem. There are good suggestions in this thread, but I suggest you use Postgres's permission system to REVOKE DROP action on production except for a very special user that can only be logged in by a human, never a script. And NE…

As a mid level developer contributing to various large corporate stacks, I would say the systems are too complex and it's too easy to break things in non obvious ways. Gone are the days of me just being able to run a simple script that accesses data read only an exports the result elsewhere as an output.

Why are those days gone? I do it all the time in an organisation with 10,000 employees. I obviously agree with the parent poster in that you should only do such things with users that have only the right amount of access, but that’s what having many users and schemas are for. I don’t, however, see why you’d ever need to complicate your stack beyond a simple python/powershell script, a correct SQL setup, an official sql driver and maybe a few stores procedures.

I build and maintain our entire employee database with a python script, from a weird non-standard XML”like” daily dump from our payment system, and a few web-services that hold employee data in other requires systems. Our IT then builds/maintains our AD from a few powershell scripts, and finally we have a range of “micro services” that are really just independent scripts that send user data changes to the 500 systems that depend on our central record.

Sure, sure, we’re moving it to azure services for better monitoring, but basically it’s a few hundred lines of scripting that, combined with AD and ADDS, does more than a 1 million USD a year license IDM.

Re: We deleted the production database by accident

#228

Post by Agentlien »

kungtotte wrote: post #206
Agentlien wrote: post #194

Earlier quoted context omitted.

Being unable to take your car when your child needs to go to the ER would be terrible. Actually getting in the car while under the influence such of stress and alcohol sounds worse. I know someone who had a glass of wine just before her daughter needed to be brought to the hospital. This was just two days ago. She simply concluded she could not drive. Luckily, she was able to get a taxi.

Maybe we should, as a society, invest in taxis equipped with medical facilities and trained personnel so that they can provide first response medical treatment while on the way to the ER. I'm sure that would save a lot of lives. An ambulatory medical service, if you will.

Sometimes ambulances are occupied and taking a taxi goes faster. Especially if it's something which isn't immediately life threatening.

I once dislocated my shoulder while on a large trampoline and was unable to get up from my hands and knees due to the intense pain whenever the trampoline wobbled. The ambulance was redirected to more serious injuries three times. I was stuck in that position waiting for two hours before it arrived.

Re: We deleted the production database by accident

#229

Post by junon »

daniellarusso wrote: post #223
theamk wrote: post #133

Earlier quoted context omitted.

Do use CSV (and other similar formats) for read-only data which fits entirely in the memory. It is great for data safety -- chown/chmod the file, and you can be sure your scripts won't touch this. And if you are accessing live instance, you can be pretty sure that you won't accidentally break it by obtaining a database lock. Now "csv" in particular is kinda bad because it never got standardized, so if you you have co…

I agree with the JSON suggestion, but what advantage is there to TSV versus CSV? I have experienced pain with both characters (tab and comma), particularly when I am not the one creating the output file.

Tabs do not appear in common literature. They're easier to justify inputs not have them in order to avoid having a quotation or escaping mess.

Commas are _way_ too common.

CSV is an awful format anyway.

Re: We deleted the production database by accident

#230

Post by junon »

pvorb wrote: post #209
toredash wrote: post #201

Earlier quoted context omitted.

2 cents his hosts file points localhost to the prod db IP

Yep. Nowadays a kubectl port-forward makes something like this all too easy. They accidentally had the kubecontext point at the production cluster instead of dev, set up the port-forward to the database, and whoops! At least that's how this could happen to me, even with my years of experience in doing unexpected things to production databases.

That whole ecosystem is a devexp nightmare. I try to stay away from it entirely having worked with it extensively.

Docker + Kubernetes are the biggest socially-acceptable hacks in the industry at the moment.

Post reply on HN