Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My manager spent $1M on a backup server that I never used

blog.dijit.sh

Post reply on HN
Print view View original

111–120 of 244 posts

Re: My manager spent $1M on a backup server that I never used

#111

Post by dijit »

Severian wrote: post #109

I can comment directly on this as I work in the backup sector. First hand knowledge, yada yada yada. Dell EMC DataDomains do have good ingest performance, you can typically throw hundreds of streams at them and they'll greedily gulp it down. And it is true that they are dog slow at restoring data. The reason? They are deduplicating appliances, you have to rehydrate the data, and this can take a very long time dependi…

Thats pretty cool, the datadomain appliance is actually really cool and I definitely appreciate its existence.

I definitely feel like we were using it wrong and I’m not entirely convinced it was my fault.

In my ideal scenario I would have had a weeks worth of point in time backups on a machine in the rack, and replicated the contents to the DD after verification.

Sadly I was denied that “in-rack” solution and was sold the datadomain solution (without it being named) as if it was simply a remote disk, not a fancy appliance.

the main point I tried (and failed) to convey in the article is that a solution can be brilliant and expensive but that doesn't mean its what you meed for the job at hand.

Re: My manager spent $1M on a backup server that I never used

#112

Post by TheRealPomax »

Good old "we spent $1m on it" fallacies. "Exactly, we spent a million on a backup solution that's objectively bad, and we spent 100 times that on the actual code we're trying to secure. So: do we want to have wasted a trivial $1m by redoing the backup part, or do we want to have wasted $100m because of a terrible pretend-backup solution? Because this should be a business no-brainer".

Re: My manager spent $1M on a backup server that I never used

#114

Post by bcrosby95 »

jeremyjh wrote: post #87
anyfoo wrote: post #52

Earlier quoted context omitted.

Care to elaborate?

Well, spending 3 months proving that MySQL can reliably store data raised an eyebrow. I mean, we are talking about gamer profiles here. Worrying about whether or not the log is on the same volume as the table really seems like a bit much in that context. Even worrying about fsync is honestly questionable.

If its linked to things like achievements and stuff, yeah, it's pretty important. I understand this isn't money they're dealing with, but it is time and I would be fairly annoyed if I had dozens of hours into a run and lost an achievement I was targeting due to some server side screw up.

I wouldn't run around cursing their name but I would probably stop playing for a period of time. Whether or not that's a problem depends on their business model.

Re: My manager spent $1M on a backup server that I never used

#115

Post by Gigachad »

dockd wrote: post #99
Gigachad wrote: post #19

Earlier quoted context omitted.

My manager isn’t a programmer but uses ChatGPT to educate himself on a topic and attempts to read the code to see what’s happening before asking one of us if anything is still unclear.

What did they do before ChatGPT? It's only been available for 3 months.

Mostly what he is trying to do is run queries on the database to export csvs. So mostly he would ask one of the devs to write a query and he would then save and tweak it himself. Now ChatGPT can mostly write these basic queries and cut down on the number of requests to devs.

Re: My manager spent $1M on a backup server that I never used

#116

Post by Severian »

dijit wrote: post #111
Severian wrote: post #109

I can comment directly on this as I work in the backup sector. First hand knowledge, yada yada yada. Dell EMC DataDomains do have good ingest performance, you can typically throw hundreds of streams at them and they'll greedily gulp it down. And it is true that they are dog slow at restoring data. The reason? They are deduplicating appliances, you have to rehydrate the data, and this can take a very long time dependi…

Thats pretty cool, the datadomain appliance is actually really cool and I definitely appreciate its existence. I definitely feel like we were using it wrong and I’m not entirely convinced it was my fault. In my ideal scenario I would have had a weeks worth of point in time backups on a machine in the rack, and replicated the contents to the DD after verification. Sadly I was denied that “in-rack” solution and was sol…

Oh yeah, for sure. I got the gist of your article. I thought it was pretty clear.

But overall, planning a backup strategy can take some time. You need to know your goals before you go buy some vendor's hardware. But, in your circumstance, you gotta do what you gotta do.

Re: My manager spent $1M on a backup server that I never used

#117

Post by roughly »

willcipriano wrote: post #110
yoyohello13 wrote: post #59

Earlier quoted context omitted.

Underrated comment. Many people believe their manager does nothing... until they become a manager and realize all the BS their manager protected them from on the daily.

So the idea isn't the manager is incompetent but his managers are so bad they are a net negative and cause so much disruption that a full time employee is required to deal with it.

Alternatively: The broader organization has requirements around accounting, planning, forecasting, resource allocation, etc. that are orthogonal to the engineer's job but critical to the company for legal, regulatory, and financial reasons, and managers exist in part to handle that stuff so that the engineer does not need to.

Re: My manager spent $1M on a backup server that I never used

#118

Post by ilyt »

dijit wrote: post #95
ilyt wrote: post #91

Earlier quoted context omitted.

Sorry but it just looks like you picked wrong solution from start and stuck with it. Vanilla pgsql backup + WAL shipping would just. work. with those constraints. Backup solutions that take time to read data and only "ramp up" once you start restoring full backup are nothing new, literally in industry for decades, before in form of tape libraries, now in form of amazon glacier and similar. Hell, backup solution where…

Very likely you're right, as mentioned in another comment (and in the article I think) I didnt know the semantics of the backup system, I’d just tested with a couple of 400G HDDs - then the NFS endpoint I was given to replace those drives was behaving differently and a deep investigation across wide timezones and uncommunicative teams began. We have the benefit of hindsight now so things can be more clear than they w…

It's particularly nice as it allows to do point in time recovery - you can tell PostgreSQL to replay WALs up to a given point so if say corruption happened because of some code bug you could play the database to the point minute before.

Backup process is two part

* archive logs as they come in - PostgreSQL have hook that will run program with the WAL segment to archive so just need to put whatever you want to use to backup it with there * start backup from PostgreSQL - that will stop it writing to the DB files and only write to WALs - and just copy the database directory files. Then tell PostgreSQL to resume writing to them. No need for anything fancy like file snapshots.

So just copying files really.

Restore is just restore the above and feed it WALs to the chosen point.

We also run slave so master being shot would only kill running transactions, fancier setups I've seen also run "delayed slave" - slave that replays non-current WAL so basically presents view of database from say 1 hour or 1 day ago. That way if something fucks up DB you already have server that is running, just need to replay WALs to the chosen point.

> I don't personally believe in backups that are never verified.

We ended up making backup job lottery. Pick a job out of system and send an email to ticketing system "hey, admin ,restore this job for testing". So far it worked.

One system also have indirectly tested restore, as the production database is routinely anonymized and fed to dev server

We've also baked it into automation we use for deploying stuff so for most things not backing up is harder than backing up. Still, accidents happened...

Re: My manager spent $1M on a backup server that I never used

#119

Post by setr »

willcipriano wrote: post #110
yoyohello13 wrote: post #59

Earlier quoted context omitted.

Underrated comment. Many people believe their manager does nothing... until they become a manager and realize all the BS their manager protected them from on the daily.

So the idea isn't the manager is incompetent but his managers are so bad they are a net negative and cause so much disruption that a full time employee is required to deal with it.

It’s the peers and stakeholders that are the problem; no one knows what they need, and everyone knows what they want, and if they aren’t responsible for the work they’ll always try to include everything and declare it a must-have. And you’ll get those who escalate over nothing, people actively working against you (because they disagree with the project goals and got overridden, or wanted to own it themselves), people who are necessary but apathetic / busy, people who are angry about something else but take it out wherever they get the chance to, changing requirements and sudden timeline shifts/deadlines because of events outside of your control… etc

At least when there’s someone with strict authority involved, these things can be resolved pretty quickly — everyone defaults to authority. But that’s rare; most projects involve multiple domains, each with authority over their dominion, and opinions on what the others need to be doing (and all trying to minimize their own responsibility and risks, by both simplifying the project, but also offloading it onto their peers).

Ultimately coordination between parties is inherently a bitch.

Re: My manager spent $1M on a backup server that I never used

#120

Post by rodgerd »

Severian wrote: post #109

I can comment directly on this as I work in the backup sector. First hand knowledge, yada yada yada. Dell EMC DataDomains do have good ingest performance, you can typically throw hundreds of streams at them and they'll greedily gulp it down. And it is true that they are dog slow at restoring data. The reason? They are deduplicating appliances, you have to rehydrate the data, and this can take a very long time dependi…

> They are best used not as _primary_ storage, but secondary (think 3-2-1 backup rule).

Yeah, these are for archives, not service restoration/disaster recovery scenarios.

Post reply on HN