Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

The absolute worst scenario happened

reddit.com

Post reply on HN
Print view View original

51–60 of 566 posts

Re: The absolute worst scenario happened

#51

Post by rebuilder »

Nextgrid wrote: post #3

This doesn’t sound that bad. Unless I’m misunderstanding something, the ERP was used as a source of truth for DNS records, and that mechanism broke. There’s no indication of actual data loss so it should still be in the ERP DB, and it’s a matter of reverse-engineering how it was all put together and then extract the data into a zone file (or into a managed DNS service such as Route 53) at least for the core domains w…

From a reply the poster made in that thread:

"So, we have backups on an encrypted drive, which we can't access (authentication is done via our custom ERP software, which is down as well - in hindsight, a very bad idea). And the backups don't work. We can rebuild the DNS servers from scratch, and the records are probably still safely stored somewhere inone of the 3 different databases our ERP software uses, but we don't have an authentication key or the scripts used to pull the DNS domain / record information from the ERP system anymore. The API's are undocumented, and there's only one key that can access those records."

Elsewhere, they say that two years ago, they found they were unable to restore from backups...

Re: The absolute worst scenario happened

#52

Post by loloquwowndueo »

michaelhah wrote: post #8

How many people had to log into their own work systems and check that things were running in order to be sure it wasn’t their employer?

I know it wasn’t mine because we still have one guy who knows how everything works :) (unless he quit over the weekend oh crap)

Re: The absolute worst scenario happened

#53

Post by viraptor »

These kind of posts are awesome tabletop scenarios. (and fuel for https://twitter.com/badthingsdaily ) Coming up with potential fixes is a good exercise.

Things I would do that I haven't seen mentioned yet:

- Grab images from the relevant servers to have current backups before messing with anything.

- Try tcpdump-ing and decoding the custom KV protocol - other databases are complicated, but things like memcached/redis/... have trivial wire protocol. Maybe it's possible to recover some data by listing all entries.

- Run `strings` on the KV binary or `file` on the data to make sure it's not just some open source project with a proprietary interface. With any luck it's just something like bdb that can be accessed externally.

- Hook up a debugger to the KV and try to recover the keys (or maybe just strace to see if they keys are in any files)

- Start tcpdump on the old DNS IPs. They mentioned they have no idea which domains they own anymore - as long as NS on customer domains wasn't trashed as well, they could collect queries coming in and recover a list from there.

- See if the customer domains can be collected from billing data / some email store of notifications.

(as much as I try to design for no disasters in production, the rush of "this is so FUBAR, what insane thing I can do to fix it" for me is an amazing feeling :) )

Re: The absolute worst scenario happened

#54

Post by protomyth »

le-mark wrote: post #14

It’s easy to imagine there are a lot of these type of time bombs out there, particularly in really old legacy systems (> 20 years for example). I was at one place where even building the application for deployment led to a one week outage when some core people were laid off.

Interestingly, some of these legacy systems run on things like IBM i (AS/400) which have an easy backup story and vendor support for recovery. I can have a new i machine receiving a backup and deployed fairly quickly (day or two) and I'm out in a rural area.

Re: The absolute worst scenario happened

#55

Post by flaxton »

Can’t you just pull the drive from the failed server, extract the DNS entries and build a new DNS server? It doesn’t sound like everything is broken, just that it is failing because the DNS server isn’t answering. Just install Linux on a new box and set up a DNS server on it. It could be done in a day I would expect. Or am I missing something?

Re: The absolute worst scenario happened

#56

Post by arethuza »

xchaotic wrote: post #47
arethuza wrote: post #29

Earlier quoted context omitted.

I had a 6 month notice period in one role, 3 months isn't uncommon for senior roles, in the UK at least.

while I agree that this is common in Europe, if the company ceases to exist, what good is the 3 months notice for the employee? If he/she has any options, I'd leave asap and not worry about amicable terms with a soon to be non-existent entity.

Fair point - in that case I suspect that would be highly dependent on whether there are any assets remaining in the company and how the local legal system treats employees as creditors.

Mind you - if the company is going down the tubes they are highly unlikely to enforce any job contracts.

Re: The absolute worst scenario happened

#57

Post by Kranar »

flaxton wrote: post #55

Can’t you just pull the drive from the failed server, extract the DNS entries and build a new DNS server? It doesn’t sound like everything is broken, just that it is failing because the DNS server isn’t answering. Just install Linux on a new box and set up a DNS server on it. It could be done in a day I would expect. Or am I missing something?

Its encrypted and the encryption keys are lost.

Re: The absolute worst scenario happened

#59

Post by buro9 »

pc86 wrote: post #9
Nextgrid wrote: post #3

This doesn’t sound that bad. Unless I’m misunderstanding something, the ERP was used as a source of truth for DNS records, and that mechanism broke. There’s no indication of actual data loss so it should still be in the ERP DB, and it’s a matter of reverse-engineering how it was all put together and then extract the data into a zone file (or into a managed DNS service such as Route 53) at least for the core domains w…

> > All domain information was wiped out and records became null. > > Every person who knew how the custom DNS system worked has left the company years ago. Our backups are fucked. Our records are wiped from all domain servers out there. Sure sounds like the data is lost to me.

To me it sounds like those people who left can command a very high day rate to help with the immediate aftermath. A high enough rate that even if they are doing hours elsewhere some will be tempted to sign in after their work hours to help out.

A company does not just stop existing or being able to function, deleting DNS records didn't empty the bank account or the ability of the accountants to be directed to pay someone... there are lots of options.

Re: The absolute worst scenario happened

#60

Post by capableweb »

flaxton wrote: post #55

Can’t you just pull the drive from the failed server, extract the DNS entries and build a new DNS server? It doesn’t sound like everything is broken, just that it is failing because the DNS server isn’t answering. Just install Linux on a new box and set up a DNS server on it. It could be done in a day I would expect. Or am I missing something?

It would work if they knew what the DNS entries were but seems they lost all DNS records and don't even know what records they have/had, so spinning up a new DNS server won't help as they don't know what it should serve.

> All domain information was wiped out and records became null [...] Our records are wiped from all domain servers out there [...] We don't even know what domains we own, the listing was hosted in the ERP which is now busted

Post reply on HN