Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My manager spent $1M on a backup server that I never used

blog.dijit.sh

Post reply on HN
Print view View original

91–100 of 244 posts

Re: My manager spent $1M on a backup server that I never used

#91

Post by ilyt »

dijit wrote: post #79
PragmaticPulp wrote: post #75

Earlier quoted context omitted.

Agreed. This reads like the $1 million system did what the company needed it to do (safely archive code to prevent more loss of old games) but it didn’t do exactly what this developer wanted. There are various good points scattered in the article for the author’s specific use case, but it’s written as if the entire company was mistaken to not make this decision revolve around this one developer.

Hi, Author here. thanks for reading my poorly written rant! The issue was that nobody ever said what the system was designed for and equally nobody was open to the idea of doing things differently until things went pop. The main thesis of the article is supposed to be that it doesn't matter what something costs or how much money is invested if it doesn't solve your needs. I wrote it a long time ago in a fit of aggrav…

Sorry but it just looks like you picked wrong solution from start and stuck with it.

Vanilla pgsql backup + WAL shipping would just. work. with those constraints.

Backup solutions that take time to read data and only "ramp up" once you start restoring full backup are nothing new, literally in industry for decades, before in form of tape libraries, now in form of amazon glacier and similar. Hell, backup solution where you can mount whole backup in directory is on the fancier side, because generally priorities are elsewhere

Re: My manager spent $1M on a backup server that I never used

#92

Post by jmull »

kubb wrote: post #8

My manager doesn't do almost anything. I wonder why they keep him around. He has like two reports and we're both fully self-directed. He can't understand technical issues, and whenever he proposes something, it's completely untenable due to his lack of understanding what's useful or possible.

If your work is getting done, things aren't falling apart, and he leaves you alone, then he's doing a great job, whatever he does. At the very least his presence will keep other managers from meddling.

Re: My manager spent $1M on a backup server that I never used

#93

Post by anyfoo »

lmm wrote: post #80
anyfoo wrote: post #44

> and begin real investigation you will quickly find that many databases that are popular are totally fine losing data. MongoDB being the most famous example that I can think off of the top of my head. Always pisses me off. University teaches the principles of ACID and how hard databases work to adhere to these principles, and then so called "NoSQL" comes along and says "lol we have eventual consistency".

> Always pisses me off. University teaches the principles of ACID and how hard databases work to adhere to these principles, and then so called "NoSQL" comes along and says "lol we have eventual consistency". 99% of the organisations that pride themselves on using "real databases" and ACID aren't actually using those guarantees or gaining anything out of them. Transactions are inherently useless in a web service, for…

> Transactions are inherently useless in a web service, for example, because you can't open the transaction on the client, so the part of the system where the vast majority of consistency issues happen (the client server communication) will always be outside the transaction boundary.

I think I need more help understanding this. In terms of ACID, a consistency issue is something like, for example: "The client ordered an item, and the count of item availability was reduced by 1, but no actual order was saved in the database. An item is now 'lost' with no trace of where it went and how it disappeared."

There is even another consistency hazard right in this very example: "The count of item availability was supposed to be reduced by 2 because 2 items were ordered by different clients at a same time, but a race condition (the two transactions both individually reading the same old value before writing back the new value) led to the item count only reduced by 1. There is now an extra item 'available' that does not exist."

The canonical example is bank accounts and money transfers, but I figured I'd use something that was more likely in the domain of "99% of organizations" (though that also includes things like ERP, where a lot of other canonical examples apply).

Care to elaborate how transactions are "inherently useless" in a web service here, and how the consistency issue happens "between client and server" communication?

Presumably, in this example, the client/server communication triggering the transaction is the order of an item. There are not that many things that can realistically go wrong in that communication (with checksums on different layers, and nowadays even cryptography, guarding against most arbitrary corruption): The order was not received, or the order was received multiple times. In both those cases, the database remains internally consistent: Yes, the order happening 0 times or 2 times may have been wrong, but the database overall shows the state of what you would expect after 0 or 2 orders. No item has been going inexplicably "missing" or "created" until the next inventory, and the external issue can just be corrected by reissuing or canceling the orders, again with the prospect of an internally consistent database. And even a corrupted order would not change that.

Re: My manager spent $1M on a backup server that I never used

#94

Post by dijit »

ilyt wrote: post #89

> The games industry is weird: It simultaneously lags behind the rest of the tech industry by half-a-decade in some areas and yet it can be years ahead in others. I'd love to see examples of "ahead" coz I literally never saw it. Game dev studios act like automated testing is a new thing... > PostgresSQL performed much better and had the additional benefit of being able to cleanly split write-ahead logs (which are lar…

The “years ahead” was described in the article. We had what amounts to a very well oiled kubernetes installation with mTLS, but on Windows, in C++ and 10 years ago (before Kubernetes was a thing).

Everything else you say is true, I could have chosen another solution for backups but the tradeoff between backup speed (replaying WAL can be time consuming) vs database load (full backups delay replication to replicas) was made before I knew the read/write characteristics of the datadomain. (I didn't even know it was a datadomain- nobody told me anything except giving me an NFS mount point until I started having problems, at which point I spent weeks debugging with a storage engineer from Montreal- this article is a 6-8month span of back and forth distilled).

Regardless: restore times are important, as the wise sages of our industry once said; you don't have a backup until it is tested in a restore - which is what this system was effectively doing. How would you test without reading?

Re: My manager spent $1M on a backup server that I never used

#95

Post by dijit »

ilyt wrote: post #91
dijit wrote: post #79

Earlier quoted context omitted.

Hi, Author here. thanks for reading my poorly written rant! The issue was that nobody ever said what the system was designed for and equally nobody was open to the idea of doing things differently until things went pop. The main thesis of the article is supposed to be that it doesn't matter what something costs or how much money is invested if it doesn't solve your needs. I wrote it a long time ago in a fit of aggrav…

Sorry but it just looks like you picked wrong solution from start and stuck with it. Vanilla pgsql backup + WAL shipping would just. work. with those constraints. Backup solutions that take time to read data and only "ramp up" once you start restoring full backup are nothing new, literally in industry for decades, before in form of tape libraries, now in form of amazon glacier and similar. Hell, backup solution where…

Very likely you're right, as mentioned in another comment (and in the article I think) I didnt know the semantics of the backup system, I’d just tested with a couple of 400G HDDs - then the NFS endpoint I was given to replace those drives was behaving differently and a deep investigation across wide timezones and uncommunicative teams began.

We have the benefit of hindsight now so things can be more clear than they were at the time, nonetheless:

WAL shipping without ever reading sounds hopeful, I don't personally believe in backups that are never verified.

Re: My manager spent $1M on a backup server that I never used

#96

Post by ilyt »

nitwit005 wrote: post #82

> Data Consistency as a Requirement Probably only sort of consistent, I imagine? Game servers tend not to support seamless failover. If the server crashes at the wrong moment, data is going to be lost, regardless of how politely behaved the data backend is. That is, if you kill a boss and successfully pick up an item, you know it'll be saved to DB. If the game crashes before you can pick it up, it's probably just gon…

> Probably only sort of consistent, I imagine? Game servers tend not to support seamless failover. If the server crashes at the wrong moment, data is going to be lost, regardless of how politely behaved the data backend is.

You could just have a let's say "user profile service" where all of the transactions about user profile (items, XP, etc.) go, on top of internal game server data.

That way the important stuff could be send immediately, like "epic or above item drops", and everything else either in batched update (XP, achievements etc.) or periodically (stuff like player rearranging inventory) or at end of the session.

Re: My manager spent $1M on a backup server that I never used

#97

Post by dijit »

jeremyjh wrote: post #87
anyfoo wrote: post #52

Earlier quoted context omitted.

Care to elaborate?

Well, spending 3 months proving that MySQL can reliably store data raised an eyebrow. I mean, we are talking about gamer profiles here. Worrying about whether or not the log is on the same volume as the table really seems like a bit much in that context. Even worrying about fsync is honestly questionable.

If you buy something and we grant it to you and your profile gets corrupted.

We are sued.

Better not risk it.

Also Gamers are the walking definition of a hostile customer. Give them no quarter to call you incompetent.

(also, we are talking enormous binary blobs, not easy to recover from if partially written- most databases don't store large rows in-line for performance reasons)

Re: My manager spent $1M on a backup server that I never used

#98

Post by optimalsolver »

kubb wrote: post #8

My manager doesn't do almost anything. I wonder why they keep him around. He has like two reports and we're both fully self-directed. He can't understand technical issues, and whenever he proposes something, it's completely untenable due to his lack of understanding what's useful or possible.

"When you do things right, people won't be sure you've done anything at all."

- Futurama

Re: My manager spent $1M on a backup server that I never used

#99

Post by dockd »

Gigachad wrote: post #19
kubb wrote: post #8

My manager doesn't do almost anything. I wonder why they keep him around. He has like two reports and we're both fully self-directed. He can't understand technical issues, and whenever he proposes something, it's completely untenable due to his lack of understanding what's useful or possible.

My manager isn’t a programmer but uses ChatGPT to educate himself on a topic and attempts to read the code to see what’s happening before asking one of us if anything is still unclear.

What did they do before ChatGPT? It's only been available for 3 months.

Re: My manager spent $1M on a backup server that I never used

#100

Post by anyfoo »

jeremyjh wrote: post #87
anyfoo wrote: post #52

Earlier quoted context omitted.

Care to elaborate?

Well, spending 3 months proving that MySQL can reliably store data raised an eyebrow. I mean, we are talking about gamer profiles here. Worrying about whether or not the log is on the same volume as the table really seems like a bit much in that context. Even worrying about fsync is honestly questionable.

So "nothing makes sense from a system developers perspective" because gamer profiles are supposedly not valuable, and it's fine to have a system where they may get lost or corrupted, instead of trying to solve the problem with the premise that they may not?

Putting aside that you seem to criticize that premise instead of the system development approach that fulfills the premise, I suspect that the resulting mess of support requests, manual intervention, and community unrest Ubisoft (or any other MMORPG developer) would find themselves in, if, say, "the shiny sword of atrocious doom" that players spent 240 hours sinking their life into obtaining were to disappear regularly, is probably not worth it.

I understand this despite not doing any online gaming myself.

Post reply on HN