Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My manager spent $1M on a backup server that I never used

blog.dijit.sh

Post reply on HN
Print view View original

71–80 of 244 posts

Re: My manager spent $1M on a backup server that I never used

#71

Post by dijit »

jtchang wrote: post #35

How does one force exponential backoff of clients without taking in any load on themselves? Is it some sort of TCP window congestion control futzing? But even that wouldn't work since the client can do whatever the heck it wants.

Hi, I’m the author of the article and I’m horrified by how many spelling mistakes I made. I must have written this in a rage.

To answer your question, since we had control of the clients (since its a game) we used a proof of work challenge on TLS handshake which increased in complexity the more failed attempts you gave us.

Very cheap on the server, very expensive on the client, which effectively rate limited connection attempts.

Re: My manager spent $1M on a backup server that I never used

#72

Post by dijit »

treis wrote: post #47
znpy wrote: post #9

The author makes a good progression in their reasoning but then somehow arrives to the wrong conclusion: > So when someone says that Amazon has invested a lot of money into security I think about the fact that Ubisoft spent $1M on a backup solution that didn’t work for the game that would have had the best use of it. The missing point here is thst ubisoft is/was spending money for something outside of its domain of e…

The whole post is clickbait. The million dollar solution was to store source code that they never ever wanted to lose. It (presumably) worked for that. What it didn't work for was backing up user profiles and being able to quickly restore them.

The environment that the datadomain was hosted in was not allowed to host source code or raw assets, but the bank vault thing was because of lost source code.

The issue is that most games just fire off into the void if they need to backup and restores are never tested.

Re: My manager spent $1M on a backup server that I never used

#73

Post by madrox »

In my observation, the game industry and tech industry are like humans and apes: they have the same ancestor, but at some point in our ancient past we diverged. You rarely see crossover between the two these days, and I doubt we'll see any in another decade.

What I think that means is there's probably a huge opportunity selling five year old tech ideas to the gaming industry

Re: My manager spent $1M on a backup server that I never used

#74

Post by voidfunc »

NikolaNovak wrote: post #48
kubb wrote: post #8

My manager doesn't do almost anything. I wonder why they keep him around. He has like two reports and we're both fully self-directed. He can't understand technical issues, and whenever he proposes something, it's completely untenable due to his lack of understanding what's useful or possible.

It is definitely possible they indeed don't do anything. Fwiw though - a decade ago when I was a sysadmin I had a manager that I was certain never did anything. And then he was replaced. And then... We realized how much politics, uncertainty, churn, screaming, changing requirements, ambiguous priorities and other carp he protected us from :-/ Not saying it's the case with your manager. But managers have duties roles…

The only bad managers I've ever really had are the ones that don't realize they're responsible just as much as their reports when shit goes sideways. To that point, I've had technical and non-technical managers and it doesn't matter, we can speak a different language but as long as we realize we're on the same boat and heading in more or less the same direction things are great. This requires trust, understanding of goals not implementation, and empathy.

Almost every bad manager I've had basically lacked at least two of those last three things if not all three.

I will say, the one big advantage of a technical manager is they can often be used as a good rubber duck for logistics around changes. Should we do {A, B, C} or {X, Y, Z}? You can get into the technical with them and explain the intricate trade-offs of the decision and how that might impact things. Their technical and political knowledge of the situation can give very good clarifying answers.

Re: My manager spent $1M on a backup server that I never used

#75

Post by PragmaticPulp »

wongarsu wrote: post #67

Wait, so Ubisoft spent $1M on a company wide backup system that apparently worked great, it's just that this sysadmin wanted incremental backups and the $1M system wasn't built to support that (blazingly fast ingest, slow online reads). So the sysadmin had to fight a lot to get dedicated hardware to do incremental backups, and eventually got it. I'm not sure Ubisoft is in the wrong at any point here? Everything else…

Agreed. This reads like the $1 million system did what the company needed it to do (safely archive code to prevent more loss of old games) but it didn’t do exactly what this developer wanted.

There are various good points scattered in the article for the author’s specific use case, but it’s written as if the entire company was mistaken to not make this decision revolve around this one developer.

Re: My manager spent $1M on a backup server that I never used

#77

Post by dijit »

znpy wrote: post #9

The author makes a good progression in their reasoning but then somehow arrives to the wrong conclusion: > So when someone says that Amazon has invested a lot of money into security I think about the fact that Ubisoft spent $1M on a backup solution that didn’t work for the game that would have had the best use of it. The missing point here is thst ubisoft is/was spending money for something outside of its domain of e…

I wrote the article in a fit of anger because someone (on hackernews) said that Amazon spent a lot of money on security therefore its better than whatever [you] could possibly need.

The main thesis of the post is that sometimes incentives are not aligned and you still need to think critically about what you bought and what its intended purpose is, the cost of a thing does not mean it fits your need.

You can pull a train with a Bugatti Veyron, but its not the right tool just because it costs as much or more than a train does.

Re: My manager spent $1M on a backup server that I never used

#79

Post by dijit »

PragmaticPulp wrote: post #75
wongarsu wrote: post #67

Wait, so Ubisoft spent $1M on a company wide backup system that apparently worked great, it's just that this sysadmin wanted incremental backups and the $1M system wasn't built to support that (blazingly fast ingest, slow online reads). So the sysadmin had to fight a lot to get dedicated hardware to do incremental backups, and eventually got it. I'm not sure Ubisoft is in the wrong at any point here? Everything else…

Agreed. This reads like the $1 million system did what the company needed it to do (safely archive code to prevent more loss of old games) but it didn’t do exactly what this developer wanted. There are various good points scattered in the article for the author’s specific use case, but it’s written as if the entire company was mistaken to not make this decision revolve around this one developer.

Hi, Author here. thanks for reading my poorly written rant!

The issue was that nobody ever said what the system was designed for and equally nobody was open to the idea of doing things differently until things went pop.

The main thesis of the article is supposed to be that it doesn't matter what something costs or how much money is invested if it doesn't solve your needs.

I wrote it a long time ago in a fit of aggravation over someone on hackernews waxing poetic about how much amazon invested in their security- without much regard as to what it was actually spent on.

Re: My manager spent $1M on a backup server that I never used

#80

Post by lmm »

anyfoo wrote: post #44

> and begin real investigation you will quickly find that many databases that are popular are totally fine losing data. MongoDB being the most famous example that I can think off of the top of my head. Always pisses me off. University teaches the principles of ACID and how hard databases work to adhere to these principles, and then so called "NoSQL" comes along and says "lol we have eventual consistency".

> Always pisses me off. University teaches the principles of ACID and how hard databases work to adhere to these principles, and then so called "NoSQL" comes along and says "lol we have eventual consistency".

99% of the organisations that pride themselves on using "real databases" and ACID aren't actually using those guarantees or gaining anything out of them. Transactions are inherently useless in a web service, for example, because you can't open the transaction on the client, so the part of the system where the vast majority of consistency issues happen (the client server communication) will always be outside the transaction boundary. ACID fanboys love to talk about how all the big name internet companies are built on RDBMSes, and fail to mention that most of them were built on MySQL 3 which never actually had working ACID in the first place.

If MongoDB had come first and SQL/ACID RDBMSes had come after, we'd recognise them for what they are: a grossly overengineered non-solution to the wrong problem.

Post reply on HN