Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

My manager spent $1M on a backup server that I never used

blog.dijit.sh

Post reply on HN
Print view View original

121–130 of 244 posts

Re: My manager spent $1M on a backup server that I never used

#121

Post by guidedlight »

I worked for a university that spent about a million dollars on data storage for a research project. They estimated that they required about a petabyte of storage, so a big expensive storage array was purchased.

In the end the project used just a few terabytes of storage because they miscalculated their requirements.

It was a bit of a running joke that they spent a million dollars, when a $150 SSD would have been fine. They ended up repurposing the storage array (which is when I got involved).

Luckily everyone has now moved to cloud, and life is better.

Re: My manager spent $1M on a backup server that I never used

#122

Post by worthless-trash »

jtchang wrote: post #35

How does one force exponential backoff of clients without taking in any load on themselves? Is it some sort of TCP window congestion control futzing? But even that wouldn't work since the client can do whatever the heck it wants.

I dont think its possible to do it without "any load" The hardware grabbing the packet, putting it into the socket buffers, analysing if its valid, requires some work.

Re: My manager spent $1M on a backup server that I never used

#123

Post by worthless-trash »

nickpeterson wrote: post #56
labster wrote: post #5

Earlier quoted context omitted.

Yeah, but this article is about keeping backups, and Heroku deletes data from paying customers without notifying them first.

Heroku is freeing customers of the burden of compliance with gdpr.

Bless them.

Re: My manager spent $1M on a backup server that I never used

#124

Post by Godel_unicode »

nickpeterson wrote: post #23
MathMonkeyMan wrote: post #16

Earlier quoted context omitted.

I'm beginning to think that making more money by becoming a better developer is hard, and that managing, while difficult in a totally different way, is less hard. At least for the money.

I think people often convince themselves that becoming a manager is a path for those who couldn’t hack it. In reality it’s a completely different set of problems. If you’re a decent developer with phenomenal soft skills, there is a very good chance you’d be better in management or sales than writing code.

Your conclusion sounds an awful lot like the people in your first sentence are right. Anecdotally, this is what I’ve seen. As people progress on the engineering side and the work gets more demanding, increasingly people bail out and make the transition to management.

Re: My manager spent $1M on a backup server that I never used

#125

Post by ilyt »

dijit wrote: post #94
ilyt wrote: post #89

> The games industry is weird: It simultaneously lags behind the rest of the tech industry by half-a-decade in some areas and yet it can be years ahead in others. I'd love to see examples of "ahead" coz I literally never saw it. Game dev studios act like automated testing is a new thing... > PostgresSQL performed much better and had the additional benefit of being able to cleanly split write-ahead logs (which are lar…

The “years ahead” was described in the article. We had what amounts to a very well oiled kubernetes installation with mTLS, but on Windows, in C++ and 10 years ago (before Kubernetes was a thing). Everything else you say is true, I could have chosen another solution for backups but the tradeoff between backup speed (replaying WAL can be time consuming) vs database load (full backups delay replication to replicas) was…

> The “years ahead” was described in the article. We had what amounts to a very well oiled kubernetes installation with mTLS, but on Windows, in C++ and 10 years ago (before Kubernetes was a thing).

I mean we had those things back then too, that's why I was curious. But then I guess we're a bit of minority, we have near-everything under configuration management for over a decade now.

Doing it under windows does seem like an achievement, it isn't exactly OS nice for tinkering.

> Everything else you say is true, I could have chosen another solution for backups but the tradeoff between backup speed (replaying WAL can be time consuming) vs database load (full backups delay replication to replicas) was made before I knew the read/write characteristics of the datadomain. (I didn't even know it was a datadomain- nobody told me anything except giving me an NFS mount point until I started having problems, at which point I spent weeks debugging with a storage engineer from Montreal- this article is a 6-8month span of back and forth distilled).

Seems you're not the only one:

https://www.dcac.com/syndication/if-you-thought-database-res...

https://community.spiceworks.com/topic/1868887-slow-recovery...

I honestly expected the usual "cherry picking files is slow but if you tell the backup software to do full restore it goes somewhat quickly" but it appears I misjudged how shitty the "enterprise" backup software can really be.

I wanted to say "a lot of fault lies on Ubisoft for not documenting upfront the workings and quirks of the system" but if you had similar speed of restores there is no excuse for that and no amount of warnings can fit something that bad...

There are days where I wish company I work for was 100x bigger so I could excuse going "fuck that shit, we will just make our own backup system"... we once priced migration from OSS software to Veeam and it costed more than the running cost for the servers... and few racks of new servers.

Anyway, our solution when we had that problem one time (backup too big to restore quickly except in "DC burning" cases) was 2 tiered backup, some "cheapest used box with many HDDs" acting as first tier for backup with second tier being the usual data store. Which seems like what happened in your case in the end.

And yeah, we also had cases where big fuckups caused management to finally spend the money.

Re: My manager spent $1M on a backup server that I never used

#126

Post by ilyt »

munificent wrote: post #101
ilyt wrote: post #89

> The games industry is weird: It simultaneously lags behind the rest of the tech industry by half-a-decade in some areas and yet it can be years ahead in others. I'd love to see examples of "ahead" coz I literally never saw it. Game dev studios act like automated testing is a new thing... > PostgresSQL performed much better and had the additional benefit of being able to cleanly split write-ahead logs (which are lar…

> I'd love to see examples of "ahead" coz I literally never saw it. Game developers were deeply aware of data-oriented design and optimizing code and data structures for efficient CPU cache usage well before I saw most other areas of industry aware of it. You can find counter-examples, of course, but overall, this is an area of performance that many game developers understand in their bones, and many programmers outs…

Depends on which side you look from. If you look from embedded realtime development side it might be game developers being behind and rest of the industry REALLY behind.

Game is essentially soft realtime system that needs to deliver frame every 16.6ms or gamer is unhappy. Very few pieces of software does optimization like game development does because it is just not that often needed.

Re: My manager spent $1M on a backup server that I never used

#127

Post by Godel_unicode »

rqtwteye wrote: post #50
nickpeterson wrote: post #42

Earlier quoted context omitted.

I think the role of managers should be facilitating productivity. If things are in peoples way, help them.

In reality in most larger companies managers mainly do reporting. There is not much emphasis on productivity.

The sad part is they think that all the reporting and metrics, and the juking of the stats which goes along with it, is making things better. In reality it’s generally keeping things the same but with a pretty coat of paint.

Re: My manager spent $1M on a backup server that I never used

#128

Post by lmm »

dragonwriter wrote: post #107
lmm wrote: post #80

Earlier quoted context omitted.

> Always pisses me off. University teaches the principles of ACID and how hard databases work to adhere to these principles, and then so called "NoSQL" comes along and says "lol we have eventual consistency". 99% of the organisations that pride themselves on using "real databases" and ACID aren't actually using those guarantees or gaining anything out of them. Transactions are inherently useless in a web service, for…

> Transactions are inherently useless in a web service, for example, because you can’t open the transaction on the client (1) There is no reason you couldn’t , in a stateful, connected web app (or a stateful web service) open backend DB transactions controlled by activity on the client. You probably don’t want to, because, “ew, stateful”, and would prefer to use compensation strategies to deal with business transacti…

> (1) There is no reason you couldn’t, in a stateful, connected web app (or a stateful web service) open backend DB transactions controlled by activity on the client. You probably don’t want to, because, “ew, stateful”, and would prefer to use compensation strategies to deal with business transactions that span beyond a database transactions, but you could.

You'd have to break the rules of how web systems are meant to work. HTTP requests are supposed to be independent (and may arrive out of order with each other etc.), holding a long-lived connection between server and client has limited support.

> The backends very often need a consistent view of state and to be able to assure that a sequence of mutating operations against the DB are applied all or nothing;

You never need that if these things are the result of a fixed client request; the set of things to be done can't change, so eventual consistency will serve the purpose. The case where you need ACID is where there's a complex back and forth to be had with the thing that's driving the change within the same transaction - and to the extent that "the thing that's driving the change" is solely on the backend, it's not really a web system.

> “Consistency issues” in the ACID sense do not (cannot, in fact, since “consistency” is a property of database state) happen anywhere other than inside the database. Client server communications have all kinds of issues, but ACID;s C is not one of them.

Small-c consistency (actually I was thinking mainly in the CAP sense). Things like requesting to make a change based on stale data.

Re: My manager spent $1M on a backup server that I never used

#129

Post by bg24 »

NikolaNovak wrote: post #48
kubb wrote: post #8

My manager doesn't do almost anything. I wonder why they keep him around. He has like two reports and we're both fully self-directed. He can't understand technical issues, and whenever he proposes something, it's completely untenable due to his lack of understanding what's useful or possible.

It is definitely possible they indeed don't do anything. Fwiw though - a decade ago when I was a sysadmin I had a manager that I was certain never did anything. And then he was replaced. And then... We realized how much politics, uncertainty, churn, screaming, changing requirements, ambiguous priorities and other carp he protected us from :-/ Not saying it's the case with your manager. But managers have duties roles…

While I agree with your assessment, the lifespan of such companies (lot of red tape) will get shorter and shorter.

Re: My manager spent $1M on a backup server that I never used

#130

Post by Godel_unicode »

optimalsolver wrote: post #98
kubb wrote: post #8

My manager doesn't do almost anything. I wonder why they keep him around. He has like two reports and we're both fully self-directed. He can't understand technical issues, and whenever he proposes something, it's completely untenable due to his lack of understanding what's useful or possible.

"When you do things right, people won't be sure you've done anything at all." - Futurama

This quote has, pound for pound, provided more comfort to terrible leaders than any other. It’s also just not true, good leadership is very obvious.
Post reply on HN