My manager spent $1M on a backup server that I never used • Hacker News Board

Re: My manager spent $1M on a backup server that I never used

#211

Post by tagyro » Fri, Mar 03, 2023, 2:41 PM UTC

xgbi wrote: post #190

tagyro wrote: post #185

Reading this, one might think that this happens only at X or Y company, but in my experience, this is a lot more common. One example: at a previous job, we spent $1M and almost 1 year to build a (new version of a) process (as in, a website) that served less than 50 customers. The project was launched to great fanfare internally (talks, videos, the whole package) but created no value, even those 50 customers didn't tr…

Yes I confirm, this looks like the story of my life :-)

I'm sorry to hear that, sending you a virtual hug

Re: My manager spent $1M on a backup server that I never used

#212

Post by ilyt » Fri, Mar 03, 2023, 2:45 PM UTC

aprdm wrote: post #144

ilyt wrote: post #118

Earlier quoted context omitted.

It's particularly nice as it allows to do point in time recovery - you can tell PostgreSQL to replay WALs up to a given point so if say corruption happened because of some code bug you could play the database to the point minute before. Backup process is two part * archive logs as they come in - PostgreSQL have hook that will run program with the WAL segment to archive so just need to put whatever you want to use to…

I believe games usually have assets too and not just metadata, would you put gbs of data in Postgres row ?

Assets don't go into database lmao.

Re: My manager spent $1M on a backup server that I never used

#213

Post by ilyt » Fri, Mar 03, 2023, 2:45 PM UTC

xupybd wrote: post #150

ilyt wrote: post #118

Earlier quoted context omitted.

It's particularly nice as it allows to do point in time recovery - you can tell PostgreSQL to replay WALs up to a given point so if say corruption happened because of some code bug you could play the database to the point minute before. Backup process is two part * archive logs as they come in - PostgreSQL have hook that will run program with the WAL segment to archive so just need to put whatever you want to use to…

I'm a bit green when it comes to db back ups. Do you have anything I can read on this approach it sounds really promising but I don't think I understand it?

The docs are good:

https://www.postgresql.org/docs/current/continuous-archiving...

Re: My manager spent $1M on a backup server that I never used

#214

Post by ilyt » Fri, Mar 03, 2023, 2:52 PM UTC

tinco wrote: post #170

dijit wrote: post #94

Earlier quoted context omitted.

The “years ahead” was described in the article. We had what amounts to a very well oiled kubernetes installation with mTLS, but on Windows, in C++ and 10 years ago (before Kubernetes was a thing). Everything else you say is true, I could have chosen another solution for backups but the tradeoff between backup speed (replaying WAL can be time consuming) vs database load (full backups delay replication to replicas) was…

> The “years ahead” was described in the article. We had what amounts to a very well oiled kubernetes installation with mTLS, but on Windows, in C++ and 10 years ago (before Kubernetes was a thing). The revolutionary thing about Kubernetes was not what it did, but that it was an enterprise supported open source version of what everyone was doing internally. And not tailored to a specific domain but fully generic. Bot…

> Both you, I and the other person replying to your comment were running a bespoke system like this in 2014. That's just 3 people semi-randomly meeting in a thread. If you imagine how much engineering work was saved by Kubernetes since then it's mind boggling.

Not if you're one running the k8s cluster. K8S benefits are wholly on dev deploying apps side, not ops.

Hell, in early versions k8s "automation" to build the cluster was so bad it made clusters that would self-destruct after a year, because there was no proper cert management built in. Even now you have huge amounts of code dedicated purely to "care and feeding" of k8s cluster. Sure, you can deploy one with one command, but if something goes wrong or something needs to be debugged you're jumping into a swamp.

We already had CM-automated CA so implementing K8S wasn't really a problem for us (just a bunch of learning) but it absolutely overcomplicates everything involved because vast majority of apps just isn't big enough to reap the benefits.

Especially when you need to herd a bunch of daemons just to have proper monitoring or logging inside it. Or how many apps now need a "sidecar" just to get some stats in it and you have 20 different apps that only job is "fetch some stats from app that doesn't support and push it into "

Re: My manager spent $1M on a backup server that I never used

#215

Post by dijit » Fri, Mar 03, 2023, 3:55 PM UTC

yathaid wrote: post #149

dijit wrote: post #77

Earlier quoted context omitted.

I wrote the article in a fit of anger because someone (on hackernews) said that Amazon spent a lot of money on security therefore its better than whatever [you] could possibly need. The main thesis of the post is that sometimes incentives are not aligned and you still need to think critically about what you bought and what its intended purpose is, the cost of a thing does not mean it fits your need. You can pull a tr…

"The main thesis of the post is that sometimes incentives are not aligned" Sure, but your example of AWS and security is exactly the opposite where incentives are aligned. "but its not the right tool just because it costs as much or more than a train does. " Yes, and you didn't investigate the impedance mismatch between your tool (PgBackRest) and your backup system. What this has to do with your stray bullet to AWS I…

>where incentives are aligned.

If I had been streaming and not reading then Ubisofts solution would have been the right one.

But the situation was as such: I didn't know what the incentives or intentions were; and others didn't either because all they thought was that "the investment had been made".

Turning off your brain is the wrong solution, you have to check that your incentives are the same as Amazons

Re: My manager spent $1M on a backup server that I never used

#216

Post by mikepurvis » Fri, Mar 03, 2023, 4:03 PM UTC

JohnFen wrote: post #108

mikepurvis wrote: post #33

Earlier quoted context omitted.

Having been both, being a really good manager is hard in a very different set of ways. Depending what itch you're trying to scratch, and what opportunities are available in your org, it might be just as rewarding to be a tech lead, feature lead, mentor, cross-team liaison, steering committee member, or possibly other roles rather than team lead / manager.

Yes, this. The only time I'm OK with being in a manager role is in my own company. Working for someone else, you couldn't pay me enough to do that job. That's because I'm a dev to the core, that's where my interest and skills are. Being a manager is a different beast entirely and, while I can do it, I find it very unpleasant.

I think what I struggled with most in my time managing a small team was feeling that I lacked agency.

Particularly with "soft" objectives like improving morale, building team cohesiveness, helping people set/advance career goals, etc, I just didn't feel like I had a ton in terms of levers to pull and resources to draw on. And I lacked the experience and training to know how I could either advocate for those resources, or achieve the objectives in ways that wouldn't have a lot of cost and disruption attached. The leadership course I took was focused on personality type assessments, and conflict resolution as a very abstract kind of exercise ("try to see and validate all sides, practice non-violent communication") rather than a concrete, like, here's how to navigate when your direct report says they're quitting if they doesn't get a 20% raise, and upper management says lol no way.

Anyway, the pay scale at my org has equivalent IC roles at each manager grade, and in the end I've found I can have more of the type of influence and leadership that I desire by exerting soft power over the decision making processes in the areas that I most care about— which are mostly related to technical direction and strategy.

Re: My manager spent $1M on a backup server that I never used

#217

Post by mikepurvis » Fri, Mar 03, 2023, 4:35 PM UTC

csa wrote: post #54

mikepurvis wrote: post #33

Earlier quoted context omitted.

Having been both, being a really good manager is hard in a very different set of ways. Depending what itch you're trying to scratch, and what opportunities are available in your org, it might be just as rewarding to be a tech lead, feature lead, mentor, cross-team liaison, steering committee member, or possibly other roles rather than team lead / manager.

> tech lead, feature lead, mentor, cross-team liaison, steering committee member All roles that young folks dream about becoming when they grow up. Grandma: What do you want to be when you grow up, honey? Granddaughter: A CROSS-TEAM LIASON!!! Yay!!!

I mean... the whole point is that that stuff is not a titled role. It's responsibility you can choose to take on as a senior IC that helps extend your reach, influence, and visibility into what is going on elsewhere in the company, making your voice that much more valuable and insightful when you're pulled into those meeting to talk long term technical direction and strategy.

I don't think it's too hard to imagine how to explain this in a way that might excite a child, but at the end of the day, they look to us for their cues on whether to be excited about Daddy's promotion, whether it's to "Team Lead" or "Staff Engineer".

Re: My manager spent $1M on a backup server that I never used

#218

Post by Mavvie » Fri, Mar 03, 2023, 4:52 PM UTC

tsegratis wrote: post #201

Mavvie wrote: post #151

Earlier quoted context omitted.

I see what you mean, but with ACID you can tell the user right away if the order was successfully placed. With eventual consistency you don't know how long it'll take so you either show them a loading spinner that takes an indeterminate amount of time, or what? Not saying that the spinner solution is wrong (as it definitely scales better), but the immediate feedback is something you lose with eventual consistency, by…

Im no expert but my recent conclusion from writing an ordering system is customers can instruct their banks to roll back many payments a month or even a year after the fact So actual ACID transactional consistency requires a year. So after basic checks just say you'll give them what they want and accept payment -- since you'll need failure recovery anyway for lack of delivery, refunds, acts of God etc So accept and p…

The idea I've seen repeated here a lot is to aim for strong consistency internal to your system, even though the real world/things external to your system will be eventually consistent.

So the inventory create order interactions are internal and can be strongly consistent, while the realities of the banking system are external and will be eventually consistent.

Just because you can't make everything strongly consistent doesn't mean you shouldn't bother at all; it can still eliminate entire classes of errors (in this case, errors around inventory being over-consumed leading to cancelled orders are completely avoided with ACID)

edit: I'm also no expert when it comes to distributed systems. Most of my experience is with transactional systems which I know quite well, and distributed is still a mystery to me. I am very open to new ideas around it and don't have a ton of confidence in my comments here.

Re: My manager spent $1M on a backup server that I never used

#219

Post by Mavvie » Fri, Mar 03, 2023, 5:09 PM UTC

lmm wrote: post #205

Mavvie wrote: post #151

Earlier quoted context omitted.

I see what you mean, but with ACID you can tell the user right away if the order was successfully placed. With eventual consistency you don't know how long it'll take so you either show them a loading spinner that takes an indeterminate amount of time, or what? Not saying that the spinner solution is wrong (as it definitely scales better), but the immediate feedback is something you lose with eventual consistency, by…

> I see what you mean, but with ACID you can tell the user right away if the order was successfully placed. With eventual consistency you don't know how long it'll take so you either show them a loading spinner that takes an indeterminate amount of time, or what? It's doing the same thing either way though; in either world you can even wait for the order to be processed, or not. In an event sourcing system you'll com…

> in an ACID RDBMS you'll wait potentially arbitrarily long for your commit to execute (and maybe if you're lucky your database has got a deadlock detector, but what are you going to if it tells you you hit a deadlock? 100% of the time I've seen the answer is "backoff and retry").

I don't agree with the conclusions from this. re: deadlocks, these can be prevented as they are only possible in certain situations, and have mitigations (keep transactions short, never acquire the same locks in a different order, ...). For something "simple" like atomically decrementing an inventory count and then inserting a new row into an orders table, deadlocks (or locking problems at all) are not possible. Of course as you make your system more complex it becomes more likely though, and that's a very fair argument of why ACID systems won't scale as well in general.

But I would still maintain that, even if you have a commit that times out, it does so in an atomic way. It takes at most your "statement timeout" (should be a few seconds probably), and then you can (in deterministic time) show the user an error message. This is still an improvement for the user experience over showing a "your order has been placed" message and then later cancelling it due to overconsumption of inventory.

I appreciate your replies by the way! I haven't been convinced yet that eventual consistency can provide an equally good experience as ACID for this use case, but you've made me think about things in a new way.

Re: My manager spent $1M on a backup server that I never used

#220

Post by dragonwriter » Fri, Mar 03, 2023, 5:25 PM UTC

lmm wrote: post #205

Mavvie wrote: post #151

Earlier quoted context omitted.

I see what you mean, but with ACID you can tell the user right away if the order was successfully placed. With eventual consistency you don't know how long it'll take so you either show them a loading spinner that takes an indeterminate amount of time, or what? Not saying that the spinner solution is wrong (as it definitely scales better), but the immediate feedback is something you lose with eventual consistency, by…

> I see what you mean, but with ACID you can tell the user right away if the order was successfully placed. With eventual consistency you don't know how long it'll take so you either show them a loading spinner that takes an indeterminate amount of time, or what? It's doing the same thing either way though; in either world you can even wait for the order to be processed, or not. In an event sourcing system you'll com…

There is a difference between theoretical worst case and practical experience. Whether the direct-update experience or the event sourcing experience on this point is better depends on a lot of factors (many of which are dimensions of scale.) Neither is categorically ideal, you've got to understand the particular application.

Of course, when using an ACID RDBMS, you can also very trivially use a CQRS/event-sourcing approach for some flows with append-only event log tables and a separate process applying them to query tables that other clients only read and direct update for other flows.