Working on complex systems: What I learned working at Google • Hacker News Board

Re: Working on complex systems: What I learned working at Google

#111

Post by econ » Thu, May 15, 2025, 7:20 PM UTC

dietr1ch wrote: post #107

vkou wrote: post #101

Earlier quoted context omitted.

If the case is as simple as you describe, surely there's more than one owner of the code that can approve this, if one guy is being unreasonable. If there is actually only one owner that can approve changes to the package, there's something really weird and unusual about that project setup, or it's someone's internal hobby project that they wrote five years ago and semi-maintain, in which case, I have to wonder why y…

> If someone absolutely insists on being an asshole, escalate. That's doubling down on time spent on contributing back. It's usually cheaper to workaround the issue once you notice it'll be way harder than it should be (not hard at all).

I would think the person is more interesting and more relevant than the button. One doesn't create hurdles when I'm trying to work. You just don't do it.

I've allowed people to build a whole obstacle course one time. Decades later it stil has me randomly burst out in laughter. It's like hoarding technical debt until nothing in the code base makes sense or even looks familiar. You just don't do that...

Re: Working on complex systems: What I learned working at Google

#112

Post by prmph » Thu, May 15, 2025, 7:28 PM UTC

nottorp wrote: post #2

Let's add a post scriptum: Whatever you're working on, your project is not likely to be at Google's scale and very unlikely to be a "complex system".

Complex is orthogonal to Large. Some small to medium scale systems address an incredibly complex problem space. Some large systems are solving relatively simple problems. Of course I do agree that size introduces it's own complexity.

Re: Working on complex systems: What I learned working at Google

#113

Post by amdivia » Thu, May 15, 2025, 7:35 PM UTC

RenThraysk wrote: post #44

There is a certain amount of irony when the cookie policy agreement is buggy on a story about complicated & complex systems. Clicking on "Only Necessary" causes the cookie policy agreement to reappear.

Had the same issue

Re: Working on complex systems: What I learned working at Google

#114

Post by pfannkuchen » Thu, May 15, 2025, 7:53 PM UTC

miki123211 wrote: post #86

repeekad wrote: post #36

Earlier quoted context omitted.

The phrase thrown around was “collaboration headwind”, the idea was if project success depends on 1 person with a 95% chance of success, project success also had a 95% chance. But if 10 people each need to succeed at a 95% chance, suddenly the project success likelihood becomes 60%… In reality, lazy domain owners layered on processes, meetings, documents, and multiple approvals until it took 6 months to change the te…

Another side of this coin is that the expected payoff from a project depends on how many unrelated projects your organization is engaging in , which is deeply counterintuitive to most people. Every project carries with it three possibilities: that of success, where the company makes money, that of failure, where the company does not, and that of a "critical failure", where the project goes so wrong that it results in…

This is really insightful, I hadn’t considered this dynamic before.

I wonder if a related intensifier is that as a company grows larger it tends to follow the letter of the law rather than the spirit of the law, which results in less buffer space between the behavior and the law (and hence higher lawsuit risk).

Re: Working on complex systems: What I learned working at Google

#115

Post by pfannkuchen » Thu, May 15, 2025, 7:54 PM UTC

YZF wrote: post #108

miki123211 wrote: post #86

Earlier quoted context omitted.

Another side of this coin is that the expected payoff from a project depends on how many unrelated projects your organization is engaging in , which is deeply counterintuitive to most people. Every project carries with it three possibilities: that of success, where the company makes money, that of failure, where the company does not, and that of a "critical failure", where the project goes so wrong that it results in…

I like the ideas here but I think the actual chance of getting sued for $3b is so small as to be negligible in the context of costs. It's also questionable how much the additional process/overhead moves the needle on that chance. Larger companies also have various "shields" against these sorts of lawsuits. E.g. they lobby politicians, they employ lawyers, they have legal and IP protection. Just like anything else in…

Getting sued for $3B maybe, but what about getting fined $3B? Such as by the EU.

Re: Working on complex systems: What I learned working at Google

#116

Post by yodsanklai » Thu, May 15, 2025, 8:04 PM UTC

neilv wrote: post #84

> My immediate reaction in my head was: "This is impossible". But then, a teammate said: "But we're Google, we should be able to manage it!". "We can do it!" confidence can be mostly great. (Though you might have to allow for the possibility of failure.) What I don't have a perfect rule for is how to avoid that twisting into arrogance and exceptionalism. Like, "My theory is correct, so I can falsify this experiment."…

Sorry to say that, but this sounds a bit like a fantasy. I think the vast majority of Google employees don't see themselves as particularly brillant or special. Even there, lots of people have imposter syndrome.

Actually, I've found this is a constant in life, whatever you achieve, you end up in a situation where you're pretty average among your peers. You may feel proud to get into Google for a few months, and then you're quickly humbled down.

Re: Working on complex systems: What I learned working at Google

#117

Post by vkou » Thu, May 15, 2025, 8:36 PM UTC

dietr1ch wrote: post #105

specialist wrote: post #97

Earlier quoted context omitted.

Do you mean the relevant code area(s) didn't have (sufficient) tests? You're being asked to backfill those missing tests in addition to your fix?

Yes. I've experienced pushback from obvious fixes with requests to formally test their code for the first time. All because it may break someone. Even when I presented a real defect based on docs/comments and fixed it. You'd think that if they truly cared about breakages they'd already have some tests for it from where I can easily start.

> All because it may break someone. Even when I presented a real defect based on docs/comments and fixed it.

It's great that you found a bug and fixed it.

The problem is, how do you know that there are no other regressions?

Code is a liability. Once you check it in, the team that owns it is responsible for it. Untested code is a liability of unknown scope. It's quite understandable why they don't want to accept someone's contributions, when the contributor isn't the one who will ultimately be dealing with any of the consequences. If you think they are being mean and lazy, imagine if the tables were reversed.

I don't accept puppies or elephants as gifts for similar reasons.

It's unfortunate that existing test coverage sucks. In this case, the best way forward should be for the team in question to improve coverage, and for you to then submit your fix + a test for it. And if they don't have budget to do this, then that sucks, but that's their call to make, and that's a signal that the project in question is abandonware.

And it's fine for a large company to have a bunch of abandonware. If it works, and produces value, the optimal amount of ongoing development effort to invest into a piece of software may, depending on the circumstances, be near-zero.

Re: Working on complex systems: What I learned working at Google

#118

Post by rawgabbit » Thu, May 15, 2025, 9:00 PM UTC

braza wrote: post #10

One of my pet peeves with the usage of complex(ity) out of the traditional time/space in computer science is that most of the time the OPs of several articles over the internet do not make the distinction between boundaried/arbitrary complexity, where most of the time the person has most of the control of what is being implemented, and domain/accidental/environmental complexity, which is wide open and carries a lot o…

If you consider their history of killing well loved products and foisting unwarranted products such as Google Plus onto customers, Google is for lack of a better word just plain stupid. Google is like a person with an IQ of 200 but would get run over by oncoming traffic because they have zero common sense.

Re: Working on complex systems: What I learned working at Google

#119

Post by carom » Thu, May 15, 2025, 9:08 PM UTC

There are typos and rough grammar in the first few paragraphs and I am actually very happy about that because I know I'm not reading LLM slop.

Re: Working on complex systems: What I learned working at Google

#120

Post by miki123211 » Thu, May 15, 2025, 9:16 PM UTC

YZF wrote: post #108

miki123211 wrote: post #86

Earlier quoted context omitted.

Another side of this coin is that the expected payoff from a project depends on how many unrelated projects your organization is engaging in , which is deeply counterintuitive to most people. Every project carries with it three possibilities: that of success, where the company makes money, that of failure, where the company does not, and that of a "critical failure", where the project goes so wrong that it results in…

I like the ideas here but I think the actual chance of getting sued for $3b is so small as to be negligible in the context of costs. It's also questionable how much the additional process/overhead moves the needle on that chance. Larger companies also have various "shields" against these sorts of lawsuits. E.g. they lobby politicians, they employ lawyers, they have legal and IP protection. Just like anything else in…

This explanation wasn't just about lawsuits.

Other things in a similar category are:

- negative media attention (media scrutiny increases proportionally to organization size)

- doing something that upsets an influential group and may have consequences for the rest of your business (think how big the outrage would have been if Apple, Google or Microsoft tried making an "Uber" app before Uber existed)

- bringing down the service which is being worked on, potentially breaking SLAs

- failing to meet customer / legal commitments, particularly in regards to internationalization, accessibility etc.

- security incidents, which are presumably a bigger deal, as your project is connected to the rest of your infrastructure

- getting cancelled online, which causes employees (unrelated to the project) to quit

- natural, random and serious consequences that result from the fact that your project needs the company to hire additional employees. E.g. there's a certain number of people willing to commit sexual assault or financial fraud in the population, and the more people you hire, the more likely it is that you get one of them.