Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Working on complex systems: What I learned working at Google

thecoder.cafe

Post reply on HN
Print view View original

61–70 of 147 posts

Re: Working on complex systems: What I learned working at Google

#61

Post by apercu »

Demiurge wrote: post #57
kubb wrote: post #24

Earlier quoted context omitted.

I’d wager 90% time spent at Google is fighting incidental organizational complexity, which is virtually unlimited.

And when you’re at a smaller company 90% of your time is fighting societal complexity, limit of which also approaches infinity, but at a steeper angle. No greater Scott’s man can tell you that the reality is surprisingly complex, and sometimes you have resources to organize and fight them, and sometimes you use those resources wiser than the other group of people, and can share the lessons. Sometimes, you just have n…

Look, I've never had to design, build or maintain systems at the scale of a FAANG, but that doesn't mean I haven't been involved in pretty complicated systems (e.g., 5000 different pricing and subsidy rules for 5000 different corporate clients with individually negotiated hardware subsidies (changing all the time) and service plans, commission structure, plus logistics, which involves not only shipping but shipping to specific departments for configuration before the device goes to the employee, etc.

Arbitrarily, 95% of the time the issues were people problems, not technical ones.

Re: Working on complex systems: What I learned working at Google

#62

Post by apercu »

repeekad wrote: post #36
kubb wrote: post #24

Earlier quoted context omitted.

I’d wager 90% time spent at Google is fighting incidental organizational complexity, which is virtually unlimited.

The phrase thrown around was “collaboration headwind”, the idea was if project success depends on 1 person with a 95% chance of success, project success also had a 95% chance. But if 10 people each need to succeed at a 95% chance, suddenly the project success likelihood becomes 60%… In reality, lazy domain owners layered on processes, meetings, documents, and multiple approvals until it took 6 months to change the te…

> lazy domain owners

Interesting. As a consultant for the most of the last 25 years, my experience is the domain owners are typically invested and have strong opinions on the things that impact their jobs.

Executive leadership, on the other hand, doesn't want to actually know the issues and eyes glaze over as they look at their watches because they have a tee time.

Re: Working on complex systems: What I learned working at Google

#63

Post by romanovcode »

teivah wrote: post #49
Pavilion2095 wrote: post #38

The cookie banner reappears indefinitely on this website when I click 'only necessary' lol.

Sorry about that, I'm my newsletter provider (Substack) which is very buggy sometimes.

Probably because it is overly complex system.

Re: Working on complex systems: What I learned working at Google

#64

Post by __MatrixMan__ »

tuyiown wrote: post #31
braza wrote: post #10

One of my pet peeves with the usage of complex(ity) out of the traditional time/space in computer science is that most of the time the OPs of several articles over the internet do not make the distinction between boundaried/arbitrary complexity, where most of the time the person has most of the control of what is being implemented, and domain/accidental/environmental complexity, which is wide open and carries a lot o…

I think this is addressed with the complex vs complicated intro. Most problems with uncontrolled / uncontrollable variables will be approached with an incremental solution, e.g. you'll restrict those variables voluntarily or involuntarily and let issues being solved organically / manually, or automatisation will be plain and simple being abandoned. This qualify as complicated. Delving in complicated problems is mostl…

I don't think it is, because the intro gets it wrong. If a problem's time or space complexity increases from O(n^2) to O(n^3) there's nothing necessarily novel about that, it's just... more.

Complicated on the other hand, involves the addition of one or more complicating factors beyond just "the problem is big". It's a qualitative thing, like maybe nobody has built adequate tools for the problem domain, or maybe you don't even know if the solution is possible until you've already invested quite a lot towards that solution. Or maybe you have to simultaneously put on this song and dance regarding story points and show continual progress even though you have not yet found a continuous path from where you are to your goal.

Climate change is both, doing your taxes is (typically) merely complex. As for complicated-but-not-complex, that's like realizing that you don't have your wallet after you've already ordered your food: qualitatively messy, quantitatively simple.

To put it differently, complicated is about the number of different domains you have to consider, complex is about--given some domain--how difficult the consideration in that domain are.

Perhaps the author's usage is common enough in certain audiences, but it's not consistent with how we discuss computational complexity. Which is a shame since they are talking about solving problems with computers.

Re: Working on complex systems: What I learned working at Google

#65

Post by ungreased0675 »

citrin_ru wrote: post #6
nottorp wrote: post #2

Let's add a post scriptum: Whatever you're working on, your project is not likely to be at Google's scale and very unlikely to be a "complex system".

For small scale one can build a simple system but I see many are trying to copy FAANG architecture anyway. IMHO it’s a fallacy - people think that if they’ll would copy architecture used by google their company will be successful like google. I think it other was around - google has to build complex systems because it has many users.

It’s an infectious disease among developers. Some people would spend weeks making a simple landing page, and it would require at least 3 different cloud services.

Re: Working on complex systems: What I learned working at Google

#66

Post by tanelpoder »

braza wrote: post #10

One of my pet peeves with the usage of complex(ity) out of the traditional time/space in computer science is that most of the time the OPs of several articles over the internet do not make the distinction between boundaried/arbitrary complexity, where most of the time the person has most of the control of what is being implemented, and domain/accidental/environmental complexity, which is wide open and carries a lot o…

Also, anything you do with enterprise (cloud) customers. People like to talk about scale a lot and data people tend to think about individual (distributed) systems that can go webscale. A single system with many users is still a single system. In enterprise you have two additional types of scale:

1) scale of application variety (10k different apps with different needs and history)

2) scale of human capability (ingenuity), this scale starts from sub-zero and can go pretty high (but not guaranteed)

Re: Working on complex systems: What I learned working at Google

#67

Post by Demiurge »

apercu wrote: post #61
Demiurge wrote: post #57

Earlier quoted context omitted.

And when you’re at a smaller company 90% of your time is fighting societal complexity, limit of which also approaches infinity, but at a steeper angle. No greater Scott’s man can tell you that the reality is surprisingly complex, and sometimes you have resources to organize and fight them, and sometimes you use those resources wiser than the other group of people, and can share the lessons. Sometimes, you just have n…

Look, I've never had to design, build or maintain systems at the scale of a FAANG, but that doesn't mean I haven't been involved in pretty complicated systems (e.g., 5000 different pricing and subsidy rules for 5000 different corporate clients with individually negotiated hardware subsidies (changing all the time) and service plans, commission structure, plus logistics, which involves not only shipping but shipping t…

I have a similar perspective. I think after a few years, it's the people things that have always been the hardest part of the job. That's probably why in the interviews, we always say things like: communication is key, culture fit, etc.

On the other hand, the good part of the job is solving complex technical problem with a team.

Re: Working on complex systems: What I learned working at Google

#68

Post by ninetyninenine »

Except computers attempt to model mathematics in an ideal world.

Unless your problem comes from something side effects on a computer that can’t be modeled mathematically there is nothing technically stopping you from modeling the problem as mathematical problem then solving that problem via mathematics.

Like the output of the LLM can’t be modeled. We literally do not understand it. Are the problems faced by the SRE exactly the same? You give a system an input of B and you can’t predict the output of A mathematically? It doesn’t even have to be a single equation. A simulation can do it.

Re: Working on complex systems: What I learned working at Google

#69

Post by blitzar »

jajko wrote: post #46
blitzar wrote: post #30

Earlier quoted context omitted.

> The idea is if someone helps you in a really big way that you’re able to reward that It never ceases to amaze me how (early) big tech embraced and even promoted things that would have been considered "career limiting" in traditional big corporations.

Don't know about your flavor of 'traditional big corporations' but my banking megacorp has internal reward system across various 'virtues' for a decade+ at least. Its not direct reward -> money link (thats rather for hiring success), it just helps you create sort of karma, and when bonuses, raises and promotions are considered then this is taken into account. Since that process is invisible to those being measured yo…

Big bank. Management theory at the time was to create competition between the silos for resources, time, budget, headcount, good desk locations in the bi-annual room desk shuffle, bonuses and even time of day from management. Even sales and trading - the most symbiotic of functions competed.

Re: Working on complex systems: What I learned working at Google

#70

Post by dietr1ch »

moregrist wrote: post #59
pclmulqdq wrote: post #56

Earlier quoted context omitted.

There's a culture of "I won't approve this unless it does something for me " at Google. So now changing the text on a button comes with 2 minor refactors, 10 obvious-but-ignored bugfixes, and 5 experiments that it is actually better.

While this sounds pretty frustrating, there is at least a small upside: at least you get to the obvious-but-ignored bugfixes. Most smaller places don’t have the bandwidth and many larger ones don’t have the desire. I’m not sure if that makes up for bugs potentially introduced in the refactors, though.

Well, when the owner asks for a whole test suite that didn't exist to get a fix in, what most likely happens is that you just wasted your time in a draft CL that will get lost.
Post reply on HN