Live data from Hacker News

Software Engineering at Google (2017)

arxiv.org

291–300 of 319 posts

Re: Software Engineering at Google (2017)

#291

What I don't understand is how they accomplish larger collaborative changes. The paper says: "Almost all development occurs at the 'head' of the repository, not on branches." Googler Rachel Potvin made an even stronger statement in her presentation about "The Motivation for a Monolithic Codebase" [1]: "Branching for development at Google is exceedingly rare [..]" In the related ACM paper she published with Josh Leven…

None of the other replies try to explain specifics of how this works, so let me illustrate an example of two teams collaborating to add Feature X to the monorepo without branching:

1) Team A checks in their code to provide Feature X. Their code is not used anywhere in the codebase yet, however full unit test coverage exists for the public API; this is required for code review.

2) Team B checks in their code to turn on Feature X in their product, gated under a command-line flag which by default uses the old behavior.

3) Team B checks in an integration test that flips the flag and makes sure everything works as planned.

4) If Team B requires changes to Feature X to get expected behavior, they communicate those changes to Team A and someone from either team (using available human resources) makes the changes.

5) Team B checks in a small change to flip the flag by default.

6) Team B monitors their product. If things go awry, only the very latest change is reverted and repeat (4).

7) Once stability is achieved, Team B checks in a change to remove the flag.

Re: Software Engineering at Google (2017)

#292
post #5

Buried in the "2.11 Frequent rewrites" section, but a great hack for "productivity via a sense of ownership": "In addition, rewriting code is a way of transferring knowledge and a sense of ownership to newer team members. This sense of ownership is crucial for productivity: engineers naturally put more effort into developing features and fixing problems in code that they feel is “theirs”."

Disclaimer: I work for Google. I speak only for myself.

In my honest opinion, frequent rewrites are by-and-large a disastrously bad idea, for several reasons. If there is one thing I would change about Google, it would be to slow down the frenetic pace of change inside. Rewrites just make the pace of change untenable. And I say this as one who is totally part of the problem: I helped rewrite significant parts of V8, the JS VM in Chrome, particularly the optimizing JIT compiler, TurboFan. (Don't get me wrong--I am not knocking any one specific project, my coworkers, even my leadership, etc). I've been at Google 9 years, and I don't know how barely any of it works anymore.

1. The assumption that requirements and environment around software change so frequently that it must be burned down to the ground and rewritten is a big part of the problem. Why do the requirements of software change? A. Scale. B. Because the software around it changed. Bingo.

2. Rewrites actively destroy institutional expertise. Instead of learning more as time goes on, engineers' knowledge becomes obsolete as old systems are constantly changing and rewritten for unclear benefit. Experts can no longer rely on their knowledge for more than a couple of years. This is extremely bad for critical pieces of infrastructure. In short, no one ever masters anything. This is due not just to incentives but due to change itself.

3. No one ever has time to do an in-depth followup study on whether the rewritten artifact was better than the original. Instead people go on their gut feeling of having rewritten something they often did not write themselves (and did not fully understand) with something new and shiny of their own creation. The justification of the outcome is done, in short, by the people who have a big vested interest in declaring success. (And yes, me too).

4. The idea that the software requirements keep changing around software is promulgated by the exact same people who never spend any time up front simply writing requirements down. Well, no fracking wonder the requirements seem to change somewhere in the middle or years later: they were never anticipated in the first place! We'd do better overall if the industry in general did some good ole requirements engineering. Most people I talk to have never even heard of this. Instead, we never have any time to stop and think about doing things right, but we always find time to rewrite from scratch.

5. Zero incentive to do things right. As a field, as industry, we are actually not very serious about writing good software. Instead, we're just going to trash it after 5 years. So software is constantly bad. But the next rewrite!

The drive for rewrites is mostly a swindle in my opinion. The reality is that some software needs to be shot in the head, some needs to be rewritten, but most software needs to be just maintained. That means bugfixes, performance improvements, scalability improvements, and sometimes, yes, refactoring too. But bugfixes and incremental performance improvements don't get anyone promoted. Even more cynically, but very realistically, "old" software that is maintained by experts means a dependency on those experts, and they end up being expensive. Corporations hate when their employees have job security! Rewrites are A.) driven by an influx of young talent who want to make their mark, B.) incentivized by the promotion process and C.) driven by a corporate pressure (everywhere, not just Google) to make sure that programmers and software are commoditized to avoid dependencies, bus factor, and job security.

Re: Software Engineering at Google (2017)

#293
post #204

Earlier quoted context omitted.

Are you allowed to share if the internal repository a custom version control system or is it one of the open source ones?

It is, like most of the tools in Google/Facebook/..., a custom solution. These companies need their custom solution because of their scale.

It begs the question : are big companies like Atlassian, GitHub or say VolksWagen required, by their scale, to have a slick solution in order to perform on the international market?

Re: Software Engineering at Google (2017)

#294
post #206

Earlier quoted context omitted.

People who like multirepos are always saying how easy it is to pin dependencies but like you I haven’t seen anyone doing it right since I left Google. The monorepo third-party system works well in practice. Ps thanks for getting scipy into third_party all those years ago.

Pinning dependencies is easy, and gets one stuck on Java 1.3 and IE 6. Since the library maintainer doesn't know it broke dependent code, the dependent code will be unlikely to have a smooth upgrade path. It's just a matter of having the luxury of picking up a time to pay the cost of dependency upgrades. Which cost may be high in a world where your dependencies also pin dependencies, likely at different versions. The…

That’s a good point. Do we have any solution from companies who use git/multirepos for their libraries? Is there anyway to compile refactored libraries against existing code?

Re: Software Engineering at Google (2017)

#295
post #9

Earlier quoted context omitted.

Thanks for highlighting this. To me it seems an important idea that contradicts conventional wisdom, similar in the way that most people over-encourage DRY, blind to the fact it increases coupling.

I frequently find myself drastically refactoring code to understand it. I don't commit those changes because it's not worth the effort to justify the cleanup to people who treat these rules as gospel. Apparently me spending half a day reading code is no big deal but cleaning it up is a waste of time. Shrug.

Take a moment to consider that someone else in your team might actually understand the code in question, and when you rewrite it (and commit), you destroy their knowledge about how it works. Sure, from your perspective, you made the code better, easier to understand. From their perspective, even if they do the code review themselves, they will not understand it as thoroughly as you do, and they will be burdened with both the knowledge of how it used to be, and how it is now. The net result can be negative, and eventually no one understands any code that they didn't refactor in the last ~6 months, because if it's any older than that, someone's rewritten it.

Sometimes you gotta sit down and bite the bullet and read and ask questions.

Re: Software Engineering at Google (2017)

#296
post #16
post #5

Buried in the "2.11 Frequent rewrites" section, but a great hack for "productivity via a sense of ownership": "In addition, rewriting code is a way of transferring knowledge and a sense of ownership to newer team members. This sense of ownership is crucial for productivity: engineers naturally put more effort into developing features and fixing problems in code that they feel is “theirs”."

I’m not convinced this works so great for Google. Some rewrites are very noticeable as a user, and things in the UI frequently shift around for no discernible reason. Perhaps worse, they seem unable to get below a certain level of bugginess in products like Google Maps and Gmail. Perhaps because a new round of rewrites always introduces new bugs before all of the old ones ever get fixed. Perhaps their metrics tell th…

As jwz put it, "It hardly seems worth even having a bug system if the frequency of from-scratch rewrites always outstrips the pace of bug fixing. Why not be honest and resign yourself to the fact that version 0.8 is followed by version 0.8, which is then followed by version 0.8?"

Re: Software Engineering at Google (2017)

#297
post #222

Earlier quoted context omitted.

Wait, huh? How does DRY increase coupling? I mean, I guess the duplicate code/class is now coupled to the two places that use it, but I have a hard time seeing how that is worse than two duplicate instances of the code.

Here is what I've seen (and been guilty of): Two pieces of code in different parts of the codebase are very similar. They have nothing to do with each other - even semantically. But the code is very similar. So someone thinks this is code duplication and creates a function/class/whatever that both pieces of code can use. Repeat all over the place. Then one day, one of those two places needs custom behavior. I can eit…

This gets back to the https://en.wikipedia.org/wiki/Open%E2%80%93closed_principle. I should be able to override just the behavior I want to change, rather than permanently edit the shared implementation eliminating the behavior you want.

Re: Software Engineering at Google (2017)

#298

Earlier quoted context omitted.

(Genuine question.) Do you think "small changes" shouldn't be "tacked on in the easiest places"? I'll try to give an example I hope is realistic: One of the heaviest things an application can get is a complete theme system. Suppose you don't have one. It's not a requirement. Adding a theme system when there is none is months of work and might impact basically every line of code that displays anything. So you're not d…

Of course this is just my opinion, but in many situations I think tacking one small change on to the easiest place is fine. My comment earlier wasn't meant as a criticism of that practice. It's just I would consider refactoring after this happens multiple times: at some point, all those little tweaks can add up enough that the original design of the "core" code gets lost in the noise. When exactly is the right time c…

Thanks, this answers my question. I had always thought documenting half-assed solutions was particularly important not just for anyone looking at the code (to understand the whole code quickly, if they're new to it, make changes, etc - or even if the author comes back 6 months later, to be reminded of the assumptions that led to the hacks) but because whoever finally sits down to architect the rewrite will have some indications of the "hacks" (degrees of freedom) that must be made possible in the proper version. I thought they (or I) would sit down, have all the documented hacks on one side of their table, then use them to architect a nice solution that the hacks can then use instead, kind of like a requirements document.

This was my thinking on a lot of projects. But you know what? That rewrite never became necessary!

So not only did I not start with the "right" architecture - I didn't end with it either!

That's what "many small hacks in the easiest of places" reminded me of and I wondered if you in fact do approve of it. It has always seemed fine for me. Just no problem at all. But with documentation right there and the worse the hack, the clearer the documentation right there explaining and justifying it, up to and including "I don't know why this works but this system call makes the next line succeed, whereas removing it causes the next line to fail sometimes - this is tested in testxzy." Obviously a hack, a terrible hack if you don't know why it works. And a project can end up with a lot of these.

Re: Software Engineering at Google (2017)

#299

Earlier quoted context omitted.

Here is what I've seen (and been guilty of): Two pieces of code in different parts of the codebase are very similar. They have nothing to do with each other - even semantically. But the code is very similar. So someone thinks this is code duplication and creates a function/class/whatever that both pieces of code can use. Repeat all over the place. Then one day, one of those two places needs custom behavior. I can eit…

This gets back to the https://en.wikipedia.org/wiki/Open%E2%80%93closed_principle . I should be able to override just the behavior I want to change, rather than permanently edit the shared implementation eliminating the behavior you want.

A better solution is "Just don't couple the two pieces of code!"

It takes a fair amount of work to design a good class where one can easily apply the open-closed principle. And one should where it's needed. But tying together two completely unrelated parts of your code base with such a class just because the two pieces of code are almost identical is the wrong approach. Then going ahead and designing it for the open-closed principle merely adds complexity.

Re: Software Engineering at Google (2017)

#300

Earlier quoted context omitted.

Google has 7 products with 1+ billion users.[1] 1. https://www.popsci.com/google-has-7-products-with-1-billion-...

Now let's look at the ration billion-user-products versus the total number of products. That'll give you the ration of employees having a huge impact: https://en.wikipedia.org/wiki/List_of_Google_products And we're not even looking at the number of active users per product versus the total amount of gmail ID's enabled for all google products by default (e.g. Google+)

> Now let's look at the ration billion-user-products versus the total number of products. That'll give you the ration of employees having a huge impact

That logic doesn't make any sense. No company allocates staff equally to all their products. More popular products (or higher revenue-generating products) are always better-staffed than less popular products.

Post reply on HN