Live data from Hacker News

An ex-Googler’s guide to dev tools

about.sourcegraph.com

51–60 of 219 posts

Re: An ex-Googler’s guide to dev tools

#51

While it is always interesting to read what successful company is up to, I have always been wondering what is the motivation behind ex employees writing memoirs about what was going on in the bedroom. For one, the possibility of such thing happening in the future keeps companies in check, so the company must ensure they don't do evil but also that reveals any weak points to the competition. Google is of course one of…

I think it's just useful as an idea what others are doing and why. Bigger companies had more time to work on their processes and approaches. They usually also have an interest in increasing their developer productivity.

It's more to see the lay of the land than anything else - might be something very cool for you to pick up. Most of it will be an overkill for smaller teams.

Re: An ex-Googler’s guide to dev tools

#52
I think what's missing from the tools discussion is the infrastructure that the tools sit on top of:

1) CitC (Client in the Cloud). Mounts your development environment on FUSE filesystems that exist in the cloud. The entire monorepo is mapped into your CitC directory. You can access it from your desktop shell, your home laptop, or a Web Browser based IDE. Any edits you make are overlaid onto the (readonly) source repository, looking seemless and creating reviewable changelists on the fly. Effortless sharing of editing between multiple machines. ObjFS, which also sits in your client, allows blaze(bazel) build artifacts to be shared as well, between clients, even between users. In other words, if I work on 3 machines, I don't need to "check out" my work 3 times. In fact, I almost never "check out" anything at all. I work in a single monorepo with Mercurial, edit files, which produce reviewable changelists against the main repo. I don't need to decide what files to check out or track, nor decide which machine I will work on, and I often switch between IntelliJ locally, IntelliJ via Chrome Remote Desktop on my office computer, and a VS-Code like Web IDE.

2) Skyframe (https://bazel.build/designs/skyframe.html). Imagine parsing the entire monorepo and every single BUILD file into a massive pre-processed graph that knows every possible build target and its dependencies. This allows ultra-efficient determination of "what do I need to rebuild? what tests need to be re-run" across all of Google. I guess the closest thing to this is MvnRepository.net or BinTray, but Skyframe doesn't just parse the stuff and give you a search box, it informs CI tools.

3) Citc/Critique extensions to Mercurial -- take a chain of commits and make them a single code review, or take a a chain of commits and make them into a stacked chain of code reviews.

4) Critique presubmit tools (e.g. errorprone, tricorder, etc). Google has a huge number of analysis tools can run on every review update, for bugs, security problems, privacy problems, optimizations, data-races, etc. Yes, these are usually available outside, but it's just so easy to enable them internally compared to doing it on GitHub. Lots of other codehealth tools, for automatically applying fixes, removing unused code, auto-updating build files with correct dependencies.

5) Forge -- basically Blaze's remote build execution (what Bazel calls RBEs). Almost every build at Google is extremely parallelized, and if you need to run flake tests, running a suite of tests 10,000 times is almost as fast as running it once.

6) Monitoring's been mentioned, but monitoring combined with CodeSearch hasn't been touched on. Depending on configuration, you can often see from Critique or CodeSearch what release or running server code ended up in and what happened to it (did it cause bugs?). CodeSearch has an insane number of overlays, it can even overlay Google's Sentry-like exception logger being able to tell you about how many times some line of code produced a crash.

A lot of Googlers use maybe 25% of all of the features in CodeSearch and Critique.

Here's a in-depth article from Mike Bland https://mike-bland.com/2012/10/01/tools.html

Re: An ex-Googler’s guide to dev tools

#53
post #15

Earlier quoted context omitted.

Note that Critique (Google's main code review UI) doesn't have the same issues that Gerrit apparently has.

Critique seems a well-guarded secret inside Google. Can't find more information on it. No images. Interesting

Here I'm sitting here thinking could I draw it from memory?

Re: An ex-Googler’s guide to dev tools

#54
post #8

Isn't code search more table stakes nowadays? Both github and gitlab have it. There isn't anything said about end to end, integration or functional testing here. I'm in a world where everyone hacks their own system together onto the same runtime, leading to some wonky outcomes and lots of operational support. Would be interesting if there was a 'google' way to do it.

I don't know what Google's internal code search is like but if you want to see Chromium's code search it's here https://cs.chromium.org also android's https://cs.android.com/ So you can compare

Nice. It looks like a subset of internal tool. Even my favorite shortcut is working (l r - reference to current line in current commit)

https://source.chromium.org/chromium/chromium/src/+/master:i...

Re: An ex-Googler’s guide to dev tools

#55

The article is based on the assumption that what developers absolutely need are the overengineered and bureaucratic tools of a big monopoly company wealthy enough to hire the absolutely best for even the most menial task.

What are the tools used in the most cash-strapped companies facing fierce competition yet prevailing?

Re: An ex-Googler’s guide to dev tools

#56
post #33

(Googler here) The article is a good description of the tools, and I was not aware of the blaze clones plz and pants. One thing that is maybe not obvious: For an API author, code search in combination with a monorepo and the somewhat hermetic universe which is Google's code base provides immediate access to all uses of a library. You can see what worked well and what didn't, and it enables effective refactorings. Tha…

Can I ask a monorepo question: what do you when you’re trying to upgrade dependency XYZ, and a moderate or large refactor is required in another team’s code?

Usually that problem is dealt with combination of:

* Automation.

* Ability to change other people's code safely, possibly gated by code review approval. Cultural and technical (e.g. unit tests must exist and be maintained) barriers may apply.

* Making incremental changes that are backward compatible (at least until you migrated all clients so you can cleanup later)

I recommend this talk https://youtu.be/tISy7EJQPzI (even if the scope is broader than your question)

(ex-googler)

Re: An ex-Googler’s guide to dev tools

#58

Earlier quoted context omitted.

> 1. Why does Google develop their own tools? Because it benefits Google. > 2. Why doesn't Google open source them? Because that would benefit the competition.

Disclaimer, I work for Google, but what follows is my personal opinion. Google open-sources a lot: Tensorflow, K8S, Apache Beam, ... . And even if it doesn't end up as fully open sourced project, Google still releases white-papers on the subject that allows startups to create something similar (Cockroach-DB for instance). However, while I admit that some decisions might be made to avoid benefitting competition (I thi…

Same reason stuff gets deprecated. Just maintaining the code against the dependency tree is work. Trying to make it functional in the real world that doesn't have the 50 billion libraries it relied on that are also internal only (or entire systems) is work ^ work.

Unless its low level infrastructure or based on a research paper, its not an easy task.

Re: An ex-Googler’s guide to dev tools

#59

Refreshing article! For people with less time check out https://github.com/jhuangtw/xg2xg which tracks open source implementations of Google internal tools (also linked in the article) Also I have been looking to move away from Makefiles and bash scripts and Bazel does come up more often. What are your experiences working it and how does it compare to Blaze?

Can't say much about Bazel. I understand it's used at Google for their increasingly polyglot builds. I just needed to isolate the incantations to get j2cl + closure-compiler (= their migration target for gwt apps) running together in a somewhat quality-assured way such that I don't run into problems related to non-aligned versions since closure-compiler, while very powerful, has always been something of a bitch to setup and run. Ended up analyzing Bazel logs to get at the needed command-line parameters, configs, etc. Would have preferred a doc about running j2cl/c-c outside Bazel rather than bundling it, since both of these programs are still just Java command-line apps. As I understood more of Bazel, I didn't find it half bad. That says a lot since my tolerance for over-reaching build tools is near zero - a couple months ago I even did the opposite of what you're suggesting and migrated my old Java builds from maven (and ant) to straight Makefiles, for a significant win in clearness, robustness, performance, scriptability, and network traffic.

Re: An ex-Googler’s guide to dev tools

#60
post #53

Earlier quoted context omitted.

Critique seems a well-guarded secret inside Google. Can't find more information on it. No images. Interesting

Here I'm sitting here thinking could I draw it from memory?

It went through a re-design recently. I don't think I would call it a well-guarded secret though, if you look at Phabricator's Differential, it is very similar [0].

[0] https://www.phacility.com/phabricator/differential/

Post reply on HN