Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Google copybara: moving code between repositories

github.com

Post reply on HN
Print view View original

51–60 of 70 posts

Re: Google copybara: moving code between repositories

#51

Post by bruckie »

dijit wrote: post #45

Ha! This post comes right at the time where I finally got around to open sourcing the patches I made to provide Perforce support which I use at my gamedev studio[0]. I find it kind of funny that perforce support was not included, only git support seems to exist in any meaningful way: despite the primary use of copybara being for releasing internal google code (which lives in Piper, a fork of Perforce). I actually got…

One reason there's no Perforce support for Gerrit/Rietveld is that Google doesn't use those tools for changes to code that's stored in Piper. Instead, they use Critique:

- https://abseil.io/resources/swe-book/html/ch19.html

- https://read.engineerscodex.com/p/how-google-takes-the-pain-...

I haven't found anything external that's as good, and am astounded that GitHub's incredibly lackluster PR review tooling is acceptable to most people. If anyone is aware of something in Critique's league, I'd love to hear about it!

Edit: I tried to do a reasonably thorough survey a couple of years ago when I left Google and https://codeapprove.com/ was the closest I found, but there were still many gaps.

Re: Google copybara: moving code between repositories

#52

Post by eddd-ddde »

madeofpalk wrote: post #28
rnagulapalle wrote: post #20

Earlier quoted context omitted.

The one-way pattern is actually how Google uses it internally too, syncing outward from their monorepo to GitHub. Bidirectional gets messy because transforms (path remapping, file exclusions, header stripping) are easy to apply in one direction but can't always be cleanly inverted. When both sides have diverged, Copybara's baseline tracking starts producing confusing results because semantically equivalent commits ge…

> The one-way pattern is actually how Google uses it internally too, syncing outward from their monorepo to GitHub Do they not support contributions on the public repos back into the internal monorepo?

The "supported" workflow is you keep your source of truth in either the monorepo or the external repo. Then you export the current state of the source of truth to keep the mirrors up to date. Then, since we can assume the mirrors are up to date, the inverse transform can be applied to import change requests from the mirrors.

It works well when the assumptions hold, that there isn't large divergence on either side. It can actually be largely automated.

Re: Google copybara: moving code between repositories

#53

Post by eddd-ddde »

alok-g wrote: post #14

Interesting. Anyone knows how this compares to using git submodules and subtrees? I had used those to create separate repo for website artifacts while the same also remain plugged into the webapp dev repo. (Both sides remain modifiable and changes mergeable to the other side.) Thx.

The main benefits you get are transformations. You can leverage tooling to automate include remapping and things like that.

But it's definitely not geared towards forks, but rather mirrors with deterministic and invertible transforms.

Re: Google copybara: moving code between repositories

#58

Post by krick »

zem wrote: post #42
krick wrote: post #39

I get it that there are use-cases for this, but it's surprising to learn that apparently use-case space is big enough for it to invite a creation of a dedicated tool. I mean that the fact you need it is a bit shameful on its own, no? Usually, when you need to reuse the code between the projects, you try to extract it as a separate library / module. The copying between repos is just a lazy solution, because "ain't nob…

it's useful for cases like google's, where they mirror internal code to github or vice versa, and the two versions need a bit of mechanical work every time they are synced (e.g. slightly different tree layout conventions, internal code or docs that you don't want to include in the github version, stripping of references to other internal stuff like bug IDs from comments, etc).

But if you are gonna extract and open-source the whole self-contained tool, why not just do that and then install in whatever project like you install any other 3rd party tool?

Re: Google copybara: moving code between repositories

#60

Post by bellowsgulch »

I already shared git-fetch-file in another comment here, but the .git-remote-files manifests seem a lot nicer than whatever this thing does:

    [file "lib/util.py" from "https://github.com/example/tools.git"]
    commit = a1b2c3d4e5f6789abcdef0123456789abcdef01
    branch = master
    comment = Common utility function

    [file "config.json" from "https://github.com/example/tools.git"]
    commit = b2c3d4e5f6789abcdef0123456789abcdef012
    branch = master
    target = vendor
    comment = Configuration from tools repo

    [file "helper.js" from "https://github.com/another/project.git"]
    commit = c3d4e5f6789abcdef0123456789abcdef0123
    branch = main
    comment = Helper from another project
vs

    core.workflow(
        name = "default",
        origin = git.github_origin(
        url = "https://github.com/google/copybara.git",
        ref = "master",
        ),
        destination = git.destination(
            url = "file:///tmp/foo",
        ),

        # Copy everything but don't remove a README_INTERNAL.txt file if it exists.
        destination_files = glob(["third_party/copybara/**"], exclude = ["README_INTERNAL.txt"]),

        authoring = authoring.pass_thru("Default email "),
        transformations = [
            core.replace(
                    before = "//third_party/bazel/bashunit",
                    after = "//another/path:bashunit",
                    paths = glob(["**/BUILD"])),
            core.move("", "third_party/copybara")
        ],
    )
There seems to be an absolute ton of reference at https://github.com/google/copybara/blob/master/docs/referenc..., whereas I feel like all the people using git-fetch-file just want files from other repos, and sometimes to make some changes on those.
Post reply on HN