Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Announcing GVFS: Git Virtual File System

blogs.msdn.microsoft.com

Post reply on HN
Print view View original

211–220 of 287 posts

Re: Announcing GVFS: Git Virtual File System

#211

Post by kevincox »

ianopolous wrote: post #52

Couldn't they use git over IPFS?

No. The problem isn't only the storage or fetching of the files (this is the easy bit :) ), it's the operations that detect changes in the working tree. If you have a large tree scanning it becomes slow.

Using a vfs allows you to track which files have changed so that these operations no longer need to scan. Now they are O(changed files) which is generally small.

Now IPFS has a vfs, but it is just a simple read/write interface. This vfs needs slightly more logic to do things like change the base revision and track changes.

Re: Announcing GVFS: Git Virtual File System

#212

Post by kentonv »

greg7mdp wrote: post #130

This is similar to what Google uses internally. See http://cacm.acm.org/magazines/2016/7/204032-why-google-store... : "Most developers access Piper through a system called Clients in the Cloud, or CitC, which consists of a cloud-based storage backend and a Linux-only FUSE13 file system. Developers see their workspaces as directories in the file system, including their changes overlaid on top of the full Piper reposit…

Google's Piper is impressive (I used it), but it emulates Perforce. Having something Git-based is a lot more exciting. Hope someone ports it to platforms other than Windows...

Re: Announcing GVFS: Git Virtual File System

#213

Post by EuAndreh »

vvanders wrote: post #155
gvb wrote: post #42

Using git with large repos and large (binary blob) files has been a pain point for quite a while. There have been several attempts to solve the problem, none of which have really taken off. I think all the attempts have been (too) proprietary – without wide support, it doesn’t get adopted. I'll be watching this to see if Microsoft can break the logjam. By open sourcing the client and protocol, there is potential... O…

That still only solves half the problem with large binary blobs. The other half is that almost all of the binary formats can't be merged and so you need a mechanism to lock them to prevent people from wiping out other people's changes. Unfortunately that runs pretty much counter the idea of DCVS.

When git-annex finds a conflict it can't solve, it gives back to you the two versions of the same file with the SHA of the original versions suffixed.

This way you can look at both and resolve the conflict.

Re: Announcing GVFS: Git Virtual File System

#214

Post by kentonv »

general_ai wrote: post #195
greg7mdp wrote: post #130

This is similar to what Google uses internally. See http://cacm.acm.org/magazines/2016/7/204032-why-google-store... : "Most developers access Piper through a system called Clients in the Cloud, or CitC, which consists of a cloud-based storage backend and a Linux-only FUSE13 file system. Developers see their workspaces as directories in the file system, including their changes overlaid on top of the full Piper reposit…

Google is far more advanced than this. They have one giant monorepo (Piper) that's backed by Bigtable (or at least it was, when I was there). Piper was mostly created in response to Perforce's inability to scale and be fault tolerant. Until Piper came along, they would have to periodically restart The Giant Perforce Server in Mountain View. Piper is 24x7x365 and doesn't need any restarts at all. But the key bit here…

> Every single ex-Googler misses it quite a bit.

I dunno. I don't miss the 1-minute incremental builds. (Maybe they've improved since I left, though.)

BTW Forge is not just the test runner, but the thing that runs all build tasks, farmed out to all servers. Blaze interprets the build language and does dependency tree analysis but then hands off the tasks to Forge. Blaze has been (partially) open sourced: https://bazel.build/

Re: Announcing GVFS: Git Virtual File System

#215

Post by general_ai »

kevincox wrote: post #204
general_ai wrote: post #195

Earlier quoted context omitted.

Google is far more advanced than this. They have one giant monorepo (Piper) that's backed by Bigtable (or at least it was, when I was there). Piper was mostly created in response to Perforce's inability to scale and be fault tolerant. Until Piper came along, they would have to periodically restart The Giant Perforce Server in Mountain View. Piper is 24x7x365 and doesn't need any restarts at all. But the key bit here…

I think this would be better the piper+citc. While the virtual filesystem aspect is nice the perforce model is far inferior to git's. (IMO) Of course Google has tools on top of it. But it's not fair to compare a VCS and interface to it to a complete development infrastructure. Hell, with the content addressing of git it would even make it easy to build something similar.

> perforce model is far inferior to git's

That's just, like, your opinion, man. There are other bits of infra that integrate with it quite nicely, and would integrate with something like Git quite poorly. One of those things is their code review system. The closest thing I could find to it outside Google is Gerrit, but it's a tremendous pain to set up and use, and it's but a pale shadow of Google's internal tool (Critique).

And also, one does not preclude another: Google has a git wrapper on top of Piper, so you can spend your entire Google career not even touching Piper directly if that's what you prefer. And Piper went beyond the "Perforce model" in ways I can't disclose here.

Re: Announcing GVFS: Git Virtual File System

#216

Post by kentonv »

d0vs wrote: post #206
general_ai wrote: post #195

Earlier quoted context omitted.

Google is far more advanced than this. They have one giant monorepo (Piper) that's backed by Bigtable (or at least it was, when I was there). Piper was mostly created in response to Perforce's inability to scale and be fault tolerant. Until Piper came along, they would have to periodically restart The Giant Perforce Server in Mountain View. Piper is 24x7x365 and doesn't need any restarts at all. But the key bit here…

> Google's dev infra is pretty amazing and it's at least a decade ahead of anything else I've seen. Every single ex-Googler misses it quite a bit. This may be naive but why not recreate it as an open source project?

Blaze has been: https://bazel.build/

Forge and Piper are built on Google's internal tech stack and designed for Google's production infrastructure, so open sourcing them would be a very big project. I think it would be a lot more likely for them to be offered as a service -- and that might be more useful to users anyway, since you'd be able to share resources with everyone else doing builds, rather than try to get your own cluster running which might sit idle a lot of the time. Of course, there are privacy issues, etc.

(Disclaimer: I'm purely speculating. I left Google over four years ago, and have no idea what the tools people are up to today.)

Re: Announcing GVFS: Git Virtual File System

#217

Post by wyldfire »

krupan wrote: post #205
greg7mdp wrote: post #130

This is similar to what Google uses internally. See http://cacm.acm.org/magazines/2016/7/204032-why-google-store... : "Most developers access Piper through a system called Clients in the Cloud, or CitC, which consists of a cloud-based storage backend and a Linux-only FUSE13 file system. Developers see their workspaces as directories in the file system, including their changes overlaid on top of the full Piper reposit…

Holy cow, it sounds like they reinvented Clearcase!

Agreed, same w/GVFS IMO.

I ruminated about ccase/git elsewhere in this thread: https://news.ycombinator.com/item?id=13560108

Re: Announcing GVFS: Git Virtual File System

#218

Post by general_ai »

kentonv wrote: post #214
general_ai wrote: post #195

Earlier quoted context omitted.

Google is far more advanced than this. They have one giant monorepo (Piper) that's backed by Bigtable (or at least it was, when I was there). Piper was mostly created in response to Perforce's inability to scale and be fault tolerant. Until Piper came along, they would have to periodically restart The Giant Perforce Server in Mountain View. Piper is 24x7x365 and doesn't need any restarts at all. But the key bit here…

> Every single ex-Googler misses it quite a bit. I dunno. I don't miss the 1-minute incremental builds. (Maybe they've improved since I left, though.) BTW Forge is not just the test runner, but the thing that runs all build tasks, farmed out to all servers. Blaze interprets the build language and does dependency tree analysis but then hands off the tasks to Forge. Blaze has been (partially) open sourced: https://baze…

I know. :-)

Re: Announcing GVFS: Git Virtual File System

#220

Post by EuAndreh »

Ajedi32 wrote: post #63
cies wrote: post #55

Earlier quoted context omitted.

I think Joey Hess' attempt at "solving the problem" deserves a mention. It is open source (GPLV3) licensed. [not proprietary] Written in Haskell. [cool aid] Currently has 1200+ stars on Github and is part of at least Ubuntu ( http://packages.ubuntu.com/search?keywords=git-annex ) since 12.04. [shows something for support and adoption] edit: Link to Github https://github.com/joeyh/git-annex -- thanks dgellow

For the problem of large files I think Git LFS has largely won out over git annex, mostly because it's natively supported by GitHub and GitLab and requires no workflow changes to use.

As of version 6 of git annex, the only thing an unlocked repo need other than the usual workflow is a `git annex sync`, that could be easily configured as a push hook.
Post reply on HN