Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Announcing GVFS: Git Virtual File System

blogs.msdn.microsoft.com

Post reply on HN
Print view View original

201–210 of 287 posts

Re: Announcing GVFS: Git Virtual File System

#201

Post by kevincox »

I love this approach. From working at Google I appreciate the virtual filesystem, it makes a lot of things a lot easier. However all my repos are large enough to fit on a single machine so I wish there was a mode where it was backed by a local repository, however the filesystem allows git to avoid tree scans.

Basically most operations in git are O(modified files) however there are a few that are O(working tree size). For example checkout and status were mentioned by the article. However these operations can be made to O(modified) files if git doesn't have to scan the working tree for changes.

So pretty much I would be all over this if:

- It worked locally.

- It worked on Linux.

Maybe I'll see how it's implemented and see if I could add the features required. I'm really excited for the future of this project.

Re: Announcing GVFS: Git Virtual File System

#202

Post by ajross »

kentt wrote: post #22

It's disappointing that all the comments are so negative. This is a great idea and solves a real problem for a lot of use cases. I remembering years ago Facebook says it had this problem. A lot of the comments were centered around that you could change your codebase to for what git can do. I'm glad there's another option now.

> This is a great idea and solves a real problem for a lot of use cases.

I don't know if "a lot" is the right qualifier. Solitary repos of millions of files have scalability problems even outside the source control system (I mean: how long does it take your workstation to build that 3.5 million-file windows tree?)

A full Android system tree is roughly the same size and works fine with git via a small layer of indirection (the repo tool) to pull from multiple repositories. A complete linux distro is much larger still, and likewise didn't need to rework its tooling beyond putting a small layer of indirection between the upstream repository and the build system.

Honestly I'd say this GVFS gadget (which I'll fully admit is pretty cool) exists because Microsoft misapplied their source control regime.

Re: Announcing GVFS: Git Virtual File System

#203

Post by vvanders »

to3m wrote: post #176
vvanders wrote: post #155

Earlier quoted context omitted.

That still only solves half the problem with large binary blobs. The other half is that almost all of the binary formats can't be merged and so you need a mechanism to lock them to prevent people from wiping out other people's changes. Unfortunately that runs pretty much counter the idea of DCVS.

I always wonder why this never gets discussed much. We seem to have tons of solutions for storing large files outside the repo, but so what? OK, so I don't deny that storage still isn't cheap enough to just say "oh well" to the idea of a multi-TByte repo, so it's certainly solving a problem. But there's still another major problem left! Don't their artists and designers use version control too? Maybe they just have o…

Yeah, best solution I've seen(but haven't had a chance to use in anger) is P4-Fusion(P4 backend, git views).

Perforce was always the gold standard for stuff like this. Did a great job at not only providing locking but stuff like seamless proxies and other solutions to common problems in that domain(like a usable UI).

Re: Announcing GVFS: Git Virtual File System

#204

Post by kevincox »

general_ai wrote: post #195
greg7mdp wrote: post #130

This is similar to what Google uses internally. See http://cacm.acm.org/magazines/2016/7/204032-why-google-store... : "Most developers access Piper through a system called Clients in the Cloud, or CitC, which consists of a cloud-based storage backend and a Linux-only FUSE13 file system. Developers see their workspaces as directories in the file system, including their changes overlaid on top of the full Piper reposit…

Google is far more advanced than this. They have one giant monorepo (Piper) that's backed by Bigtable (or at least it was, when I was there). Piper was mostly created in response to Perforce's inability to scale and be fault tolerant. Until Piper came along, they would have to periodically restart The Giant Perforce Server in Mountain View. Piper is 24x7x365 and doesn't need any restarts at all. But the key bit here…

I think this would be better the piper+citc. While the virtual filesystem aspect is nice the perforce model is far inferior to git's. (IMO) Of course Google has tools on top of it. But it's not fair to compare a VCS and interface to it to a complete development infrastructure. Hell, with the content addressing of git it would even make it easy to build something similar.

Re: Announcing GVFS: Git Virtual File System

#205

Post by krupan »

greg7mdp wrote: post #130

This is similar to what Google uses internally. See http://cacm.acm.org/magazines/2016/7/204032-why-google-store... : "Most developers access Piper through a system called Clients in the Cloud, or CitC, which consists of a cloud-based storage backend and a Linux-only FUSE13 file system. Developers see their workspaces as directories in the file system, including their changes overlaid on top of the full Piper reposit…

Holy cow, it sounds like they reinvented Clearcase!

Re: Announcing GVFS: Git Virtual File System

#206

Post by d0vs »

general_ai wrote: post #195
greg7mdp wrote: post #130

This is similar to what Google uses internally. See http://cacm.acm.org/magazines/2016/7/204032-why-google-store... : "Most developers access Piper through a system called Clients in the Cloud, or CitC, which consists of a cloud-based storage backend and a Linux-only FUSE13 file system. Developers see their workspaces as directories in the file system, including their changes overlaid on top of the full Piper reposit…

Google is far more advanced than this. They have one giant monorepo (Piper) that's backed by Bigtable (or at least it was, when I was there). Piper was mostly created in response to Perforce's inability to scale and be fault tolerant. Until Piper came along, they would have to periodically restart The Giant Perforce Server in Mountain View. Piper is 24x7x365 and doesn't need any restarts at all. But the key bit here…

> Google's dev infra is pretty amazing and it's at least a decade ahead of anything else I've seen. Every single ex-Googler misses it quite a bit.

This may be naive but why not recreate it as an open source project?

Re: Announcing GVFS: Git Virtual File System

#207

Post by kevincox »

alkonaut wrote: post #185
cafebabbe wrote: post #94

My sysadmin: "we won't switch to git because it can't handle binary files and our code base is too big" Our whole codebase is 800MB.

Our codebase (latest tree) is similar, but switching to git it's the total history size that is the problem. Our history is well over 25GB which git doesn't handle very gracefully.

History shouldn't be a problem, you can do a shallow checkout. But you will have to store the working tree at least on your workstation.

This solves the next scaling problem of avoiding managing the whole working tree. (without requiring narrow clones which have significant downsides)

Re: Announcing GVFS: Git Virtual File System

#208

Post by EuAndreh »

gvb wrote: post #42

Using git with large repos and large (binary blob) files has been a pain point for quite a while. There have been several attempts to solve the problem, none of which have really taken off. I think all the attempts have been (too) proprietary – without wide support, it doesn’t get adopted. I'll be watching this to see if Microsoft can break the logjam. By open sourcing the client and protocol, there is potential... O…

git-annex is, IMHO, by far the best solution.

Pros of git-annex:

- it is conceptually very simple: use symlinks instead of ad-hoc pointer files, virtual files system, etc. to represent symbolic pointer that point to the actual blob file;

- you can add support for any backend storage you want. As long as it support basic CRUD operations, git-annex can have it as a remote;

- you can quickly clone a huge repo by just cloning the metadata of the repo (--no-content in git-annex) and just download the necessary files on-demand;

And many other things that no other attempt even consider having, like client-side encryption, location tracking, etc.

Re: Announcing GVFS: Git Virtual File System

#210

Post by general_ai »

d0vs wrote: post #206
general_ai wrote: post #195

Earlier quoted context omitted.

Google is far more advanced than this. They have one giant monorepo (Piper) that's backed by Bigtable (or at least it was, when I was there). Piper was mostly created in response to Perforce's inability to scale and be fault tolerant. Until Piper came along, they would have to periodically restart The Giant Perforce Server in Mountain View. Piper is 24x7x365 and doesn't need any restarts at all. But the key bit here…

> Google's dev infra is pretty amazing and it's at least a decade ahead of anything else I've seen. Every single ex-Googler misses it quite a bit. This may be naive but why not recreate it as an open source project?

Because very few people have a need to support a billion-LOC monorepo on which 30K engineers make tens of thousands of commits daily. That's where this system shines.

For smaller projects, Git+Bazel (open source, non-distributed version of Blaze) works fine if you're working with C++, and other build systems work OK as well, if you're working with other languages.

Post reply on HN