I taught a bucket to speak Git
tigrisdata.com
I taught a bucket to speak Git
1–10 of 20 posts
Re: I taught a bucket to speak Git
#2Came here for a five-gallon bucket hooked to Dulwich (archiving rain?), Slightly disappointed :)
Go Git and Dulwich and friends are indeed fun tech.
Re: I taught a bucket to speak Git
#3Most of the pain here is the typical set of issues people run into trying to make S3 a filesystem as-is, common with S3FS-family approaches.
ZeroFS (https://github.com/Barre/zerofs) is 9P/NFS/NBD over S3 on an LSM. Point stock go-git, or just /usr/bin/git, at a mount and skip the gymnastics. Rename is a metadata op in the keyspace, so you get it atomic on any S3, no Tigris-specific X-Tigris-Rename needed.
Different point on the spectrum, but less square-peg, also most probably much, much faster (it works great on linux-sized repos) :)
Re: I taught a bucket to speak Git
#4Most of the pain here is the typical set of issues people run into trying to make S3 a filesystem as-is, common with S3FS-family approaches. ZeroFS ( https://github.com/Barre/zerofs ) is 9P/NFS/NBD over S3 on an LSM. Point stock go-git, or just /usr/bin/git, at a mount and skip the gymnastics. Rename is a metadata op in the keyspace, so you get it atomic on any S3, no Tigris-specific X-Tigris-Rename needed. Different…
Author of the article here. I'm aware of ZeroFS and other similar approaches (such as something internal at Tigris that will become public at a later date), this was more of an experiment to see how far you can get with stuff I already had "on the shelf". I am going to be improving this a fair bit; I just need to plan out what I'm gonna work on and figure out the best times to stream it, etc.
Re: I taught a bucket to speak Git
#5If you want to store a git repo on S3, you can that with git-annex[1] today. It can do client side encryption and large files as well.
Re: I taught a bucket to speak Git
#6Most of the pain here is the typical set of issues people run into trying to make S3 a filesystem as-is, common with S3FS-family approaches. ZeroFS ( https://github.com/Barre/zerofs ) is 9P/NFS/NBD over S3 on an LSM. Point stock go-git, or just /usr/bin/git, at a mount and skip the gymnastics. Rename is a metadata op in the keyspace, so you get it atomic on any S3, no Tigris-specific X-Tigris-Rename needed. Different…
I wouldn’t call it gymnastics. The surprising part of the article was that Git itself is an object store that happens to use a filesystem for persistence, but an S3 bucket might actually be more suitable than a .git directory on POSIX.
Re: I taught a bucket to speak Git
#7This was really thought provoking — it made me realize that Git just happens to use a filesystem for persistence, but doesn’t necessarily have to. A POSIX filesystem might not even be the best way to store a git repo. Makes me wonder: what else could speak Git + POSIX? Redis? Postgres? IPFS is a fun one — it’s already content addressed.
Re: I taught a bucket to speak Git
#8I did something similar, though a full reimplementation of a git and git-lfs library in Elixir. Still a work in progress though as the S3 backend isn't quite complete and there are performance problems doing some git things through S3.
https://anvil.fangorn.io/fangorn/ex_git_objectstore
The documentation isn't quite correct, but it's getting there
Re: I taught a bucket to speak Git
#9Most of the pain here is the typical set of issues people run into trying to make S3 a filesystem as-is, common with S3FS-family approaches. ZeroFS ( https://github.com/Barre/zerofs ) is 9P/NFS/NBD over S3 on an LSM. Point stock go-git, or just /usr/bin/git, at a mount and skip the gymnastics. Rename is a metadata op in the keyspace, so you get it atomic on any S3, no Tigris-specific X-Tigris-Rename needed. Different…
I wouldn’t call it gymnastics. The surprising part of the article was that Git itself is an object store that happens to use a filesystem for persistence, but an S3 bucket might actually be more suitable than a .git directory on POSIX.
[deleted]
Re: I taught a bucket to speak Git
#10Most of the pain here is the typical set of issues people run into trying to make S3 a filesystem as-is, common with S3FS-family approaches. ZeroFS ( https://github.com/Barre/zerofs ) is 9P/NFS/NBD over S3 on an LSM. Point stock go-git, or just /usr/bin/git, at a mount and skip the gymnastics. Rename is a metadata op in the keyspace, so you get it atomic on any S3, no Tigris-specific X-Tigris-Rename needed. Different…
I wouldn’t call it gymnastics. The surprising part of the article was that Git itself is an object store that happens to use a filesystem for persistence, but an S3 bucket might actually be more suitable than a .git directory on POSIX.
[deleted]