Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Supercharging the Git Commit Graph IV: Bloom Filters

blogs.msdn.microsoft.com

Post reply on HN
Print view View original

21–30 of 44 posts

Re: Supercharging the Git Commit Graph IV: Bloom Filters

#21

Post by tzahola »

dingo_bat wrote: post #19
jaytaylor wrote: post #2

This is interesting work. I've never noticed the referenced git operation as being slow. In my experience, git log -- has always seemed instantaneous or nearly so. It's quick even on code bases that are quite large (years of work from large teams, 10's of thousands of commits). Maybe data and performance information on before vs. after bloom filters would help clarify the specific design goals. I love seeing bloom fi…

> I've never noticed the referenced git operation as being slow On our monorepo, a simple 'git log -- random_file.cpp' can take anywhere from 10s to a couple of minutes. Edit: This is on VMware instances with 16 Xeon cores and 32 gigs of memory with SSD backed storage. My puny laptop would probably struggle to even clone the repo.

Magnificent! Which company is this?

Re: Supercharging the Git Commit Graph IV: Bloom Filters

#23

Post by ninkendo »

kanox wrote: post #22

Can you make `$ git branch` fast? Because it takes 5-10 seconds for me on linux.

How big of a repository? It should be a near-instantaneous operation.

I’m in a repo with 40GB in .git (hundreds of thousands of commits) and `git branch newbranch` is pretty much immediate.

Re: Supercharging the Git Commit Graph IV: Bloom Filters

#24

Post by stolee »

kanox wrote: post #22

Can you make `$ git branch` fast? Because it takes 5-10 seconds for me on linux.

Do you mean `git branch --contains` or `git branch -vv`? There are many options to `git branch` that cause Git to be very slow.

The commit-graph feature in general will make these faster by reducing time spent parsing commits. You can compute a commit-graph right now if you have Git 2.18 installed: https://blogs.msdn.microsoft.com/devops/2018/06/25/superchar...

Generation numbers will make these operations much faster in Git 2.19. Here are the related commits:

`git branch --contains`: https://github.com/git/git/commit/f9b8908b85247ef60001a683c2...

`git tag --contains`: https://github.com/git/git/commit/819807b33f820dc17d96f04374...

Re: Supercharging the Git Commit Graph IV: Bloom Filters

#25

Post by influx »

avar wrote: post #13
influx wrote: post #8

Earlier quoted context omitted.

Odd that people are including Twitter as a data point in the monorepo debate. I was there when they transitioned to a Git monorepo, and it nearly killed the companies productivity. Imagine having to torrent a starter pack of the repo, then trying to sync and failing multiple times. Then after it syncs, it could take minutes to do common operations such as change a branch, or check in a file. Not to mention many devel…

[Source: I contribute to Git and talked to Twitter's "git guys" at the time at Git Merge, am not affiliated with Twitter in any way] My understanding is that they didn't cargo-cult anything. They made a conscious choice at Twitter to pursue the monorepo model. One reason they mentioned is that while they have a lot of services running all over the place, those services tend to heavily use the same underlying base lib…

Sure, that was the justification, but when you literally can’t switch branches without waiting for a minute, I fail to see the benefit.

As I recall the developers authoring the custom git patches left the company around that time. I don’t know what happened after that. There was talk of moving to mercurial with facebooks patches, but I had already left.

Re: Supercharging the Git Commit Graph IV: Bloom Filters

#26

Post by [deleted] »

masklinn wrote: post #12
jaytaylor wrote: post #2

This is interesting work. I've never noticed the referenced git operation as being slow. In my experience, git log -- has always seemed instantaneous or nearly so. It's quick even on code bases that are quite large (years of work from large teams, 10's of thousands of commits). Maybe data and performance information on before vs. after bloom filters would help clarify the specific design goals. I love seeing bloom fi…

> has always seemed instantaneous or nearly so. It's quick even on code bases that are quite large (years of work from large teams, 10's of thousands of commits). The issue tends to be mostly graph log, note that while this applies to all commit walks it's work mostly done in the context of the commit-graph feature as graphlogging can get fairly slow on big complex repos (mega-scale commit counts). For example on $da…

[deleted]

Re: Supercharging the Git Commit Graph IV: Bloom Filters

#28

Post by mwcampbell »

dingo_bat wrote: post #19
jaytaylor wrote: post #2

This is interesting work. I've never noticed the referenced git operation as being slow. In my experience, git log -- has always seemed instantaneous or nearly so. It's quick even on code bases that are quite large (years of work from large teams, 10's of thousands of commits). Maybe data and performance information on before vs. after bloom filters would help clarify the specific design goals. I love seeing bloom fi…

> I've never noticed the referenced git operation as being slow On our monorepo, a simple 'git log -- random_file.cpp' can take anywhere from 10s to a couple of minutes. Edit: This is on VMware instances with 16 Xeon cores and 32 gigs of memory with SSD backed storage. My puny laptop would probably struggle to even clone the repo.

Sounds like your team would benefit from GVFS [1].

[1]: https://github.com/Microsoft/GVFS

Re: Supercharging the Git Commit Graph IV: Bloom Filters

#29

Post by Heliophite »

ythn wrote: post #27

Coming from a game programming background - before clicking the link I was expecting a VFX "bloom filter" effect on a visual commit graph where "light sources" are exaggerated and seem to "glow".

That was my reaction as well. "Oh, neat, visualization!" ".. oh, the óther bloom."

Re: Supercharging the Git Commit Graph IV: Bloom Filters

#30

Post by Sir_Cmpwn »

mwcampbell wrote: post #28
dingo_bat wrote: post #19

Earlier quoted context omitted.

> I've never noticed the referenced git operation as being slow On our monorepo, a simple 'git log -- random_file.cpp' can take anywhere from 10s to a couple of minutes. Edit: This is on VMware instances with 16 Xeon cores and 32 gigs of memory with SSD backed storage. My puny laptop would probably struggle to even clone the repo.

Sounds like your team would benefit from GVFS [1]. [1]: https://github.com/Microsoft/GVFS

No sane team is running on Windows, so probably not.
Post reply on HN