My first thought, as suggested by some on the list, was modularization. Redstone's response (that the 1.3 million files are essentially all interdependent) terrifies me.
Facebook hit git performance issue on large repository
211–217 of 217 posts
Re: Facebook hit git performance issue on large repository
#212Earlier quoted context omitted.
Mac OS X's FSEvents API has something similar to that. When you create a FSEvent listener you can pass in an old event ID so the system can give you all the stuff that happened while you weren't listening [1]. Apple uses this for Time Machine (and I suspect Spotlight, too). [1] https://developer.apple.com/library/mac/#documentation/Darwi...
What happens if a file is created and deleted multiple times? How does this avoid doing a complete walk of FS state and being O(size) itself?
It does not actually tell you what files got changed; it tells you what directories saw at least one change.
Programs will still have to inspect those directories to find out what file(s) changed.
To quote https://developer.apple.com/library/mac/#documentation/Darwi...:
To better understand this technology, you should first understand what it is
not. It is not a mechanism for registering for fine-grained notification of
filesystem changes. It was not intended for virus checkers or other technologies
that need to immediately learn about changes to a file and preempt those changes
if needed. [...]
The file system events API is also not designed for finding out when a
particular file changes. For such purposes, the kqueues mechanism is more
appropriate.
The file system events API is designed for passively monitoring a large tree of
files for changes. The most obvious use for this technology is for backup
software. Indeed, the file system events API provides the foundation for Apple’s
backup technology.
Re: Facebook hit git performance issue on large repository
#213Earlier quoted context omitted.
In violent agreement here. Git and HG: 1. Require you to be sync'ed to tip before pushing. 2. Cannot selectively check out files. The former means that in any reasonably sized team, you will be forced to sync 30 times a day, even if you are the only one editing your section of the source tree. The latter means that Joe who is checking in the libraries for (huge open source project) for some testing increases everyone…
Require you to be sync'ed to tip before pushing. That's not the case. In fact, in the context of Linux kernel development, there's many emails on LKML where Linus is telling someone that they shouldn't be merging random-kernel-of-the-day into their development branch.
It's required to maintain a linear history, no?
Re: Facebook hit git performance issue on large repository
#214Earlier quoted context omitted.
There's no need to spill internal processes and configurations. The fellow said he had a synthetic repo that he used to benchmark various operations. Surely whatever generated that test repo can scale it up or down to whatever size they like, so you can benchmark at various points and collect the data that would tell us if there is some horrible non-linear scaling going on under the covers.
Right now it sounds like he's just trying to see what the possible solutions for his issues are. If he can provide additional benchmarks, etc., great. But he's under no obligation to provide any more than he has. Once there's a solution, then maybe.
Of course he's not under any obligation to provide any more info than he has. But given that he already has the test harness setup, and that only he has access to the hardware on which his benchmarks ran, it seems that he could easily enable more people to help him by providing additional data points.
Re: Facebook hit git performance issue on large repository
#215Earlier quoted context omitted.
If you limit change propagation, your changes won't propagate as fast. That goes for bugs and bug fixes. I can certainly see why you would have the latter propagated instantaneously, or close to it. There's also the point that if you don't propagate change to everybody at the same time, you'll have dozens of slightly different versions of those projects across your company. The question of submodules vs. large repo i…
No, you're misunderstanding what I'm saying with regards to publishing stable changes. Say you have two branches: master and next. Stable work goes in master, unstable work goes in next. When the code is ready for consumption, you merge it into master. Anyone who is using a project has it setup as a submodule. They add post/pre-commit hooks to update all project submodules. These submodules pull from master. This way…
I do understand what you mean just fine. Except it doesn't work that way. If I have a critical bug fix in libA, I need it rolled out now. What's more, I need it rolled out across all other projects that use libA, immediately. And no, I don't want to work until all projects committed a change of their own.
Even more, I (or my team) are not the only ones working on libA. Others are too. So keeping changes in 'next' and pushing to master only on occasion doesn't help much. Yes, it keeps non-working patches out - but that's what local branches on your machine are for.
(I'm not even going to mention the issue of merge conflicts. If you work on a massive scale, the longer you stay in a branch, the more likely you are to get a merge conflict. There's easily the chance to go into a several weeks long merge hell. Pull from master, resolve conflicts, run local tests - oh wait, master is already updated by somebody else)
Re: Facebook hit git performance issue on large repository
#216Re: Facebook hit git performance issue on large repository
#217Earlier quoted context omitted.
None of what you mention here precludes breaking up the code into many smaller repositories, and then having them all linked together in one super-repository. Then tags at the super-repository level can record the exact state of all submodules. It's not about not checking the other modules out; you can make this the standard behavior, sure. Instead it's about having git manage reasonable sized blocks of the code base…
Three big problems with a split up codebase: 1) Instead of doing one large release every week (which facebook does: http://www.facebook.com/video/video.php?v=10100259101684977 ) you now have dozens or hundreds of smaller releases, a lot more heterogeneity to test for. 2) If you have inter-dependencies on modules you have to grapple with the "diamond dependency" problem. Say module A depends on module B and C, and sup…
I disagree.
There are ALREADY interdependency issues if you're using git. Anyone could have changed anything before you did your last commit. If you pull and then push your changes without running tests, you're already risking breaking the build.
If a diamond dependency conflict came up, it shouldn't ever be committed to the TRUNK. Whoever made the change that causes a conflict should ideally discover that conflict before they submit it to TRUNK. But it's still not relevant to the decision to have one big repo or a hundred smaller ones, since the exact same problem can come up in either case if you fail to test before you submit.
With the proper workflow, having a lot of smaller trees can be functionally equivalent to having one massive tree. Checking in your branch to the TRUNK would require that you have the latest version of the TRUNK, and if you don't, then you'd need to pull the latest, just like how git works now. And updates to the TRUNK ARE atomic: Before you update TRUNK, it's pointing to the previously working version, and after you update, it's pointing to the one you just tested to make sure it still works.
And, just like now, a developer should therefore test to make sure the merge works after they've grabbed the latest.
It sounds like you're assuming that, if I update one of the submodules, it could break TRUNK? That can't happen without someone trying to commit it to TRUNK. Git remembers a specific commit for each submodule, and doesn't move forward to a new commit without being told to.