Hacker News Board Hacker News, browsed like a classic message board
Live data from Hacker News

Microsoft Office migration from Source Depot to Git

danielsada.tech

Post reply on HN
Print view View original

41–50 of 290 posts

Re: Microsoft Office migration from Source Depot to Git

#41

Post by dshacker »

MBCook wrote: post #31

Could someone explain the ideas of forward integration and reverse integration in Source Depot? I’d never heard of Source Depot before today.

RI/FI is similar to having long-lived branches in Git. Imagine you have a "develop-word" branch in git. The admins for that branch would merge all of the changes of their code to "main" and from "main" to their long lived branches. It was a little bit different than long-lived git branches as they also had a file filter (my private branch only had onenote code and it was the "onenote" branch)

Re: Microsoft Office migration from Source Depot to Git

#42

Post by dshacker »

3eb7988a1663 wrote: post #10

We communicated the same information through multiple channels: weekly emails, Teams, wiki docs, team presentations, and office hours. The rule: if something was important, people heard it at least 3 times through different mediums. If only this were standard. Last week I received the only notification that a bunch of internal systems were being deleted in two weeks . No scream test, no archiving, just straight delet…

Even with this, there were many surprised people. I'm still amazed at all of the people that can ignore everything and just open their IDE and code (and maybe never see teams or email)

Re: Microsoft Office migration from Source Depot to Git

#43

Post by dshacker »

90s_dev wrote: post #16

In about 2010, I briefly had a contract with a security firm with one dev, and there was no source control, and everything written was in low quality PHP. I quit after a week.

php_final_final_v2.zip shipped to production. A classic. I had a similar experience with https://www.ioncube.com/ php encryption. Everything encrypted and no source control.

Re: Microsoft Office migration from Source Depot to Git

#44

Post by dshacker »

TowerTall wrote: post #38
golergka wrote: post #29

Earlier quoted context omitted.

It takes less than an hour on my third world apartment wifi to download Call of Duty Modern Warfare remake which is over 200 gygabytes. Since we're not talking about remote work here, I think Microsoft offices and servers (probably on local network) might have managed similar bandwidth back then.

There is a lot more to it than that. Check out "The largest Git repo on the planet" by Brian Harry who was in charge of the git migration and Azure DevOps (Microsoft's pendant to GitHub) https://devblogs.microsoft.com/bharry/the-largest-git-repo-o...

> For context, if we tried this with “vanilla Git”, before we started our work, many of the commands would take 30 minutes up to hours and a few would never complete. The fact that most of them are less than 20 seconds is a huge step but it still sucks if you have to wait 10-15 seconds for everything. When we first rolled it out, the results were much better. That’s been one of our key learnings. If you read my post that introduced GVFS, you’ll see I talked about how we did work in Git and GVFS to change many operations from being proportional to the number of files in the repo to instead be proportional to the number of files “read”. It turns out that, over time, engineers crawl across the code base and touch more and more stuff leading to a problem we call “over hydration”. Basically, you end up with a bunch of files that were touched at some point but aren’t really used any longer and certainly never modified. This leads to a gradual degradation in performance. Individuals can “clean up” their enlistment but that’s a hassle and people don’t, so the system gets slower and slower.

Great quote from him here.

Re: Microsoft Office migration from Source Depot to Git

#45

Post by firesteelrain »

sherdil2022 wrote: post #4
mrheosuper wrote: post #2

[flagged]

If it were that simple, would 100s of engineers spend so much time and effort? They did what they have to and spent the time and energy to maintain some semblance of commit and change history.

GP has a valid point. We had a Git repo managed in BitBucket that was gigantic because it contained binary files and the team didn’t know about LFS and storing them in an external tool like Artifactory. So checkouts took forever and even with shallow clones it took forever. With a CI/CD system running constantly and tests needing constant full coverage and hundreds of developers well it eats into developers time. We can’t just prune all the branches well because of compliance rules.

So we ended up removing all the binary artifacts before cloning into a new repo then making the old repo as read only.

Microsoft seemed to want to mirror everything rather than keep source depot alive.

We had another case where we had a subversion system that went out of security compliance that we simply ported to our git systems and abandoned it.

So my guess is they wanted everything to look the same and not just importing the code.

Re: Microsoft Office migration from Source Depot to Git

#46

Post by mikepurvis »

dshacker wrote: post #41
MBCook wrote: post #31

Could someone explain the ideas of forward integration and reverse integration in Source Depot? I’d never heard of Source Depot before today.

RI/FI is similar to having long-lived branches in Git. Imagine you have a "develop-word" branch in git. The admins for that branch would merge all of the changes of their code to "main" and from "main" to their long lived branches. It was a little bit different than long-lived git branches as they also had a file filter (my private branch only had onenote code and it was the "onenote" branch)

I've long wanted a hosted Git service that would help me maintain long lived fork branches. I know there's some necessary manual work that is occasionally required to integrate patches, but the existing tooling that I'm familiar with for this kind of thing is overly focused on Debian packaging (quilt, git-buildpackage) and has horrifyingly poor ergonomics.

I'd love a system that would essentially be a source control of my patches, while also allowing a first class view of the upstream source + patches applied, giving me clear controls to see exactly when in the upstream history the breakages were introduced, so that I'm less locking in precise upstream versions that can accept the patches, and more actively engaging with ranges of upstream commits/tags.

I can't imagine how such a thing would actually be commercially useful, but darned if would be an obvious fit for AI to automatically examine the upstream and patch history and propose migrations.

Re: Microsoft Office migration from Source Depot to Git

#47

Post by dybber »

MBCook wrote: post #31

Could someone explain the ideas of forward integration and reverse integration in Source Depot? I’d never heard of Source Depot before today.

We had a similar setup, also with a homegrown VCS developed internally in our company, where I sometimes acted as branch admin. I’m not sure it worked exactly like Source Depot, but I can try to explain it.

Basically instead of everyone creating their own short-lived branches (expensive operation), you would have long-lived branches that a larger group of people would commit to (several product areas). The branch admins job was then to get the work all of these people forward integrated to a branch upwards in the hierarchy. This was attempted a few times per day, but if tests failed you would have to reach out to the responsible people to get those test fixed. Then later, when you get the changes merged upwards, some other changes have also been made to the main integration branch, and now you need to pull these down into your long lived branch - reverse integration - such that your branch is up to date with everyone else in the company.

Re: Microsoft Office migration from Source Depot to Git

#48

Post by xmprt »

o11c wrote: post #30
com2kid wrote: post #8

Earlier quoted context omitted.

At one point source depot was Toincredibly advanced, and there are still features that it had that git doesn't. Directory mapping being a stand out feature! Being able to only pull down certain directories from a depot and also remap where they are locally, and even have the same file be in multiple places. Makes sharing dependencies across multiple projects really easy, and a lot of complicated tooling around "monor…

As always, git's answer to the problem is "stop being afraid of `git submodule`." Cross-repo commits are not a problem as long as you understand "it only counts as truly committed if the child repo's commit is referenced from the parent repo".

> it only counts as truly committed if the child repo's commit is referenced from the parent repo

This is a big problem in my experience. Relying on consumers of your dependency to upgrade their submodules isn't realistic.

Re: Microsoft Office migration from Source Depot to Git

#49

Post by bsder »

yjftsjthsd-h wrote: post #18
bsder wrote: post #9

Earlier quoted context omitted.

> git supported directory mapping. Is this a "git" failure or a "Linux filesystems suck" failure? It seems like "Linux fileystems" are starting to creak under several directions (Nix needing binary patching, atomic desktops having poor deduplication, containers being unable to do smart things with home directories or too many overlays). Would Linux simply sucking it up and adopting ZFS solve this or am I missing some…

What would ZFS do for those issues? I guess maybe deduplication, but otherwise I'm not thinking of anything that you can't do with mount --bind and overlays (and I'm not even sure ZFS would replace overlays)

Snapshots seems to be a cheap feature in ZFS but are expensive everywhere else, for example.

OverlayFS has had performance issues on Linux for a while (once you start composing a bunch of overlays, the performance drops dramatically as well as you start hitting limits on number of overlays).

Re: Microsoft Office migration from Source Depot to Git

#50

Post by carlual »

dshacker wrote: post #39
carlual wrote: post #24

> Authenticity mattered more than production value. Thanks for sharing this authentic story! As an ex-MSFT in a relatively small product line that only started switching to Git from SourceDepot in 2015, right before I left, I can truly empathize with how incredible a job you guys have done!

Yeah, it was a whole journey. I can't believe it happened. Thanks for your comment.

Thank you! Btw, it reminds me of the book "Showstopper" about the journey of releasing Windows NT; highly recommended!
Post reply on HN