Brian Harry, a Microsoft executive over Visual Studio Team Services, writes about developer tools, Git at scale, and open sourcing Microsoft projects.
HN commenters mostly cite his post on Windows moving to one huge Git repo with GVFS as evidence of how Git can be scaled.
This is a bit alarmist. If github went away or became like Sourceforge of old then we could just move to something else or do self-hosting again. Lose a chunk of the internet by moving popular software like Handbrake to GitHub? how could that happen? Most multiplatform open source software like this have copies of their source code across thousands of linux distribution mirrors.
Also, google code never went away. It just stopped working as an active platform, but Google still keeps the archive of what already existed there, to this day:
https://code.google.com/archive/
Microsoft is doing the same after shutting down their Code Plex because of moving toward GitHub:
https://blogs.msdn.microsoft.com/bharry/2017/03/31/shutting-down-codeplex/
> At that time, CodePlex.com will start serving a read-only lightweight archive that will allow you to browse through all published projects – their source code, downloads, documentation, license, and issues – as they looked when CodePlex went read-only. You’ll also be able to download an archive file with your project contents, all in common, transferrable formats like Markdown and JSON. Where possible, we’ll put in place redirects so that existing URLs work, or at least redirect you to the project’s new homepage on the archive. And, the archive will respect your “I’ve moved” setting, if you used it, to direct users to the current home of your project.
If there is anything to lose after GitHub's shutdown at some distant point in the…
Scalability to large numbers of files and large numbers of changesets.
For the large number of files one https://code.facebook.com/posts/218678814984400/scaling-mercurial-at-facebook/ is one source. There's some earlier discussion of the same issue at https://news.ycombinator.com/item?id=3549679 that goes into some technical details.
There has been some work on git since then to address some of those issues (e.g. see https://blogs.msdn.microsoft.com/bharry/2017/02/03/scaling-git-and-some-back-story/ ) but it's not clear to me that it helped enough to catch up to where Mercurial is for large repos.
For large numbers of changesets, just try running "log" or "annnotate" on any file with a long history in git. I just did this simple experiment:
1) hg clone https://hg.mozilla.org/mozilla-central/
2) git clone https://github.com/mozilla/gecko-dev.git
3) (cd mozilla-central && time hg log dom/base/nsDocument.cpp)
4) (cd gecko-dev && time git log dom/base/nsDocument.cpp)
It's not quite apples to apples because the git repo there has some pre-mercurial CVS history in it. But note that I'm not even using --follow for git and the file _has_ been renamed after the mercurial repo starts, so git is actually finding fewer commits than mercurial is here.
Anyway, if I do the above log calls a few times to make sure the caches are warm, I end up seeing times in the 8s range for git and the 0.8s range (yes, 10x faster) for mercurial.
That all said, most repos do not have millions…
their earlier blogpost goes into it a bit https://blogs.msdn.microsoft.com/bharry/2017/02/03/scaling-git-and-some-back-story/ :
The first big debate was – how many repos do you have – one for the whole company at one extreme or one for each small component? A big spectrum. Git is proven to work extremely well for a very large number of modest repos so we spent a bunch of time exploring what it would take to factor our large codebases into lots of tenable repos. Hmm. Ever worked in a huge code base for 20 years? Ever tried to go back afterwards and decompose it into small repos? You can guess what we discovered. The code is very hard to decompose. The cost would be very high. The risk from that level of churn would be enormous. And, we really do have scenarios where a single engineer needs to make sweeping changes across a very large swath of code. Trying to coordinate that across hundreds of repos would be very problematic.
After much hand wringing we decided our strategy needed to be “the right number of repos based on the character of the code”. Some code is separable (like microservices) and is ideal for isolated repos. Some code is not (like Windows core) and needs to be treated like a single repo.
I can't see it being easy to take a 20 year massive code base and split it up into components that are so isolated enough they can live in different repositories smoothly.
As they say in: https://blogs.msdn.microsoft.com/bharry/2017/02/03/scaling-git-and-some-back-story/
The first big debate was – how many repos do you have – one for the whole company at one extreme or one for each small component? A big spectrum. Git is proven to work extremely well for a very large number of modest repos so we spent a bunch of time exploring what it would take to factor our large codebases into lots of tenable repos. Hmm. Ever worked in a huge code base for 20 years? Ever tried to go back afterwards and decompose it into small repos? You can guess what we discovered. The code is very hard to decompose. The cost would be very high. The risk from that level of churn would be enormous. And, we really do have scenarios where a single engineer needs to make sweeping changes across a very large swath of code. Trying to coordinate that across hundreds of repos would be very problematic.
Quoting Brian Harry from a comment response at https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-largest-git-repo-on-the-planet/
"It depends a great deal on the operation. SourceDepot was much faster at some things – like “sd opened”, the equivalent of “git status”. sd opened was < .5s. git status is at 2.6s now. But SD was much slower at some other things – like branching. Creating a branch in SD would take hours. In Git, it's less than a minute. I saw a mail from one of our engineers at one point saying they'd been putting off doing a big refactoring for 9 months because the branch mechanics in SD would have been so cumbersome and after the switch to Git they were able to get the whole refactoring done in a topic branch in no time.
On an operation, by operation basis, SD is still much faster than our Git/GVFS solution. We're still working on it to close the gap but I'm not sure it will ever get as fast at everything. The broader question, though is about overall developer productivity and we think we are on a path to winning that."
> It has nothing to do with the Windows filesystem; Git simply cannot support a 5 GB working tree on any filesystem.
Can you provide a reference? I was searching a bit and only things I found was bugs in windows[1] for git lfs.
> You can call this "pathological" but this throws a lot of shade on monorepos without much critical examination of how or when they might be useful.
Windows codebase has 3.5 million files and its repo is 300GB in size. It is not normal. This is google or MS type of problem and not average git user. MS instead changing workflow decided to create GVFS[2]
[1] https://github.com/git-lfs/git-lfs/issues/2434
[2] https://blogs.msdn.microsoft.com/bharry/2017/05/24/the-largest-git-repo-on-the-planet/