Microsoft's Azure DevOps team posts product updates and engineering notes on Git performance, Azure Pipelines, Repos and Boards.
HN links it for Git engineering posts like Scalar and the commit graph, plus reactions to Azure Repos' default branch rename, which many commenters disliked.
> Are there some fundamental aspects of git that would make it either very difficult to improve that, or that would sacrifice some important benefits if they were made?
I can't speak to improving git, but I think some light on this area can be shed by Linus' tech talk at Google in 2007.
1. Linus says there's a specific focus on full history and content, not files ... so it's a deliberate, different axis of focus than file count:
https://youtu.be/4XpnKHJAok8?t=2586
... AND it's a specific pitfall to avoid when using Git:
https://youtu.be/4XpnKHJAok8?t=4047
2. As Linus tells it, Git appears to be designed specifically for project maintenance while not getting in the way of individual commits and collaboration. But the global history and more expensive operations on things like "who touched this line" are deliberate so lines of a function are tracked across all moves of the content itself.
Maintainer tool enablement: https://youtu.be/4XpnKHJAok8?t=3815
Content tracking slower than file-based "who touched this": https://youtu.be/4XpnKHJAok8?t=4071
===
I have no answer, but ...
Practically, I've used lazy filesystems both for Windows-on-Git via GVFS [1][2] and Google's monorepo jacked into a mercurial client (I think that's what it is?). Both companies have made this work, but as Linus says, a lot of the stuff just doesn't work well with either system.
Windows-on-Git still takes a lot of time overall, and stacking > 10 patches of an exploratory refactor with the…
Some examples, in no particular order.
Hampering the productivity:
- Review messages get sent out before review is actually finished. It should be sent out only once the reviewer has finished the work.
- Code reviews are implemented in a terrible way compared to GitHub or GitLab.
- Re-requesting a review once you did implemented proposed changes? Takes a single click on GitHub, but can not be done in Azure DevOps. I need to e.g. send a Slack message to the reviewer or remove and re-add them as reviewer. - Knowing to what line of code a reviewer was giving feedback to? Not possible after the PR got updated, because the feedback of the reviewer sticks to the original line number, which might now contain something entirely different. - Reviewing the commit messages in a PR takes way too many clicks. This causes people to not review the commit messages, letting bad commit messages pass and thus making it harder for future developers trying to figure out why something got implemented the way it did. Examples:
- Too many clicks to review a commit message: PR -> Commits -> Commit -> Details - Comments on a specific commit does not shown in the commits PR - Unreliable servers. E.g. "remote: TF401035: The object '<snip>' does not exist.\nfatal: the remote end hung up unexpectedly" happens too often on git fetch. Usually works on a 2nd try.
- Interprets IPv6 addresses in commit messages as emoji. E.g. fc00::6:100:0:0 becomes fc00::60:0.
- Can not cancel a stage…
Thanks for the link to the documentation. That is updated with every major Git version, and can be used to track what features are present. Also, the release notes can be helpful.
In particular, the recent Git v2.27.0 release does include an implementation of Bloom filters with speedups for `git log` and `git blame`. You need to manually run the command to make it work. The version I prefer is this:
> git commit-graph write --changed-paths --reachable
After that first write (which writes filters for every reachable commit) you can do a smaller write by adding `--split` to write incrementally [2]
[2] https://devblogs.microsoft.com/devops/updates-to-the-git-commit-graph-feature/
By writing these filters, you will speed up most `git log` and `git blame` calls. There is an improvement coming in the next version that includes speedups for `git log -L`.
Caveat: The biggest reason these improvements have not been widely advertised is that the user experience has not been completely smoothed out. In particular, you can only write the changed-path Bloom filters using the command(s) above. If a commit-graph is written during GC (due to `gc.writeCommitGraph` config setting) then the filters will disappear. Similar for `fetch.writeCommitGraph`. We plan to have these resolved in time for v2.28.0, along with more performance improvements.
(Full disclosure: I am a contributor to Git, Scalar, and VFS for Git, which are referenced by the article.)
microsoft: in 2017, the Windows git repo contained ~3.5 million files & used 300GB when checked out as a git repo [1], In 2020, the Office git repo contained ~3 million files [2].
facebook: In 2011, the main site had around 9.2 million lines of code "excluding numerous backend services" [3]. In 2014 the android codebase was 4 million lines of code and the main site without backend code was 62 million lines of code [3]. In 2014, when facebook were talking about scaling mercurial, the main source repo was described as "many times larger than even the Linux kernel, which checked in at 17 million lines of code and 44,000 files in 2013". Around 2018 (?) facebook started using "eden", a fork of mercurial [5].
If we make the working assumption that expected lines-of-code / file of 2013 linux kernel is representative of all repos, then that gives around 386 lines-of-code / file.
loc(microsoft) >= loc(windows in 2017) + loc(office in 2020) ≈ 386 loc / file * (3.5m + 3m) ≈ 2.5 billion lines of code
The linked blog posts from microsoft & facebook talking about scaling version control systems are fairly interesting!
[1] https://devblogs.microsoft.com/bharry/the-largest-git-repo-on-the-planet/
[2] https://devblogs.microsoft.com/devops/introducing-scalar/
[3] https://www.quora.com/How-many-lines-of-code-is-Facebook
[4] https://engineering.fb.com/core-data/scaling-mercurial-at-facebook/
[5] https://github.com/facebookexperimental/eden
That’s quite a few assumptions, at least one of which likely isn’t correct; at Microsoft, the .git directory doesn’t contain ”all the git files, including file versions […] and branche”
https://devblogs.microsoft.com/devops/optimizing-git-beyond-gvfs/:
”GVFS allows our developers to simply not download most of those 3.5 million files during a clone, and instead simply page in the (comparatively) small portion of the source tree that a developer is working with.”
Having said that, the linked https://docs.microsoft.com/en-us/azure/devops/learn/git/gvfs-design-history claims Windows gives a size about half that 500GB for the Windows source code:
”the Windows repo, which has over 3 million files in the working directory, totalling 270GB of source files. That’s 270GB in the working directory, at the tip of master. To clone this repo, you would have to download a packfile that is about 100GB in size, which would take hours. And once you’ve succeeded in cloning it, local git operations like checkout (3 hours), status (8 minutes), and commit (30 minutes) would still take way too long to run”.
Microsoft had a bunch of solutions to handle their massive Windows repo: VFS for Git (GVFS), Scalar, and now it has a bunch of MS specific patches on top of the official git client, but apparently that one is also not required any more as partial clone is now supported on azure as well (which is another such implementation from Microsoft employees that made it to both GitHub and upstream git).
So yeah, solved problem thanks to Microsoft. Solved multiple times in fact, and because it's Microsoft, only not all had Linux installers: https://github.com/microsoft/scalar/issues/323#issuecomment-927865734
https://github.blog/2020-01-17-bring-your-monorepo-down-to-size-with-sparse-checkout
https://devblogs.microsoft.com/devops/introducing-scalar/
https://github.com/microsoft/git
https://devblogs.microsoft.com/devops/git-partial-clone-now-supported-in-azure-devops/