> The developers making Microsoft Windows use Git

https://arstechnica.com/gadgets/2018/03/building-windows-4-m...

Also, even after learning a fair amount of git, I still find I spend a noticeable amount of time dealing with it. I don't remember spending a lot of time on any of the VCS systems I've used in the past. They just stayed out of my way and let do my thing.

To answer your question though - Microsoft has a lot of git extensions that they are slowly submitting upstream. Hence git is usable for their megarepo.

Look up GVFS on GitHub, very cool work.

Same for Twitter (they are not submitting upstream though as far as I know)

Similar for fb/mercurial

Prime example: the git staging area/cache/index needs to die. Git would be half as difficult to use with fewer code shredding surprises. This abomination is a prime example of badly exposed internal structure. Everything feature that is crammed into this whatever-the-hell-that-is could be replaced by a vastly superior solution which does not require anything like it.

Says who? Branches and commits are cheap, it's how we can undo ourselves and freely experiment.

I agree that untested commits into a publishing/release branch shouldn't exist: all commits there should be merges from dev branches, but to say every commit should be tested is utter bollocks and denies us the advantages of cheap branching and commits.

Either they need to be squash+merge or the dev branches need to end up with working commits before the merge. Otherwise your life will become hell the first time you need to bisect.

Microsoft works around several of the issues there by using GVFS. Also, at Microsoft scale, everything "sort of sucks", there's just no silver bullets. You take one of the least bad options and put all the effort you can towards making it work as well as you can.

At "Microsoft scale," you have the resources to purpose build anything you need from scratch for any scale you're working at... therefore if anything "sort of sucks" it's because:

a) It's not worth the money/resources. (The "let it suck" approach)

or

b) Nobody cares (The "acceptance that we suck, at 'scale'" approach)

There's a fair bit of excitement around the new, exciting, open source friendly Microsoft with built-in Linux kernel emulation, an embrace of git and a huge release of open source tools...

Honestly though, I'm not buying it, the issue I've had with Microsoft over the years doesn't just stem from the shitty software or aggressive business practices... it's the deep rooted culture of mediocrity it promotes.

People who belong to that church believe it's ok to build shitty software because doing it right is too hard. Why root-cause an issue when you can just script reboots? It's too hard anyway, we're at "Microsoft scale."

The rise of the internet and the companies that grew around it showed us that if you have a culture of giving a shit, you can build really complex things "at scale" that aren't complete shit.

I worry that the new Microsoft will be a different kind of trojan horse for the OSS world. It won't be "embrace, extend, extinguish" it will be more like a social media psyops campaign that beats it into everyone's heads that now we're at "Microsoft scale" it's ok for everything to "kinda suck", and if we're not careful... everything will.

But it was specifically designed for the Linux kernel.

`git status` still takes over a second for me though, oh well...

‘git show-ref -s | git commit-graph write —stdin-commits'

how do you automatically update this?

That's completely unnecessary and way too frequent. Some where else (reddit I think?) the authors noted that they'd like to have it run alongside GCs in the next version. So running it as pre-auto-gc is a better idea.

You don't need to run this with every commit, but maybe once after a big fetch or right after a new clone.

https://www.fossil-scm.org/index.html/timeline

This is the amount of commits that goes into Linux every ~5months.

Has anyone done any meaningful performance comparison between fossil and git?

There are 2 different conversations happening.

You seemed to be responding to ggp (rakoo) "sqlite db vs files".

However masklinn was responding to gp (Aissen) question of "Fossil vs Git" performance and a charitable reading would be comparing the Fossil algorithms (also affected by combination with underlying SQLite db engine algorithm) for commit searches, graph traversal, etc. In that case, the high number of commits and total repository size to stress test Fossil/Git would be very relevant.

An example post about Fossil scaling: https://www.fossil-scm.org/xfer/technote/be8f2f3447ef2ea3344...

- on filesystem vs sqlite (put git files in sqlite): there's a good benchmark on https://www.sqlite.org/fasterthanfs.html claiming sqlite is up to 35% faster than fs. I'd like to see the same benchmark with git's file pattern; also, it's a known issue with git that it was written for linux first, hence optimized against Linux (relatively) good fs performance (vs Windows and Mac at the time). Same with most OSS build systems that (over)use process forking, which is also very optimized in Linux.

- on Fossil vs git (why bother putting git files in sqlite and not directly jump to fossil?): that was my comment, and it relates to subject of this article (the commit-graph). I'm wondering if Fossil has seen the optimization that git has, with regards to number of commits, considering that sqlite is the only high-profile project that uses it. Maybe performance is supposed to be taken care of by the sqlite database itself ?

Edit: see https://www.fossil-scm.org/xfer/doc/trunk/www/stats.wiki for a fossil-provided performance analysis, that does a comparison with CVS (!).

I don't see git ever changing its file format, but I do see another tool that imports everything from git and gives you a read-only sqlite db where you can do whatever you want, including displaying a graph quickly as the post advertises.

If there was a stronger graph database store and graph query language for consideration than SQL you might be on to something. SQL isn't a great fit here either.

I'm not saying SQL is the best way to store and query DAGs; any graph database would be better. All I'm saying is that SQL is probably better at designing and maintaining a solution than what git does with its custom file format and custom code.

I'm only comparing what the pile-of-files that git currently is and a full-fledged SQL database. None is perfect, but one feels overall easier than the other.

The SQL standards define a query language, not a storage format. There are SQL databases that themselves optimize their internal storage structures into "piles of files". In fact, most have at one point or another. SQLite is an intentional outlier here; it's part of why SQLite exists.

There's nothing stopping anyone from building an SQL query engine that executes over a git database, for what that is worth. Because you can't execute SQL queries against it today doesn't really say anything at all about whether or not git's database storage format is insufficient or not.

All of that is also before you even start to get into the weeds about standards compliance in the SQL query language itself and how very little is truly compliant between database engines, as they all have slightly different dialects due to historic oddities. Or the weeds that there's never been a good interchange format between SQL database storage formats other than overly verbose DDL and INSERT statement dumps. That again are sometimes subject to compatibility failures if trying to migrate between database engines, due to dialectal differences. Including what should be incredibly fundamental things like making sure that foreign key relationships import and index correctly, without data loss or data security issues, because even some of that is dialectal and varies between engines (drop keys, ignore keys, read keys, make sure everything is atomically transacted to the strongest transaction level available in that particular engine, etc).

Git's current pile of files may not be better than "a full-fledged SQL database", that's a long and difficult academic study to undertake, but a "a full-fledged SQL database" isn't necessarily the best solution just because it has a mostly standard query language, either.

Well, there is this: https://www.loc.gov/preservation/resources/rfs/data.html

Also, the on-disk format for SQLite has been extended, but has not fundamentally changed since version 3.0.0 was released on 2004-06-18. SQLite version 3.0.0 can still read and write database files created by the latest release, as long as the database does not use any of the newer features. And, of course, the latest release of SQLite can read/write any database. There are over a trillion SQLite databases in active use in the wild, and so it is important to maintain backwards compatibility. We do test for that.

The on-disk format is well-documented (https://sqlite.org/fileformat2.html) and multiple third parties have used that document to independently create software that both reads and writes SQLite database files. (We know this because they have brought ambiguities and omissions to our attention - all of which have now been fixed.)

> Pile-of-Files Formats. Sometimes the application state is stored as a hierarchy of files. Git is a prime example of this, though the phenomenon occurs frequently in one-off and bespoke applications. A pile-of-files format essentially uses the filesystem as a key/value database, storing small chunks of information into separate files. This gives the advantage of making the content more accessible to common utility programs such as text editors or "awk" or "grep". But even if many of the files in a pile-of-files format are easily readable, there are usually some files that have their own custom format (example: Git "Packfiles") and are hence "opaque blobs" that are not readable or writable without specialized tools. It is also much less convenient to move a pile-of-files from one place or machine to another, than it is to move a single file. And it is hard to make a pile-of-files document into an email attachment, for example. Finally, a pile-of-files format breaks the "document metaphor": there is no one file that a user can point to that is "the document".

More precisely:

> But even if many of the files in a pile-of-files format are easily readable, there are usually some files that have their own custom format (example: Git "Packfiles") and are hence "opaque blobs" that are not readable or writable without specialized tools.

What is advocated here is to transform the pile-of-files in a single SQLite database accessed through SQL queries. So instead of having only a few binary blob, transform everything in a binary blob and force the use of one specialized tool for everything.

> It is also much less convenient to move a pile-of-files from one place or machine to another, than it is to move a single file.

This is not true.

> And it is hard to make a pile-of-files document into an email attachment, for example.

I would not trust someone that had just sent his git repo over email.

> Finally, a pile-of-files format breaks the "document metaphor": there is no one file that a user can point to that is "the document".

A VCS will track source files. Maybe their argument is true for other applications, but for a VCS this is plain useless.

Indeed having only an SQL connector accessing a database is a unified interface to the file. But unifying this to the user means that you have to move the complexity further down, as explained:

> But an SQLite database is not limited to a simple key/value structure like a pile-of-files database. An SQLite database can have dozens or hundreds or thousands of different tables, with dozens or hundreds or thousands of fields per table, each with different datatypes and constraints and particular meanings, all cross-referencing each other, appropriately and automatically indexed for rapid retrieval, and all stored efficiently and compactly in a single disk file. And all of this structure is succinctly documented for humans by the SQL schema.

Yeah, and I don't want to have this complexity managed by a single "entity", I want to have several different tools available to do whichever kind of work I need to do. If I'm working on graphs and need to store them, I would prefer having the ability to read my file directly in my other tools for graph analysis / debugging without having to take the intermediate step of connecting to the SQL database, or redefining a way to work with the SQL paradigm to adapt my file format to the "dozens or hunders or thousands of different tables, fields per table, each with different datatypes".

This point is even more salient regarding grep / awk. The author obviously prefer using the query language of his choice and disregards the variety of tools to work on text, but there are many, many tools available to do all kind of work on it, and believing that

> An SQLite database file is not an opaque blob. It is true that command-line tools such as text editors or "grep" or "awk" are not useful on an SQLite database, but the SQL query language is a much more powerful and convenient way for examining the content, so the inability to use "grep" and "awk" and the like is not seen as a loss.

Is just nonsense. Passing on the file edition conveniently put under the rug, querying the text is usually only the beginning, usually someone wants to parse the output and act upon it, maybe even put back some modified version (sed), and so on.

The author just seems close-minded and living in his own world, unable to imagine that other people might want to work differently.

This reminds me a lot of his rant against git and for fossil, with the exact same bad faith arguments and lack of knowledge about other ways to do things.

- I could probably plug that into something else with more ease than something that is git-specific - I have more flexibility for querying out of the box, without learning the specifics of each subcommand. The full SQL language is there at my disposal for outputting exactly what I need

Unless you're intending to run joins on git data, were exactlty do you see any fundamental difference between running CRUD operations via an SQL interface or just importing/exporting a file?

create-graph = "!f() { git show-ref -s | git commit-graph write --stdin-commits ; } ; f"

And even then you still really only need run it once per repository, you can just cd/paste/return; cd/paste/return; … Hell, you'd probably have an easier time writing a script which looks for all git repositories and runs the command versus having to manually visit each and do so.

As a former BitKeeper developer, this is a key person for Microsoft to have on hand to improve git. BitKeeper got about 10X faster after it was used on the Linux kernel and many of the key performance wins were due to better graph traversal algorithms. Rick was a wizard at that sort of thing. The other was memory layout optimizations and caching. (my contribution)

So Rick made sure we walked the graph as little as necessary and I made sure the graph had an extremely compact representation containing as little information as possible and then once a target commit is found it is looked up in another store.

However, this quote was disturbing. "There are a few Git features that don’t work well with the commit-graph, such as shallow clones, replace-objects, and commit grafts. If you never use any of those features, then you should have no problems!" The joy of not having to write commercial software! Reading between the lines it appears they changed the default output order for 'git log' or some internal API and then didn't bother to fix the cases that depending on the old order.

These interactions with grafts, replace-objects, and shallow clones are one reason 2.18 does not create and manage this file automatically. The commit-graph file works by representing the commit relationships in a new file, and if that file exists, we treat that as the truth. The commit grafts, replace-objects, and shallow clones use another set of special files to track commits whose parents have been modified in special ways. If you would like to see our progress on integrating these features together, please see this thread on the Git mailing list: https://public-inbox.org/git/20180531174024.124488-1-dstolee...

Yes, my reply was unnecessarily disparaging. Overall this looks like a cool feature. Perhaps a stopgap solution if for those commands to just delete your cache. But then your repository will get mysteriously slower. I just need to think of this as a technology demonstration.

* yes, it's open source, but being open source and "you can add any feature yourself" doesn't imply there is a momentum behind and a kind of "directed" force to move it forward

    [alias]
        l = log --date-order --date=iso --graph --full-history --all --pretty=format:'%x08%x09%C(red)%h %C(cyan)%ad%x08%x08%x08%x08%x08%x08%x08%x08%x08%x08%x08%x08%x08%x08%x08 %C(bold blue)%aN%C(reset)%C(bold yellow)%d %C(reset)%s'

    *       8264cf2a 2018-06-28 yebyen (origin/dev, dev) ...
    |\
    | *     df6e4c49 blabla commit messages
    | *     195d8924 commit messages
    |/
    | *     96e74a5c (branch-label) commit message
    | |\
    | | *   b2786c07 blabla
    | | *   85721288 blabla

I think I found it here[1]

[1]: https://stackoverflow.com/a/16735971/661659

Also, I don't encourage rebasing, especially to junior developers. I realize everyone has different preferences, but a messy graph is useful and there are other tools like `git log --first-parent` for getting "clean" baselines from the graph. The great thing about using a graph in the first place is there are a lot of traversal options and you don't have to micromanage where the trees are to still see the forest.

Our team is actually really small, and we like to make sure everyone knows about the hassles involved in putting together a release and doing a complete code review when it's needed. The main reason I encourage rebasing is because it helps avoid a messy graph, and a messy graph makes it immediately much harder to do a rebase across any number of merges, or any non-trivial span of time.

So in other words, I like to selfishly expect the other developers on my team to do rebases at the appropriate times, in order to preserve my own capability to easily do rebases when needed. (We also learned last week that git rebase has a --preserve-merges option that I feel foolish to not have known about sooner. We've wiped out so many merge commits unnecessarily.)

We treat the master branch as carved in stone like anyone should, but other branches should clearly flow their merges only one way (features into releases, or features into environments), and if a branch hasn't actually been included in a release tag, it's considered fair game for rebasing. It helps us to prevent our three developers' sometimes too many concurrent trains of thought from resulting in equally too many confusing merges, or an intractable number of HEADs to manage and organize into releases.

One of the places we struggle is that we're not all-the-way onboard with CI/CD processes, but we do the best we can so that whenever support for that kind of thing materializes, we will mostly not need to change our processes at all and can obtain the benefits of that kind of tooling as quickly as possible.

I tend to disagree on this as well. The best person to merge a conflict is the developer creating the conflict in the first place, as soon or nearly as soon as they create the conflict, because they are most likely to know why the conflict exists in the first place. "Merge early, merge often."

Delaying "reverse" merges until the last possible second means you often don't have the integration expertise needed without research involved, even if it was "you" that introduced the conflict you may have moved on to other problems since and not recall why you did something one way or another, or which bits are important to integrate.

Delaying those merges as rebases I feel is even worse, because not only do you not have the resources to know why an integration needs to be made, you also aren't recording a history of the merge conflicts you saw in the rebase such that you can easily revisit the integration if you made an integration mistake (which does happen, because we are, after all, only human).

My advice tends to be to use a good PR system of some sort that makes your proper, code reviewed "forward" merges clear and obvious, and then don't worry about a mess of other merges "underneath" that top-level of strong PR merges. (Hence the key suggestion is that `git log --first-parent` is your best friend if you want a "clean" view of the DAG. It gives you a linear list of just your PR merges in master, or branch work and PR merges to that branch in any other branch.) Also, yes, CI/CD are really good ideas.

This is a good point. Mistakes are made during merge conflict resolution. But if you are tracking your upstream when you develop a long-lived feature branch, and rebasing when there are changes to the base, and actually comparing your rebased feature branches to the version that you had before you force push over the old remote version, then the only thing you really need to track is "does my diff still look like the change I intended to make."

The most valuable skill we are learning from Git is how to avoid merge conflicts altogether, and it tends to be more a human problem than a technological issue. (You don't avoid merge conflicts with some special commit strategy, you do it by ensuring that two people are not actively changing the same part of the codebase unless it's absolutely necessary.)

Also, every rebased branch is its own likely source of merge conflicts for people working on "differently rebased" versions of the same code.

> The most valuable skill we are learning from Git is how to avoid merge conflicts altogether, and it tends to be more a human problem than a technological issue. (You don't avoid merge conflicts with some special commit strategy, you do it by ensuring that two people are not actively changing the same part of the codebase unless it's absolutely necessary.)

Definitely. Communication is a key. It's also why I think "merge early, merge often" works better, because it forces communication as early and often as possible ("I'm seeing a merge conflict with this work you are doing, can you explain it to me?"). Keep branches as short-lived as possible, and try to avoid "separate but equal" work where you can't comingle features and have to intentionally fence your branches from integrating with each other. Merge feature branches between each other, even, to keep communication up. Find tools like feature flags that allow you to "ship" to master "unfinished" work faster to save yourself from trying to integrate long running branches after the fact. (If you are going to let "marketing" choose which features "ship" in a version, it is much nicer to do so by flipping flags in a dashboard somewhere, maybe even one Marketing can use themselves, than to try to furiously merge long-running feature branches at release time.)

The amount of uhh "pucker" I'll feel when we release next month, and certain pieces of code are touching prod for the first time, is much higher than I'd like.

The only reason I can sleep at night while spending a whole year with only hotfixes and minor feature addons for the last release going to prod, is because we've spent much too much time at all layers of the testing pyramid, and every time I've made a change that should probably break some tests, it actually breaks a few more tests than I expected (so I know that our coverage is pretty good.) Every time we write a brand new feature, it absolutely gets a matching Cucumber feature test in plain english that we run before and after every merge. And that's not even to mention unit testing.

So we can be reasonably confident that everything we've ever promised, is still true. Our test suite takes over an hour to run at this point, which is way longer than it should. But it helps us catch the bugs, and well before they start to have "piled up."

I'd feel marginally better if we knew what code doesn't run in prod because we could turn it on and off, and see those feature tests failing (or simply exclude them, since they would also be tagged with the feature label.)

It's interesting that some people feel feature flags are complicated enough to make "Feature flags as-a-Service" businesses a thing, like LaunchDarkly. I looked at what they are offering and thought it would serve us well, but we haven't actually started using feature flags for anything.

Rebasing is a fascinating footgun. I'd rather a messy story that includes details of how someone screwed up, then fixed their mistakes, than an entire branch I need to cherry pick in a hazmat suit because a developer rebased the wrong thing.

Strongly disagree. I am mostly interested in reviewing your best work, and ensuring that it is correct, as much as I enjoy hearing the story about how you got there. When we spend a lot of time on specs and ensuring they are correct, it's mostly important that I can review your change and compare it to the spec to ensure correctness. A commit that was wrong, but was corrected before it ever went to prod, doesn't really help me at all during code reviews.

I agree that 'git stash' and 'git commit --amend' are also important skills to have, but I prefer to make small, incremental commits that are obviously correct changes in isolation, even if they are not very interesting, so long as they are moving something toward the goal. Then I'll condense them into commits that look something like "the whole change I intended to make" – but that's maybe once it's fully formed and the tests are actually passing.

I don't like to keep a lot of small commits around, again selfishly and simply because they can make rebasing harder. It is telling how many things I do in service of keeping it easy to rebase a changeset. It's not an unobtrusive feature and there's definitely merit to the argument that it's a nice gun to shoot yourself in the foot with.

There is a happy medium between squashing all commits within a feature branch, and keeping every "boring, but obviously correct" incremental change intact forever. A change is only interesting in isolation if it might be reverted, or if I read it in isolation and for whatever reason strongly believe that I might ever want to read it in isolation again.

We have the power of a DAG to both record and hide all the mess. I can take my messiest repository and with a couple DAG traversal options still get that clean "story" back out of it. On the flipside with rebase you are absolutely erasing history and there is no way to dig deeper into any information lost or discarded in that rebase. I can give you what you want to see from my repositories, but you can't give me what I sometimes want to see from yours.

Merge commits are a great place to tell the story of "the whole change I intended to make". A good PR system even encourages and automates exactly that. (It also makes it easier to review the branch as a whole, or incrementally, or what have you in between to suit your needs/interests.)

I don't outlaw rebasing locally. There are absolutely cases in which a local rebase is necessary, and if a developer feels comfortable with rebasing I'm not going to stop them from doing it in a branch they control.

But I also don't encourage rebasing. I'd rather see how the sausage was made, ugly as it was, as it does tell a story even if you may not find it an "interesting" story, and sometimes in code archaeology or debugging nightmares you do need to dive into tiny incremental trivia.

The main point I'm trying to make is that developers should not have to feel compelled by some unspoken pressure to make fewer commits, knowing that once they are made, some day, someone will be stuck reading or replaying them. Commit history is malleable, and unpublished commits are mostly free to be thought of as exactly like soup; free to stir at any time, and ladle into portions as needed.

Sometimes I read a commit in a chain of commits that I made and, though it was interesting enough to commit for some reason, I can immediately tell that the moment I've just replayed will never be a point in time that I'll ever want to replay again. I can rebase in that moment and potentially save repeating some wasted time for "future-me."

Sometimes I even know at the time of making the commit that it will not be interesting ever again, but that at that moment, I'm less likely to make an error if I just `git add -p` the "done part," commit that, and keep the WIP unstaged. Maybe the change is so boring that I'll actually write "squash" or "fixup" as the commit message, and then I'll be sure to follow my own advice some time soon, when it's not so interruptive to my flow, and I will squash that commit before it gets published or merged.

I love descriptive commit messages, but I also love totally obvious, eminently readable code. And I feel like sometimes the message I need to get out to developer members of my team is that just one "readable" or patently obvious block of code is easily worth ten descriptive commit messages.

There also may be times when "nobody is reading your commit history" is a little demoralizing but also exactly what a developer needs to hear. "We don't need a novel about each commit, write just what you need to and spend that time you saved making the code itself actually better."

As great as Github is, I've never understood why they still have such a hard-to-read, horizontal commit graph while competitors like Stash/bitbucket/gitlab have all had beautiful vertical graphs (like shown in this article) for as long as I can remember. I think this is especially valuable for newbies who are less inclined to get similar viz at the command line, but still useful for vets when they (inevitably) end up in weird branch situations.

Makes me sad I'll never do anything like this.