F=mirroring mirroring gitolite servers

Mirroring a repo is simple in git; you just need code like this in a post-receive hook in each repo:

#!/bin/bash
git push --mirror slave_user@mirror.host:/path/to/repo.git
# if running gitolite, the $GL_REPO variable could be useful:
# git push --mirror slave_user@mirror.host:/repo/base/path/$GL_REPO.git

For a lot of people, though, mirroring is more than just 'backup', and their needs are complex enough that setup is hard.

#mirrwhy_ why

Gitolite's mirroring used to be very rigid -- one master, any number of slaves, but the slaves are identical copies of the master. No variations allowed.

It's now been reworked to be much more flexible, to cater to almost any kind of setup you need. Here're some advantages:

As you can see, this is a bit more than a backup solution ;-)

RULE NUMBER ONE!

RULE OF GIT MIRRORING: users should push directly to only one server! All the other machines (the slaves) should be updated by the master server.

If a user pushes directly to one of the slaves, those changes will get wiped out on the next mirror push from the real master server.

Corollary: if the primary went down and you effected a changeover, you must make sure that the primary does not come up in a push-enabled mode when it recovers.

Getting around rule number one: see the section on "redirecting pushes" later.

concepts and terminology

Servers can host 3 kinds of repos: master, slave, and local.

setting up mirroring

F=mirrcautions IMPORTANT cautions

F=mirrsetup setup and usage

server level setup

To start with, assign each server a short name. We will use 'frodo', 'sam', and 'gollum' as examples here.

  1. Generate ssh keys on each machine. Copy the .pub files to all other machines with the appropriate names. I.e., frodo should have sam.pub and gollum.pub, etc.

    Warning: server keys are different from user keys. Do NOT attempt to (re-)use a server key for normal gitolite operations, as if the server were a normal "user"; it won't work.

  2. Install gitolite on all servers, under some 'hosting user' (we'll use git in our examples here). You need not use the same hosting user on all machines.

    It is not necessary to use the same "admin key" on all the machines. However, if you do plan to mirror the gitolite-admin repo also, they will eventually become the same anyway. In our example, frodo does mirror the admin repo to sam, but not to gollum. (Can you really see frodo or sam trusting gollum?)

  3. Now copy hooks/common/post-receive.mirrorpush from the gitolite source, and install it as a custom hook called post-receive; see [here][customhooks] for instructions.

  4. Edit ~/.gitolite.rc on each machine and add/edit the following lines. The GL_HOSTNAME variable must have the correct name for that host (frodo, sam, or gollum), so that will definitely be different on each server. The other line can be the same, or may have additional patterns for other git config keys you have previously enabled. See [here][rsgc] and the description for GL_GITCONFIG_KEYS in [this][rcsecurity] for details.

    $GL_HOSTNAME = 'frodo';     # will be different on each server!
    $GL_GITCONFIG_KEYS = "gitolite.mirror.*";
    

    (Remember the "rc" file is NOT mirrored; it is meant to be site-local).

    Note: if GL_HOSTNAME is undefined, you cannot push to repos which have the 'gitolite.mirror.master' config variable set. (See 'details' section below for more info on this variable).

    If you wish, you can also add this hostname information to the GL_SITE_INFO variable in the rc file. See the rc file documentation for more on that.

  5. On each machine, add the keys for all other machines. For example, on frodo you'd run these two commands:

    gl-tool add-mirroring-peer sam.pub
    gl-tool add-mirroring-peer gollum.pub
    
  6. Create "host" aliases on each machine to refer to all other machines. See [here][sshhostaliases] for what/why/how.

    The host alias for a host (in other machines' ~/.ssh/config files) MUST be the same as the GL_HOSTNAME in the referred host's ~/.gitolite.rc. Gitolite mirroring requires this consistency in naming; things will NOT work otherwise.

    For example, if machine A's ~/.gitolite.rc says $GL_HOSTNAME = 'frodo';, then all other machines must use a host alias of "frodo" in their ~/.ssh/config files to refer to machine A.

Once you've done this, each host should be able to reach the other hosts and get a response back. For example, running this on sam:

ssh frodo info

should get you

Hello sam, I am frodo.

Check this command from everywhere to everywhere else, and make sure you get expected results. Do NOT proceed otherwise.

repository level setup

Setting up mirroring at the repository level instead of at the "entire server" level gives you a lot of flexibility (see "discussion" section below).

The basic idea is to use git config variables within each repo (gitolite allows you to create them from within the gitolite.conf file so that's convenient), and use these to specify which machine is the master and which machines are slaves for the repo.

Let's say frodo and sam are internal servers, while gollum is an external (and therefore less trusted) server that has agreed to help us out by mirroring one of our high traffic repos. We want the following setup:

So here's how our example would go:

  1. Clone frodo's and sam's gitolite-admin repos to your workstation, then add the following lines to both their gitolite.conf files:

    repo ip1 gitolite-admin
        config gitolite.mirror.master   =   "frodo"
        config gitolite.mirror.slaves   =   "sam"
    
    
    repo ip2
        config gitolite.mirror.master   =   "sam"
        config gitolite.mirror.slaves   =   "frodo"
    

    You also need normal access control lines for ip1 and ip2; I'm assuming you already have them elsewhere, at least on frodo. (What you have on sam won't matter in a few minutes, as you will see!)

    Commit and push these changes.

  2. There are a couple of quirks to keep in mind when you make changes to the gitolite-admin repo's config.

  3. That completes the setup of the gitolite-admin and the internal project repos. We'll now setup things for the open source project, "os1".

    On frodo's gitolite-admin clone, add the following lines to conf/gitolite.conf, then commit and push:

    repo os1
        config gitolite.mirror.master   =   "frodo"
        config gitolite.mirror.slaves   =   "sam gollum"
    

    Also, send the same lines to gollum's administrator and ask him to add them into his conf/gitolite.conf file, commit, and push.

F=mirrsync commands to (re-)sync mirrors

You don't have to put all the slaves in gitolite.mirror.slaves. For example, let's say you have some repos that are very active, and two of your mirrors that are halfway across the world are getting pushed very frequently. But you don't need those mirrors to be that closely updated, perhaps because they are halfway across the world and those guys are asleep ;-)

Or maybe there was a network glitch and even the default slaves are now lagging, so they need to be manually synced.

Or a slave realised that one of its repos is lagging for some reason, and wants to request an immediate update.

Whatever the reason, you need ways to sync a repo from a command line. Here are ways to do that:

  1. On the master server, you can start a background job to mirror a repo. The command/syntax is

    gl-mirror-shell request-push reponame [list of keys/slaves]
    

    The list at the end is optional, and can be a mix of slave names or your own gitolite mirror config keys. (Yes, you can have any key, named anything you like, as long as it starts with gitolite.mirror.).

    If the list is not supplied, the gitolite.mirror.slaves key is used.

    Keys can have values that in turn contain a list of keys/slaves. The list is recursively expanded but recursion is not detected. Order is preserved while duplicates are removed. If you didn't get that, see the example :-)

    Warning: the gitolite.mirror.slaves key should have only hosts, no keys, in it.

    The program exits with a return value of "1" if it found no slaves in the list passed, otherwise it fires off the background job, prints an informative message, and exits with a return value of "0".

    We'll take an example. Let's say your gitolite config file has this:

    repo ip1
        config gitolite.mirror.master       =   "frodo"
        config gitolite.mirror.slaves       =   "sam merry pippin"
        config gitolite.mirror.hourly       =   "sam legolas"
        config gitolite.mirror.nightly      =   "gitolite.mirror.hourly gimli"
        config gitolite.mirror.all          =   "gitolite.mirror.nightly gitolite.mirror.hourly gitolite.mirror.slaves"
    

    Then the following commands have the results described in comments:

    gl-mirror-shell request-push ip1
    # which is the same as:
    gl-mirror-shell request-push ip1 gitolite.mirror.slaves
    # pushes to sam, merry, pippin
    
    
    gl-mirror-shell request-push ip1 gollum
    # pushes only to gollum.  Note that gollum is not a member of any of
    # the slave lists we defined.
    
    
    gl-mirror-shell request-push ip1 gitolite.mirror.slaves gollum
    # pushes to sam, merry, pippin, gollum
    
    
    gl-mirror-shell request-push ip1 gitolite.mirror.slaves gitolite.mirror.hourly
    # pushes to sam, merry, pippin, legolas
    
    
    gl-mirror-shell request-push ip1 gitolite.mirror.all
    # pushes to sam, legolas, gimli, merry, pippin
    

    The last two examples show recursive expansion with order-preserving duplicate removal (hey there's now a published conference paper on gitolite, so we have to use jargon somewhere or they won't accept follow-on papers!).

    If you do something like this:

    config gitolite.mirror.nightly      =   "gimli gitolite.mirror.nightly"
    

    or this:

    config gitolite.mirror.nightly      =   "gimli gitolite.mirror.hourly"
    config gitolite.mirror.hourly       =   "legolas gitolite.mirror.nightly"
    

    you deserve what you get.

  2. If you want to start a foreground job, the syntax is gl-mirror-shell request-push ip1 -fg gollum. Foreground mode requires one (and only one) slave name -- you cannot send to an implicit list, nor to more than one slave.

  3. Cronjobs and custom mirroring schemes are now very easy to do. Use either of the command forms above and write a script around it. Appendix A contains an example setup.

  4. Once in a while a slave will realise it needs an update, and wants to ask for one. It can run this command to do so:

    ssh sam request-push ip2
    

    If the requesting server is not one of the slaves listed in the config variable gitolite.mirror.slaves on the master, it will be rejected.

    This is always a foreground push, reflecting the fact that the slave may want to know why their push errored out or didn't work last time or whatever.

#ad/m-dtls details

F=mirrconf the conf/gitolite.conf file

One goal I have is to minimise the code changes to "core" gitolite due to this, so all repo-specific mirror settings are stored as git config variables (you know you can specify git config variables in the gitolite config file right?). These are:

F=mirrredirect redirecting pushes

Please read carefully; there are security implications if you enable this for mirrors NOT under your control.

When a user pushes to a non-native repo, it is possible to transparently redirect the push to the correct master server. This is a very neat feature, because now all your users just use one URL (the mirror nearest to them). They don't need to know where the actual master is, and more importantly, if you and the other admins change it, they don't need to know it changed!

The gitolite.mirror.redirectOK config variable decides where this redirection is OK. If it is set to 'true', any valid 'slave' can redirect an incoming non-native push from a developer. Otherwise, it contains a list of slaves that are permitted to redirect pushes (this might happen if you don't trust some of your slaves enough to accept a redirected push from them).

Warning: like gitolite.mirror.slaves, this key also should have only hosts, no keys, in it.

This check needs to pass on both the master and slave servers; both have a say in deciding if this is allowed. (The master may have real reasons not to allow this; see below. I cannot think of any real reason for the slave to disable this, but it's there in case some admin doesn't like it).

There are some potential issues that you MUST consider before enabling this:

example setups

Here are some samples of what is possible.

F=mirrnonauto non-autonomous

In this setup, the slave server is under the same "management" as the master. All repos, including gitolite-admin are mirrored, and each slave is an exact replica of the master. Since the admin repo is mirrored, authentication info is identical across all servers, and it is safe to use redirected pushes. (This was the only type of mirroring possible in the old mirroring code in gitolite).

Install gitolite on all servers. Then add these lines to the top of all admin repos and push them all. This sets up the config for mirroring all repos.

repo @all
    config gitolite.mirror.master = "frodo"
    config gitolite.mirror.slaves = "sam gollum"

Once they're all pushed, sync the admin repo once:

# on master server
gl-mirror-shell request-push gitolite-admin

Since authentication is also being mirrored, you can take advantage of redirected pushing if you wish:

repo @all
    config gitolite.mirror.redirectOK = "true"

F=mirrnonautolocal non-autonomous with local repos

As above, but you want to allow each slave server to have some repos be "local" to the server (not be mirrored), for whatever reason. Different slaves may have different needs, so this really means that the same gitolite.conf should behave differently on each server -- something which till now was impossible.

Well what's life without a new feature once in a while? The string "HOSTNAME" is now specially treated in an include filename. If it is seen without any alphanumeric characters or underscores next to it on either side, it is replaced by the value of $GL_HOSTNAME.

Setup the config as in the previous setup except that you shouldn't use repo @all now; instead, you'll have to name the repos to be mirrored in some way. Make sure gitolite-admin is in this list. Complete the mirror setup (including the first-time sync command) like before.

Now add the line include "HOSTNAME.conf" to the end of conf/gitolite.conf, and create new files, conf/frodo.conf, conf/sam.conf, etc., with appropriate content.

That's it. When this config is pushed, each machine will have an effective config that consists of the main file, with the correct HOSTNAME.conf included (and all the others ignored) when the include statement is reached.

F=mirrsemiauto semi-autonomous

So far, the "central" admin still has control over the gitolite.conf file and all repos created. Sometimes it's easier to give control over parts of the configuration to people at the mirror sites. To keep it simple, each admin will be able to do whatever they want to directories within a subdirectory of the same name as the hostname.

You can combine the "HOSTNAME" feature above with [delegation][deleg]. Let's say the admin for sam is a user called "gamgee", and the admin for gollum is "smeagol".

Add this to your conf file:

@sam    =   sam/..*
@gollum =   gollum/..*

Then use NAME/ rules (see the delegation doc for details) and allow gamgee to write only conf/sam.conf, and smeagol to write only conf/gollum.conf.

Now in the main config file, at the end (or wherever you wish), add one line:

subconf "HOSTNAME.conf"

F=mirrauto autonomous

In many ways this is the simplest setup.

The slave server belongs to someone else. Their admin has final say on what goes into their gitolite-admin repo and thus their server's config. The gitolite-admin repo is NOT mirrored, and mirroring of individual repos (i.e., actual config lines included) is by negotiation/agreement between the admins.

Authentication info is not common. The master has no real control over who can read the repos on the slave. Allowing redirected pushes is not a good idea, unless you have other means of trust (administrative, contractual, legal, etc.)

Best for open source projects with heavy "fetch" load compared to "push".

F=mirrdisc discussion

problems with the old mirroring model

The old mirroring model had a single server as the master for all repositories. Slaves were effectively only for load-balancing reads, or for failover if the master died.

This is not good enough for corporate setups where the developers are spread fairly evenly across the world. Some repos need to be closer to some teams (NUMA is a good analogy).

A model where different repos are "mastered" in different cities is much more efficient here.

The old model had other rigidities too, though they're not really problems, as such:

the new mirroring model

In the new model, servers can be much more independent and autonomous than in the old model. (Don't miss the side note in the 'repository level setup' section if you prefer the old model).

The new model has a few pros and cons. The pros come from the flexibility and freedom that mirrors servers get, and the cons come from authorisation being more rigorously checked (for example, a slave will only accept a push if its configuration also says that the sending server is indeed the master for this repo).

appendices

F=mirrcron appendix A: example cronjob based mirroring

Let's say you have some repos that are very active. You're pushing halfway across the world every few seconds, but those slaves do not need to be that closely updated, perhaps because they are halfway across the world and those guys are asleep ;-)

You'd like to update them once an hour instead. Here's how you might do that.

First add this line to the configuration for those repos:

config gitolite.mirror.hourly = "slave1 slave2 slave3"

Then write a cron job that looks like this (untested).

#!/bin/bash

REPO_BASE=`${0%/*}/gl-query-rc REPO_BASE`

cd $REPO_BASE
find . -type d -name "*.git" -prune | while read r
do
    # get reponame as gitolite knows it
    r=${r:2}
    r=${r%.git}

    gl-mirror-shell request-push $r gitolite.mirror.hourly

    # that command backgrounds the push, so you'd best wait a few seconds
    # before hitting the next one, otherwise you'll have all your repos
    # going out at once!
    sleep 10
done

F=mirrparanoia appendix B: efficiency versus paranoia

If you're paranoid enough to use mirrors, you should be paranoid enough to use the receive.fsckObjects setting. However, informal tests indicate a 40-50% CPU overhead from this. If you're ok with that, make the appropriate adjustments to GL_GITCONFIG_KEYS in the rc file, then add this to your gitolite.conf file:

repo @all
    config receive.fsckObjects = "true"

Personally, I just set git config --global receive.fsckObjects true, since those servers aren't doing anything else anyway, and are idle for long stretches of time. It's upto you what you want to do here.