Phil Booth

Existing by coincidence, programming deliberately

Build a better release script

Automating your release process saves time, eliminates tedious busywork and reduces the likelihood of mistakes when cutting a new release. There are plenty of off-the-shelf solutions available, but this post will show how easy it is to build your own release script and why the end result can be better than using a generic, third-party option. Throughout the post I'll use the case study of some recent work we did to automate the release process for FxA, to provide concrete examples of what I'm talking about.

Why

There's nothing wrong with off-the-shelf solutions per se, and if they work for you that's great. But many projects have local idiosyncracies, which a generic script can't cater to by definition. Perhaps the generic option doesn't do some things your project needs, or does them differently, or it does other things you don't want a release script to do.

On FxA, after migrating all of our services to a monorepo, we wanted a single script that works equally well for our JavaScript and Rust codebases, now they all live in one place. We also need to co-ordinate version numbers across public and private forks of our repository, because some security-sensitive commits only exist in the private fork. And we have an FxA-specific step of opening a deployment ticket in Bugzilla for our Ops team, copying notes into it from various sources.

Where to begin

You should start by making a list of everything that you and/or your team have to do when it's time to cut a new release. Include every detail, even if there are things you don't think can be automated. Once you have the complete list, try to arrange it into a rough chronological order. Some things can happen in parallel and the order won't matter for those, but try to organise items so they hang together coherently in a single thread.

For FxA, our list looked roughly like this:

  1. Make sure all the pull requests labelled for the release have been merged. Check that no-one has anything else they want to land.

  2. Create a branch to cut the release from.

  3. Bump the version strings for all services that have changed since the last release. For JavaScript projects the versions are stored in JSON files, but for Rust they're stored in TOML files.

  4. Update each change log with details of all the commits made since the last release.

  5. Commit those changes.

  6. Create a git tag.

  7. Create another release branch from the private fork.

  8. Merge the public release branch into the private release branch.

  9. Push the public branch and tag to the public remote.

  10. Push the private branch and tag to the private remote.

  11. Check the respective builds have passed in CI and images have been uploaded to DockerHub.

  12. Open a deployment ticket, with links to the tags, the builds, the change logs, deployment notes and any requests for QA.

  13. Open pull requests to merge the release branches back to their respective trunks.

We also had a slight variation to that list for our patch releases, where a pre-existing release branch is used and there's no need to open a new deployment ticket. We wanted a script to handle both workflows.

Decide on scope

When you have your complete list, you can go through each item and decide whether it's in scope for automation, or out of scope. The in-scope items are going to form the body of your script, so the main qualification for an item being in scope is deciding whether it can be automated. The out-of-scope items won't be ignored completely though. The script can emit reminders and even commands to copy/paste, so people are less likely to forget them.

For FxA, items 2 through 10 were deemed to be in scope for automation, leaving items 1, 11, 12 and 13 as manual steps requiring a human to help out.

Pick a language

Next, you should decide what language you're going to work in. This decision should take into account your own or your team's competencies, but also the nature of the tasks being automated.

If you're not considering POSIX-compatible shell syntax at this point, I'd like to make a case for it. The Bourne shell is virtually ubiquitous in modern development environments, with interpreters widely available across a variety of environments on Linux, MacOS and Windows. Writing shell code that works with git is immediately intuitive because it uses the exact same interface you're already familiar with from the command line. And other UNIX commands such as cut, grep, awk and sed make working with git's output straightforward too.

For FxA, /bin/sh was an obvious choice for all of the reasons above. The only downside was that not everyone felt fully comfortable maintaining a shell script. To mitigate that we commented the script heavily.

All of the code examples that follow are taken from the script we wrote for FxA. If you want to read the whole thing in its entirety before diving any deeper, you can find it here.

Enforce pre-requisites

The first thing your script should do is check pre-requisite conditions and abort if any of them aren't met. Aborting early means your script is less likely to fail part-way through, leaving someone's local tree in a bad state.

For FxA, these pre-requisites include:

For the command-line argument, we use the value patch to indicate a patch-level bump and the absence of any argument indicates a major release, or "train" in FxA-speak. The code for checking it looks like this:

case "$1" in
  "")
    BUILD_TYPE="Train"
    ;;
  "patch")
    BUILD_TYPE="Patch"
    ;;
  *)
    echo "Release aborted: Invalid argument \"$1\""
    exit 1
    ;;
esac

Keep an eye out for that BUILD_TYPE variable as it will make a number of appearances in later steps.

To check there are no uncommitted changes locally, we use git status. Passing it the --porcelain argument makes it return parseable output and if that result is not the empty string, we error out:

STATUS=`git status --porcelain`
if [ "$STATUS" != "" ]; then
  echo "Release aborted: You have uncommited changes"
  exit 1
fi

There's a small refactoring we can make here, extracting a function to abort the script instead of repeating the pattern of echo then exit every time we want to fail:

abort() {
  echo "Release aborted: $1."
  exit 1
}

So the previous status check now looks like this:

STATUS=`git status --porcelain`
if [ "$STATUS" != "" ]; then
  abort "You have uncommited changes"
fi

To check that some commits have been made since the last tag, there are a couple of different options.

Firstly, you could just get the most recent tag that is reachable from the current HEAD:

LAST_TAG=`git describe --tags --first-parent --abbrev=0`

The --first-parent argument prevents tags from merged branches being selected, which you may or may not want depending on how branching is done in your repository.

The alternative approach is to sort all tags alphanumerically and then pick the last one. This works if your tags aren't necessarily arranged chronologically in your history:

LAST_TAG=`git tag -l --sort=version:refname | tail -1`

For FxA, we actually use one or the other, depending on what type of release it is:

if [ "$BUILD_TYPE" = "Train" ]; then
  LAST_TAG=`git tag -l --sort=version:refname | tail -1`
else
  LAST_TAG=`git describe --tags --first-parent --abbrev=0`
fi

This allows us to tag patches for older releases, without more recent tags causing a problem.

Regardless of how you identify the last tag, you can check for intervening commits with git log:

COMMITS=`git log $LAST_TAG..HEAD`
if [ "$COMMITS" = "" ]; then
  abort "I see no work"
fi

Pull from remote branches

It's easy to forget to pull from remote branches before cutting a release, so it makes sense to have the script do that job for you too. But some care has to be taken when doing so, because the local repository might be on a different branch when the script is executed. And at least in FxA's case, the remote branch may or may not exist yet, depending on what type of release it is. If we're creating a new release branch from scratch, we want to pull from master, but if we're bumping an existing release, we want to pull from the release branch. And before we can do any of that, we need to work out what the name of the release branch actually is.

In FxA, release branches are named like train-$TRAIN, where $TRAIN is the train number from the version string. The version string is encoded in the tags, which are of the form v1.$TRAIN.$PATCH, e.g. at the time of writing the current tag is v1.138.4. The tags in our private fork also have a -private suffix, e.g. v1.138.4-private. So we can get the name of the release branch from the LAST_TAG variable that we set earlier, by splitting it into its constituent parts using cut:

MAJOR=`echo "$LAST_TAG" | cut -d '.' -f 1 | cut -d 'v' -f 2`
TRAIN=`echo "$LAST_TAG" | cut -d '.' -f 2`
PATCH=`echo "$LAST_TAG" | cut -d '.' -f 3 | cut -d '-' -f 1`

Here we've set MAJOR to the substring between the v and the first period, TRAIN to the substring between the first and second periods, and PATCH to the substring after the second period (and before the hyphen, if one exists).

Now that we've broken the version string into its constituent parts, we can determine the name of the release branch, depending on what type of build it is:

case "$BUILD_TYPE" in
  "Train")
    NEW_TRAIN=`expr $TRAIN + 1`
    ;;
  "Patch")
    NEW_TRAIN="$TRAIN"
    ;;
esac
RELEASE_BRANCH="train-$NEW_TRAIN"

So for major releases, we increment the train number using expr and for patches, we just re-use the current train number.

Now we know the branch name, we can check which branch we're on locally. If we're already on the release branch, we just need to pull from origin to get the latest changes. If we're not, we can look for a remote release branch instead. If one exists use that, otherwise create a new branch from master:

CURRENT_BRANCH=`git branch --no-color | grep '^\*' | cut -d ' ' -f 2`

if [ "$CURRENT_BRANCH" = "$RELEASE_BRANCH" ]; then
  git pull origin "$RELEASE_BRANCH" > /dev/null 2>&1 || true
else
  RELEASE_BRANCH_EXISTS=`git branch --no-color | awk '{$1=$1};1' | grep "^$RELEASE_BRANCH\$"` || true

  if [ "$RELEASE_BRANCH_EXISTS" = "" ]; then
    git fetch origin $RELEASE_BRANCH > /dev/null 2>&1 || true

    REMOTE_BRANCH="origin/$RELEASE_BRANCH"
    REMOTE_BRANCH_EXISTS=`git branch --no-color -r | awk '{$1=$1};1' | grep "^$REMOTE_BRANCH\$"` || true

    if [ "$REMOTE_BRANCH_EXISTS" = "" ]; then
      echo "Warning: $RELEASE_BRANCH branch not found on local or remote, creating one from master."
      git checkout master > /dev/null 2>&1
      git pull origin master > /dev/null 2>&1
      git checkout -b "$RELEASE_BRANCH" > /dev/null 2>&1
    else
      git checkout --track -b "$RELEASE_BRANCH" "$REMOTE_BRANCH" > /dev/null 2>&1
    fi
  else
    git checkout "$RELEASE_BRANCH" > /dev/null 2>&1
    git pull origin "$RELEASE_BRANCH" > /dev/null 2>&1 || true
  fi
fi

There's a few things worth calling out here:

There is a problem introduced by this code in that it changes the local branch, so if the script aborts the user will find themselves in a different state to when they ran the script. Clearly that's unacceptable, so we can fix it by moving the assignment to CURRENT_BRANCH to the very beginning of the script, then changing our earlier definition of the abort function like so:

abort() {
  git checkout "$CURRENT_BRANCH" > /dev/null 2>&1
  echo "Release aborted: $1."
  exit 1
}

Bump version strings

Now we're on the right branch and have pulled latest changes, we can get on with updating the version strings. We already broke the current version string down into its constituent parts and bumped the train number, so it's a minor tweak to revisit that code and bump the patch level too:

case "$BUILD_TYPE" in
  "Train")
    NEW_TRAIN=`expr $TRAIN + 1`
    NEW_PATCH=0
    ;;
  "Patch")
    NEW_TRAIN="$TRAIN"
    NEW_PATCH=`expr $PATCH + 1`
    ;;
esac

Then we can recombine the parts and generate our new version string:

NEW_VERSION="$MAJOR.$NEW_TRAIN.$NEW_PATCH"

We're going to use sed to update the version in a bunch of places, so we need to turn the old version string into a regular expression:

LAST_VERSION_REGEX="$MAJOR\\.$TRAIN\\.$PATCH"

We have to deal with two levels of character escaping here, because we want to escape the period in the regex using a backslash but to do that in a shell script, we must first escape the backslash itself using another backslash.

Updating the version strings then happens in two parts.

First we have to loop through each of the directories in our repo. For this we're going to assume the existence of a function called update that we'll define in a moment:

DIRECTORIES="packages/fxa-auth-db-mysql
packages/fxa-auth-server
packages/fxa-content-server
packages/fxa-customs-server
packages/fxa-email-event-proxy
packages/fxa-email-service
packages/fxa-event-broker
packages/fxa-profile-server"

while read -r DIRECTORY; do
  update "$DIRECTORY"
done <<< "$DIRECTORIES"

Then for each directory we're going to look for files where version strings might be stored and use sed to update them. We do that in the implementation of update like this:

update() {
  if [ -f "$1/package.json" ]; then
    sed -i.release -e "s/$LAST_VERSION_REGEX/$NEW_VERSION/g" "$1/package.json"
    rm "$1/package.json.release"
  fi

  if [ -f "$1/Cargo.toml" ]; then
    sed -i.release -e "s/$LAST_VERSION_REGEX/$NEW_VERSION/g" "$1/Cargo.toml"
    rm "$1/Cargo.toml.release"
  fi
}

Note the -i option to sed is optional in some environments but we choose to include it for maximum portability.

The nice thing with this pattern is its ambivalence about the target language. If the repo grows to include projects from other languages like Ruby or Python, it's simple to add more invocations of sed as needed.

Update change logs

For the change logs, we'll extend our implementation of the update function to loosely parse the first line of each commit message for the target directory, then write a sorted list of commits to each package's CHANGELOG.md.

FxA follows the Angular.js conventions for formatting commit messages. This means the first line of the commit should be of the form:

type(scope): summary message

type is limited to a strict set of values, the most interesting of which are feat, fix, perf, refactor and revert. In our change logs, we want to group commit messages under each of those headings to make it easier for readers to find particular changes they might be looking for.

We'll also include the commit hash with the summary message, to help anyone who wants to link a change from the log to the specific point in git's history.

To do this, we'll add the following code to the body of update:

LOCAL_COMMITS=`git log $LAST_TAG..HEAD --no-color --pretty=oneline --abbrev-commit -- "$1"`

while read -r COMMIT; do
  HASH=`echo "$COMMIT" | cut -d ' ' -f 1`
  MESSAGE=`echo "$COMMIT" | cut -d ':' -f 2- | awk '{$1=$1};1'`
  TYPE=`echo "$COMMIT" | cut -d ' ' -f 2 | awk '{$1=$1};1' | cut -d ':' -f 1 | cut -d '(' -f 1 | awk '{$1=$1};1'`
  SCOPE=`echo "$COMMIT" | cut -d '(' -f 2 | cut -d ')' -f 1 | awk '{$1=$1};1'`

  if [ "$SCOPE" = "$COMMIT" ]; then
    SCOPE=""
  fi

  if [ "$SCOPE" != "" ]; then
    SCOPE="$SCOPE: "
  fi

  case "$TYPE" in
    "")
      # Ignore blank lines
      ;;
    "Merge")
      # Ignore merge commits
      ;;
    "Release")
      # Ignore release commits
      ;;
    "feat")
      if [ "$FEAT_SUMMARY" = "" ]; then
        FEAT_SUMMARY="### New features\n"
      fi
      FEAT_SUMMARY="$FEAT_SUMMARY\n* $SCOPE$MESSAGE ($HASH)"
      ;;
    "fix")
      if [ "$FIX_SUMMARY" = "" ]; then
        FIX_SUMMARY="### Bug fixes\n"
      fi
      FIX_SUMMARY="$FIX_SUMMARY\n* $SCOPE$MESSAGE ($HASH)"
      ;;
    "perf")
      if [ "$PERF_SUMMARY" = "" ]; then
        PERF_SUMMARY="### Performance improvements\n"
      fi
      PERF_SUMMARY="$PERF_SUMMARY\n* $SCOPE$MESSAGE ($HASH)"
      ;;
    "refactor")
      if [ "$REFACTOR_SUMMARY" = "" ]; then
        REFACTOR_SUMMARY="### Refactorings\n"
      fi
      REFACTOR_SUMMARY="$REFACTOR_SUMMARY\n* $SCOPE$MESSAGE ($HASH)"
      ;;
    "revert")
      if [ "$REFACTOR_SUMMARY" = "" ]; then
        REVERT_SUMMARY="### Reverted changes\n"
      fi
      REVERT_SUMMARY="$REVERT_SUMMARY\n* $SCOPE$MESSAGE ($HASH)"
      ;;
    *)
      if [ "$OTHER_SUMMARY" = "" ]; then
        OTHER_SUMMARY="### Other changes\n"
      fi
      OTHER_SUMMARY="$OTHER_SUMMARY\n* $SCOPE$MESSAGE ($HASH)"
      ;;
  esac

  if [ "$FEAT_SUMMARY" != "" ]; then
    FEAT_SUMMARY="$FEAT_SUMMARY\n\n"
  fi

  if [ "$FIX_SUMMARY" != "" ]; then
    FIX_SUMMARY="$FIX_SUMMARY\n\n"
  fi

  if [ "$PERF_SUMMARY" != "" ]; then
    PERF_SUMMARY="$PERF_SUMMARY\n\n"
  fi

  if [ "$REFACTOR_SUMMARY" != "" ]; then
    REFACTOR_SUMMARY="$REFACTOR_SUMMARY\n\n"
  fi

  if [ "$REVERT_SUMMARY" != "" ]; then
    REVERT_SUMMARY="$REVERT_SUMMARY\n\n"
  fi

  if [ "$OTHER_SUMMARY" != "" ]; then
    OTHER_SUMMARY="$OTHER_SUMMARY\n\n"
  fi

  SUMMARY="$FEAT_SUMMARY$FIX_SUMMARY$PERF_SUMMARY$REFACTOR_SUMMARY$OTHER_SUMMARY"
  if [ "$SUMMARY" = "" ]; then
    SUMMARY="No changes.\n\n"
  fi

  awk "{ gsub(/^## $LAST_VERSION/, \"## $NEW_VERSION\n\n$SUMMARY## $LAST_VERSION\") }; { print }" "CHANGELOG.md" > "CHANGELOG.md.release"
  mv "CHANGELOG.md.release" "CHANGELOG.md"
done <<< "$LOCAL_COMMITS"

We use cut a number of times to pull out the different components of the commit message and awk is used like before to trim spaces from the resulting strings. We then collect the formatted messages into different variables and write them to the correct point in the change log using awk.

This code also uses a variable LAST_VERSION that we haven't defined yet, so we should scoot back to where we defined LAST_VERSION_REGEX and add this line alongside it:

LAST_VERSION="$MAJOR.$TRAIN.$PATCH"

Tag the release

At this point, the local tree will contain uncommitted changes so we need to commit them:

git commit -a -m "Release $NEW_VERSION"

Then we can create the tag:

NEW_TAG="v$NEW_VERSION"
git tag -a "$NEW_TAG" -m "$BUILD_TYPE release $NEW_VERSION"

Lastly, we mustn't forget to return the user to their original branch:

git checkout "$CURRENT_BRANCH" > /dev/null 2>&1

Talk to the user

In many of the commands above, we sent the output to /dev/null and the user doesn't have any idea what has happened. So we should tell them that the script finished successfully and how they can check what's changed:

echo
echo "Success! The release has been tagged locally but it hasn't been pushed."
echo "Before pushing, you should check that the changes appear to be sane."
echo
echo "Branch:"
echo
echo "  $RELEASE_BRANCH"
echo
echo "Tag:"
echo
echo "  $NEW_TAG"

We can also tell them what their next steps are:

echo
echo "When you're ready to push, paste the following lines into your terminal:"
echo
echo "git push origin $RELEASE_BRANCH"
echo "git push origin $NEW_TAG"
echo
echo "After that, you must open a pull request to merge the changes back to master:"
echo
echo "  https://github.com/mozilla/fxa/compare/$RELEASE_BRANCH?expand=1"

Putting it all together

This is the full output we see when we run the script we built for FxA:

~/c/fxa (train-138) $ ./release.sh patch
[train-138 1d8914565] Release 1.138.5
 23 files changed, 44 insertions(+), 16 deletions(-)

Success! The release has been tagged locally but it hasn't been pushed.
Before pushing, you should check that the changes appear to be sane.
At the very least, eyeball the diffs and git log.
If you're feeling particularly vigilant, you may want to run some of the tests and linters too.

Branches:

  train-138
  train-138-private

Tags:

  v1.138.5
  v1.138.5-private

When you're ready to push, paste the following lines into your terminal:

git push origin train-138
git push origin v1.138.5
git push private train-138-private
git push private v1.138.5-private

After that, you must open pull requests in both the public and private repos to merge the changes back to master:

  https://github.com/mozilla/fxa/compare/train-138?expand=1
  https://github.com/mozilla/fxa-private/compare/train-138-private?expand=1

Don't forget to leave a comment in the deploy bug.

Include links to the tags:

### Tags
* https://github.com/mozilla/fxa/releases/tag/v1.138.5
* https://github.com/mozilla/fxa-private/releases/tag/v1.138.5-private

There will be bugs

You can see the full release script for FxA here. We fixed a number of issues to reach that point, so it might be instructive to link to some examples of things we got wrong along on the way: