Existing by coincidence, programming deliberately
Automating your release process saves time, eliminates tedious busywork and reduces the likelihood of mistakes when cutting a new release. There are plenty of off-the-shelf solutions available, but this post will show how easy it is to build your own release script and why the end result can be better than using a generic, third-party option. Throughout the post I'll use the case study of some recent work we did to automate the release process for FxA, to provide concrete examples of what I'm talking about.
There's nothing wrong with off-the-shelf solutions per se, and if they work for you that's great. But many projects have local idiosyncracies, which a generic script can't cater to by definition. Perhaps the generic option doesn't do some things your project needs, or does them differently, or it does other things you don't want a release script to do.
On FxA, after migrating all of our services to a monorepo, we wanted a single script that works equally well for our JavaScript and Rust codebases, now they all live in one place. We also need to co-ordinate version numbers across public and private forks of our repository, because some security-sensitive commits only exist in the private fork. And we have an FxA-specific step of opening a deployment ticket in Bugzilla for our Ops team, copying notes into it from various sources.
You should start by making a list of everything that you and/or your team have to do when it's time to cut a new release. Include every detail, even if there are things you don't think can be automated. Once you have the complete list, try to arrange it into a rough chronological order. Some things can happen in parallel and the order won't matter for those, but try to organise items so they hang together coherently in a single thread.
For FxA, our list looked roughly like this:
Make sure all the pull requests labelled for the release have been merged. Check that no-one has anything else they want to land.
Create a branch to cut the release from.
Bump the version strings for all services that have changed since the last release. For JavaScript projects the versions are stored in JSON files, but for Rust they're stored in TOML files.
Update each change log with details of all the commits made since the last release.
Commit those changes.
Create a git tag.
Create another release branch from the private fork.
Merge the public release branch into the private release branch.
Push the public branch and tag to the public remote.
Push the private branch and tag to the private remote.
Check the respective builds have passed in CI and images have been uploaded to DockerHub.
Open a deployment ticket, with links to the tags, the builds, the change logs, deployment notes and any requests for QA.
Open pull requests to merge the release branches back to their respective trunks.
We also had a slight variation to that list for our patch releases, where a pre-existing release branch is used and there's no need to open a new deployment ticket. We wanted a script to handle both workflows.
When you have your complete list, you can go through each item and decide whether it's in scope for automation, or out of scope. The in-scope items are going to form the body of your script, so the main qualification for an item being in scope is deciding whether it can be automated. The out-of-scope items won't be ignored completely though. The script can emit reminders and even commands to copy/paste, so people are less likely to forget them.
For FxA, items 2 through 10 were deemed to be in scope for automation, leaving items 1, 11, 12 and 13 as manual steps requiring a human to help out.
Next, you should decide what language you're going to work in. This decision should take into account your own or your team's competencies, but also the nature of the tasks being automated.
If you're not considering
POSIX-compatible shell syntax
at this point,
I'd like to make a case for it.
The Bourne shell is virtually ubiquitous
in modern development environments,
with interpreters widely available
across a variety of environments
on Linux, MacOS and Windows.
Writing shell code that works with git
is immediately intuitive
because it uses the exact same interface
you're already familiar with
from the command line.
And other UNIX commands
such as cut,
grep,
awk
and sed
make working with git's output
straightforward too.
For FxA,
/bin/sh was an obvious choice
for all of the reasons above.
The only downside was that
not everyone felt fully comfortable
maintaining a shell script.
To mitigate that
we commented the script heavily.
All of the code examples that follow are taken from the script we wrote for FxA. If you want to read the whole thing in its entirety before diving any deeper, you can find it here.
The first thing your script should do is check pre-requisite conditions and abort if any of them aren't met. Aborting early means your script is less likely to fail part-way through, leaving someone's local tree in a bad state.
For FxA, these pre-requisites include:
Parsing a command-line argument that indicates whether the release is a major version or a patch-level bump.
Checking the local tree contains no uncommitted changes.
Checking some commits have been made since the last tag.
For the command-line argument,
we use the value patch
to indicate a patch-level bump
and the absence of any argument
indicates a major release,
or "train" in FxA-speak.
The code for checking it
looks like this:
case "$1" in
"")
BUILD_TYPE="Train"
;;
"patch")
BUILD_TYPE="Patch"
;;
*)
echo "Release aborted: Invalid argument \"$1\""
exit 1
;;
esac
Keep an eye out for that BUILD_TYPE variable
as it will make a number of appearances
in later steps.
To check there are no uncommitted changes locally,
we use git status.
Passing it the --porcelain argument
makes it return parseable output
and if that result is not the empty string,
we error out:
STATUS=`git status --porcelain`
if [ "$STATUS" != "" ]; then
echo "Release aborted: You have uncommited changes"
exit 1
fi
There's a small refactoring
we can make here,
extracting a function to abort the script
instead of repeating the pattern
of echo then exit
every time we want to fail:
abort() {
echo "Release aborted: $1."
exit 1
}
So the previous status check now looks like this:
STATUS=`git status --porcelain`
if [ "$STATUS" != "" ]; then
abort "You have uncommited changes"
fi
To check that some commits have been made since the last tag, there are a couple of different options.
Firstly,
you could just get
the most recent tag
that is reachable
from the current HEAD:
LAST_TAG=`git describe --tags --first-parent --abbrev=0`
The --first-parent argument
prevents tags from merged branches
being selected,
which you may or may not want
depending on how branching is done
in your repository.
The alternative approach is to sort all tags alphanumerically and then pick the last one. This works if your tags aren't necessarily arranged chronologically in your history:
LAST_TAG=`git tag -l --sort=version:refname | tail -1`
For FxA, we actually use one or the other, depending on what type of release it is:
if [ "$BUILD_TYPE" = "Train" ]; then
LAST_TAG=`git tag -l --sort=version:refname | tail -1`
else
LAST_TAG=`git describe --tags --first-parent --abbrev=0`
fi
This allows us to tag patches for older releases, without more recent tags causing a problem.
Regardless of how you identify the last tag,
you can check for intervening commits
with git log:
COMMITS=`git log $LAST_TAG..HEAD`
if [ "$COMMITS" = "" ]; then
abort "I see no work"
fi
It's easy to forget
to pull from remote branches
before cutting a release,
so it makes sense
to have the script
do that job for you too.
But some care has to be taken when doing so,
because the local repository
might be on a different branch
when the script is executed.
And at least in FxA's case,
the remote branch may or may not exist yet,
depending on what type of release it is.
If we're creating a new release branch from scratch,
we want to pull from master,
but if we're bumping an existing release,
we want to pull from the release branch.
And before we can do any of that,
we need to work out what the
name of the release branch actually is.
In FxA,
release branches are named like train-$TRAIN,
where $TRAIN is the train number
from the version string.
The version string is encoded in the tags,
which are of the form v1.$TRAIN.$PATCH,
e.g. at the time of writing
the current tag is v1.138.4.
The tags in our private fork
also have a -private suffix,
e.g. v1.138.4-private.
So we can get the name of the release branch
from the LAST_TAG variable
that we set earlier,
by splitting it into its constituent parts
using cut:
MAJOR=`echo "$LAST_TAG" | cut -d '.' -f 1 | cut -d 'v' -f 2`
TRAIN=`echo "$LAST_TAG" | cut -d '.' -f 2`
PATCH=`echo "$LAST_TAG" | cut -d '.' -f 3 | cut -d '-' -f 1`
Here we've set
MAJOR to the substring
between the v and the first period,
TRAIN to the substring
between the first and second periods,
and PATCH to the substring
after the second period
(and before the hyphen,
if one exists).
Now that we've broken the version string into its constituent parts, we can determine the name of the release branch, depending on what type of build it is:
case "$BUILD_TYPE" in
"Train")
NEW_TRAIN=`expr $TRAIN + 1`
;;
"Patch")
NEW_TRAIN="$TRAIN"
;;
esac
RELEASE_BRANCH="train-$NEW_TRAIN"
So for major releases,
we increment the train number
using expr
and for patches,
we just re-use the current train number.
Now we know the branch name,
we can check which branch we're on locally.
If we're already on the release branch,
we just need to pull from origin
to get the latest changes.
If we're not,
we can look for
a remote release branch instead.
If one exists use that,
otherwise create a new branch
from master:
CURRENT_BRANCH=`git branch --no-color | grep '^\*' | cut -d ' ' -f 2`
if [ "$CURRENT_BRANCH" = "$RELEASE_BRANCH" ]; then
git pull origin "$RELEASE_BRANCH" > /dev/null 2>&1 || true
else
RELEASE_BRANCH_EXISTS=`git branch --no-color | awk '{$1=$1};1' | grep "^$RELEASE_BRANCH\$"` || true
if [ "$RELEASE_BRANCH_EXISTS" = "" ]; then
git fetch origin $RELEASE_BRANCH > /dev/null 2>&1 || true
REMOTE_BRANCH="origin/$RELEASE_BRANCH"
REMOTE_BRANCH_EXISTS=`git branch --no-color -r | awk '{$1=$1};1' | grep "^$REMOTE_BRANCH\$"` || true
if [ "$REMOTE_BRANCH_EXISTS" = "" ]; then
echo "Warning: $RELEASE_BRANCH branch not found on local or remote, creating one from master."
git checkout master > /dev/null 2>&1
git pull origin master > /dev/null 2>&1
git checkout -b "$RELEASE_BRANCH" > /dev/null 2>&1
else
git checkout --track -b "$RELEASE_BRANCH" "$REMOTE_BRANCH" > /dev/null 2>&1
fi
else
git checkout "$RELEASE_BRANCH" > /dev/null 2>&1
git pull origin "$RELEASE_BRANCH" > /dev/null 2>&1 || true
fi
fi
There's a few things worth calling out here:
Some of these commands can legitimately fail
and we don't want them to abort the script
if we're running it with set -e.
Appending || true takes care of this.
They're quite noisy on the console
and we don't want the output from our script
getting too cluttered.
To keep them quiet,
we redirect stdout to the null device
using > /dev/null.
In the cases where commands can fail,
we also redirect stderr
using 2>&1.
When running git branch
we specify the --no-color option
to prevent any control codes
from leaking into the output.
Otherwise they'd break our assumptions with grep,
where we use ^ and $ to specify
strict start/end-of-string matches.
awk '{$1=$1};1' is a pattern
you'll see repeated a lot,
used to trim any space
from the start or end of a string.
The single quotes are important
because they prevent shell-expansion of $1,
passing it through to awk unadulterated.
There is a problem introduced by this code
in that it changes the local branch,
so if the script aborts
the user will find themselves
in a different state
to when they ran the script.
Clearly that's unacceptable,
so we can fix it by moving
the assignment to CURRENT_BRANCH
to the very beginning of the script,
then changing our earlier definition
of the abort function
like so:
abort() {
git checkout "$CURRENT_BRANCH" > /dev/null 2>&1
echo "Release aborted: $1."
exit 1
}
Now we're on the right branch and have pulled latest changes, we can get on with updating the version strings. We already broke the current version string down into its constituent parts and bumped the train number, so it's a minor tweak to revisit that code and bump the patch level too:
case "$BUILD_TYPE" in
"Train")
NEW_TRAIN=`expr $TRAIN + 1`
NEW_PATCH=0
;;
"Patch")
NEW_TRAIN="$TRAIN"
NEW_PATCH=`expr $PATCH + 1`
;;
esac
Then we can recombine the parts and generate our new version string:
NEW_VERSION="$MAJOR.$NEW_TRAIN.$NEW_PATCH"
We're going to use sed
to update the version in a bunch of places,
so we need to turn the old version string
into a regular expression:
LAST_VERSION_REGEX="$MAJOR\\.$TRAIN\\.$PATCH"
We have to deal with two levels of character escaping here, because we want to escape the period in the regex using a backslash but to do that in a shell script, we must first escape the backslash itself using another backslash.
Updating the version strings then happens in two parts.
First we have to loop through
each of the directories in our repo.
For this we're going to assume
the existence of a function called update
that we'll define in a moment:
DIRECTORIES="packages/fxa-auth-db-mysql
packages/fxa-auth-server
packages/fxa-content-server
packages/fxa-customs-server
packages/fxa-email-event-proxy
packages/fxa-email-service
packages/fxa-event-broker
packages/fxa-profile-server"
while read -r DIRECTORY; do
update "$DIRECTORY"
done <<< "$DIRECTORIES"
Then for each directory
we're going to look for files
where version strings might be stored
and use sed to update them.
We do that
in the implementation of update
like this:
update() {
if [ -f "$1/package.json" ]; then
sed -i.release -e "s/$LAST_VERSION_REGEX/$NEW_VERSION/g" "$1/package.json"
rm "$1/package.json.release"
fi
if [ -f "$1/Cargo.toml" ]; then
sed -i.release -e "s/$LAST_VERSION_REGEX/$NEW_VERSION/g" "$1/Cargo.toml"
rm "$1/Cargo.toml.release"
fi
}
Note the -i option to sed
is optional in some environments
but we choose to include it
for maximum portability.
The nice thing with this pattern
is its ambivalence about the target language.
If the repo grows
to include projects
from other languages
like Ruby or Python,
it's simple to add
more invocations of sed
as needed.
For the change logs,
we'll extend our implementation
of the update function
to loosely parse
the first line of each commit message
for the target directory,
then write a sorted list of commits
to each package's CHANGELOG.md.
FxA follows the Angular.js conventions for formatting commit messages. This means the first line of the commit should be of the form:
type(scope): summary message
type is limited
to a strict set of values,
the most interesting of which
are feat, fix, perf, refactor and revert.
In our change logs,
we want to group commit messages
under each of those headings
to make it easier for readers
to find particular changes
they might be looking for.
We'll also include the commit hash with the summary message, to help anyone who wants to link a change from the log to the specific point in git's history.
To do this,
we'll add the following code
to the body of update:
LOCAL_COMMITS=`git log $LAST_TAG..HEAD --no-color --pretty=oneline --abbrev-commit -- "$1"`
while read -r COMMIT; do
HASH=`echo "$COMMIT" | cut -d ' ' -f 1`
MESSAGE=`echo "$COMMIT" | cut -d ':' -f 2- | awk '{$1=$1};1'`
TYPE=`echo "$COMMIT" | cut -d ' ' -f 2 | awk '{$1=$1};1' | cut -d ':' -f 1 | cut -d '(' -f 1 | awk '{$1=$1};1'`
SCOPE=`echo "$COMMIT" | cut -d '(' -f 2 | cut -d ')' -f 1 | awk '{$1=$1};1'`
if [ "$SCOPE" = "$COMMIT" ]; then
SCOPE=""
fi
if [ "$SCOPE" != "" ]; then
SCOPE="$SCOPE: "
fi
case "$TYPE" in
"")
# Ignore blank lines
;;
"Merge")
# Ignore merge commits
;;
"Release")
# Ignore release commits
;;
"feat")
if [ "$FEAT_SUMMARY" = "" ]; then
FEAT_SUMMARY="### New features\n"
fi
FEAT_SUMMARY="$FEAT_SUMMARY\n* $SCOPE$MESSAGE ($HASH)"
;;
"fix")
if [ "$FIX_SUMMARY" = "" ]; then
FIX_SUMMARY="### Bug fixes\n"
fi
FIX_SUMMARY="$FIX_SUMMARY\n* $SCOPE$MESSAGE ($HASH)"
;;
"perf")
if [ "$PERF_SUMMARY" = "" ]; then
PERF_SUMMARY="### Performance improvements\n"
fi
PERF_SUMMARY="$PERF_SUMMARY\n* $SCOPE$MESSAGE ($HASH)"
;;
"refactor")
if [ "$REFACTOR_SUMMARY" = "" ]; then
REFACTOR_SUMMARY="### Refactorings\n"
fi
REFACTOR_SUMMARY="$REFACTOR_SUMMARY\n* $SCOPE$MESSAGE ($HASH)"
;;
"revert")
if [ "$REFACTOR_SUMMARY" = "" ]; then
REVERT_SUMMARY="### Reverted changes\n"
fi
REVERT_SUMMARY="$REVERT_SUMMARY\n* $SCOPE$MESSAGE ($HASH)"
;;
*)
if [ "$OTHER_SUMMARY" = "" ]; then
OTHER_SUMMARY="### Other changes\n"
fi
OTHER_SUMMARY="$OTHER_SUMMARY\n* $SCOPE$MESSAGE ($HASH)"
;;
esac
if [ "$FEAT_SUMMARY" != "" ]; then
FEAT_SUMMARY="$FEAT_SUMMARY\n\n"
fi
if [ "$FIX_SUMMARY" != "" ]; then
FIX_SUMMARY="$FIX_SUMMARY\n\n"
fi
if [ "$PERF_SUMMARY" != "" ]; then
PERF_SUMMARY="$PERF_SUMMARY\n\n"
fi
if [ "$REFACTOR_SUMMARY" != "" ]; then
REFACTOR_SUMMARY="$REFACTOR_SUMMARY\n\n"
fi
if [ "$REVERT_SUMMARY" != "" ]; then
REVERT_SUMMARY="$REVERT_SUMMARY\n\n"
fi
if [ "$OTHER_SUMMARY" != "" ]; then
OTHER_SUMMARY="$OTHER_SUMMARY\n\n"
fi
SUMMARY="$FEAT_SUMMARY$FIX_SUMMARY$PERF_SUMMARY$REFACTOR_SUMMARY$OTHER_SUMMARY"
if [ "$SUMMARY" = "" ]; then
SUMMARY="No changes.\n\n"
fi
awk "{ gsub(/^## $LAST_VERSION/, \"## $NEW_VERSION\n\n$SUMMARY## $LAST_VERSION\") }; { print }" "CHANGELOG.md" > "CHANGELOG.md.release"
mv "CHANGELOG.md.release" "CHANGELOG.md"
done <<< "$LOCAL_COMMITS"
We use cut a number of times
to pull out the different components
of the commit message
and awk is used like before
to trim spaces from the resulting strings.
We then collect
the formatted messages
into different variables
and write them to the correct point
in the change log
using awk.
This code also uses a variable LAST_VERSION
that we haven't defined yet,
so we should scoot back
to where we defined LAST_VERSION_REGEX
and add this line
alongside it:
LAST_VERSION="$MAJOR.$TRAIN.$PATCH"
At this point, the local tree will contain uncommitted changes so we need to commit them:
git commit -a -m "Release $NEW_VERSION"
Then we can create the tag:
NEW_TAG="v$NEW_VERSION"
git tag -a "$NEW_TAG" -m "$BUILD_TYPE release $NEW_VERSION"
Lastly, we mustn't forget to return the user to their original branch:
git checkout "$CURRENT_BRANCH" > /dev/null 2>&1
In many of the commands above,
we sent the output to /dev/null
and the user doesn't have any idea
what has happened.
So we should tell them
that the script finished successfully
and how they can check
what's changed:
echo
echo "Success! The release has been tagged locally but it hasn't been pushed."
echo "Before pushing, you should check that the changes appear to be sane."
echo
echo "Branch:"
echo
echo " $RELEASE_BRANCH"
echo
echo "Tag:"
echo
echo " $NEW_TAG"
We can also tell them what their next steps are:
echo
echo "When you're ready to push, paste the following lines into your terminal:"
echo
echo "git push origin $RELEASE_BRANCH"
echo "git push origin $NEW_TAG"
echo
echo "After that, you must open a pull request to merge the changes back to master:"
echo
echo " https://github.com/mozilla/fxa/compare/$RELEASE_BRANCH?expand=1"
This is the full output we see when we run the script we built for FxA:
~/c/fxa (train-138) $ ./release.sh patch
[train-138 1d8914565] Release 1.138.5
23 files changed, 44 insertions(+), 16 deletions(-)
Success! The release has been tagged locally but it hasn't been pushed.
Before pushing, you should check that the changes appear to be sane.
At the very least, eyeball the diffs and git log.
If you're feeling particularly vigilant, you may want to run some of the tests and linters too.
Branches:
train-138
train-138-private
Tags:
v1.138.5
v1.138.5-private
When you're ready to push, paste the following lines into your terminal:
git push origin train-138
git push origin v1.138.5
git push private train-138-private
git push private v1.138.5-private
After that, you must open pull requests in both the public and private repos to merge the changes back to master:
https://github.com/mozilla/fxa/compare/train-138?expand=1
https://github.com/mozilla/fxa-private/compare/train-138-private?expand=1
Don't forget to leave a comment in the deploy bug.
Include links to the tags:
### Tags
* https://github.com/mozilla/fxa/releases/tag/v1.138.5
* https://github.com/mozilla/fxa-private/releases/tag/v1.138.5-private
You can see the full release script for FxA here. We fixed a number of issues to reach that point, so it might be instructive to link to some examples of things we got wrong along on the way: