Time passes and it's easy to forget about all the cool stuff you did or lessons you learned. But source control makes it easy to go back and remind yourself, so I thought I'd take a moment to look back and summarise my past year in code.
January kicked things off with some improvements to our Redis abstraction and a number of small fixes to the Firefox Accounts metrics infrastructure.
In mozilla/fxa-auth-server#2254, I changed the serialisation format of the session token data that we store in Redis, reducing the average payload size by about 16%. The approach I took was to use arrays with fixed indices so that we could stop paying for property keys. I also investigated storing gzipped data instead, but felt that the loss of eyeballability of the data at rest wasn't worth the ~25% reduction in size it would have garnered.
That work was then continued in mozilla/fxa-auth-server#2257, where I added code to prune expired session tokens from Redis. Together, these two changes solved a real operational problem we were suffering from, where we had to keep scaling up the size of our Redis instance and our costs were going up commensurately.
In mozilla/fxa-amplitude-send#38, I wrote a script to import FxA marketing events to Amplitude, so they could be analysed alongside the rest of our product metrics. It entailed some fairly hairy regex work and, as ever, we made sure not to send personally-identifiable information to 3rd-party services. That extends as far as hashing our own uid prior to sending, to make it impossible for anyone to cross-reference metrics data with information from our production database. We do this even though production db access is locked down to just a few members of our Ops team.
In all, there were 16 notable changesets in January:
In February I went on holiday to Barbados!
But before that, I also managed to get some coding done. Again it was mostly metrics-related, althought none of it hugely interesting.
I wrote a script to import
complete event to Amplitude
from Firefox telemetry data.
But, as it stands,
we still haven't enabled this in production
because it generates a lot of events
and needs more metadata
to be really useful.
And in mozilla/fxa-auth-server#2282, I eliminated some duplication that achieved a net deletion of 620 lines of code. I love deleting code even more than I love writing it!
February saw 20 changesets in total:
March saw fixes for a couple of interesting security vulnerabilities.
In mozilla/fxa-auth-server#8da511c8 and mozilla/fxa-content-server#e73873cd, I added safeguards to prevent unescaped input reaching our back-end via the user agent string. The user agent parser we use includes some regexes that propagate parts of the input string to the output. The new safeguards ensure that such input gets dropped at point of entry. Unfortunately the contextual discussion for this vulnerability is not public, but people with appropriate permissions can read more in bug 1445629.
I wrote a
to ensure our communication
with other back-end services
did not leave us open
to a node.js request-splitting exploit.
Also related to this
were some small validation fixes
The discussion around these changes
can be found in bug 1447452.
Elsewhere, in philbooth/bfj#e2e320db, I added a nice feature that allows JSON streams to be asynchronously parsed for interesting subtrees, without needing to load the entire tree into memory. I blogged more about this feature here.
In April, I started work on mozilla/fxa-email-service, which would prove to be my main focus for most of the year. It was a rare opportunity for some greenfield development on Firefox Accounts and a chance for me to drive a project from start to finish.
The rationale behind the work was to decouple our authentication server from Amazon SES. That would open up the possibility of richer bounce-handling behaviour and let us fall back to alternative email providers for problematic domains or addresses.
The most enjoyable aspect for me was that it was to be written in Rust. Learning Rust was the best part of my job in 2018 and a direction I'm keen to pursue further in the future.
Apart from that, there was one interesting feature in mozilla/fxa-auth-server#2401, where I added logic to check the available budget for Amazon SNS before deciding whether to display our "Connect Another Device" form after a user signs in to FxA. We'd found ourselves running out of SMS budget in production a few times, which led to a poor user experience when the Firefox install link that we send out via text message failed to send. Detecting that state ahead of time allowed us to provide a more coherent user experience.
In May, I moved all of my personal repositories to GitLab. As a proponent of open-source software, I like the idea of hosting my own open-source code on a service that is itself open-source and I'd wanted to move for a while, but the inertia of already having everything on GitHub was difficult to get past. Rumours of a Microsoft buyout were a sufficient nudge though and, as it turned out, migrating was really easy. GitLab pulled everything across, including issues and so on, at the touch of a button. Ultimately I plan to move to a self-hosted GitLab instance running under this domain, but that's a topic for another blog post.
On the work front, May was all about the email service. Looking back at some of those changes now, my inexperience with Rust at the time is clear to see. But the broad direction stands up well, such as mozilla/fxa-email-service#34 where I leaned on the trait system to implement a Sendgrid provider that had interface-compatibility with our SES provider.
These were the changesets from May:
June was a difficult month, as my Dad's health deteriorated rapidly and he finally lost his long, grim battle against a cancer that started in his pancreas, then spread to his liver, stomach and lungs. It seems strange to write about that in a blog post about programming, but it felt strange at the time to just continue on with work and side-projects, while more important stuff was happening in real life.
That dissonance is reflected in the changesets for June, which were mostly trivial and relatively few in number:
In July, work continued on the email service and started on a Rust port of the Firefox Sync storage server.
An interesting change related to the former was the implementation of dynamic email config in mozilla/fxa-auth-server#2535. This was a big deal because previously all our configuration data was static and required a deployment by the Ops team for us to make changes. Co-opting Redis for this was something of an ad hoc approach, but worked well and allowed us to make rapid config changes such as diverting email for particular users via a different provider, after they reported that they weren't receiving any email from Firefox Accounts at all.
Amongst the usual array of other fixes,
I addressed a curious
that was reported by Sentry.
It indicated that somehow,
one of the session token objects
cached in Redis
was invalid JSON when we came to unpack it.
After failing to identify any possible ways
for such data to arrive in the data store,
and bearing in mind there had been a single case
of this error
across hundreds of millions of session tokens,
I chalked it up to "operational weirdness"
then added some logic to purge the offending data
if it happened again.
The error never recurred
and it remains an unsolved mystery.
There were 27 changesets in July:
I wrote a script
of our MySQL stored procedures
This was an idea I had
during the postmortem meeting
for a Firefox Accounts database outage
that we suffered in production.
When I suggested it during the meeting
I got some doubtful responses,
but I decided to try it out anyway
and it turned out pretty well.
You can read more about it
in my blog post
on the subject.
I also started working on Hawk authentication for the Rust syncstorage port in mozilla-services/syncstorage-rs#20. It was the first time I'd really looked at the Hawk protocol in depth, so there were a lot of kinks to iron out during code review and the PR didn't actually land until September.
We had something of an indian summer in September and it was proper beach weather where I live on the south coast of England:
For Firefox Accounts, September saw deployment of the email service to production. The deployment went smoothly and over following weeks we were able to use the work I'd done on dynamic email configuration to stage a gradual rollout to increasing percentages of our userbase.
In October, I dedicated a week of leave to working on side-projects that I'd been neglecting for some time. Mostly I worked on pbvi, a text editor that I decided to build when I started learning Rust. I'm not sure if I'll ever finish it, but it's been a fun project and a really useful playground to help me learn the language. I also published my first Rust crate, unicode-bom, which is just a tiny little dependency that detects the unicode byte-order mark in a file or byte array. And I spent a little bit of time on fxabot, a modified Hubot instance that can broadcast deployment info to our IRC and Slack channels.
Back at work,
with the email service deployed,
it was an opportunity to pick up
a number of lower-priority house-keeping tasks
that had been annoying me for a while.
I refactored from raw strings
to a strong
I automated the (de)serialisation of data
at the boundary of our Redis abstraction
and in mozilla/fxa-email-service#213
I tidied up the directory structure a bit.
It was also a chance
to lay down some groundwork
for opening up the email service
to other teams,
by starting the process of
migrating away from
the auth server's database
In November, traffic to the email service reached 100%, marking the end of our phased rollout to production. It was another month of refactorings and small-ish fixes.
I wrote an automated release-tagging script
for the email service.
It borrows from some prior art
in my personal release-tagging script,
but is tailored towards
the particular conventions
followed by the FxA repos.
Because we use structured commit messages,
it's easy to employ a combination of
to generate a readable change log
for each release.
The version number
can be bumped
then it's just a case
of committing the changes
and creating a tag.
In mozilla/fxa-email-service#236, I paid some much-needed attention to error-handling in the email service. It had grown organically into a somewhat tangled mess and was a source of irritation, so I was greatly pleased to clean and simplify it.
I refactored some settings
from strings to enums.
This work included
a nice little
to reduce duplication in the resulting code.
I absolutely love the macro system in Rust,
something that you can read more about
December was pretty quiet, because of the Mozilla All Hands and then the holidays.
On a personal note, I was really happy to study and pass my WSET level 2.
Looking back, I think it was a productive year, all things considered:
I learned Rust.
I took a new microservice for the FxA stack all the way from planning to deployment in production.
I made proactive and meaningful contributions with tangible outcomes, in response to operational issues that we hit in production.
I kept the FxA metrics infrastructure up and running, fixing issues as they showed up in the data and deploying code for new metrics as the needs arose.
I fixed a whole bunch of different issues across a whole bunch of different repositories, using a whole bunch of different languages.
Looking ahead and keeping myself honest for 2019, I'd like to continue working with Rust and go back to using a LISP again for some of my side-projects.
Away from programming, I want to pass my driving test, learn French and pass my WSET level 3.