Phil Booth

Existing by coincidence, programming deliberately

2018 retrospective

Time passes and it’s easy to forget about all the cool stuff you did or lessons you learned. But source control makes it easy to go back and remind yourself, so I thought I’d take a moment to look back and summarise my past year in code.

January

January kicked things off with some improvements to our Redis abstraction and a number of small fixes to the Firefox Accounts metrics infrastructure.

In mozilla/fxa-auth-server#2254, I changed the serialisation format of the session token data that we store in Redis, reducing the average payload size by about 16%. The approach I took was to use arrays with fixed indices so that we could stop paying for property keys. I also investigated storing gzipped data instead, but felt that the loss of eyeballability of the data at rest wasn’t worth the ~25% reduction in size it would have garnered.

That work was then continued in mozilla/fxa-auth-server#2257, where I added code to prune expired session tokens from Redis. Together, these two changes solved a real operational problem we were suffering from, where we had to keep scaling up the size of our Redis instance and our costs were going up commensurately.

In mozilla/fxa-amplitude-send#38, I wrote a script to import FxA marketing events to Amplitude, so they could be analysed alongside the rest of our product metrics. It entailed some fairly hairy regex work and, as ever, we made sure not to send personally-identifiable information to 3rd-party services. That extends as far as hashing our own uid prior to sending, to make it impossible for anyone to cross-reference metrics data with information from our production database. We do this even though production db access is locked down to just a few members of our Ops team.

In all, there were 16 notable changesets in January:

February

In February I went on holiday to Barbados!

Sunset at Drill Hall Beach, Bridgetown, Barbados

But before that, I also managed to get some coding done. Again it was mostly metrics-related, althought none of it hugely interesting.

In mozilla/fxa-amplitude-send#47, I wrote a script to import a Sync complete event to Amplitude from Firefox telemetry data. But, as it stands, we still haven’t enabled this in production because it generates a lot of events and needs more metadata to be really useful.

And in mozilla/fxa-auth-server#2282, I eliminated some duplication that achieved a net deletion of 620 lines of code. I love deleting code even more than I love writing it!

February saw 20 changesets in total:

March

March saw fixes for a couple of interesting security vulnerabilities.

In mozilla/fxa-auth-server#8da511c8 and mozilla/fxa-content-server#e73873cd, I added safeguards to prevent unescaped input reaching our back-end via the user agent string. The user agent parser we use includes some regexes that propagate parts of the input string to the output. The new safeguards ensure that such input gets dropped at point of entry. Unfortunately the contextual discussion for this vulnerability is not public, but people with appropriate permissions can read more in bug 1445629.

In mozilla/fxa-auth-server#2368, I wrote a SafeUrl class to ensure our communication with other back-end services did not leave us open to a node.js request-splitting exploit. Also related to this were some small validation fixes in mozilla/fxa-auth-server#2359 and mozilla/fxa-content-server#5996. The discussion around these changes can be found in bug 1447452.

Elsewhere, in philbooth/bfj#e2e320db, I added a nice feature that allows JSON streams to be asynchronously parsed for interesting subtrees, without needing to load the entire tree into memory. I blogged more about this feature here.

March changesets:

April

In April, I started work on mozilla/fxa-email-service, which would prove to be my main focus for most of the year. It was a rare opportunity for some greenfield development on Firefox Accounts and a chance for me to drive a project from start to finish.

The rationale behind the work was to decouple our authentication server from Amazon SES. That would open up the possibility of richer bounce-handling behaviour and let us fall back to alternative email providers for problematic domains or addresses.

The most enjoyable aspect for me was that it was to be written in Rust. Learning Rust was the best part of my job in 2018 and a direction I’m keen to pursue further in the future.

Apart from that, there was one interesting feature in mozilla/fxa-auth-server#2401, where I added logic to check the available budget for Amazon SNS before deciding whether to display our “Connect Another Device” form after a user signs in to FxA. We’d found ourselves running out of SMS budget in production a few times, which led to a poor user experience when the Firefox install link that we send out via text message failed to send. Detecting that state ahead of time allowed us to provide a more coherent user experience.

April changesets:

May

In May, I moved all of my personal repositories to GitLab. As a proponent of open-source software, I like the idea of hosting my own open-source code on a service that is itself open-source and I’d wanted to move for a while, but the inertia of already having everything on GitHub was difficult to get past. Rumours of a Microsoft buyout were a sufficient nudge though and, as it turned out, migrating was really easy. GitLab pulled everything across, including issues and so on, at the touch of a button. Ultimately I plan to move to a self-hosted GitLab instance running under this domain, but that’s a topic for another blog post.

On the work front, May was all about the email service. Looking back at some of those changes now, my inexperience with Rust at the time is clear to see. But the broad direction stands up well, such as mozilla/fxa-email-service#34 where I leaned on the trait system to implement a Sendgrid provider that had interface-compatibility with our SES provider.

These were the changesets from May:

June

June was a difficult month, as my Dad’s health deteriorated rapidly and he finally lost his long, grim battle against a cancer that started in his pancreas, then spread to his liver, stomach and lungs. It seems strange to write about that in a blog post about programming, but it felt strange at the time to just continue on with work and side-projects, while more important stuff was happening in real life.

That dissonance is reflected in the changesets for June, which were mostly trivial and relatively few in number:

July

In July, work continued on the email service and started on a Rust port of the Firefox Sync storage server.

An interesting change related to the former was the implementation of dynamic email config in mozilla/fxa-auth-server#2535. This was a big deal because previously all our configuration data was static and required a deployment by the Ops team for us to make changes. Co-opting Redis for this was something of an ad hoc approach, but worked well and allowed us to make rapid config changes such as diverting email for particular users via a different provider, after they reported that they weren’t receiving any email from Firefox Accounts at all.

Amongst the usual array of other fixes, in mozilla/fxa-auth-server#2550 I addressed a curious 500 error that was reported by Sentry. It indicated that somehow, one of the session token objects cached in Redis was invalid JSON when we came to unpack it. After failing to identify any possible ways for such data to arrive in the data store, and bearing in mind there had been a single case of this error across hundreds of millions of session tokens, I chalked it up to “operational weirdness” then added some logic to purge the offending data if it happened again. The error never recurred and it remains an unsolved mystery.

There were 27 changesets in July:

August

During August, I finished off the work for dynamic email configuration in mozilla/fxa-auth-server#2571, mozilla/fxa-auth-server#2574 and mozilla/fxa-auth-server#2576.

In mozilla/fxa-auth-db-mysql#392, I wrote a script to run EXPLAIN checks of our MySQL stored procedures in CI. This was an idea I had during the postmortem meeting for a Firefox Accounts database outage that we suffered in production. When I suggested it during the meeting I got some doubtful responses, but I decided to try it out anyway and it turned out pretty well. You can read more about it in my blog post on the subject.

I also started working on Hawk authentication for the Rust syncstorage port in mozilla-services/syncstorage-rs#20. It was the first time I’d really looked at the Hawk protocol in depth, so there were a lot of kinks to iron out during code review and the PR didn’t actually land until September.

August changesets:

September

We had something of an indian summer in September and it was proper beach weather where I live on the south coast of England:

Bournemouth Beach in September

For Firefox Accounts, September saw deployment of the email service to production. The deployment went smoothly and over following weeks we were able to use the work I’d done on dynamic email configuration to stage a gradual rollout to increasing percentages of our userbase.

September changesets:

October

In October, I dedicated a week of leave to working on side-projects that I’d been neglecting for some time. Mostly I worked on pbvi, a text editor that I decided to build when I started learning Rust. I’m not sure if I’ll ever finish it, but it’s been a fun project and a really useful playground to help me learn the language. I also published my first Rust crate, unicode-bom, which is just a tiny little dependency that detects the unicode byte-order mark in a file or byte array. And I spent a little bit of time on fxabot, a modified Hubot instance that can broadcast deployment info to our IRC and Slack channels.

Back at work, with the email service deployed, it was an opportunity to pick up a number of lower-priority house-keeping tasks that had been annoying me for a while. For instance, in mozilla/fxa-email-service#196 I refactored from raw strings to a strong EmailAddress type, in mozilla/fxa-email-service#200 I automated the (de)serialisation of data at the boundary of our Redis abstraction and in mozilla/fxa-email-service#213 I tidied up the directory structure a bit. It was also a chance to lay down some groundwork for opening up the email service to other teams, by starting the process of migrating away from the auth server’s database in mozilla/fxa-email-service#203.

October changesets:

November

In November, traffic to the email service reached 100%, marking the end of our phased rollout to production. It was another month of refactorings and small-ish fixes.

In mozilla/fxa-email-service#246, I wrote an automated release-tagging script for the email service. It borrows from some prior art in my personal release-tagging script, philbooth/please-release-me, but is tailored towards the particular conventions followed by the FxA repos. Because we use structured commit messages, it’s easy to employ a combination of git log, cut and awk to generate a readable change log for each release. The version number can be bumped in Cargo.lock and Cargo.toml using sed, then it’s just a case of committing the changes and creating a tag.

In mozilla/fxa-email-service#236, I paid some much-needed attention to error-handling in the email service. It had grown organically into a somewhat tangled mess and was a source of irritation, so I was greatly pleased to clean and simplify it.

In mozilla/fxa-email-service#249, mozilla/fxa-email-service#255 and mozilla/fxa-email-service#256, I refactored some settings from strings to enums. This work included a nice little enum_boilerplate! macro to reduce duplication in the resulting code. I absolutely love the macro system in Rust, something that you can read more about here.

November changesets:

December

December was pretty quiet, because of the Mozilla All Hands and then the holidays.

In mozilla/fxa-email-service#261, I implemented a serializable regex type. And in actix/actix-web#637, I contributed a fix to support custom values for the Content-Type header.

On a personal note, I was really happy to study and pass my WSET level 2.

December changesets:

Conclusion

Looking back, I think it was a productive year, all things considered:

Looking ahead and keeping myself honest for 2019, I’d like to continue working with Rust and go back to using a LISP again for some of my side-projects.

Away from programming, I want to pass my driving test, learn French and pass my WSET level 3.