About a month ago,
we had an outage
caused by a slow-running query in MySQL.
This particular slow query
wasn't spotted when it was deployed
because it depended on data
inserted by client browsers
and the related preference in Firefox
was not enabled at that point.
A few weeks after it shipped,
the client pref was flipped on
and as the table grew,
it slowed down MySQL increasingly
until the whole of Firefox Accounts
And of course,
because Sod's Law
is one of the fundamental forces of nature,
this happened late on a Friday night.
There were some complicating factors
that slowed down diagnosis,
but it's also fair to say
we could have caught it at source
with an EXPLAIN of the offending query
during code review.
Because of that,
I decided to try and automate
EXPLAIN checks for our MySQL queries.