Editors Note: This one’s just a quick – and much delayed – little bit of story-time, and a word of warning to remember to pin your dependency versions… and to tag your docker releases.
Banned Word Tracker went down last week – in a way I couldn’t recover, and just had to fix – when publishing an extremely small change. What the hell happened, and how can I stop it from happening again?
Just how small was this change?
The “offending PR” was this one: “Alert Text Cleanup.” A pretty inconspicuous name, but what did it do? It sought to condense the alert messaging, since with more potential bans there’s a larger chance of triggering multiple at once. Compare below:
CI says we’re good to go… almost
Because I’m incredibly risk-averse, I run TravisCI on all my PRs, as well as my master branch, to make sure I don’t merge in anything broken. Unsurprisingly, all the tests were green on this PR. After all, it was just a copy change.
Except for the second part of CI: the DockerHub build. It seemed to be failing to install “ast3” when building the image.
ast3/Python/ast.c: In function ‘decode_bytes_with_escapes’:
ast3/Python/ast.c:4534:24: error: too many arguments to function ‘_PyBytes_DecodeEscape’
PyObject *result = _PyBytes_DecodeEscape(s, len, NULL, 0, NULL,I wasn’t sure if I was even using this library, so I deleted it from the requirements and went on my way. All the lights turned green, no tests failed and the build worked, so I went ahead and merged it in. Time to deploy and kick back while people enjoy the more trim messaging, right?
Deployment’s failing now?
Surprising only myself, that requirement was, in fact, necessary for running the actual discord bot. Whether it was required by discord.py, or the startup script, I’m not entirely sure. Either way though, it certainly didn’t run now.
The obvious fix would’ve been to revert to the last version of the docker image. Of course, I haven’t been tagging any of my releases, I’ve just been overriding the “latest” tag with each build. With that regrettable failure behind me, my next thought was to make a revert PR, and go from there.
Of course, that revert ended up having the same docker-build failure as the original PR. What gives? Without a quick fix available, it was time to solve things under the gun.
Whoops, I was using the wrong image
A quick bit of research later, I learned that a new version of Python had been released not 5 days prior to this PR. My dockerfile was based off of the Python3 image, instead of an actual pinned version. My solution ended up being to update my Dockerfile to base the images off of python:3.8.6 instead python:3.
With that quick change, the bot was able to build, deploy, and eventually run, without any problems. So let that be a reminder to my future self to pin every dependency (not just the library versions), and to figure out how to tag releases built by dockerhub.