Skip to content

Zero-Downtime Deployments for Node Apps With Docker Compose · Part 9 of 11 in the series

Part 9: When a Deploy Goes Wrong

Watch a broken version fail without a single visitor noticing, teach the script to remember an image that failed, and build a way back: one click names an older image production again.

Quest objectives

  • Tell a version that cannot start properly from one that runs but is wrong
  • Keep the script from trying the same broken image every minute
  • Break the health check on purpose and watch the old version keep running
  • Roll back by moving the production tag to an older image
  • Know what a rollback does not bring back

Sooner or later you will push a version that should not have gone live. There are two kinds of them. The first kind cannot even start properly: it crashes, or it runs but cannot reach its database. The health check catches those. The second kind starts fine, reports that it is healthy, and then does the wrong thing: a broken form, a wrong text, a page that looks odd. No check catches those. People do.

This part deals with both.

The broken container is lifted away again. The old one never stopped working, and its visitors noticed nothing.
The broken container is lifted away again. The old one never stopped working, and its visitors noticed nothing.

A version that never gets healthy

For the first kind, most of the work is done. In step 4 the script waits for the new container to report healthy, and if it does not, the script removes it and keeps the old one.

One problem is left, and the timer from the last part created it. A minute later the script runs again, finds that production is still not the image that is running, and tries the same broken image again: start, wait a minute, remove. And again, every minute, until someone pushes a fix.

So the script should remember an image that failed. That takes four small changes to deploy/deploy.sh. First, add this after the check that ends with Nothing new:

deploy/deploy.sh
# An image that failed before is not tried again every minute.
if [ "$wanted" = "$(cat .failed 2> /dev/null || true)" ]; then
  say "Still the image that failed, waiting for a new one"
  exit 0
fi

fail() {
  say "$1"
  echo "$wanted" > .failed
  exit 1
}

A file called .failed will hold the ID of the image that did not work. If the image the registry offers is that one, the script says so and ends. The part behind cat keeps things quiet on the days when there is no such file. fail is a helper for everything that can go wrong from here on: it says what happened, notes the image's ID in the file, and ends the script with an error.

Second, step 2 gets a new ending:

deploy/deploy.sh
# 2. Bring the database up to date, with the new image's migrations.
say "New image found, updating the database"
docker compose run --rm -T web node src/migrate.js || fail "The database update failed, the old version keeps running"

The two bars mean "if the command before failed": a migration that does not work now goes through fail as well. Third, replace the block after the waiting loop in step 4:

deploy/deploy.sh
if [ "$status" != healthy ]; then
  docker logs --tail 20 "$new"
  docker rm --force "$new" > /dev/null
  fail "The new container did not get healthy, the old version keeps running"
fi

And fourth, near the end, add a line after the one that clears old images:

deploy/deploy.sh
docker image prune --force > /dev/null
rm -f .failed

A deploy that worked forgets the failure before it. This is the whole script now:

deploy/deploy.sh
#!/bin/sh
# Brings the newest image online without a gap: the new container starts beside the old
# one, has to prove it is healthy, gets the visitors, and only then the old one leaves.
set -eu
cd "$(dirname "$0")"
# Compose reports every step it takes. The log should hold only what matters.
export COMPOSE_PROGRESS=quiet

say() { echo "$(date +%H:%M:%S) $*"; }

# 1. Fetch the image the registry calls "production" and compare it with what is running.
docker compose pull --quiet web
image=$(grep "^IMAGE=" .env | cut -d = -f 2-):production
wanted=$(docker image inspect --format '{{.Id}}' "$image")
old=$(docker compose ps --quiet web)
running=""
if [ -n "$old" ]; then running=$(docker inspect --format '{{.Image}}' "$old"); fi

if [ "$wanted" = "$running" ]; then
  say "Nothing new"
  exit 0
fi

# An image that failed before is not tried again every minute.
if [ "$wanted" = "$(cat .failed 2> /dev/null || true)" ]; then
  say "Still the image that failed, waiting for a new one"
  exit 0
fi

fail() {
  say "$1"
  echo "$wanted" > .failed
  exit 1
}

# 2. Bring the database up to date, with the new image's migrations.
say "New image found, updating the database"
docker compose run --rm -T web node src/migrate.js || fail "The database update failed, the old version keeps running"

# 3. Start a container from the new image beside the old one.
say "Starting the new container"
count=$(echo $old | wc -w)
docker compose up --detach --no-deps --no-recreate --scale web=$((count + 1)) web
new=$(docker compose ps --quiet web | grep --invert-match --fixed-strings "${old:-none}")

# 4. Wait until it reports healthy, for a minute at most.
status=starting
for attempt in $(seq 1 30); do
  status=$(docker inspect --format '{{.State.Health.Status}}' "$new")
  if [ "$status" = healthy ]; then break; fi
  sleep 2
done

if [ "$status" != healthy ]; then
  docker logs --tail 20 "$new"
  docker rm --force "$new" > /dev/null
  fail "The new container did not get healthy, the old version keeps running"
fi

# 5. Send the visitors to the new container.
name=$(docker inspect --format '{{.Name}}' "$new" | cut -c 2-)
echo "reverse_proxy $name:3000" > caddy/upstream.caddy
docker compose up --detach caddy
docker compose exec -T caddy caddy reload --config /etc/caddy/Caddyfile
say "Visitors now reach $name"

# 6. Let the old container finish its open requests, then remove it.
if [ -n "$old" ]; then
  docker stop "$old" > /dev/null
  docker rm "$old" > /dev/null
fi
docker image prune --force > /dev/null
rm -f .failed

say "Version $(docker exec "$new" printenv APP_VERSION | cut -c 1-7) is live"

Copy it to the server. There is nothing to restart: the timer picks up the new file at its next round.

~/projects/guestbook
$ scp deploy/deploy.sh deploy@guestbook.example.com:guestbook/

Try it

Break the app on purpose, in a way the health check must notice. In src/db.js, change ping so that it asks a table that does not exist:

src/db.js
export async function ping() {
  await pool.query('select 1 from nowhere');
}

Commit this one file and push:

~/projects/guestbook
$ git commit --message "Break the health check on purpose" src/db.js$ git push

Naming the file at the end commits only that file. The changed script stays out of it, because this is a commit you are going to take back in a moment.

Watch the pipeline: it is green. The tests pass, because no test of yours talks to a database, so the image is built and called production. Tests are the first safety net, and this version fell through it. Now follow the journal on the server, with the command from the last part:

deploy@guestbook: ~
Nov 17 10:50:00 guestbook systemd[839]: Starting guestbook-deploy.service - Deploy the newest guestbook image...Nov 17 10:50:00 guestbook deploy.sh[5512]: 10:50:00 New image found, updating the databaseNov 17 10:50:01 guestbook deploy.sh[5570]: Database is up to dateNov 17 10:50:02 guestbook deploy.sh[5512]: 10:50:02 Starting the new containerNov 17 10:51:03 guestbook deploy.sh[5731]: Guestbook listening on port 3000Nov 17 10:51:03 guestbook deploy.sh[5512]: 10:51:03 The new container did not get healthy, the old version keeps runningNov 17 10:51:03 guestbook systemd[839]: guestbook-deploy.service: Main process exited, code=exited, status=1/FAILURENov 17 10:51:03 guestbook systemd[839]: guestbook-deploy.service: Failed with result 'exit-code'.Nov 17 10:51:03 guestbook systemd[839]: Failed to start guestbook-deploy.service - Deploy the newest guestbook image.Nov 17 10:51:03 guestbook systemd[839]: Starting guestbook-deploy.service - Deploy the newest guestbook image...Nov 17 10:51:03 guestbook deploy.sh[5790]: 10:51:03 Still the image that failed, waiting for a new oneNov 17 10:51:03 guestbook systemd[839]: Finished guestbook-deploy.service - Deploy the newest guestbook image.

Read it line by line. The script finds the new image and starts a container from it. Then nothing happens for a minute: the script is asking for healthy, thirty times. The line Guestbook listening on port 3000 is the last thing the broken container said. It did start, but its health check never got a good answer. The script removes it and says that the old version keeps running, and systemd notes that this round ended with an error. The timer's next round came due while all this took its minute, so the script runs again at once, sees the same image, recognises it and leaves it alone.

And the site?

~/projects/guestbook
$ curl https://guestbook.example.com/health{"status":"ok","version":"e07b6d2"}

Still the version from before, still healthy. A broken version was pushed, built and offered to the server, and no visitor ever met it.

Now the fix. git revert makes a new commit that undoes the last one:

~/projects/guestbook
$ git revert --no-edit HEAD$ git push

The pipeline builds a new image. It has another ID than the one in .failed, so the script deploys it as usual, as guestbook-web-4, and removes the note.

A version that runs but is wrong

The second kind is harder, because everything looks fine to the machines. The container is healthy and the swap went through, and then someone writes to you that the form looks strange.

The quickest cure is to go back to a version that worked, and you prepared for that in part 4 without knowing it: the registry still holds every image under the ID of its commit. Going back means giving the name production to one of those older images again. The server follows that name and nothing else, so it deploys the old image like any new one, without a gap.

Say the friendlier placeholder from the last part turned out to be a mistake, and you want the version before it back. Find the ID of its commit:

~/projects/guestbook
$ git log --format='%H %s' -57a1c3e94d0b2f58c6e1a9d3b7f4c2e8a5d0b6f19 Revert "Break the health check on purpose"2b9d4f8e6a0c3d7b1f5e9a2c4d8b0e6f3a7c1d95 Break the health check on purposee07b6d2c9f1a4e8b3d5c7a0f2e6b9d1c4a8f3e57 A friendlier placeholderc41f0a9b7e3d6c2f8a1b5e9d0c4f7a3e6b2d8c10 A shorter button8d3e5b7a1c9f04e62b7d5a3c8e1f6b09d4a2c7e5 Say whose guestbook it is

The version before is A shorter button, and the long row of letters and digits in front of it is its full commit ID. On GitHub you will paste it in a moment; on GitLab the commit's title is enough to find it. Now build the way back:

I work with

GitHub

Create a second workflow, .github/workflows/rollback.yml:

.github/workflows/rollback.yml
name: Rollback

on:
  workflow_dispatch:
    inputs:
      commit:
        description: The full commit ID of the version to bring back
        required: true
        type: string

permissions:
  packages: write

env:
  IMAGE: ghcr.io/your-name/guestbook
  COMMIT: ${{ inputs.commit }}

jobs:
  rollback:
    runs-on: ubuntu-latest
    steps:
      - name: Log in to the registry
        run: echo "${{ secrets.GITHUB_TOKEN }}" | docker login ghcr.io --username ${{ github.actor }} --password-stdin
      - name: Call the old image "production" again
        run: |
          docker pull "$IMAGE:$COMMIT"
          docker tag "$IMAGE:$COMMIT" "$IMAGE:production"
          docker push "$IMAGE:production"

Change IMAGE to your own name, as in part 4. What is new here:

  • workflow_dispatch means that this workflow never starts by itself. It gets a button on GitHub, and inputs puts a text field beside it, for the commit ID.

  • There is nothing to build and nothing to test, so the one job has three commands after the login: fetch the image of that commit, give it the name production as well, and push that name.

  • What you type into the field reaches the commands through env, as the variable COMMIT, in quotes. Text typed by a person is never pasted straight into a command: someone could type a command of their own into that field.

GitLab

In .gitlab-ci.yml, add a third stage at the top:

.gitlab-ci.yml
stages:
  - test
  - build
  - rollback

And add this job at the end of the file:

.gitlab-ci.yml
back-to-this-version:
  stage: rollback
  image: docker:29-cli
  services:
    - docker:29-dind
  variables:
    DOCKER_TLS_CERTDIR: '/certs'
  rules:
    - if: $CI_COMMIT_BRANCH == $CI_DEFAULT_BRANCH
      when: manual
      allow_failure: true
  script:
    - echo "$CI_REGISTRY_PASSWORD" | docker login $CI_REGISTRY --username $CI_REGISTRY_USER --password-stdin
    - docker pull $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA
    - docker tag $CI_REGISTRY_IMAGE:$CI_COMMIT_SHA $CI_REGISTRY_IMAGE:production
    - docker push $CI_REGISTRY_IMAGE:production

What is new here:

  • when: manual means that this job never starts by itself. Every pipeline on main now ends with a job that waits for a click.

  • allow_failure: true keeps the pipeline green while the job waits. Without it, every pipeline would be shown as blocked.

  • There is nothing to build and nothing to test, so the job has three commands after the login: fetch the image of this pipeline's commit, give it the name production as well, and push that name.

Commit and push, together with the changed deploy script:

~/projects/guestbook
$ git add .$ git commit --message "Remember failed images, and a way back [skip ci]"$ git push

The words [skip ci] in a commit message tell GitHub and GitLab not to start a pipeline for this push. Nothing in the app changed, so there is nothing to build and nothing to deploy. Now go back:

I work with

GitHub

Open the Actions tab of your repository, pick Rollback in the list on the left, and press Run workflow. Paste the full commit ID of A shorter button into the field and start it. Shortly after, the registry calls that image production again.

GitLab

Open Build → Pipelines and find the pipeline of the commit you want back. Every pipeline that runs from now on has the stage rollback with a play button on back-to-this-version: press it, and shortly after the registry calls that image production again.

The pipelines you have so far ran before the job existed, the one of A shorter button included, so they do not have the button. For such a pipeline, open it and press the retry button on its build job instead. That builds the same commit once more, replaces its image in the registry with the fresh build and calls it production: the long way round to the same place.

The server needs no telling. At its next look into the registry it finds that production is not what is running (Caddy's two lines are left out here):

deploy@guestbook: ~
Nov 17 11:14:01 guestbook deploy.sh[6820]: 11:14:01 New image found, updating the databaseNov 17 11:14:02 guestbook deploy.sh[6820]: Database is up to dateNov 17 11:14:02 guestbook deploy.sh[6820]: 11:14:02 Starting the new containerNov 17 11:14:05 guestbook deploy.sh[6820]: 11:14:05 Visitors now reach guestbook-web-5Nov 17 11:14:05 guestbook deploy.sh[6820]: 11:14:05 Version c41f0a9 is live

Reload the guestbook: the old placeholder is back, and the footer says c41f0a9. For the server this was a deploy like any other, with the health check, the swap and a kind goodbye to the container that had to leave.

One thing is still to do. Your main branch holds the newer commits, mistake included, and the next push will build all of it and move production forward again. A rollback buys you time. Use it to repair the code, or to take the mistake back with git revert and its commit ID, and push when it is right.

What you have now

Three safety nets, one behind the other. A failing test stops the pipeline before an image exists. A version that does not get healthy never meets a visitor, and is not tried a second time. And a version that is healthy but wrong is one click away from being replaced by the one before it.

The warning above names the one thing that can still spoil a deploy or a rollback: the database. In the next part you change a table while the guestbook stays online, in three small deploys.

Comments

The comments are GitHub Discussions: sign in with GitHub to write one.

The comments come from GitHub, through giscus. Loading them sends your IP address to giscus.app and GitHub.