Skip to content

Zero-Downtime Deployments for Node Apps With Docker Compose · Part 11 of 11 in the series

Part 11: A Message After Every Deploy

The server deploys by itself, so nobody is watching. Let the deploy script post a line to your chat when a version goes live and when one fails, and look back at the whole pipeline.

Quest objectives

  • Understand what a webhook is and create one in Discord or Slack
  • Post a message from the command line with curl
  • Let the deploy script report every success and every failure
  • Make sure a chat that is down never stops a deploy
  • See the whole pipeline in one piece and know where to go next

Your server fetches new versions by itself. That was the point of part 8, and it has a price: nobody is watching. The pipeline turns green when the image is in the registry, long before the server has done anything with it. If the new version then never gets healthy, the old one keeps running, which is what you want, and the reason sits in a log on the server until somebody looks.

A failed deploy should come to you. A line in a chat is enough, and it costs a few lines of shell.

A webhook

A webhook is a web address that does something with whatever is sent to it. Chat services hand them out for channels: send a short text to the address, and it appears as a message. No account for the server, no bot, no library.

This part uses Discord. In your Discord server, open Server Settings, go to Integrations and click Create Webhook (if the server already has webhooks, the button is under Webhooks and reads New Webhook). Give it a name such as Guestbook, pick the channel the messages should go to, and copy the webhook URL.

Try it on the server, which is where the messages will come from. Put your own address in place of the one here:

deploy@guestbook: ~
$ curl --header "Content-Type: application/json" --data '{"content": "Hello from the server"}' "https://discord.com/api/webhooks/1234567890/AbCdEf..."

The command prints nothing, and Hello from the server shows up in the channel. What you sent is a tiny piece of JSON with one field, content, the text of the message.

Teach the script

On the server, in ~/guestbook, open the .env file:

deploy@guestbook: ~/guestbook
$ nano .env

Add one line, with your address:

.env
DISCORD_WEBHOOK=https://discord.com/api/webhooks/1234567890/AbCdEf...

The rest happens on your computer, in deploy/deploy.sh. Three places change. First, a new function notify, right after the line that defines say:

deploy/deploy.sh
# Posts a line to the chat, if a webhook is set in .env. A failed post never stops a deploy.
notify() {
  webhook=$(grep "^DISCORD_WEBHOOK=" .env | cut -d = -f 2-)
  if [ -z "$webhook" ]; then return; fi
  curl --silent --show-error --max-time 10 \
    --header "Content-Type: application/json" \
    --data "{\"content\": \"$1\"}" "$webhook" > /dev/null || true
}

What it does:

  • It reads the address from .env, the same way step 1 reads the image's name: grep finds the line, cut takes what follows the =.

  • Without an address it returns and does nothing. The script still works on a server where nobody set one.

  • curl sends the message, as you just did by hand. It stays quiet unless something goes wrong, and gives up after ten seconds.

  • || true at the end means: whatever happened, carry on. A chat service that is down must never stop a deploy, or fail one that went fine.

The text is placed between quotation marks inside the JSON, so keep your messages free of quotation marks of their own.

Second, the function fail from part 9 gets one more line. Replace it with this:

deploy/deploy.sh
fail() {
  say "$1"
  echo "$wanted" > .failed
  notify ":x: Guestbook: $1"
  exit 1
}

The two failures the script handles go through this function: a database update that does not work, and a new container that never gets healthy. Each now reaches your chat with its reason. And because the script remembers a failed image, you get one message per broken version, not one a minute.

Third, the good news. Replace the last line of the script, the one that says the version is live, with these three:

deploy/deploy.sh
version=$(docker exec "$new" printenv APP_VERSION | cut -c 1-7)
say "Version $version is live"
notify ":white_check_mark: Guestbook: version $version is live"

The version is now kept in a variable, because it is used twice: for the log and for the chat. :x: and :white_check_mark: are the names of two emoji. The chat shows them as a red cross and a green check mark, so you can tell the two kinds of message apart without reading them.

The whole script

This is deploy/deploy.sh as it stands at the end of the series. Check yours against it:

deploy/deploy.sh
#!/bin/sh
# Brings the newest image online without a gap: the new container starts beside the old
# one, has to prove it is healthy, gets the visitors, and only then the old one leaves.
set -eu
cd "$(dirname "$0")"
# Compose reports every step it takes. The log should hold only what matters.
export COMPOSE_PROGRESS=quiet

say() { echo "$(date +%H:%M:%S) $*"; }

# Posts a line to the chat, if a webhook is set in .env. A failed post never stops a deploy.
notify() {
  webhook=$(grep "^DISCORD_WEBHOOK=" .env | cut -d = -f 2-)
  if [ -z "$webhook" ]; then return; fi
  curl --silent --show-error --max-time 10 \
    --header "Content-Type: application/json" \
    --data "{\"content\": \"$1\"}" "$webhook" > /dev/null || true
}

# 1. Fetch the image the registry calls "production" and compare it with what is running.
docker compose pull --quiet web
image=$(grep "^IMAGE=" .env | cut -d = -f 2-):production
wanted=$(docker image inspect --format '{{.Id}}' "$image")
old=$(docker compose ps --quiet web)
running=""
if [ -n "$old" ]; then running=$(docker inspect --format '{{.Image}}' "$old"); fi

if [ "$wanted" = "$running" ]; then
  say "Nothing new"
  exit 0
fi

# An image that failed before is not tried again every minute.
if [ "$wanted" = "$(cat .failed 2> /dev/null || true)" ]; then
  say "Still the image that failed, waiting for a new one"
  exit 0
fi

fail() {
  say "$1"
  echo "$wanted" > .failed
  notify ":x: Guestbook: $1"
  exit 1
}

# 2. Bring the database up to date, with the new image's migrations.
say "New image found, updating the database"
docker compose run --rm -T web node src/migrate.js || fail "The database update failed, the old version keeps running"

# 3. Start a container from the new image beside the old one.
say "Starting the new container"
count=$(echo $old | wc -w)
docker compose up --detach --no-deps --no-recreate --scale web=$((count + 1)) web
new=$(docker compose ps --quiet web | grep --invert-match --fixed-strings "${old:-none}")

# 4. Wait until it reports healthy, for a minute at most.
status=starting
for attempt in $(seq 1 30); do
  status=$(docker inspect --format '{{.State.Health.Status}}' "$new")
  if [ "$status" = healthy ]; then break; fi
  sleep 2
done

if [ "$status" != healthy ]; then
  docker logs --tail 20 "$new"
  docker rm --force "$new" > /dev/null
  fail "The new container did not get healthy, the old version keeps running"
fi

# 5. Send the visitors to the new container.
name=$(docker inspect --format '{{.Name}}' "$new" | cut -c 2-)
echo "reverse_proxy $name:3000" > caddy/upstream.caddy
docker compose up --detach caddy
docker compose exec -T caddy caddy reload --config /etc/caddy/Caddyfile
say "Visitors now reach $name"

# 6. Let the old container finish its open requests, then remove it.
if [ -n "$old" ]; then
  docker stop "$old" > /dev/null
  docker rm "$old" > /dev/null
fi
docker image prune --force > /dev/null
rm -f .failed

version=$(docker exec "$new" printenv APP_VERSION | cut -c 1-7)
say "Version $version is live"
notify ":white_check_mark: Guestbook: version $version is live"

Copy it to the server, as before:

~/projects/guestbook
$ scp deploy/deploy.sh deploy@guestbook.example.com:guestbook/

The timer picks up the new script with its next run. Nothing has to be restarted.

Try it

Any small change will do. In src/page.js, write Thank you for visiting. into the footer, in front of the word Version. Then:

~/projects/guestbook
$ git commit --all --message "Thank the visitors"$ git push

Put the terminal away. A minute or two after the pipeline is done, your chat shows a green check mark and the line Guestbook: version … is live, with the first seven characters of your commit. That is the whole way, from your keyboard to your visitors and back to you, without logging in anywhere.

If you want to see the other message too, break the health check once more as in part 9. About a minute after the server has found the image, a red cross arrives with The new container did not get healthy, the old version keeps running. Then undo the change with git revert, as you did there.

What you built

The whole series in one picture: the new container is set down beside the old one, and the visitors never stop coming.
The whole series in one picture: the new container is set down beside the old one, and the visitors never stop coming.

Look back at what stands between your keyboard and your visitors now:

  • an app that can say whether it is healthy, and that finishes its open requests before it leaves;

  • a pipeline on GitHub or GitLab that tests every push, builds an image from it and keeps it in the registry under its commit;

  • a server with Docker that runs without root, a firewall, and Caddy in front, which takes care of HTTPS by itself;

  • a server that fetches new versions itself, so that no key to it is stored anywhere else;

  • a swap in which the new container has to prove itself before a single visitor reaches it, and the old one leaves only afterwards;

  • safety nets: tests that stop the build, a health check that stops the swap, and a way back to any older version by moving one tag;

  • a pattern for changing the database in steps that every running version survives;

  • and a message that tells you how each deploy went.

None of it is a product you had to buy or a framework you have to trust. It is a pipeline file or two, one Compose file, a few lines for Caddy and a shell script you have read from top to bottom.

Where to go from here

  • Backups on a schedule. The pg_dump line from part 10 in a small script, run by a timer like the one from part 8, once a night. A backup that lies on the same server only helps half, so copy it somewhere else too.

  • A test environment. A second copy of the stack with its own domain and its own tag beside production, where a version can be looked at before your visitors get it.

  • More tests. Every test you add to the pipeline is a mistake that can no longer reach the server.

  • Your own app. The guestbook was only the example. To deploy another Node.js app this way, it needs three things: an address like /health that says whether it is ready, a handler for SIGTERM that lets it finish its work, and a Dockerfile. Everything else you have already built.

Comments

The comments are GitHub Discussions: sign in with GitHub to write one.

The comments come from GitHub, through giscus. Loading them sends your IP address to giscus.app and GitHub.