Corundite

"depends_on: condition: service_healthy" with Podman

Some docker configs use a healthcheck to make sure the database is ready for connections before starting other containers. This can be quite important, as some services fail very ungracefully if the database is refusing connections.

services:
  postgres:
    healthcheck:
      test: ["CMD-SHELL", "pg_isready -U postgresuser"]

services:
  xyz:
    depends_on:
      postgres:
        condition: service_healthy

Since podman is daemonless, this ends up being a little bit of an issue. However, the solution is suprisingly easy. For services of Type=notify, systemd expects a "ready" call when the service has finished starting. By default, podman generate systemd 1 creates Type=notify services, but sets --sdnotify to conmon. As far as I can tell, this tells systemd that the service is ready as soon as the container has started (assuming it doesn't immediately exit with an error code). This is all well and good, but we want to wait until our healthcheck succeeds. Enter --sdnotify=healthy, the solution to our problems. When combined with --health-cmd='pg_isready -U postgres', podman only sends this "ready" call when the command succeeds. You can then use --requires=postgres with your dependant container and podman generate systemd will automatically translate that into After=container-postgres.service, which will only let your container start until after the database is ready. Success!

So, in the end, our docker compose config can be translated pretty nicely to:

podman run -d \
--name=postgres \
--health-cmd='pg_isready -U postgresuser' \
--sdnotify=healthy \
...

podman run -d \
--name=xyz \
--requires=postgres \
...

I wasn't able to find much info on this, in the docs or really anywhere, so hope it helps!

Note: If you use podman generate systemd your generated service files should look something like the ones below. I've run into issues where it keeps --sdnotify=conmon for some reason, so make sure to check your postgres service file. If it still has --sdnotify=conmon under the ExecStart command, be sure to replace that with both --sdnotify=healthy and --health-cmd='pg_isready -U postgres'.

[Unit]
Description=Podman container-postgres.service
...
After=network-online.target

[Service]
...
ExecStart=/usr/bin/podman run \
        ...
        --sdnotify=healthy
        --health-cmd='pg_isready -U postgres' \
        ...
...
Type=notify
NotifyAccess=all
[Unit]
Description=Podman container-xyz.service
...
After=container-postgres.service

[Service]
...

  1. Yes, I know that podman generate systemd is technically "deprecated". More info is available here, but unless you require features from the new solution - quadlets - podman generate systemd should be called "feature-complete" rather than "deprecated" 

Thoughts? Leave a comment