A fleet upgrade is finished when the reboot audit passes

Decide what finished means before the first package

A server that has been running for months is described by two things: the configuration on its disk and the state it has built up while running. A new agent binary and upgraded packages change the first. A reboot discards the second and rebuilds it from the first, and that is the only moment you learn whether the two ever agreed. A directory someone created by hand, a traffic rule added with a command and an address a container received when it started all work until the server restarts. For that reason we do not treat an upgrade as finished when the last package is installed, or when the server answers again. It is finished when an audit after the reboot has compared the server with a list written before it.

This article follows one such upgrade across our own fleet: a handful of Linux servers on two CPU architectures, each running the Hostwatch agent and anything from a couple of containers to many more on the busiest one. The work had four stages: a new agent build, operating-system package upgrades, a reboot into the new kernel, and the audit. Two terms recur throughout. Runtime state is anything that exists only in memory, such as files under /run, which is a memory-backed filesystem, and objects held by the kernel. A boot-time owner is a piece of configuration on disk that recreates that state at every start, so that nobody has to remember it. Most of what the audit found was runtime state without a boot-time owner.

Upgrade the agent with a helper that can refuse

The agent is a single Go binary run by systemd, so replacing it takes one copy and one restart. The risk is in everything around that restart. We upgrade our own servers with a reviewed helper script that an operator runs as root over SSH, on one server per run. It is internal tooling for our fleet. It cannot be reached through MCP and it accepts no command from a remote caller. Its first job is to find a reason to stop. It refuses if the staged binary does not match the checksum the operator supplied. It also refuses if the directory for this run's receipt already exists. The receipt holds the files a run replaced and a short record of what it did, and a second run into the same directory would overwrite the saved old binary with the new one.

Next it protects work in progress. The agent can build and publish container images, and restarting it in the middle of one would leave that operation with an unknown outcome. The helper saves the server's governance policy, which is the per-server setting that decides which operations a remote MCP caller may run. It switches off operations that change the server for the duration, places a maintenance marker beside the journal of image operations and reads that journal. If another operator's marker is already there, it stops. If an operation is queued or running it also stops, and the message says that no process was stopped. Then it runs the configuration checker that ships with the new build against the live configuration, as the service's own user. A new binary that rejects the existing configuration is discovered at this point, while the old agent is still running.

Only then does it change anything. It finds the installed executable from the systemd unit, copies it into the receipt directory, stops the agent service and nothing else, writes the new binary into place atomically and starts the service. A process that systemd reports as running may not be listening yet, so the helper polls the agent's local API until the process inventory and the CPU count are reported again. If a step after the stop fails, it puts back the saved binary and configuration and starts the old agent. In every case it then removes its marker, restores the governance policy and compares each field with the saved copy. The receipt records the checksum that was installed, what the agent reported afterwards and that no application was restarted. Application containers and the shared Nginx are not touched.

Why one server goes first

The agents were upgraded one server at a time, with the public sites checked after each. The package upgrades and reboots that followed used a fixed order, starting with the server that runs the fewest containers and ending with the one that runs the most. The first server answers the general questions where a mistake is cheapest: whether the build starts on that architecture, whether the helper's checks pass against a real configuration, whether the agent reports again and whether the sites on that server still answer. Two days later, when a release added new agent operations, the new build went to a single small arm64 server. The other servers kept the previous build and did not offer the new operations until they were upgraded the same way. After that restart the control plane showed the server as degraded for a few minutes while the agent's collector caught up. We had seen the same after earlier releases, and the record notes when it cleared.

A first server cannot answer for conditions it does not have. On our busiest server the old agent had been running at memory.high, the cgroup threshold above which the kernel throttles a process and makes it give memory back. A process in that state does everything slowly, including shutting down, and the agent did not finish its graceful stop within the minute the helper allowed. The helper replaces the binary only after the stop has completed, so nothing had changed and there was nothing to roll back. We raised the live threshold to the unit's hard limit for that drain only, the agent exited cleanly, and the verified binary was installed, started and checked. The helper now raises the threshold itself before stopping and waits up to fifteen minutes. systemd reapplies the configured value when the service starts.

The same server exposed a regression in the first build we rolled out. A change to the way the agent samples host memory made one reconciliation step reject every sample, but only where a particular memory feature was configured, and only that server's sites use it. The symptom was a log line repeating every few tens of seconds that no server had logged in the days before. We corrected the comparison, added a regression test that fails without the correction and upgraded every server again, with a new receipt for each run. An order that begins with the quietest server proves the procedure. It says little about the busiest one, where both of these problems appeared, which is why reading the new agent's log for lines that were not there before belongs among the checks after each run.

Upgrade packages so that nothing restarts by surprise

With the agents done, each server took its operating-system package upgrade. The upgrade ran as a transient unit started with systemd-run, so it was not a child of the SSH session and a dropped connection could not interrupt dpkg halfway through a package. dpkg was told to keep our existing configuration files wherever a package shipped a new version of one. Afterwards we looked for files ending in .dpkg-dist, which is where dpkg leaves the packaged version when it keeps yours. There were none, so no packaged default had changed underneath a file we had edited. As with the agent, the upgrade refused to start while an image operation was active on that server.

Before and after each run, a snapshot recorded the running services, every container with its ID and start time, the failed units and the package versions. Comparing the two gave the claim that nothing broke something to stand on: on every server no running service had stopped, no container had restarted and no unit had newly failed. The needrestart tool ran in its listing mode only, so it named the daemons still using old libraries without restarting any of them. We then restarted the internal daemons that carry no site traffic, such as cron, the login manager and the SSH daemon. The SSH restart came after a configuration test, and our earlier article on safe SSH changes explains why a fresh connection is the check that counts. Nginx, the container runtime and the application units waited for the reboot.

Two programs needed more than the package manager gave them. The ClickHouse package does not restart the database server when it is upgraded, so the new version is not running until someone restarts it. We did that explicitly, waited for its ping endpoint to answer and read its log for errors. While it was down, the log shipper on another server retried from its disk buffer and stopped warning once the database returned. That log shipper, Vector, was itself due for an upgrade. Before installing it we unpacked the new version from its package and validated it against the live configuration as its own service user, including its health check against the database, for the same reason the agent's configuration is checked before the old agent stops.

Reboot one server at a time, after a preflight

Every server already needed a reboot before this work began, for earlier kernel and C library updates, and the package upgrade added a newer kernel. Once the owner had approved, the servers were rebooted one at a time in the same order. A reboot is the step where a mistake costs the most, because the server may not come back, so each began with a preflight of the things that decide whether it boots and can be reached. The bootloader's first entry had to be the newest kernel. The filesystem table had to pass findmnt --verify, with attached volumes marked nofail so that an absent volume cannot hold up the boot. The network configuration had to regenerate exactly the files the running system was using. Every running container had to have something that starts it at boot.

The reboot helper then took a wider snapshot than the one used for packages: running and exited services, containers, listening TCP ports, mounts and failed units. Like the others, it refused to run while an image operation was active. Each server was reachable over SSH again within about two minutes on its new kernel, and needrestart no longer reported an outdated kernel or service on any of them. The snapshot was taken again and compared with the first. That comparison, together with a read of every site, unit and recovery target, is what we mean by the audit. On every server it found at least one thing that needed a decision, and the sections below take them in turn.

A directory in /run that nobody recreated

Hostwatch runs a recovery watchdog as a service separate from the agent, so that it can keep observing when the agent stops. On one server the watchdog did not start after the reboot. Its systemd unit is sandboxed and names the directories it may write to with ReadWritePaths. One of them is a directory under /run that holds the lock taken by anything that writes the shared Nginx configuration. systemd builds a unit's mount namespace before it starts the process, and a listed path that does not exist makes that step fail with exit status 226/NAMESPACE. The unit failed five times in a row and systemd stopped trying.

The directory was missing because our installer had created it once, with mkdir. /run is a tmpfs, so it is empty at every boot. On the other servers the fault stayed hidden. Where the agent runs as root it recreates the directory when it starts, and on one server the watchdog happened to retry until the directory existed. The server that failed runs its agent without root privileges, so nothing put the directory back. The installer had verified the watchdog's observations on every server at installation, and that verification passed everywhere, since it ran minutes after the mkdir on a server that had not rebooted.

The repair is one line in a file under /etc/tmpfiles.d that declares the directory, its mode and its owner, in the form d /run/example 0750 root root. systemd-tmpfiles reads those files early in boot and creates what they describe before ordinary services start. Every server now has the file, and the installer writes and applies it where it used to call mkdir. For a directory that belongs to one service, the RuntimeDirectory setting in the unit does the same job and removes the directory when the service stops. A directory shared between services, like this one, needs tmpfiles.d. The rule we took away covers every path under /run that appears in a unit file, a socket setting or a lock path: something that runs at boot must create it, because a mkdir run once by an installer lasts only until the next reboot.

A queueing discipline the kernel did not keep

The second piece of missing state was in kernel memory. On our busiest server the agent limits outbound traffic with a token bucket filter. That is a queueing discipline, or qdisc, attached to the network interface with the tc command. A qdisc is a kernel object and nothing on disk describes it, so after the reboot the interface carried the default qdisc the system attaches at boot, which is fq_codel on these servers. The agent's limiter is cautious about anything it did not create. It treated the qdisc it found as someone else's, refused to manage the interface and reported a guard error. The control plane marks a server as degraded on that error, which is how the fault became visible.

We restored the previous qdisc with the same tc command the agent uses, taking the rate from the agent's saved state. Changing the queueing on the interface that carries your own SSH session deserves a way back, so the command ran behind a five-minute rollback timer that would have undone it. A fresh SSH session connected, the public sites answered, the agent confirmed that it owned the qdisc at the saved rate, and only then was the timer cancelled.

A manual step that has to be repeated after every reboot will be forgotten eventually, and the upgrade record said plainly that every reboot of a server with this limiter would need it. The agent release that followed a day later changed the limiter to adopt the default root qdisc, which gives this state a boot-time owner in the agent itself. That release's record does not include a reboot, so the next reboot is where the change gets confirmed. The pattern reaches well beyond traffic shaping. Firewall rules added with a command, sysctl values set by hand, routes and extra addresses all live in kernel memory, and each needs a unit, a configuration file or a daemon that applies it again at boot.

Active is not the same as serving

The most serious finding was on the busiest server, and every unit involved did what it was configured to do. Podman ships a service, podman-restart, that starts at boot every container whose restart policy is always or unless-stopped. Earlier operator releases had left stopped copies of previous containers on the server as rollback copies, and those copies still carried the restart policy of the originals. At boot several rollback copies started first and took the ports, and several current containers then could not start. For a few minutes some services were down and others served an older build. A rollback copy of one background scheduler also ran beside the current one, although it logged no work in that time.

A status-code check does not see this, since a previous build answers a health request as readily as the current one. What showed it was the container list compared with the snapshot taken before the reboot: names running that should not have been, and names missing that should have been there. We stopped the rollback copies and started the current containers, which brought the server back to exactly the set recorded beforehand. Every stopped rollback copy was then given the restart policy no, and the previous policies were written into the receipt. The copies remain available for a deliberate rollback and can no longer start themselves. Containers deployed by the agent were not affected, because the agent names its backup copy after the job and removes it on success. For copies made by hand, the rule is to set the restart policy to no when the copy is made.

Two smaller findings belong here too. A tunnel unit on one server forwarded a Unix socket to a Redis container at a fixed IP address. Podman assigns those addresses when containers start, and after the reboot the container had a different one. The failure surfaced on another server, where a worker reported its Redis client offline and scheduled jobs failed. The unit now resolves the address in a pre-start step ordered after podman-restart, and the check was a PING through the socket followed by the next scheduled runs. If that container is ever recreated, the tunnel has to be restarted to pick up the new address. On a third server one container turned out to have no boot unit at all, despite the preflight, and was started by hand. Our article on checking service health end to end explains how to choose a request that proves a service is doing its work.

A root filesystem filled by the database's own logs

The upgrade record noted that one server's root filesystem was nearly full and left it at that. The next day a routine health report showed only a few gigabytes free, and we looked properly. ClickHouse, which stores request evidence on that server, held several gigabytes. The evidence itself was a small part of that and already expires after three days. The rest was ClickHouse's own system log tables: traces, metrics and similar diagnostics that the database server writes about itself. They have no expiry by default, so they grow for as long as the server runs.

The package upgrade had made this worse in a way that is easy to miss. When a new ClickHouse version changes the structure of a system log table, the server renames the old table aside with a numeric suffix and starts a new one. Nothing deletes the renamed tables. Ours held a large share of the total, and no query would ever read them again. This applies to upgrades in general: a new version can leave the old version's data behind under a name that no retention rule matches, and a before-and-after comparison of services and packages will not show it.

The cleanup had two parts. First we bounded the source. A small file in ClickHouse's config.d directory gives the system log tables a one-day retention, partitioned by day, and the server was restarted once to apply it while Vector buffered to disk. Then we dropped the renamed tables. The database shrank from several gigabytes to a few hundred megabytes and free space on the root filesystem roughly tripled. Evidence retention was not changed. Other large items were left for the owner, with their sizes written down: old database copies that sit outside the managed backup store, source retained from a migration, and images that stopped rollback copies still reference. Our article on full disks covers finding what occupies a filesystem. Setting the retention before deleting is what keeps the same tables from filling the disk again.

Audit against a written list, and record what you leave

The audit worked because its questions were written down before the reboot. The snapshot supplied the containers that should be running, the listening ports, the mounts and the units that had already failed. The other half is the expected answer for each public hostname, and that column matters more than it seems. One of our endpoints accepts only WebSocket upgrades and closes a plain GET on purpose. Some API hosts return 404 at the root and 200 at their health path. Someone checking from memory raises a false alarm on those, or after a few false alarms stops looking closely. A written record also keeps known faults from being rediscovered. On one server, cloud-init units failed during boot because of a race between an interface rename and early DHCP. We confirmed that the network itself was configured and unaffected, removed the leftover DHCP client processes so that they could not deconfigure the interface, recorded the units as failed and harmless, and left them.

Hostwatch's part in this was the read side. After the reboots we read every server back through read-only MCP operations: sites for the configured sites and their containers, recovery_status for the watchdog's targets, runtimes for the container runtimes and backups for the backup state. For each installed target the watchdog reports the process state, an optional local readiness check and an optional public HTTPS check as separate results, and a check that is missing is labelled as missing. The audit also went over SSH for enabled units that were not active, timers and failed units. That is how we found a nightly database backup whose preflight had refused to run for two nights because the server lacked the memory headroom it requires at that hour. It predated the reboot. We ran it once through its own unit, verified the archive and recorded that the refusal would return until the owner decides on that server's memory budget. It returned the next night.

The division of work should be stated plainly. Hostwatch did not upgrade packages or reboot anything. That was operator work over SSH with small helper scripts, and the agent upgrade helper is our own internal tool. For customers the agent upgrade is an explicit step as well: the installer does not replace an existing agent and installs no background updater. The watchdog observes installed targets and, where an owner has enabled it, restarts them within a budget, as our article on automatic recovery describes. On the server where the watchdog itself failed to start it observed nothing, and the list of failed units is what showed it. Two things from this upgrade are still open and are written in the record for the owner: the memory budget for that backup and the large items left on the root filesystem.