On a small server, reserve space for the build first and give cache what is left
A source build needs working room that the finished image never shows. One of our small servers fell short of that room. This is how the cause was confirmed, how the space was recovered without touching rollback versions, and how the scheduled cleanup was corrected so that it continues.
Published 13 November 2026. Hostwatch engineering review; implementation limits are stated below.
A source build needs working room
One of our own servers is small. Its root disk has a few tens of gigabytes, shared by the operating system, a handful of applications and everything needed to build those applications from source on the same machine. Building in place is convenient because there is no separate build machine to maintain, but it means the disk that serves the applications is also the disk the build works on. A source build unpacks an archive, fetches dependencies, compiles, writes intermediate image layers and only then produces the final image, and for a while all of those exist together. Our earlier article, Disk full with gigabytes free, explains how to find the filesystem a failing write lands on and why the next operation needs room of its own. This article starts where that one stops: the build filesystem is known, and it is short.
The common reaction to a build that fails for lack of space is to delete until it works: prune the container runtime, clear the build cache and try again. The difficulty lies in what a broad prune counts as unused. Docker's reference for its system prune command says that the all option removes every unused image, where unused means that no container is using it. The image of the previous release matches that description exactly. It has no container and nothing is running from it, and it is also the one artifact that lets you return to yesterday's version without building it again. A server that has just run out of build space is a poor place to discover that going back now requires a build.
The order of decisions is therefore the subject here. Decide first how much room a build needs, and keep that room free. Decide next which previous versions must survive any cleanup. Whatever is left after those two decisions is available to cache, and on a small disk the honest answer can be nothing. The sections below follow one morning on that server, when the capacity check reported that a source build could not be given its space. They describe how the cause was confirmed, what was removed, what was left alone and what had to change in the scheduler afterwards.
Confirm the cause before deleting anything
The symptom was plain. The server's capacity check reported a shortfall, which means the free space on the build filesystem was below the amount a source build is required to have before it starts. A release for one of the applications was waiting with its source archive prepared. Under that check, a build that still cannot reach its required space after cleanup reports the shortfall and stays unstarted. A few hours earlier, in the night, the same check had reported zero shortfall after a reviewed cleanup. Something had used the space since, and the scheduled cleanup, which runs every six hours on this server, had not restored it.
The records answered both halves of that. Builds, publications, deployments and cleanups each leave a receipt with a time and a result, and a cleanup receipt includes the free space measured before and after the run. The receipts showed that a separately authorised release of another application had been built, published and deployed shortly after the earlier check. That is ordinary use of the disk: a new image, its intermediate layers and a new source tree. They also showed that the scheduled cleanup had run several hours later, had recovered several hundred megabytes and had then stopped at its limit with a handful of reviewed artifacts still waiting. Free space after that pass was slightly under the required amount.
The last piece was the schedule. The next run reported by the agent was a full six-hour interval after the pass that had stopped early, so nothing was going to continue the unfinished work before then. A fresh preview about an hour later measured a shortfall of under two hundred megabytes and listed about two dozen eligible artifacts, with a lower-bound estimate close to a gigabyte. This was enough to explain the shortfall without hunting for a leak. The space had gone to a real release, more than enough removable material existed to get it back, and the cleanup had stopped before reaching it and then waited as though it had finished.
What a build reserve is and how it is computed
A rule such as keeping a fixed percentage of the disk free says nothing about whether the next build fits. Hostwatch computes the requirement from the build's input instead. For approved source builds, the build reserve is six times the configured maximum expanded source size, plus one GiB, the binary gigabyte of 1,024 MiB. The maximum expanded source size is a limit in the source profile that the server's operator installs: the largest unpacked source tree a build of that service may submit. The useful property of this formula is that the input size is known before the build starts, while the size of everything the build will write is not. Because the unpacked source, fetched dependencies, intermediate layers and final image all exist at once, the size of the last finished image is a poor guide to the room the next build needs.
Two more amounts complete the calculation. Headroom is an extra margin the owner sets on top of the reserve. The minimum free space is a server-wide figure the owner wants available at all times. The admission floor, the amount that must be free before a build is admitted, is the larger of two numbers: the reserve plus headroom, and the minimum free space. Bytes promised to uploads that have started and not finished are then reserved in addition. As an illustration, take a service whose profile allows 500 MiB of unpacked source. Six times that is 3 GiB, and one more gives a reserve of 4 GiB. With 2 GiB of headroom the build asks for 6 GiB. If the server's minimum free space is 10 GiB, the floor is 10 GiB, and the headroom is not added to it a second time. A 30 MiB upload in progress raises the required total by 30 MiB.
Our server was in that situation. The owner had set a minimum free space of ten GiB and two GiB of headroom, the reserve plus headroom for the waiting service came to well under ten, and so the shortfall was measured against the minimum. The capacity view reports each part separately: the space currently available, the reserve required by the installed source profile, the headroom, the unfinished upload reservations, the required total and any shortfall. Two details prevent false comfort. Free space on another filesystem does not count toward the build filesystem's shortfall. Uploads are admitted against their own fixed safety floor while no build is running, so that a requested build can reach its pre-build cleanup check, and while a build is active a new upload must also leave that build's configured budget intact.
Cache gets a budget, and the budget can be zero
Build cache is useful and replaceable. Intermediate layers and downloaded packages let the next build skip work that an earlier build already did. When they are missing, the build repeats that work and finishes later. Losing cache costs time, whereas losing the room to build or the previous version costs the ability to release or to go back. That difference is the reason the two get different kinds of rule. The reserve and the rollback versions are requirements. Cache gets a budget: an upper limit on how much old cache the server keeps once that cache has become eligible for removal.
On this server the owner set the cache budget to zero. Our other servers keep a budget of a couple of gigabytes. The change was a single policy field, made through the same owner-confirmed operation as any other policy change, and a readback showed that nothing else in the policy had moved. Zero does not instruct the cleanup to delete everything that resembles cache, because the age rules and every protection rule still take priority. Layers that a current or rollback image depends on stay, since they belong to a protected version. Cache involved in an active or uncertain build stays. Cache whose ownership cannot be proven stays. A zero budget says only that completed, unreferenced, proven build cache is not kept for its own sake once it is eligible.
This choice has two costs. Builds on this server may run longer, since less earlier work is available to reuse. The second cost is a limit of the approach. A physical audit of the same disk, counting shared layers once, found that the largest single group in the container store was several gigabytes of historical intermediate layers, the largest of them created before builds began labelling their intermediates with strict provenance. The containers' own writable layers were tiny, so application writes did not explain it. A generic builder label does not prove who created a layer or whether anything still needs it, so those layers remain protected and keep occupying space. A zero cache budget does not reach them. Deciding what to do with them is an operator's investigation, and it has not been automated.
Rollback versions stay protected as complete sets
A rollback version is a previous successful release kept in a form that can be served again. That form is a set. It includes the source tree, the built image, the layers the image depends on and the operation records showing that the version was built and deployed successfully. An image whose base layers have been pruned, or a source tree whose image is gone, looks like a rollback in a directory listing and cannot be used as one. Hostwatch therefore considers the source, uploads, images, layer dependencies and operation metadata of a version together when it decides what is protected.
The policy keeps between two and ten successful rollback versions for a service, and two is the lowest value the setting accepts. Several things are protected regardless of that count: the current source, running deployments, the latest published version, installed baselines and anything involved in an active or uncertain operation. An owner can also pin a known version to keep it explicitly, and removing the pin removes only that extra protection. When the evidence about earlier deployments is incomplete, Hostwatch keeps additional successful versions and reports a warning. Missing history is treated as a reason to keep more.
Protection has limits that are worth stating. It cannot create versions that were never retained: if fewer earlier versions exist than the policy asks for, the policy cannot supply the missing ones. A retained image also does not make a return to it safe for the data. Whether the previous version can still read the current database is a separate question, covered in our article A manual release needs a recovery story you can prove. What the protection does provide is narrower and still valuable on a small disk. No free-space target and no cache budget makes the current version or the retained rollback versions eligible for cleanup.
What a safe cleanup removes and what it never removes
The eligible material is specific. It includes intermediate images that are proven to come from completed builds and that no retained version depends on. It includes a final image that has fallen outside the rollback count after newer releases succeeded, together with its old source tree, where the operator has opted that service into final-image retirement. It includes uploads that were never used and have passed their expiry. Source versions and final images must also be older than the minimum artifact age, which is six hours on this server, the lowest value allowed. An earlier pass that night shows the shape of a typical result: a pair of superseded intermediate images, the final image that had just become the third-oldest version of one application with its source tree, and one small unused upload.
The list of what is never removed is longer. Application data, databases, runtime volumes and backups are outside this cleanup entirely. Containers are never retired by the build-artifact helper, including stopped ones. Named, published or referenced images stay unless the installed profile can establish that disposal is safe. Unknown directories and unknown image ownership count as reasons to keep an artifact. During the morning described here, every image that was protected for unknown provenance at the first preview was still present at the last. A free-space target that cannot be met from eligible artifacts produces a reported shortfall, and nothing protected is taken to close the gap.
The removal itself is conservative. The helper rechecks runtime and filesystem identities immediately before acting, holds an exclusive maintenance gate so that no build or publication changes the picture underneath it, and removes images through the container runtime without force, leaf images first. That order has a visible consequence. An image that other images were built on cannot go until its children have gone, so a parent becomes a candidate only in a later inspection. This is why a fresh preview after a partial pass can list more candidates than the pass left behind. Estimates deserve the same care. Adding up virtual image sizes overcounts shared layers, and a preview's figure for shared layers is a lower bound. The result that counts is the measured change in available space after the run, which also includes whatever the applications wrote in the meantime.
Recover the space in reviewed passes
With the cause confirmed and the candidates listed, the space was recovered by manual cleanup. While the maintenance was in progress new source uploads were refused, which was expected during a maintenance drain, and the waiting release had not started its upload. A manual run is tied to a preview. The request carries the revision of the preview that was reviewed and a stable request identity. If references change between the review and the run, the plan is invalid and the run refuses to delete. Age eligibility is frozen at the reviewed snapshot, so an artifact that becomes old enough during the inspection waits for the next preview. A manual request does not overtake a build either. Earlier that night a reviewed cleanup had ended without deleting anything because a source build was running and held admission, and its receipt was left exactly as it was.
The first pass was partial, like the scheduled one before it. It retired about ten artifacts, recorded no uncertain deletion, recovered a couple of hundred megabytes and then reached its limit. That was enough to lift the free space above the floor, and the next preview reported zero shortfall. The same preview showed how thin the margin was: a few tens of megabytes, the same order of size as the source archive that was waiting to be uploaded, with an agent update still to install. A zero shortfall that the next upload would cancel was not a state to stop in.
A second pass was therefore authorised separately, with a new request identity and a newly reviewed revision. The first request was not replayed. Retrying a completed request returns its original receipt instead of doing new work, and only artifacts in the new review could be retired. The second pass retired everything in its plan, nearly twenty artifacts, without errors, and recovered a little under a gigabyte. Together the two passes recovered a little over a gigabyte, measured as the change in available space on the filesystem. The receipts of the earlier partial passes were kept unchanged beside the new ones, so the record still shows that the first attempts stopped early.
How a bounded pass stalled the schedule, and how it now continues
Every cleanup pass is bounded. A run is limited to 64 artifacts and about three minutes of deletion work, with inspection and individual runtime operations adding some time around that. The bound has a practical value on a server that also builds and serves. The pass holds the exclusive maintenance gate, and a short, known hold is easier to live with than an open-ended one. The morning's first manual pass stopped after about ten artifacts, far fewer than the count limit, which points to the time limit as the one this server reaches.
The defect was in what happened after a bounded pass. A pass that stops at its bound returns a partial result. The scheduler recorded that partial pass as the interval's maintenance and set the next run a full six hours later, exactly as it would have done after a pass that found nothing left to remove. On a quiet day the only effect would be a slower cleanup. On this day a release had used space, the reserve was still short, and any build requested in that window would have had to rely on its own pre-build cleanup, which is bounded in the same way.
The agent now treats that case separately. When a scheduled build-artifact pass stops only at its bound, has verified deletions, measures a positive gain in free space and still leaves less than the required reserve, the agent schedules a continuation five minutes after the pass completes. Each continuation is a new request with a fresh inspection and the same writer and provenance guards, so nothing from the earlier plan is reused. There are at most three continuations per scheduled batch, counted in a private ledger that survives an agent restart, and after that the schedule returns to its ordinary interval. The next-run time reported through the API shows the earlier deadline. Paused automation, a restored reserve, a pass without progress, any writer, runtime or uncertain error, and registry maintenance do not enter this path, because in those cases a quick retry would be either pointless or unsafe. The change came with regression tests for the deadline, the cap across restarts and the excluded outcomes.
Verify the final state, including what should not have changed
The corrected agent was installed on this one server only. The installed binary was compared with the build that had passed the tests, and the previous executable and a backup of the private configuration were kept on the server. The maintenance drain was removed, remote writes were restored, and the agent and its capture timer were confirmed active. The restart had one visible side effect. On startup the agent reconciled its receipts once, and that briefly paused the release upload that had just begun. The upload then resumed with its committed chunks and checksum intact, and no further blocking maintenance was started.
The final readback used the same preview that had reported the problem. It showed zero shortfall, with free space a little under a gigabyte above the required total, and that total included the bytes still reserved for the unfinished upload. It listed no eligible cleanup candidates. The policy was compared with the first preview of the morning and showed no difference, which matters because a shortfall can also be made to disappear by lowering a floor. Every image protected for unknown provenance at the start was still present.
The things a cleanup should not touch were then checked directly on the server instead of being inferred from cleanup receipts. For the two applications that had released most recently, the current image and both rollback images were present. All application containers were running with the same process and start identities as before the work. The registry answered an authenticated read, and no write was used for that check. The waiting release sealed its upload with the checksum verified, passed its source preview and started its build without a disk shortfall. Its publication and deployment belong to a separate operation and are not claimed here. A zero shortfall is also an admission check at one moment. It does not promise room for builds of unlimited size later, and the next release will use some of the margin again.
Write the order down as a retention policy
Hostwatch expresses this order of decisions as a build retention policy for each server. It appears in the web console under Host, Storage and cleanup, once the server's operator has installed a fixed, reviewed artifact profile and enabled the capability. A server without a profile shows the feature as unavailable, and that is not permission to clean arbitrary folders. The server defaults are an interval from one to 168 hours, a minimum artifact age from six to 720 hours, two to ten successful rollback versions, the minimum available space, the build headroom, the cache budget and an upload expiry. One switch controls automatic cleanup, both scheduled and before a build. A separate clean-before-build switch allows a cleanup when an admitted source build lacks its reserve or a source slot. Pausing automatic cleanup still lets an owner review a preview and run a manual pass.
Each approved service can inherit the server defaults, use its own rollback, age and cache rules, or be excluded. Shared runtime cache often cannot be attributed to one service, so its common rules stay conservative, excluding a service protects the shared cache that cannot be attributed, and a smaller budget for one service does not authorise deleting another service's layers. A service name alone does not establish coverage. Manually published services need fixed bindings installed by the operator, and publishing over SSH that bypasses the installed release wrapper can be observed but never becomes eligible for automatic deletion. The same state is available to an assistant through MCP: retention_preview reads capacity and candidates, while retention_set_policy, retention_pin and retention_run require an owner, write scope and explicit confirmation. Callers cannot supply filesystem paths, commands or new cleanup recipes.
Several things remain outside the policy, and no field in it reaches them: application data, databases, runtime volumes, backups and containers. Retiring a local image does not reclaim registry storage, which needs a separately installed registry target. The reserve covers source builds. It does not plan space for backups or restore tests, and the storage guide lists combined retention and restore capacity planning as follow-up work. No CI/CD or Git-triggered deployment is added by any of this. The server owner stays responsible for provisioning storage and verifying recovery. When the protected versions and the reserve do not fit on the disk together, the policy reports a concrete shortfall, and the remaining choices belong to the owner: a larger disk, fewer services building on that server, or an investigation of storage that no rule can classify.
Compute the room a build needs from its source size and keep it free. Protect the current version and at least two rollback versions as complete sets, give cache only what is left, and make sure a bounded cleanup continues until the measured reserve is back.