A manual release needs a recovery story you can prove
Recovery depends on the previous executable, compatible application data and a user flow that works after restoration. Build a practical manual release plan around evidence you can retain and test.
Published 5 November 2026. Hostwatch engineering review; implementation limits are stated below.
Begin with the outcome users need
A recovery plan should name the useful behavior it is meant to restore. For a public website, that may be a working page and account flow. For an MCP component, it may be an authenticated request that returns the expected tool result. For a background worker, it may be processing the correct queue without duplicating work. A running process is relevant evidence, but it is not the full outcome. Define the important flow before the release so the team knows what to check when the new container starts, what would count as failure and which earlier state can realistically restore that behavior.
Manual releases make this definition particularly important. A person or authorized assistant deliberately invokes each change rather than relying on a repository push to apply it. Hostwatch supports that explicit workflow through configured component targets, fresh previews and durable operation receipts. There is no installed CI/CD pipeline or background application updater. The absence of automatic deployment does not itself make a release safe; the team still needs a backup, compatibility review and meaningful verification. The advantage is a visible decision boundary where the operator can connect the proposed image, current application data and recovery method before changing the process that serves real users.
Describe the state a service actually depends on
Applications depend on more than their executable image. A service can require a database, uploaded files, a protected environment file, a runtime identity, a writable cache directory and a public route. Some of those inputs are durable business data; others can be regenerated; some are secrets whose loss makes the remaining files unusable. Write down which is which. A useful service inventory links the observed component with its dependencies and configured release profile, while the application owner explains consistency and compatibility. Without that distinction, a backup of the container image can be mistaken for a backup of the customer records stored outside it.
Docker's storage documentation describes a volume as persistent storage whose lifecycle is separate from an individual container. This is why recreating an application can preserve its data, and also why reverting the container does not automatically revert that data. A bind mount has an additional dependency on a particular host directory and its ownership. A restored file tree with the wrong user permissions may be intact but inaccessible to the application. Include these relationships in the release review. The goal is to know which state the previous and next images need, which state can change while the release runs and which state must be restored from a separate backup if recovery requires it.
A backup timestamp is only the first fact
The latest successful backup receipt is a useful starting point. It establishes that a configured backup action completed at a particular time. The recovery question goes further: which data did it include, how consistent was the copy, where is it retained and what is needed to read it? An application with active writers can require a coordinated snapshot or database-aware export. A file copy performed while the store changes may not represent a usable point in time. Record the backup method and its application assumptions alongside freshness. This makes the evidence useful to someone who did not configure the job and cannot infer consistency from a green status label.
Retention should also survive a plausible failure. A backup stored only on the same disk as the live database has a different risk profile from an independently retained copy. An encrypted archive depends on access to its decryption material. A private control plane depends on its state and protected environment configuration, including the secret used for stored credential encryption. Keep the recovery material under the organization's access policy and test that authorized administrators can actually obtain it. A backup plan is incomplete when it preserves many archive files while the only key needed to use them remains on the machine the plan is intended to replace.
Restore testing supplies different evidence
A restore test asks whether the saved material can produce a usable isolated application state. It should run away from production writers and public routes, with separate data paths and an explicit verification target. Confirm that the restored store opens, expected records are available and the application can perform an appropriate harmless flow. Record the backup identity, restore time and result. This establishes more than the archive's existence or checksum. It can reveal missing environment material, incompatible runtime assumptions or permissions that a successful backup job did not exercise. Keep the test narrow enough to understand and realistic enough to expose the dependencies the service actually needs.
Do not turn restore testing into an accidental production write. An isolated database copied from real data still needs careful access and outbound integration handling. Disable or separate actions that could send messages, consume live jobs or modify external services unless that behavior is deliberately part of the authorized test. For simpler runtime recipe validation, private empty mounts and harmless preservation markers can test mount ownership and survival without copying live customer data. Report that result as runtime validation rather than a full data restore test. The distinction lets the team combine several useful checks without overstating what any one fixture, archive or container startup has proved.
Retain the previous executable and its assumptions
An immutable previous image gives recovery a concrete executable target. Keep its digest with the runtime configuration and a description of the data version it expects. A movable tag named previous is convenient, but the digest records the actual image behind that name at the reviewed moment. Source history helps explain how the executable was produced, while the built image can shorten restoration when a fresh compilation would be slow or depend on an unavailable network. Retain both according to the service's recovery needs rather than assuming one can always substitute for the other during an incident. The important property is an available, identified result that the operator knows how to run.
Review compatibility before the release applies. If the next application performs an irreversible schema migration at startup, returning to the earlier image may fail or corrupt data even though the old container configuration is preserved. Decide whether a migration has a supported reverse path, whether the previous version can read the new schema and whether a separate restore would lose subsequent writes. Coordinate the application's writer when needed. Hostwatch's configured deployment restoration can restore a previous component after a health failure, but it does not make an application-specific schema transition reversible. That limitation belongs in the release decision, beside the proposed code change and backup evidence, while there is still time to change the plan.
Preview immediately before the manual change
A deployment preview records the configured component, source digest, ownership and current runtime revision. Carry those exact values into the explicit update or replacement request. This catches a state change that occurred after the operator began preparing the release. An assistant may have read several guides, built an image and inspected logs in the meantime; the production revision can change during that interval. A fresh preview keeps the actual change bound to recent evidence. If the state no longer matches, stop that attempt and inspect what changed. Repeating the old request with a new operation key does not make its underlying assumption current again.
Registry publication uses a separate preview and, for an existing tag, its own expected published digest. A published image can wait safely while the operator coordinates application-specific conditions, because publication does not itself recreate the live component. Record the intended sequence in the runbook: prepare, inspect, publish, preview the live target, apply and verify. Each stage should have a clear result and stable operation key where required. This makes manual operation concise without making it implicit. Another administrator can see whether the fix was only built, already published or actually serving, and can choose the appropriate next action without recreating the whole release from a conversational summary.
Interpret rollback as a failed attempt with recovery
When a new component fails its configured health verification, Hostwatch can attempt to restore the prior deployment according to the installed profile. A rolled_back receipt means that the attempted update failed and restoration succeeded. That is a useful recovery outcome, yet it is not successful delivery of the new release. Report both facts. Inspect the actual serving image and health after restoration, then verify the important user route. If the previous image is again serving correctly, the team has recovered its prior behavior and can investigate the failed candidate separately. A clear result prevents the new change from being mistakenly treated as deployed merely because the operation reached a terminal state.
Keep the failed candidate and bounded diagnostics long enough to explain the failure. It may have a missing asset, incorrect writable directory, incompatible environment assumption or genuine application defect. Correct the relevant input through the approved source or operator workflow, then prepare a new reviewed release. Do not disable the health check simply because it prevented an update from staying live. Also inspect the data compatibility question before retrying. A successful container restoration is strongest when the team's pre-release review already established that the previous executable can still use the current data. Health verification and data recovery support each other while remaining different responsibilities.
A lost response needs investigation before another change
An interrupted client connection can leave the operator unsure whether the server accepted or completed an action. Stable idempotency keys make the original operation inspectable through image_publish_status. Reuse the same key for a true retry of the same request and poll its state rather than immediately creating a new intent. A known successful receipt can settle an acknowledgment failure without another deployment. A known failed receipt can point to diagnostics. An uncertain result requires additional evidence from the actual runtime. This approach reduces avoidable duplicate work and gives a teammate a durable reference that survives the original terminal or assistant session.
An agent restart during an operation can make the outcome unknown. That label should remain visible until the operator reconciles the serving component, publication state and relevant logs. It is not a successful result and should not trigger a blind second update. Compare the observed container identity with the intended and previous images, check the public flow and preserve the timestamps around the interruption. Then decide whether any recovery action is needed through the supported controls. The infrastructure can be healthy even when the operation receipt is uncertain, or unhealthy despite a completed preparation step. The investigation should connect those facts explicitly instead of forcing them into a single optimistic status.
Measure recovery through the real application boundary
Configured health checks provide a repeatable local criterion, and representative public requests test a different boundary. A healthy local listener does not prove that DNS, certificate trust and the external proxy all lead to it correctly. After a release or restoration, inspect the actual serving image and test the route users depend on. If IPv4 and IPv6 are advertised, account for both. For an authenticated workflow, use an authorized test identity or a safe established verification path. Avoid reporting a login page as successful account recovery when the underlying data flow has not been exercised. Choose checks that match the original outcome the recovery plan named.
Watch sibling components and current host pressure during the same interval. A successful website replacement can coexist with a worker failure, an unavailable assistant endpoint or memory pressure caused by a build. Preserve the relevant timestamps so later readers know the checks describe the post-release state. When a check cannot be performed, state that limit and explain what observation is available instead. The final release report should connect the change to behavior: which component and image now serve, which user flow responded, whether persistent data remained available and whether any attempted change rolled back. This is more useful than an unqualified all green message assembled from unrelated successful requests.
Maintain a recovery record that remains usable
Operation receipts, source archives and build logs have bounded retention. Keeping everything indefinitely can eventually prevent new work or make the current recovery material difficult to identify. Archive the completed evidence needed by the organization, then use explicit cleanup for eligible superseded source trees and old operation receipts. Current, deployed and uncertain source releases remain protected by the source controls. Forgetting a receipt removes its idempotency history permanently, so never reuse that old operation key. Cleanup should be a deliberate housekeeping decision after the recovery requirement is understood, with a preserved record of the release that is actually serving the application now.
The commercial model does not change these application responsibilities. Cloud uses the workspace subscription or trial while the customer owns its servers and workloads. Private Enterprise supplies the complete central installation under its annual license without an additional Cloud subscription, and its customer also operates that central service. Neither model promises unlimited hardware capacity or invents an availability agreement from a license fee. A dependable manual release combines an identified destination, compatible executable, usable data backup, explicit application and measured recovery. The result is a system the team can explain and restore through evidence, including when the newest candidate fails and the correct outcome is a verified return to the previous version.
A recoverable release joins an identified image with compatible data, usable backup material and meaningful post-release checks. A verified restoration is a recovery result; it does not make the failed candidate a successful release.