When the monitoring agent becomes the memory pressure it reports

The agent shares the server it observes

A monitoring agent on a small server is one more process on that server. It uses the same memory, the same CPU cores and the same disk as the applications it reports on. When the server is busy, the agent usually has more to do as well: more requests to count, more history to keep, more state to save. Its cost therefore tends to rise at the moment the server has least to spare. On a machine with a few gigabytes of memory, an agent that holds several hundred megabytes is one of the larger workloads, and its behaviour under pressure belongs in the same review as any container's.

Hostwatch runs a Go agent on every server it manages. On one of our own servers, a two-core machine with a few gigabytes of memory that hosts several of our sites, the control plane kept marking the server degraded. The agent was sitting at its memory ceiling and using most of one core. The tailer, the part of the agent that follows the Nginx access log, counts as healthy only while its last read is less than five seconds old, and it kept missing that window. The other servers ran the same agent build without trouble. This article describes how we measured the problem, where the memory and CPU went, what we changed and how we checked the result. Figures are rounded, and we say which statements are measurements and which are design reasoning.

An explicit budget made the cost visible

The agent runs as a systemd service with resource controls on its unit. MemoryHigh is a soft ceiling: above it the kernel slows the service's processes and reclaims their memory aggressively. MemoryMax is the hard limit, where the out-of-memory killer acts inside the unit. CPUQuota caps the service at one core. The Go runtime has its own soft limit, GOMEMLIMIT, set below the soft ceiling so that the garbage collector works harder before the kernel starts reclaiming. Together these four settings are the agent's budget. Our earlier articles on container memory incidents and on cgroup v2 limits explain how such boundaries and pressure signals behave, so we only use them here.

The budget did what it was there for. The agent's memory stayed at the soft ceiling instead of growing into what the application containers need. The cost stayed inside the agent, where it showed up as a slow process, a stale tailer and a degraded server, all under the agent's own name. Without the limits the same growth would have been paid for by the applications, and the first symptom would probably have been a container failing for a reason nobody connected to monitoring. That is reasoning; we did not run the server without limits to find out.

Measuring a process that is already short of memory

Diagnosing an overloaded process with heavy tools can make the overload worse, so every measurement was read-only and short. We took them between fifteen and thirty minutes after a boot, from three places: the cgroup files for the agent's unit, the proc filesystem, and a private diagnostics socket on the agent that serves Go runtime statistics and the standard pprof profiles. Nothing was restarted or reconfigured in order to measure. The heap profile comes from sampling that the runtime already does, and the CPU profile ran for ten seconds.

The cgroup files showed memory at the soft ceiling. The counter of high events, which records each time the kernel throttled the group for exceeding that ceiling, was rising by more than a thousand a minute. The agent used about four fifths of a core, and its CPU quota throttled it in a large share of scheduling periods. Across the host, the pressure figures showed some task waiting for a CPU roughly two thirds of the time. The agent was also writing more than half a gigabyte to disk every minute, which adds up to several hundred gigabytes of rewrites a day.

The Go runtime statistics explained the CPU. The live heap, meaning the memory still referenced after a collection, was equal to the collector's target, and both sat just under GOMEMLIMIT. A collection started about once a second, over several million objects, and nearly half of all the CPU time the agent had used since it started had gone to garbage collection. The Go garbage collector guide describes this state: as the live heap approaches the memory limit the collector runs almost continuously, and the runtime limits it to roughly half of the available CPU time so that the program still makes progress. Our measured share was close to that cap. On the other servers the same build used a few hundred megabytes or less.

Where the memory went

A heap profile groups the memory in use by the code that allocated it. Nearly nine tenths of the agent's heap had been allocated by one function, the one that loads traffic history at startup. Traffic history is the agent's record of request counters for each site in five-minute intervals, kept for seven days, which makes about two thousand intervals. It backs the traffic views, which accept windows of up to seven days. Each interval holds one record per site, and each record carries around fifteen maps: countries, paths, sources, origin outcomes, threats and others. The history was fully decoded in memory. Map entries made up about half of it, and compressed per-country detail and the headers of maps and structures made up most of the rest.

The profile names the code and says nothing about the data, so we analysed a copy of the saved history offline, by site and by field. One busy site accounted for more than half of the file. By field, per-country detail took about seven tenths, and inside it the list of paths requested from each country took nearly two fifths of the whole history by itself. About four fifths of the history was more than a day old. The file shrank to about a sixth of its size at the fastest gzip level, which told us the content was repetitive enough for compression to be worth using.

The history did have a bound. It kept a fixed number of intervals and dropped the oldest, and intervals more than a day old were already trimmed to a limited number of paths. What it lacked was a bound in bytes. The size of one interval depends on how many sites, countries and paths appear in it, and those multiply. A server with varied traffic to several sites therefore held a history more than ten times larger than a quieter server running the same code with the same interval count. That comparison is measured: the saved history was several hundred megabytes on the busy server and a few tens of megabytes on each of the others.

Where the CPU went: saving a week every minute

The ten-second CPU profile split almost evenly in two. A little under half was the garbage collector marking the heap. About two fifths was the routine that saves traffic history. Once a minute the agent encoded all two thousand intervals as JSON, wrote them to a single state file and synced it to disk, although only the current interval had changed since the previous save. The per-country detail was held compressed in memory, but each save decompressed every entry, and the JSON encoder then compacted the result again. That last step alone was about a seventh of the agent's CPU.

These costs reinforce each other. Encoding the whole history allocates a large amount of temporary memory while the live heap is already at its limit, so the collector has more to do. The collector and the save together use up the CPU quota. The save also held the lock that request ingestion needs, so the tailer stalled while it ran, the freshness check failed, and the control plane marked the server degraded. A graceful stop took minutes. During an earlier upgrade the old agent did not finish stopping within the upgrade helper's one-minute wait, and the helper now lifts the soft ceiling to the hard limit before a stop and waits up to fifteen minutes. The profile shares and counters are measurements. The chain that links them is our reading, which the fix later tested.

A second, smaller source appeared in the allocation profile. The agent's egress guard counts outbound bytes against a traffic budget and records them in a budget ledger, a file with month and day totals and a rolling 24-hour window. The guard runs every two seconds, and each run rewrote and synced the whole ledger. The window kept one sample per run, so on the busy server it held tens of thousands of samples in a few megabytes of JSON. Rewriting that file accounted for about a fifth of all the memory the agent allocated and a few percent of its CPU. None of it stayed in the heap for long, but with the heap at its limit, the allocation rate decides how often the collector runs.

The proposals, including those we did not apply

The diagnosis ended with three proposed code changes. The first was to write closed history once. A five-minute interval stops changing after it closes, except through explicit backfills and imports, so closed intervals can be stored in files that are never rewritten, and the minute save can cover only the open part. The cost of a save then follows a few minutes of traffic instead of seven days. The second was to keep closed intervals compressed in memory and decompress them when a report asks. The third was to stop decompressing per-country detail during a save and write the compressed bytes as they are.

Two interim measures needed no release, and we applied neither. Raising GOMEMLIMIT and the soft ceiling would have made collections far less frequent. It would also have taken a couple of hundred megabytes more on a server that had about a gigabyte available and was already using swap, which moves the pressure onto the application containers. Trimming the stored history to two days while the agent was stopped would have shrunk the heap roughly in proportion, at the price of five days of per-site detail in the dashboard. That choice belongs to the server's owner.

The diagnosis also listed product choices that would shrink the history itself. Path detail per country could be kept for one or two days while country totals stay for seven. Alternatively, five-minute resolution could be kept for two days with hourly rollups beyond that, and longer ranges read from ClickHouse, which already receives that server's request evidence. Both change what an operator can drill into for older intervals, so they need the owner's decision, and neither has been implemented. The changes we released do not shorten the history.

Closed hours are written once, in segments

The released design stores closed history in segments. A segment is a gzip-compressed file holding the intervals of one clock hour. An hour counts as closed ten minutes after it ends, and it is then written to its segment once. The state file now holds only the open hours, the tailer's position in the log, recent request samples and the list of segments. The minute save rewrites that small file. A closed hour is written again only when something changed it: a log line that arrived late, a backfill, or the trimming of path detail when an interval becomes a day old. Each rewrite produces a new file and leaves the earlier one in place until the save completes.

The order of writes decides what a crash can damage. Segments are written and synced first, and the state file that lists them is renamed into place last, so the state file is the commit point. A segment that no state file lists, left by a failed save or a crash, is never read and is deleted after the next successful save. A segment that cannot be read costs its own hour, which is logged and omitted, and the other hours of the week still load. State files written by older agents load unchanged and move to segments on the first save.

The second proposal was only partly adopted, because we measured its cost. With whole intervals compressed in memory, a seven-day series took more than two seconds to compute, and a live dashboard requests it every five seconds. That would have exchanged memory pressure for CPU pressure. The release therefore compresses one thing: the route counters of intervals more than about a day old. They are the largest decoded structure, and only the path and error-group reports read them. Everything that live dashboards poll stays decoded. The third proposal was dropped, since a closed hour is now encoded once and its cost no longer recurs every minute.

The budget ledger and the memory sampler

The budget ledger received the same treatment on a smaller scale. It is now saved at most once a minute. Changes that a reader must not see late are still saved immediately: a change of meter status, a month or day reset, a new boot and a new uncertainty reserve. A failed save is retried on the next run, and a graceful stop writes whatever is pending. The rolling window is folded into one sample per minute, and each sample keeps the time of its latest addition. The window can therefore count up to a minute too much at its edge and never too little, which is the safe direction for a budget. Ledgers written by older agents are folded when they load.

Saving less often raises the question of what a crash loses. The saved interface counter and the month total are written together, so after a crash on the same boot the next reading covers the bytes that were not yet saved. After a reboot the gap since the last save is charged as uncertainty, as it was before the change. This is design reasoning backed by automated tests; we did not crash a production agent to confirm it. What we measured, on copies of live ledgers, is that every total was unchanged, the file shrank from a few megabytes to about a hundred kilobytes, and the cost of one guard run fell from about twenty milliseconds and ten megabytes of allocations to a fraction of a millisecond and about ten kilobytes.

The third fix concerns two samplers that shared one record of host memory. The full runtime sample feeds memory bursts: Hostwatch can raise a site's container memory cap from its normal limit to a configured peak when the host has enough memory available. A separate, lighter sampler reads the kernel's memory figures every few seconds. It had been added earlier so that decisions which need a fresh memory reading would not wait behind large traffic-history writes. Both advanced the same observation time. A runtime sample older than the latest kernel reading was rejected, and on the busy server that meant every reconciliation failed and new memory bursts stayed paused. Runtime samples are now ordered against the previous runtime sample only. An older one still reconciles bursts and does not replace a newer kernel reading. A regression test fails without the change, and the error did not recur after the redeploy.

Verification: synthetic data, a rehearsal and the rollout

We checked the storage change at three levels and report them separately. A test builds a synthetic week shaped like the busy server's traffic. On it, the live heap fell by about two fifths, a save went from several seconds and about two gigabytes of allocations to a few tens of milliseconds and a few megabytes, and the history on disk shrank to about a tenth. Then we rehearsed on real data: a copy of the busy server's state file was loaded on a workstation by the previous build and by the new one. Both took about twenty seconds to load. The new build's live heap was about 30 percent smaller, its first save wrote one segment for each hour of the week in under ten seconds, and its next save took a few tens of milliseconds.

The rehearsal also compared content. Every interval had identical per-site path counters before the first save, after it and after reloading from segments. Report answers agreed with two expected exceptions. One interval became exactly a day old between the two loads and was trimmed to the older path limit, which moves counts into an other-paths row and keeps the totals. The attribution, source and threat reports already differ between two identical calls on the same store, because rows with equal counts are ordered by map iteration. The release makes the path report break such ties by path, so its top rows no longer change between identical calls.

The rollout changed no configuration and restarted no application. On the smaller servers the upgrade took under twenty seconds each, and agent memory fell by between a quarter and a little over two fifths. On the busy server it took about three minutes, most of it stopping the old agent and reading the old single file one last time. Afterwards the agent used about a tenth of a core where it had used four fifths. Garbage collection fell from nearly half of the agent's CPU to under a tenth, CPU-quota throttling from about half of the scheduling periods to a few percent, and disk writes to roughly a tenth of the earlier rate. The live heap was about a third lower, no high events were recorded after warm-up, and the server reported online. The memory limits were the same as before.

What remains open, and what we take from it

Some things are not settled by this release. When the record was written, the agent's resident memory on the busy server was still close to its earlier level, because it included the one-time peak of converting the old file. The live heap was lower, and we have not yet recorded the settled resident figure. Of the remaining disk writes, traffic history is now under ten megabytes a minute, and the rest comes from other stores the agent keeps, which this release did not change. The same release carried a separate fix to how the egress limiter starts after a reboot, which does not bear on these figures. Rollback uses the previous binary, and a compressed copy of the old history was kept on the busy server for a rollback that needs all seven days.

Several lessons apply to any agent, ours or another. Give the agent an explicit budget that the kernel enforces, with the runtime's own limit set below it, and decide in advance that the agent must fit it. Measure memory by cause, which took two steps here: a heap profile to find the code, then an offline breakdown of the saved data by site and field to find the content. Bound every in-memory history in bytes as well as in entries. A fixed number of intervals or a fixed time window says little when the size of each entry depends on traffic, and a window sampled every two seconds holds thirty times as many samples as the same window folded per minute.

Prefer append-only segments to one large state file. Data that has stopped changing should be written once, the frequent save should cover only what is still open, and a single small file should act as the commit point. Look also for places where the observer amplifies the pressure it reports. In our case the freshness check that marked the server degraded was failing because of the agent's own saves, and memory bursts were paused by a sampler added to work around those saves. An operator can check any agent the same way: read the memory events and CPU throttling counters of its unit, compare its resident memory across servers running the same build, and ask what data explains the difference. In Hostwatch the visible results are the server's status and the traffic reports, and the rehearsal compared those reports before and after the change.