Log rotation without losing the tail: keeping an unprivileged reader attached to an Nginx access log

A reader has to outlive the file it reads

Traffic evidence on a self-run server usually begins as lines in an Nginx access log. A program that turns those lines into request counts, error rates and attack findings does so by following the file: it keeps the file open, reads what has been appended since its last read and remembers how far it got. The file does not live as long as the reader. Log rotation replaces it on a schedule or at a size limit, often once a day, so that old records can be compressed and eventually deleted. A reader that handles the ordinary minutes well and the rotation badly produces a history with a small hole or a small double count at the same hour every day, which is difficult to notice and more difficult to explain afterwards.

This article follows one reader across that boundary. It explains what rotation does to the file, where records are lost or counted twice, and why a reader that runs without root privileges can lose its permission to read at the moment the new file appears. It then describes how the reader on one of our own servers stopped after a rotation, how the repair was tested before it went near the live log, and how to tell an idle site from a reader that has stopped. In Hostwatch the reader is called the tailer: the part of the Go agent that follows the configured Nginx access log. Our earlier articles on building traffic history you can trust and on reading an HTTP attack explain what the resulting numbers mean. Here the subject is whether the lines reached the counter.

Rename and create, or copy and truncate

The default method in logrotate is to rename the file and create a new one. The active file, for example access.log, becomes access.log.1, and older copies move up by one; each rotated copy is called a generation in this article. A rename changes only the directory entry. The file itself, which the kernel identifies by its inode, is untouched, and a process that has it open keeps the same file under the new name. With the create directive, logrotate then makes an empty file at the original name with a stated mode, owner and group. That file is a different inode. At this point Nginx is still writing to the old one, because it opened its log once and writes through that descriptor without consulting the name again.

The postrotate script completes the rotation. For Nginx it sends the USR1 signal to the master process, either directly or through the command nginx -s reopen. The Nginx documentation describes the result: the master reopens its log files and asks the worker processes to do the same, and the workers open the new file and close the old one right away. Requests that complete between the rename and a worker's reopen are logged to the old inode, now named access.log.1. For that reason the delaycompress directive is usual for web server logs. It leaves the newest generation uncompressed until the next cycle, so late lines arrive in an ordinary file. If the reopen never happens, Nginx keeps writing to the renamed file, and if that file is later deleted its space stays allocated; our article on disk-full diagnosis covers that case.

The other method, copytruncate, exists for programs that cannot be told to reopen their log. Logrotate copies the active file into a new generation and then truncates the original to zero length in place. The name and the inode stay the same and the writer continues without knowing that anything happened. The logrotate manual states the cost plainly: there is a short interval between the copy and the truncation, and data written during it may be lost. The create directive has no effect in this mode, since no new file is made. Nginx can reopen its logs, so copytruncate has no advantage for it. A reader should still recognise a truncation, because rotation settings are sometimes changed by someone who does not know that a reader is attached.

Where records are lost or counted twice

A reader crossing a rename-and-create rotation can lose records in three places. If it moves to the new file as soon as the name points to a new inode, it misses the lines that Nginx wrote to the old file between the rename and the reopen. If it opens the new file and starts at the end, as a reader reasonably does when it first attaches to a log full of old history, it misses whatever was written to the new file before it looked. If it was behind when the rotation happened and the old generation is compressed or deleted before it catches up, the unread part is gone for that reader. Each loss is small on a quiet site and larger during a burst, which is the period an operator is most likely to examine later.

Double counting comes from losing track of identity or of position. A reader that recognises files by name sees access.log.1 appear and may treat it as new, although it is the file the reader has just finished. A reader that keeps its position in one place and its counters in another can stop between the two writes. On restart it repeats lines it had already counted or skips lines it had not, depending on which of the two was saved first. Both problems have the same remedy in principle. A file should be identified by its device and inode, and the position should be saved together with the data derived from it.

Truncation adds two more cases. The reader holds an offset into a file that has become shorter, and if it does not check, it waits at a position past the end. If the file grows beyond the old offset before the reader looks again, a comparison of sizes finds nothing wrong and the reader resumes in the middle of unrelated lines. Every reader also meets the unfinished line. A read returns a block of bytes, and the block can end partway through a record. Parsing that fragment produces an error or, worse, a shortened record that happens to parse. The saved position must therefore advance only past complete lines.

How the Hostwatch tailer crosses a rotation

The Hostwatch tailer polls the configured log path twice a second. On each poll it asks the filesystem which file the path names now and compares it with the file it has open, by device and inode. If the two differ, it opens the path again. The file it had been reading is not closed. It is marked as retired and kept open beside the new one, and on every poll the tailer reads the retired files first and the current file after them, within a fixed budget of bytes per poll. The new file is read from its first byte, since everything in it was written after the rotation. The old file is read onward from the last complete line already counted, so the lines Nginx wrote between the rename and the reopen arrive through the old descriptor.

The tailer's position, which the code calls a cursor, is a byte offset that advances only when a complete line has been processed. Bytes after the last newline are held as pending and reported as pending; they are not parsed until the rest of the line arrives. A line longer than one mebibyte is discarded and counted as an oversized record. A line that is not valid JSON, has no timestamp, or carries a timestamp more than five seconds in the future is counted as a parse error. In both cases the tailer records a gap. A gap, in this article, is a recorded point at which the tailer can no longer vouch that it has seen everything. Recording one increases a counter and stores the time and the reason.

Nothing tells a reader that the writer has finished with the old file. Nginx sends no acknowledgement, and the tailer cannot rule out a late write to the old inode. It therefore keeps a retired file open until thirty seconds have passed without new bytes, then closes it and records a gap whose reason says the retired log was closed after a bounded wait. This is deliberate. A routine rotation adds one to the gap count, and coverage, described below, reads partial for the following minute, because the tailer does not claim a continuity it cannot prove. An unfinished line left in a retired file at that point is counted as one dropped record. The tailer keeps at most two retired files; if rotations come faster than that, the oldest is closed early and the gap says so.

Truncation, restarts and the saved cursor

For copytruncate and any other rewrite in place, the tailer inspects the current file before each read. Beside the offset, the cursor holds an anchor: a SHA-256 digest of up to the last 128 bytes before the offset. If the file is now shorter than the position the tailer has read to, or the bytes before the offset no longer match the anchor, the file has been truncated or rewritten. The anchor is what catches a file that has already grown past the old offset. The tailer then starts again at the beginning of the file and records a gap whose reason states that the lost bytes are unknown. It does not try to recover the difference from the rotated copy, so the number of lines missed in a copytruncate rotation stays unknown.

The cursor is saved in the same state as the traffic counters and not in a separate position file, so the two cannot disagree after a crash: lines read after the last save are read again and counted once. For each open file the saved cursor holds the device and inode, the offset and the anchor. When the agent starts, it looks for each saved file by that identity, first at the configured path and then among the uncompressed rotated generations in the same directory, verifies the anchor and continues from the offset. An agent that was stopped across a rotation therefore resumes both generations. If a generation has been compressed or deleted in the meantime, or its anchor no longer matches, the tailer records a gap and carries on. On its first attachment to a log, with no saved cursor, it starts at the end of the file and records the skipped history as a gap without replaying old lines into the totals.

A new file starts with new permissions

Rotation changes more than content. The new file takes its mode, owner and group from the create directive, commonly read and write for the web server's user, read for a log-reading group and nothing for anyone else. A reader that runs as root never notices. A reader that runs under a dedicated unprivileged account needs something more, and the usual way to let one account read one file is an access control list, or ACL: an extra entry on the file that grants read permission to a named user. An ACL is stored with the inode. When logrotate renames the file, the entry goes with it to access.log.1. The file created at the old name is a new inode and has no such entry, unless the program that creates it copies one or the directory carries a default ACL. A grant made once by hand lasts until the first rotation.

The broader remedies are easy, and each grants more than the reader needs. Adding the account to the log-reading group opens every log that group can read. A default ACL on the directory applies to every file created there afterwards, including the logs of other sites and the error logs. Running the agent as root removes the question together with the sandbox. The access log deserves a narrow grant because of what it holds. Hostwatch's log format passes raw query strings to the local agent, which keeps parameter names and allowlisted values and redacts the rest before retaining anything, so the file on disk can contain tokens and search text. Our operating notes for this arrangement exclude both the directory default ACL and the group membership.

The narrow remedy is to restore the grant at each rotation, inside the existing logrotate stanza. Logrotate creates the new file before it runs the postrotate script. A few lines at the top of that script, ahead of the command that makes Nginx reopen, check that the path is a regular file and not a symbolic link, then run setfacl to add a read-only entry for the one account on that one path. The path is written out, because under sharedscripts the script's argument is the whole pattern. If setfacl fails, the script exits with its error, so logrotate reports the failure and nothing proceeds silently. The log directory has to be owned by root and not writable by the reader's account, since the hook runs as root and must not be led to another file. The lines are merged into the existing stanza; a second stanza for the same log would overlap the first.

Telling no traffic from a reader that stopped

A flat line on a request chart has two possible meanings, and the chart cannot say which one applies. Either no requests arrived, or requests arrived and the reader did not see them. The difference has to come from the reader, as a statement about its own state that is published beside the data. The first part of that statement is whether the reader is alive and attached. The Hostwatch tailer counts as healthy only when its last poll could reach the configured log and that poll is less than five seconds old. An idle site keeps passing this test, because every poll succeeds, finds nothing new and renews the time of the last read. A reader whose process has stalled fails it within seconds; our article on the monitoring agent's memory budget describes a tailer that went stale that way. A reader whose log cannot be opened fails it as well.

The second part is coverage, which the tailer reports as one of three states. Unknown means the tailer is not healthy, so nothing can be said about the present. Partial means it is healthy but has bytes it has not yet processed, including an unfinished line, or has had less than one minute of unbroken operation since it started or since the last gap. Complete means healthy, nothing unread and at least a minute of continuity. A server with no access log configured reports a fourth value, not configured, so that the absence of HTTP data is not read as a fault. A quiet night on a working reader therefore appears as complete coverage with zero requests, and the zero can be believed because of the coverage beside it.

The remaining figures explain a state that is not complete. Unread bytes are the distance between the cursor and the end of each file the tailer has open. Lag is reported while unread bytes exist and is the age of the newest record processed so far. Parse errors and dropped records count lines that were read and could not be used, which usually means the log format and the agent no longer agree; each rejected line also records a gap, so coverage stays partial while they keep arriving. The gap count, with the time and reason of the latest gap, shows where continuity was broken. The figures have to be read together. Unread bytes are measured on open files, so a reader that cannot open the new log may show none, and in that case the health flag and the coverage state carry the message.

The collector that stopped after a rotation

One of our servers runs the agent under a dedicated unprivileged account inside a systemd sandbox. Its permission to read the access log was a single ACL entry, set once on the file that was active at the time. A few minutes after a scheduled daily rotation, fleet status reported incomplete observation for that server, and the service groups hosted on it inherited the state. The agent itself answered, and every systemd unit registered on the server was running. What had stopped was HTTP collection, which Hostwatch reports as the state of the server's HTTP collector: the tailer's newest record was several minutes old and nothing was arriving after it.

The cause was in the permissions of the new file. The new access log had the mode, owner and group from the create directive and no ACL entry for the agent's account. The previous generation, renamed by the rotation, was still readable by it. From the tailer's side the sequence had gone as designed up to one step. The path named a new inode, the attempt to open it was refused, and the old descriptor went on working, because permission is checked when a file is opened and the ACL had stayed with the old inode. Through that descriptor the tailer could still read whatever Nginx wrote to the old file before reopening. After the reopen, Nginx wrote only to a file the tailer could not open.

The repair consisted of two changes. A read-only entry for the agent's account was placed on the active access log, which let the tailer open the file without the agent being rebuilt or restarted. The hook described above was merged into the existing Nginx stanza, after the file is created and ahead of the existing reopen command. The rotation schedule, the number of generations kept and the compression settings stayed as they were. No second stanza, directory ACL, group membership or additional socket permission was introduced, and the agent was not made root. The rotated generation was left with the standard owner, group and mode, so readers already approved through the group kept their access and no other user gained any. Its content was not rewritten.

Qualifying the repair with real rotations

A logrotate change is awkward to test. The honest test is a rotation, and forcing one on a live log to find out whether a hook contains a mistake puts at risk the records the change is meant to protect. The edited stanza was therefore qualified on a fixture before it was installed: a copy in an isolated directory with only the log path replaced, the hooks that act on the live Nginx disabled, its own logrotate state file, and synthetic records in place of real ones. Logrotate then performed two real rotations of the fixture. The second rotation shows what the first cannot: that the grant is applied again, that the generation rotated first comes through its delayed compression intact, and that the generation still kept uncompressed remains readable.

The assertions concerned the agent's account. It could open the new active file and the rotated one for reading, and attempts to open them for writing failed. Neither file carried any permission for other users. An unrelated log placed in the same fixture received no ACL, which shows that the hook touches one path. The older, compressed generation matched the records originally written to it. The complete system configuration was then checked with logrotate's debug mode, which reads the configuration and reports what it would do without changing any log. Two earlier qualification attempts were rejected and corrected before the one that counted; they are kept in the record as rejected attempts and were not reported as a successful installation. No rotation of the live log and no Nginx reopen was forced, and the process IDs recorded beforehand for the agent, the ingress and the application units were unchanged afterwards.

After the change the collector was read again. It was healthy with complete coverage, no unread bytes, no lag and no parse errors, and the gaps recorded during the interruption were still in its history. Because the tailer reads a new file from its first byte, the requests that Nginx had logged while the file could not be opened were still in it. By design they are read late and counted in the five-minute interval of their own timestamp, not at the time of the repair. A fleet read through MCP then showed every Linux server and every service group online, and each readiness probe configured for that server was healthy. One check lies outside that record. It does not include the first scheduled rotation of the live log under the new stanza, so the same collector read has to be repeated after that rotation.

What Hostwatch reports, and what stays with the operator

The agent returns the collector's state with each fleet health read: the health flag, the coverage state and its reason, the time of the last read and of the newest record, the start of the current unbroken period, unread and pending bytes, lag, the counts of parsed, dropped, oversized and unparseable records, and the gap count with the time of the latest gap. The control plane marks a server that has sites and a configured access log as degraded when the collector is unhealthy or its coverage is anything other than complete, and the service groups on that server inherit the state. The fleet view in the console shows the coverage state with unread bytes, lag, dropped records and parse errors, and the reason beneath them. A server without a configured access log is shown as not configured and its request figures as unavailable, not as zero.

A degraded server in this sense is a statement about observation. Missing read access is a coverage failure and does not establish that the applications behind Nginx failed, so collector continuity and the readiness of services are read separately. Detection that depends on the log is also affected: automatic protection actions require complete log coverage and are withheld without it. What deserves attention is a state that does not clear. Hostwatch does not reset the gap count after a repair or move late records to the time they were read, and an interval lost to truncation or to a generation that was compressed before it was read stays unknown.

The rest belongs to the operator. Request counts exist only for virtual hosts that include the Hostwatch JSON access-log format, and the format and the agent should be deployed together. The raw log needs its own rotation schedule, a short retention period and disk headroom, and it should stay out of general log shipping because of the query strings in it. For an agent that runs without root, the hook is a fragment that a root operator merges into the existing stanza by hand; the agent does not edit logrotate configuration or grant itself access. Before the edit, back up the configuration and save the numeric ACLs. Afterwards, validate with debug mode, qualify the stanza on a fixture and read the collector's state after the first real rotation. The guide to reading traffic, clients and mitigation describes how missing intervals appear in the traffic views.