Which service owns that process? Reading CPU and memory on a shared Linux server

A process list shows what is running

On a server that runs one application, a process list is easy to read. The busiest process belongs to the application, and whatever it uses is the application's cost. A server that runs several applications is harder. The list may show six processes called node, three called python and a dozen database workers, and nothing in those names says which site or component each one belongs to. A process name is a short label that the program sets for itself, and two unrelated applications built on the same runtime set the same one. During an incident the operator needs to know which service to inspect, whom to tell and which release to suspect, and the list alone does not say.

This article uses a few terms throughout. The owner of a process is the service responsible for it: an application component, a shared piece of infrastructure such as a web server, or the operating system itself. Ownership evidence is the set of observed and recorded facts that connects a process to its owner. A snapshot is one reading of the process list at one moment. The sections below explain what makes a process identifiable, which ownership evidence can be trusted, how to read CPU and memory figures once the owner is known, and what to do when no owner can be established. The last two sections describe how the Hostwatch server task manager presents this evidence and where its limits are.

A PID needs a boot ID and a start time

A process ID, or PID, is a number the kernel assigns when a process starts. The kernel takes PIDs in sequence from a limited range and begins again at the low end when the range is used up, so a number that belonged to one process is later given to another. On a server that starts many short commands, such as health checks, scheduled jobs and build steps, this happens routinely. A PID therefore identifies a process only while that process is alive. A note saying that a certain PID held a lot of memory last night describes whatever had that number at the time, and the process that has it today may be unrelated.

The risky case is the short gap between reading and acting. A tool lists the processes and then reads more detail about each one. In between, a process exits and its number is given to a new one. The detail now describes the newcomer, while the name and the figures describe the process that left. If the second read was the ownership lookup, the tool attributes one service's memory to another service. If the next step is a signal, it reaches a process nobody chose. Linux added process file descriptors, known as pidfds, for this reason: they let a program hold a reference to one specific process instead of a reusable number.

A reliable process identity combines three values. The start time, which the kernel records in /proc/PID/stat as a count of clock ticks since boot, separates two processes that held the same PID at different times. That count begins again at every boot, so the identity also needs the boot ID, a random identifier the kernel generates each time it starts. Boot ID, PID and start time together name one process for practical purposes. A tool that samples processes should read this identity before it samples, read it again afterwards and drop any row where the two readings differ. An operator can keep the same habit by writing the start time beside every PID in an incident note; the ps command prints it in its lstart column.

Cgroups, systemd units and container IDs as ownership evidence

Two obvious clues are weak. The process name can be set and changed by the program itself. The parent PID looks more promising and breaks easily: when a parent exits, its children are adopted by PID 1 or by a nearer process that has asked to adopt orphans, and many daemons detach from their parent on purpose. A worker whose parent is PID 1 has lost the trail back to whatever started it.

A control group, or cgroup, is a better record. The kernel places every process in a cgroup, a child process starts in its parent's cgroup, and moving a process elsewhere requires write access to the destination, which an unprivileged process does not have outside a part of the tree that was handed to it. The kernel keeps this membership, and it survives the death of the parent. On a systemd host, the processes of each service live in a cgroup named after the service's unit, by default in the system slice, so the path in /proc/PID/cgroup names the unit that manages the process. Container runtimes do the same for containers: the cgroup of a containerized process normally carries the full container ID, 64 hexadecimal characters long. Running systemctl status with a PID prints the unit that contains that process.

The cgroup answers half of the question. It says which unit or container the process is in. Which application that unit or container belongs to is something the kernel cannot know. That second link has to be recorded by someone who does know: this unit is the API of one product, this container or compose project serves another site. We call such a record a binding. Trustworthy ownership needs both halves, an observed cgroup membership and a binding that matches it exactly. A binding with no observed process is only a statement of intent. An observed unit with no binding tells the operator where to look and leaves the application unknown.

A delegated cgroup must not pass for a host unit

The host's service manager does not write the whole cgroup tree. systemd can delegate a part of the tree to another manager, which then creates and names cgroups inside that part as it likes. Every logged-in user normally has a user manager with a delegated subtree, and a user can define a personal unit with any name, including nginx.service or the name of a database unit. A container that runs its own service manager can build a complete tree of slices and services inside its own scope. The cgroup paths of such processes end in familiar unit names.

A tool that matches on the last component of the path, or searches the path for a known unit name, attributes those processes to the host unit of the same name. An unprivileged account's program then appears in the process list as shared infrastructure, with the standing that label carries. A real problem inside a container can be charged to the host's web server in the same way. The defense is to read the position as well as the name. A host service normally sits directly in the host's system slice. A cgroup with the same name deeper down, inside a user's subtree or a container's scope, belongs to that user or that container and should be attributed there or left without an owner.

Container evidence needs the same strictness. Container names are reused by successive instances, and the short ID that command-line tools print is only the beginning of the full one. The match that holds up is the full ID from the process's cgroup against the full ID reported by the runtime, together with a binding that names that container or its compose project for a service. A name that looks similar or a truncated ID is not enough, and a process matched that loosely should be shown with an unknown owner.

CPU percentages when one core is 100%

Process tools commonly count one logical CPU core as 100%. A process with one busy thread shows about 100%, and a process with several busy threads shows more, up to 100% multiplied by the number of cores. The same figure means different things on different machines: 150% is three quarters of a two-core server and less than a tenth of a sixteen-core one. Host charts usually follow the other convention, in which 100% is the whole machine. Comparing a process figure with a host figure means converting one of them first. A reader who skips the conversion is alarmed by 180% on a large machine, or overlooks a single-threaded program that has reached its own ceiling of 100% while the host chart shows plenty of idle capacity.

A percentage is also a rate, and a rate needs two readings. The kernel keeps a running total of the CPU time each process has used. A tool reads the total twice and divides the difference by the time between the readings, so the interval decides what the number says. A one-second sample catches a spike that a five-minute average hides. The ps command divides the total CPU time by the whole lifetime of the process, so a worker that idled for a week and has been spinning for the last minute shows almost nothing. A process that could not be read twice has no rate, and a tool should show that as unknown. Printing zero would tell the operator that the process is quiet, which nobody measured.

The figure for a process also differs from the figure for its service. A service with twenty workers at 8% each uses more CPU than a service with one worker at 90%, and a list sorted by process puts the second service first. Adding up the processes of one unit gives a fair estimate for that sample. Whether a CPU limit is holding the service back is a separate question, answered by the quota and throttling counters of its cgroup. Our article on reading cgroup v2 limits covers those denominators, so we leave them out here.

Resident memory cannot be summed into used RAM

Resident set size, or RSS, is the part of a process's memory that is currently held in physical RAM. It includes pages the process shares with others. A shared library is loaded once and counted in the RSS of every program that uses it. Workers forked from one parent keep sharing the pages that have not been modified since the fork. A database that keeps one large buffer for all its worker processes shows that buffer, or the part each worker has touched, in the RSS of each worker. Ten workers that each report two gigabytes may occupy little more than two gigabytes between them.

Adding RSS across processes therefore counts shared pages several times, and the total can exceed the RAM installed in the machine. The sum also leaves things out: file cache that no process has mapped, the kernel's own structures and memory held by temporary filesystems. It is neither an upper nor a lower estimate of memory in use. The machine-wide figures come from the kernel's own accounting: total and free memory, its estimate of the memory available for new work, and swap. Swap is space on disk and belongs in its own column. For one service, the memory counter of its cgroup is a better starting point, because each page is charged to a single cgroup; it also includes file cache the service has read in. The kernel can report a proportional figure per process that divides shared pages among their users, but it is costly to read and requires privileged access to the processes of other accounts.

RSS remains useful for comparison. It ranks processes, shows one process growing between two snapshots and exposes a difference between two servers that run the same build. It supports the statement that a process is large and leaves open how much memory the server would get back if that process stopped. Questions about a service's memory limit, about pressure and about an out-of-memory kill need the cgroup and kernel evidence described in our article on container memory incidents.

Kernel threads and PID 1 belong to the operating system

Some rows in a process list are not programs in the usual sense. Kernel threads are tasks the kernel runs for its own work, such as reclaiming memory, writing out modified pages and finishing deferred interrupt handling. They have no user memory, so they show no resident size, and ps prints their names in square brackets. Their CPU time is work done on behalf of something else. A busy memory-reclaim thread points to memory pressure, and busy worker threads often point to heavy disk or network activity. The useful response is to find the workload that causes the work. A kernel thread cannot be stopped from user space, and no application owns it.

The brackets are a display convention, and a program can arrange to be displayed the same way. Software that wants to hide has used that trick, which makes the name a poor test here as well. The kernel marks its own threads with a flag in each task's flags field, which /proc/PID/stat exposes, and a classifier should rely on that flag. PID 1 is the other special case. It is the first process, the service manager on a systemd host and the ancestor that adopts orphans, and the kernel treats its exit as fatal for the whole system. Kernel threads and PID 1 belong in a protected category of their own, so that a tool never offers them as candidates for maintenance.

When the owner is unknown

An unknown owner is a statement about the evidence, and most processes in that state have ordinary explanations. Someone started a command by hand in an SSH session and it is still running. A scheduled job launched a script. A package installed its own daemon. A service was deployed before anyone recorded which application it belongs to. The absence of a binding gives no grounds for calling a process unused, and a tool that labels such a process as safe to remove is guessing.

The investigation can be done without changing anything. Record the process identity first, with the time of the observation. Read the cgroup path, which shows whether the process sits in a host unit, a login session, a user manager or a container. Ask systemd which unit contains it and read that unit's definition. Compare the start time with deployments, logins and scheduled jobs. Check the account it runs as, the executable behind it and the sockets it listens on. An executable that has been deleted from disk is normal after a package upgrade and still deserves a second look. Command lines and environments often contain credentials, so copy into the note only what it needs.

Stopping the process first is tempting and usually a mistake. The kill destroys the state that would have explained it, a supervisor may start it again within seconds, and something that looked idle may be a tunnel, a backup or a migration that another system depends on. The investigation should end in one of three ways. The owner is found and recorded, so that the next snapshot attributes the process. The process turns out to be unwanted and is removed through whatever manages it, by the person responsible for that service. Or the evidence suggests an intrusion, which makes it a security incident with its own procedure, where preserving the state of the machine matters more than ending one process.

How the Hostwatch task manager presents the evidence

In Hostwatch, the task manager opens from a server's Overview. Selecting CPU shows the processes sorted by their observed CPU use. Selecting Memory also explains physical RAM, swap and the RSS inventory, and keeps them as separate figures. Refresh takes a new snapshot from the server. The list can be searched by name, PID, service unit or recorded service, filtered by role and sorted by CPU or RSS. Expanding a row shows its identity, ownership evidence, service bindings, dependency coverage and maintenance guidance. One core counts as 100%, and a dash in the CPU column means the value was not measured.

Every row has one of four roles. Required system marks PID 1 and kernel threads; kernel threads are recognized by the kernel's own flag. Shared infrastructure is used for a configured shared Nginx, and the row names the services recorded as its consumers. That list is limited to installed bindings and is no proof that nothing else uses the web server. Application service requires both halves described earlier: the process is observed in a host service cgroup or carries a full container ID, and that unit, container or compose project is registered for a service. Unattributed means those facts have not been established. A cgroup in a delegated user subtree is not accepted as a host unit. On Linux, each row is bound to boot ID, PID and start time. The identity is checked before and after sampling and again after the ownership read, and a process that exited or changed identity in between is left out.

For a process in a systemd unit, the expanded row also reports the unit's Requires, Wants, BindsTo and PartOf relationships and the service units that hold those relationships to it. Start ordering is kept apart, since a unit that merely starts after another does not need it. Relationships that could not be read are shown as unknown. These are relationships between systemd units, which is less than a map of everything an application depends on. The same snapshot is available to an authorized MCP client through the read operation host_processes, called with the node's ID. Clients receive process names, PID and start identity, CPU, RSS and ownership evidence. Command arguments, environment values, working directories and executable paths are omitted from the response.

The observer, coverage and limits

The measurements come from a separate program, the SweepLoom observer, which the Linux installer places beside the agent together with the observer's own license and source archive. Existing installations receive it through an explicit upgrade. The agent runs the observer from a fixed path with one fixed argument, so no request can add to its command line. A manifest next to it pins the observer's SHA-256 and source identity, and both files must be regular files owned by root that group and others cannot write. Without that rule, any account able to replace the file could have its code run whenever someone opened the task manager. A run is limited to four seconds and 8 MiB of output, so a stuck or noisy observer cannot hold up the agent. When the observer is missing or its installation is invalid, the agent reads RSS natively and reports CPU as unknown.

A snapshot returns at most 500 rows, taken in order of RSS, together with the number of processes that were readable and the number returned. RSS totals by process name cover every readable process. The console shows the first part of the list and loads more rows on request. Its coverage section states how many visible processes had their CPU sampled and which limitations apply. Three limits matter in practice. A short-lived process, or one that the agent is not permitted to read, can be absent, so the list never proves that nothing else is running. Because rows are cut by memory, on a server with more than 500 readable processes a process with little or no resident memory, including a busy kernel thread, can fall outside the returned rows. A short sample shows what a process used just now and does not establish a service's peak demand.

The task manager has no kill or delete control, and the agent has no endpoint that removes a process. A row's role and guidance explain what was observed. Stopping a service remains a decision for its owner, made through the service's own controls. Service recovery is a separate feature: an independent watchdog can restart installed targets within a bounded policy, as our article on automatic recovery and the guide describe. Attribution is only as complete as the bindings an operator has installed. Hostwatch does not infer ownership from process names, so an unregistered service stays Unattributed until someone records it. Since the response leaves out arguments and paths, investigating such a process further means reading them on the server with administrative access.