1. Probe
The coordinator addresses a probe job to every machine that takes work now (online, not paused, inside its hours, lending model stages). Each worker starts the stage it would use — Metal on a Mac, WebGPU on a graphics card, the portable CPU kernels elsewhere — on a few layers, within the memory its owner lends, and returns two numbers per backend: the bytes it can hold for layers and their K/V cache, and the milliseconds one layer takes in a verify pass.
2. Plan
The planner splits the layers into consecutive runs, one per machine, so that every run fits its machine’s offer — memory first — and the slowest stage is as fast as possible. If the machines together cannot hold the model, it says so before anything starts: “the machines hold 22 of 28 layers”. Machines can also compute one stage together, each taking a share of every layer (tensor parallelism), when their links are fast.
3. Start the stages
Each planned machine receives a stage job. Its worker starts the stage on exactly the memory it lends, publishes the address it listens on, and keeps the job leased while it serves. A machine that has no copy of the model fetches only its own layers from the coordinator’s model server.
4. Answer
For each step the coordinator embeds the tokens, sends the hidden state through the stages in order (raw FP16 over TCP), and reads the logits from its own LM head. To hide the round trips it proposes several tokens at once — by looking them up in the conversation, or with a small draft model on its own GPU — and verifies all of them in one pass through the stages. Accepted tokens are exactly the ones greedy decoding would have chosen; a rejected one is replaced by the model’s own choice. Nothing is sampled or approximated.
Exactness
Every stage computes with fixed arithmetic: the CPU kernels give the same bits on any machine and thread count, and a GPU stage gives the bits of a pass of one token alone however many it verifies at once. A distributed run therefore matches the same placement computed in one process, bit for bit — measured on 63 of 63 logits rows for Qwen3-4B across two Linux machines. Different kinds of hardware may round differently; the answer you get is still the model’s greedy decoding, and the same every time for the same placement.
What goes through Hostwatch, and what does not
| Data | Path |
|---|---|
| Jobs and their small files | Through Hostwatch: probe inputs and results, stage inputs, where each stage listens. Up to 4 MiB each, 256 MiB per organization, removed after a week. |
| Settings and status | Through Hostwatch: each worker’s heartbeat every 15 seconds brings back its owner’s settings. |
| Hidden states | Directly between the coordinator and the stages. They encode the prompt as it passes through the layers. |
| Weights | From the coordinator’s model server (or the machine’s own copy) directly to the machine that computes them. |
Because the computation flows directly, the coordinator and the machines must reach each other: one local network, a VPN such as WireGuard or Tailscale, or open ports. A relay through Hostwatch for machines behind different routers is planned.
Speed, honestly
A pipeline does not make one answer faster than the fastest single machine that could hold the whole model; it makes models possible that no single machine can hold, and lets idle memory do useful work. Each extra hop costs a network round trip per verify pass, which speculation amortizes over several tokens. Machines on one memory bus (containers on one host) share its bandwidth, so they cannot decode faster together than one of them alone; separate machines can.