Monitoring#

What to collect from a swarm and what to read in it. Commands and outputs here follow the conventions of the operations guide.

Before you start#

• The agents' envelopes: the wrapper's files, one per agent, or the client's transcripts.

Summarise the envelopes#

Every command and every tool result is an envelope, so a log of envelopes is the monitoring data. A command prints its envelope; the MCP server keeps none of its answers, so what agents' calls did is read from the files that the wrapper keeps, one per agent, or from the client's transcripts. A line of those files holds its envelope twice, as an object and, escaped, as text; these commands read the object, which has no escaped quotes, and need no jq. In the wrapper's directory:

cd ~/.local/state/swarmops/envelopes
for f in *.jsonl; do
  printf '%-22s calls %s, failed %s, recovered %s, warnings %s\n' "${f%.jsonl}" \
    "$(grep -c '"structuredContent"' "$f")" "$(grep -c '"isError":true' "$f")" \
    "$(grep -c '"recovered"' "$f")" "$(grep -c '"warnings":' "$f")"
done
cat *.jsonl | grep -o '"code":"E_[A-Z_]*"' | sort | uniq -c | sort -rn
cat *.jsonl | grep -o '"elapsed_ms":[0-9]*' | cut -d: -f2 | sort -n | tail -3

agent_worker-1         calls 7, failed 1, recovered 0, warnings 0
agent_worker-2         calls 6, failed 2, recovered 0, warnings 0
      1 "code":"E_PATH_NOT_FOUND"
      1 "code":"E_CONFLICT"
      1 "code":"E_CAS_MISMATCH"
19
23
31

The first command counts, for each agent, the calls answered, the failed ones, the writes that repaired what a dead writer left (meta.recovered) and the results that carry warnings; the second counts the failures by code; the third lists the three slowest calls by meta.elapsed_ms.

What to read in them#

Errors by code#

Expected now and then, and handled by the agent: E_CONFLICT (exit 3), E_CAS_MISMATCH (re-read, then land). A rising count of E_BUSY or E_CAS_RETRY_EXHAUSTED (exit 4) means more contention than the lock serves: too many writers, a slow disk, or something holding the lock (a backup). Act at once on E_IO, E_IO_TRANSIENT (their details name the file operation and the store file), E_STORE_CORRUPT and E_INTERNAL, and on an MCP server that exits.

meta.recovered#

meta.recovered on a write: what it repaired of a writer that died mid-write. Agents get killed, so it appears now and then; often, it means something keeps killing writers.

warnings#

warnings in a result: what an operation that succeeded still wants you to know, such as a commit that is durable but whose follow-up work failed (the next write finishes it), or an export that wrote commits anew because their history was imported with a loss.

meta.elapsed_ms#

meta.elapsed_ms of commits and lands: the lock's wait is in it.

verify --deep#

problems and findings empty, on a schedule and on every backup.

Disk#

Free space on the store's filesystem, and the size of objects/, objects/tmp/ and meta/derived/, which only grow. meta/derived/ grows with the exports: the first export of a history writes one file of derived git ids per object it exports, about as many files as objects/ holds, each taking a whole filesystem block (4 KiB on ext4) for its 72 or 73 bytes. At 470 commits the store took 57 MB in 8,854 files after the import, and 92 MB in 17,232 files after the first export.

The store keeps one file per object and one per derived value, so count its files and the filesystem's inodes as well: ext4 fixes the number of inodes when the filesystem is made, and a store of small files can use them up before it uses the space. Making room says what may go.

find "$SWARMOPS_REPO" -type f | wc -l
df --output=iused,iavail,ipcent "$SWARMOPS_REPO"

91
  IUsed     IFree IUse%
3021323 241161717    2%

The git side#

Each export's exported, branch_moved and rewritten (0 unless the history was imported with a loss), any E_GIT_IMPORT_DIVERGED or E_GIT_EXPORT_DIVERGED (someone wrote the trunk, or an export waits for its review) or E_GIT_WRITE, how long the review branch has been ahead of the trunk (git -C "$GIT_SIDE" log --oneline stable..review lists what waits), and the CI runs on the review branch and the deployment runs on the trunk.

op log#

Who moved which ref, for moves no one expected.