blobhub-worker instance ls | show | stop inventory every worker on this machine and let you stop one.
None of the three resolves a credential or calls the API — all three read the filesystem directly, which
is what makes them usable when the API is the thing you suspect.
Synopsis
show and stop both take a required <selector> — see Selector grammar below.
What counts as an instance
Discovery walks~/.blobhub-worker/profiles/* and ~/.blobhub-worker/keys/* — every state
tree on the machine, not a scan for lock files. A clean shutdown unlinks
instance.yaml, so a lock-only scan would be blind to every worker that stopped properly, and to the
config, cursors and thread state it left behind — exactly what “what was it doing?” is asking about. A
tree with no lock at all is reported, not omitted.
Each tree is in exactly one of five states:
stale (pid reused) exists because pids get recycled: a lock naming a pid that is alive but belongs to
some other program is not a worker to trust, and it is not the same thing as stale, where the pid is
simply gone. See Why stop can refuse for how that distinction is made.
running is the default when the check can’t be sure, not a guarantee. The demotion to stale (pid reused) only fires on a positive “this is not a worker” — where ps can’t answer at all (missing,
non-zero, timed out), the tree still reports running, because treating an unverifiable pid as not
running risks a second worker landing on the same tree and corrupting its cursors. The only place that
distinction surfaces is stop --json’s pid_verified field (true/false/null) — see
instance stop below.
Selector grammar
One argument, matched against every discovered tree and lock:
An instance id may be given as a unique prefix, with or without the
ins_ prefix. Matches are unioned:
none of them → INSTANCE_NOT_FOUND; more than one → INSTANCE_AMBIGUOUS, naming every candidate. A
profile name that happens to prefix a key digest collides into INSTANCE_AMBIGUOUS too — disambiguate
with key:.
The selector is required on both show and stop; neither defaults to “the only one running”, because
that would make the target of a stop depend on whatever else happens to be running at the time.
instance ls
--running filters to running only. With no state trees
at all, ls prints no worker state trees on this machine instead of an empty table.
The TREE column carries two optional annotations, both derived from the credential store and dropped
silently if it is missing or unreadable:
key:<digest> (=name)— a key tree whose digest matches a stored profile’s key. This is the everyday shape of running withBLOBHUB_API_KEYexported alongside a stored profile of the same key: the worker runs from the key tree instead of the profile’s, and this annotation is the way to notice.<profile> (orphan)— a profile tree with no stored profile of that name. The credential was deleted; the state tree was not.
show or stop as key:<digest>.
--json prints {"instances": [...]}, one object per row with instance_id, tree, kind, label,
path, pid, version, uptime_seconds, state, and state_label.
instance show
Every source is read defensively: an unreadable file degrades that block to a note (or an
unreadable
row for one section or thread) rather than failing the whole command. A thread’s alias, when not recorded
on the object itself, is rebuilt from its path under the job’s threads/ directory.
show works on a stopped tree, and that is the point of it — not a tolerated edge case. The header
reads Instance: - followed by the bare state (stopped, stale, …) instead of running (up …), and
the Process: line is omitted since there is no pid to report. Everything else — Tree, User, Config,
Sections, Threads — is the state the worker left behind, which is otherwise the only way to answer “what
was it doing when it died” for a headless worker.
A refused section (attachment.status: refused) prints its error code in place of the attach time — a
section that never attached is the most common reason a running worker looks idle.
--json prints the same dict this text rendering is built from: the top-level instance/process fields,
plus tree, user, and config objects and sections/threads arrays.
instance stop
SIGTERM reaches the same handler start installs for Ctrl-C — sections drain, in-flight turns finish,
the lock is released. See start → Shutdown. This holds identically for a
--tui worker: the drain runs unbounded, the same as headless, so --timeout means the same thing in
both modes — how long stop itself waits before giving up (or escalating with --force), not a bound on
the worker’s own drain. (Pressing q inside the dashboard is a separate, deliberately bounded path — see
start → Shutdown — but that only matters at the keyboard, not to stop.)
Outcomes by state, all five handled deliberately:
On timeout: without
--force, INSTANCE_STOP_TIMEOUT and the worker is left running. With --force,
SIGKILL, then up to 5 more seconds, then INSTANCE_STOP_TIMEOUT if it is somehow still alive.
A --force stop leaves the lock behind. SIGKILL never runs the process’s cleanup, so
instance.yaml outlives it. That is safe — the pid is dead, so the next start reclaims the tree — but
instance ls will show a stale row until then, and the command says so at the time rather than leaving
it to be discovered.
If the process has already exited between the state check and the signal, that counts as success. Any
other signalling failure — a PermissionError for a worker owned by another user is the realistic one —
is reported as INSTANCE_SIGNAL_FAILED.
--json prints the same payload stop returns, for example after a SIGTERM:
pid_verified is tri-state — true, false, or null when ps could not answer — and rides along on
the sigterm and sigkill outcomes only; the stale-lock-removal outcome has no pid to verify and omits
the field.
Why stop can refuse
instance.yaml records a pid, and the operating system recycles pids. Before signalling, stop asks
ps what that process actually is, judging argv[0] — the console script, or a Python interpreter
running -m blobhub_worker — never the whole command line. A pid that is alive but positively not a
worker is stale (pid reused), and stop refuses it:
ps cannot answer at all (absent, non-zero, timed out), stop proceeds anyway and prints a warning
first. That is what plain kill does unconditionally, and a stop command that stops working because a
system tool is missing would be worse than one that occasionally signals without having verified.
Errors
These are raised by
instance show/instance stop, not by start’s preflight — full descriptions
alongside the process-level codes: Reference → Instance command
codes.