- Two runs act on the same state at once.
- An event lands while a run is busy and is never seen.
- A run’s own writes wake it again.
- A burst of writes starts a burst of runs.
- A cursor kept somewhere drifts from the truth.
What a run can count on
- It is woken, not fed. An event says something changed here. A run reads the current state and acts on it;
the batch tells it where to look (which aliases, items and executions) and is not the state. Two hundred writes to
one alias are one entry with
distinct_by. - One lane runs one run at a time.
concurrencydivides a listener’s events in a session into lanes:serialhas one lane andkeyedone per key value, each with at most one run in flight.parallelruns up tomax_parallelat once and waits when its slots are full (below). - A matching event reaches a run at least once, unless its batch is consumed without one. Delivery is at least
once: a rare event reaches two runs, an event that commits late
can arrive in a later batch than events with larger ids, and batches are not globally ordered. A declared
retrydelivers a failed run’s batch again (below). A listener takes effect up to about ten seconds after it is saved, and events routed before then are not delivered to it. A batch that isskipped(depth,session_not_open,unmatchedorthrottled), ends inaccess_error, or comes due while the listener is switched off in the session is consumed and never runs; every one but the switched-off batch leaves an outcome that says why. A reconcile catches up on the next batch. - It keeps no cursor. The platform holds the listener’s position in the stream. A run that wants more than its
batch reads the stream itself with
list_session_events;batch.first_idandbatch.last_idare ids it can pass asafterorbefore. - It is not woken by itself. Everything the run writes, and the run’s own
execution.ended, carries the listener assource.listenerand never matches it. - It is bounded. A burst becomes one batch per window, up to
max_eventsentries; a chain of listeners stops atworkflow.listener_chain_depth; fires are throttled atworkflow.listener_fires_per_minute. See Limits.
Concurrency
Events that match while their lane is busy wait in its next batch: under
serial and keyed while the lane’s one run
is in flight, under parallel while all max_parallel slots are taken, and under keyed also while max_parallel
keys are running. A workflow run holds its slot until its execution ends (completed, failed or stopped), and the
waiting batch is claimed once a slot is free. A thread_item or object action frees its slot as soon as its
outcome is recorded. An event the key path does not reach falls in one shared lane.
What you write
A reconcile: given the batch and what it reads now, write the state that should follow. Running it twice on the same state changes nothing, which is what makes at-least-once delivery, a failed run and a retried batch safe. Not needed, and not to be written:- a cursor, since the platform holds the position;
- a lease, since a
serialorkeyedlane never has two runs in flight; - a sleep-and-poll loop, since the listener wakes the run;
- an item posted to remember that an event was seen, since a repeated event is reconciled to the same result.
When a run fails
Its batch is consumed unless the listener declaresretry. With retry, a run whose execution ends failed has its
batch delivered again, up to attempts times. The retry after a run whose batch.retry is n waits min(delay_seconds x 2^n, 900) seconds: the first retry, after the first delivery (n = 0), waits delay_seconds, the second twice that.
Each repeat is a batch of its own with its own outcome and receipt, carrying retry, and the failed run’s slot is held
through the wait. Nothing else is retried: not a completed or stopped run, and not a batch that started no run. The
fields and the limits are in the listener body. The failure shows three
ways:
- The execution’s status.
list_executions {session_id, listener}lists the runs a listener started. A run that was started and then failed has the outcomesuccess, since the action was accepted; its own status is the result. - The outcome on the listener’s state in the session, read with
list_session_listeners:invocation_errorwhen the action or its template failed,refusedwhen a limit refused it. A retry’s outcome carriesretry, which says which repeat it is (outcomes). - A receipt, when the listener declares one: a thread item with the outcome’s
status, posted forsuccess,invocation_error,refusedandthrottled.
One writer per alias
The guarantees are per listener. Two listeners, or a listener and a person, may still write the same thing. The rule an application follows is one writer per alias: each listener owns its key space, and a person’s input arrives as items on a thread rather than as overwrites of a listener’s objects. Where two writers must share one graph element, the graph’sif_rev guards each write (Component Code). A session
object is replaced by each write to it.
A clock
Time is the one thing events cannot say: “nobody answered for thirty minutes” is the absence of an event. A reactive application gets a clock from a schedule that writes an object, withinvocation_target_type: "object", every fifteen minutes:
ops/clock, in its filter, beside everything else it reacts to:
object.written on the stream. It enters the same batch and run slot as every other event, so it
never starts a run beside one already in flight. A schedule that started the workflow directly would bypass the slot
and need a lease back. A schedule’s strings are literal, so the clock writes the same value on every tick. A periodic
sweep is the same schedule pointed at a sweep alias, such as Missions’ app/missions/sweep.
Use cases
A time-based rule, such as a cell idle past its deadline, needs no listener of its own: the next run that reaches the
cell decides it, whatever started that run, and a sweep request starts one on demand. No second listener catches a run’s
end either, because the application starts no workflow but its own, and a run’s own events never match the listener that
started it. The orchestrator’s batch stays lossless, with no
distinct_by: keeping only the newest event per alias
would drop the cell ids an earlier graph.changed in the window named. Its retry: {attempts: 2} means a failed run’s
batch comes again after 30 s, then after 60 s; a batch whose third run fails is consumed, and an operator can request a
sweep.
