You signed in with another tab or window.Reload to refresh your session.You signed out in another tab or window.Reload to refresh your session.You switched accounts on another tab or window.Reload to refresh your session.Dismiss alert
<p>The<ahref="https://clojure.github.io/core.async/flow.html">flow</a> library enables a strict separation application logic from the deployment concerns of topology, execution, communication, lifecycle, monitoring and error handling.</p>
<p>You provide logic to flow in the form of<em>step-fns</em>, which are wrapped into running processes, executing in a loop. Flow manages the life cycle of the process and handles incoming and outgoing messages by putting or taking them on channels. Step-fns do not access channels directly or hold state, making them easy to test in isolation and reuse.</p>
<p>The describe arity must return a static description of the step-fn’s :params, :ins, and :outs. Each of these is a map of name (a keyword) to docstring.</p>
12
12
<p>For example, the describe arity might return this description for a simple step-fn:</p>
<p>The init arity is called once by the process to takes a set of args from the flow def (corresponding to the params returned from the describe arity) and returns the init state of the process.</p>
20
+
<h3><ahref="#transition-step-fn-state-transition-state"id="transition-step-fn-state-transition-state"></a>transition: (step-fn state transition) -> state’</h3>
21
+
<p>The transition arity is called any time the flow or process undergoes a lifecycle transition (::flow/start, ::flow/stop, ::flow/pause, ::flow/resume). The description arity takes the current state and returns an updated state to be used for subsequent calls.</p>
22
+
<p>The step-fn should use the transition arity to coordinate the creation, pausing, and shutdown of external resources in a process.</p>
23
+
<h3><ahref="#transform-step-fn-state-input-msg-state-out-id-msgs"id="transform-step-fn-state-input-msg-state-out-id-msgs"></a>transform: (step-fn state input msg) -> [state’ {out-id<ahref="msgs">msgs</a>}]</h3>
24
+
<p>The transform arity is called in a loop by the process for every message received on an input channel and returns a new state and a map of output cids to messages to return. The process will take care of sending these messages to the output channels. Output can be sent to none, any or all of the :outsenumerated, and/or an input named by a<ahref="pid inid">pid inid</a> tuple (e.g. for reply-to), and/or to the ::flow/report output. A step need not output at all (output or msgs can be empyt/nil), however an output<em>message</em> may never be nil (per core.async channels).</p>
25
+
<p>The step-fn may throw excepitons from any arity and they will be handled by flow. Exceptions thrown from the transition or transform arities, the exception will be logged on the flow’s :error-chan.</p>
<p>The process state is a map. It can contain any keys needed by the step-fn transition and transform arities. In addition, there are some flow-specific keys, described here.</p>
22
28
<p><code>::flow/pid</code> is added to the state by the process based on the name supplied in the flow def.</p>
23
29
<p><code>::flow/in-ports</code> and<code>::flow/out-ports</code> are maps of cid to external channel, optionally returned in the initial state from the init arity. The in-ports and out-ports are used to connect source and sink processes to external channels. These channels must be provided by the step-fn and returned in the init arity map, either by creating the channel or using a channel passed in via the flow def init args for the process. The flow does not manage the lifecycle of these channels.</p>
24
30
<p><code>::flow/input-filter</code>, a predicate of cid, can be returned in the state from any arity to indicate a filter on the process input channel read set. For example, a step-fn that is waiting for a response from multiple inputs might remove the channels that have already responded from the read-set until responses have been received from all.</p>
<p>The transition arity is called any time the flow or process undergoes a lifecycle transition (::flow/start, ::flow/stop, ::flow/pause, ::flow/resume). The description arity takes the current state and returns an updated state to be used for subsequent calls.</p>
27
-
<p>The step-fn should use the transition arity to coordinate the creation, pausing, and shutdown of external resources in a process.</p>
<p>The transform arity is called in a loop by the process for every message received on an input channel and returns a new state and a map of output cids to messages to return. The process will take care of sending these messages to the output channels. Output can be sent to none, any or all of the :outsenumerated, and/or an input named by a<ahref="pid inid">pid inid</a> tuple (e.g. for reply-to), and/or to the ::flow/report output. A step need not output at all (output or msgs can be empyt/nil), however an output<em>message</em> may never be nil (per core.async channels).</p>
30
-
<p>The step-fn may throw excepitons from any arity and they will be handled by flow. Exceptions thrown from the transition or transform arities, the exception will be logged on the flow’s :error-chan.</p>
<p>Some additional helpers exist to create step-fns from other forms:</p>
33
33
<ul>
34
-
<li><code>lift*->step</code> - given a fn f taking one arg and returning a collection of non-nil values, creates a step-fn as needed by a process, with one input and one output (named :in and :out), and no state</li>
34
+
<li><code>lift*->step</code> - given a fn f taking one arg and returning a collection of non-nil values, creates a step-fn as needed by a process launcher, with one input and one output (named :in and :out), and no state</li>
35
35
<li><code>lift1->step</code> - like<code>lift*->step</code> but for functions that return a single value (when<code>nil</code>, yield no output)</li>
36
36
<li><code>map->step</code> - given a map with keys<code>:describe</code>,<code>:init</code>,<code>:transition</code>,<code>:transform</code> corresponding to the arities above, create a step-fn.</li>
37
37
</ul>
38
-
<h3><ahref="#creating-a-process"id="creating-a-process"></a>Creating a process</h3>
39
-
<p>Processes can be created using the<ahref="https://clojure.github.io/core.async/clojure.core.async.flow.html#var-process">process</a> function, which takes a step-fn, and an option map with keys:</p>
38
+
<h3><ahref="#creating-a-process-launcher"id="creating-a-process-launcher"></a>Creating a process launcher</h3>
39
+
<p>Process launchers can be created using the<ahref="https://clojure.github.io/core.async/clojure.core.async.flow.html#var-process">process</a> function, which takes a step-fn, and an option map with keys:</p>
40
40
<ul>
41
41
<li><code>::workload</code> - one of<code>:mixed</code>,<code>:io</code> or<code>:compute</code></li>
42
42
<li><code>:compute-timeout-ms</code> - if :workload is :compute, this timeout (default 5000 msec) will be used when getting the return from the future - see below</li>
43
43
</ul>
44
-
<p>A :workload supplied as an option to process will override any :workload returned by the :describe fn of the process. If neither are provded the default is :mixed.</p>
44
+
<p>A :workload supplied as an option to<code>process</code> will override any :workload returned by the :describe fn of the process launcher. If neither are provded the default is :mixed.</p>
45
45
<p>In the :workload context of :mixed or :io, this dictates the type of thread in which the process loop will run,<em>including its calls to transform</em>.</p>
46
46
<p>When :io is specified, transform should not do extensive computation.</p>
47
47
<p>When :compute is specified, each call to transform will be run in a separate thread. The process loop will run in an :io context (since it no longer directly calls transform, all it does is I/O) and it will submit transform to the :compute executor then await (blocking, for compute-timeout-ms) the completion of the future returned by the executor. If the future times out it will be reported on ::flow/error.</p>
48
48
<p>When :compute is specified transform must not block!</p>
49
+
<p>Note that process launchers are defined by the<ahref="https://clojure.github.io/core.async/clojure.core.async.flow.spi.html#var-ProcLauncher">ProcLauncher</a> protocol. While you will typically use<code>process</code> to create a process launcher, advanced uses may also implement the protocol directly.</p>
<p>Because the step-fn is called in a loop, it is a good practice to define the step-fn in a var and use the var (<code>#'the-fn</code>) instead of the function value itself (<code>the-fn</code>). This practice supports interactive development by allowing the var to be rebound from the repl while the flow is running.</p>
<p>The step-fns are how you supply code for each process in the flow. The other thing you must supply is the flow configuration that ties together theprocs and the connections between them.</p>
53
+
<p>The step-fns are how you supply code for each process in the flow. The other thing you must supply is the flow configuration that ties together theproc launchers and the connections between them.</p>
53
54
<p>This flow definition is supplied to the<ahref="https://clojure.github.io/core.async/clojure.core.async.flow.html#var-create-flow">create-flow</a> function and consists of a map with<code>:procs</code>,<code>:conns</code>, and optionally some workflow executors.</p>
54
-
<p>The<code>:procs</code> is a map of pid -> proc-def. The proc-def is a map with<code>:proc</code> (the processfunction), the<code>:args</code> (passed to the init arity of the step-fn), and the<code>:chan-opts</code> which can be used to specify channel properties.</p>
55
+
<p>The<code>:procs</code> is a map of pid -> proc-def. The proc-def is a map with<code>:proc</code> (the processlauncher), the<code>:args</code> (passed to the init arity of the step-fn), and the<code>:chan-opts</code> which can be used to specify channel properties.</p>
55
56
<p>The<code>:conns</code> is a collection of<code>[[from-pid outid] [to-pid inid]]</code> tuples. Inputs and outputs support multiple connections. When an output is connected multiple times, every connection will get every message, per<code>core.async/mult</code>.</p>
56
57
<p>An example flow definition might look like this for a flow with two procs where the in-chan and out-chan are being passed through the source and sink args:</p>
<li><ahref="https://clojure.github.io/core.async/clojure.core.async.flow.html#var-pause-proc">pause-proc</a> - Pauses a single proc</li>
73
74
<li><ahref="https://clojure.github.io/core.async/clojure.core.async.flow.html#var-resume-proc">resume-proc</a> - Resumes a single proc</li>
74
75
</ul>
75
-
<p>You can also use these functions to ping the running processesare return their current state and status:</p>
76
+
<p>You can also use these functions to ping the running processesand return their current state and status:</p>
76
77
<ul>
77
78
<li><ahref="https://clojure.github.io/core.async/clojure.core.async.flow.html#var-ping">ping</a> - Pings all procs and returns a map of their status</li>
78
79
<li><ahref="https://clojure.github.io/core.async/clojure.core.async.flow.html#var-ping-proc">ping-proc</a> - Pings a single proce by pid and returns a map of status</li>