Repository files navigation

• Scrape web sources and send emails based on some content in them
• Receive a URL, run youtube-dl and save the video locally
• Periodically download Roam database
• Some sort of Twitter list archiving?
• Scrape websites and populate a local database
• Fetch filled brokerage orders
• Interactive Slack/Discord bots
• Multi-stage ETL pipelines

This is somewhat out of date.

• Inputs
  • from POST to an endpoint
  • send events based on some periodic check that triggers when it sees a condition
  • trigger events unconditionally on a schedule
• Actions
  • Spawn docker containers (and/or Nomad jobs?)
  • Query HTTP endpoints
  • Run some local command
  • Link actions to accounts when required
  • Embedded JavaScript for actions
  • Return data for use by the next task in the chain
• Data Schemas
  • Each action can specify the types of data that it accepts
  • Duck typing for events
• State machines will take data from an event, modify it somehow, and pass it on
  • Embed JavaScript to write state machine logic
  • Persistent context for state machines
  • Optional schema input
  • Optional type checking on state machine return value
• Tasks
  • Trigger a task based on events
  • Run actions
  • Run state machines in response to events
  • Sequences - Tasks can be chained together and optionally pass information between them (file locations, etc.)
  • Tasks can clone themselves, and further inputs for that clone are routed to them. This will probably involve interaction with another process set up to be aware of how this works.
• Templates for events, actions, and state machines
• Extensive logging of events, actions, etc.

• Tasks made up of linked blocks that run as a DAG

• MVP of UI for editing tasks
• MVP of UI for editing actions and inputs.
• Periodic tasks
• Task logic can be written in Javascript.
• Implement last missing pieces of queue behavior.

• Add all missing tests
• Read-only Web UI with events timeline
• JavaScript execution as part of state machines, actions, and inputs.
• Set up foundations of serialized long-running tasks in V8.

• Events can be triggered by REST endpoint
• Tasks consists of one event that triggers one or more actions
• Actions can run local scripts
• Log everything that happens
  • Log inputs and actions to Postgres tables
  • Trigger Discord webhooks
• Events go into a queue for processing
• Actions are triggered from a queue