Repository files navigation

Honeydew ("Honey, do!") is a pluggable job queue and worker pool for Elixir, focused on at-least-once execution.

defmoduleMyWorkerdodefdo_a_thingdoIO.puts"doing a thing!"endend:ok=Honeydew.start_queue(:my_queue):ok=Honeydew.start_workers(:my_queue,MyWorker):do_a_thing|>Honeydew.async(:my_queue)# => "doing a thing!"

Isolation

• Jobs are run in isolated one-time-use processes.
• Optionally stores immutable state loaned to each worker (a database connection, for example).
• Initialized Worker

Strong Job Custody

• Jobs don't leave the queue until either they succeed, are explicitly abandoned or are moved to another queue.
• Workers are issued only one job at a time, no batching.
• If a worker crashes while processing a job, the job is reset and a "failure mode" (e.g. abandon, move, retry) is executed. (The default failure modeis to abandon the job.)
• Job Lifecycle

Clusterable Components

• Queues, workers and your enqueuing processes can exist anywhere in the BEAM cluster.
• Global Queues

Plugability

• Queues,workers,dispatch strategies,failure modes and success modes are all plugable with user modules.
• No forced dependency on external queue services.

Batteries Included

• Mnesia Queue, for in-memory/persistence and simple distribution scenarios. (default)
• Ecto Queue, to turn an Ecto schema into its own work queue, using your database.
• Fast In-Memory Queue, for fast processing of recreatable jobs without delay requirements.
• Can optionally heal the cluster after a disconnect or downed node when using aGlobal Queue.
• Delayed Jobs
• Exponential Retry, even works with Ecto queues!

Easy API

• Jobs are enqueued usingasync/3 and you can receive replies withyield/2, somewhat likeTask.
• API Overview
• Hex Docs

The Ecto Queue is designed to painlessly turn your Ecto schema into a queue, using your repo as the backing store.

• You don't need to explicitly enqueue jobs, that's handled for you (for example, sending a welcome email when a new User is inserted).
• Eliminates the possibility of your database and work queue becoming out of sync
• As the database is the queue, you don't need to run a separate queue node.
• You get all of the high-availability, consistency and distribution semantics of your chosen database.

Check out the includedexample project, and its README.

In your mix.exs file:

defpdepsdo[{:honeydew,"~> 1.5.0"}]end

If you're using the Mnesia queue (the default), you'll need tell your release system to include the:mnesia application, and you'll have to decide how you're going to create your on-disk schema files, which needs to be done while mnesia isnot running.

If you use mnesia outside of Honeydew, you'll want to use the:extra_applications configuration key in your mix.exs file, as well as manually creating your mnesia schema with:mnesia.create_schema(nodes) in an iex session in production:

defapplicationdo[extra_applications:[:mnesia]]end

Otherwise, if Honeydew is the only user of mnesia, you can let Honeydew manage it by simply using the:included_applications key instead.

defapplicationdo[included_applications:[:mnesia]]end

• Check out theexamples.
• Enqueue jobs withHoneydew.async/3, delay jobs by passingdelay_secs: <integer>.
• Receive responses withHoneydew.yield/2.
• Emit job progress withprogress/1
• Queue/Worker status withHoneydew.status/1
• Suspend and resume withHoneydew.suspend/1 andHoneydew.resume/1
• List jobs withHoneydew.filter/2
• Move jobs withHoneydew.move/2
• Cancel jobs withHoneydew.cancel/2

The rest of the README is broken out into slightly more digestiblesections.

Also, check out the README files included with each of theexamples.

It's worth keeping abreast with theCHANGELOG