Repository files navigation

• Foreword
• Install
• Upgrading
• Usage and Examples
  • ECMAScript modules (ESM)
  • CommonJS (CJS)
• Instance Options
• Job Options
• Job Interval and Timeout Values
• Listening for events
• Custom error/message handling
• Cancellation, Retries, Stalled Jobs, and Graceful Reloading
• Interval, Timeout, Date, and Cron Validation
• Writing jobs with Promises and async-await
• Callbacks, Done, and Completion States
• Long-running jobs
• Complex timeouts and intervals
• Custom Worker Options
• Using functions for jobs
• Typescript and Usage with Bundlers
• Concurrency
• Plugins
  • Available Plugins
  • Creating plugins for Bree
• Real-world usage
• Contributors
• License

Bree was created to give you fine-grained control with simplicity, and has built-in support for workers, sandboxed processes, graceful reloading, cron jobs, dates, human-friendly time representations, and much more.

We recommend you to query a persistent database in your jobs, to prevent specific operations from running more than once.

Bree does not force you to use an additional database layer ofRedis orMongoDB to manage job state.

In doing so, you should manage boolean job states yourself using queries. For instance, if you have to send a welcome email to users, only send a welcome email to users that do not have a Date value set yet forwelcome_email_sent_at.

npm:

npm install bree

yarn:

yarn add bree

To see details about upgrading from the last major version, please seeUPGRADING.md.

IMPORTANT:Bree v9.0.0 has several breaking changes, please seeUPGRADING.md for more insight.

NOTE:Bree v6.5.0 is the last version to support Node v10 and browsers.

The example below assumes that you have a directoryjobs in the root of the directory from which you run this example. For example, if the example below is at/path/to/script.js, then/path/to/jobs/ must also exist as a directory. If you wish to disable this feature, then passroot: false as an option.

Inside thisjobs directory are individual scripts which are run usingWorkers per optional timeouts, and additionally, an optional interval or cron expression. The example below contains comments, which help to clarify how this works.

The optionjobs passed to a new instance ofBree (as shown below) is an Array. It contains values which can either be a String (name of a job in thejobs directory, which is run on boot) OR it can be an Object withname,path,timeout, andinterval properties. If you do not supply apath, then the path is created using the root directory (defaults tojobs) in combination with thename. If you do not supply values fortimeout and/norinterval, then these values are defaulted to0 (which is the default for both, seeindex.js for more insight into configurable default options).

We have also documented allInstance Options andJob Options in this README below. Be sure to read those sections so you have a complete understanding of how Bree works.

// app.mjsimportBreefrom'bree';constbree=newBree({// ... (see below) ...});// top-level await supported in Node v14.8+awaitbree.start();// ... (see below) ...

Please reference the#CommonJS example below for more insight and options.

// app.jsconstpath=require('path');// optionalconstms=require('ms');constdayjs=require('dayjs');constGraceful=require('@ladjs/graceful');constCabin=require('cabin');// requiredconstBree=require('bree');//// NOTE: see the "Instance Options" section below in this README// for the complete list of options and their defaults//constbree=newBree({//// NOTE: by default the `logger` is set to `console`// however we recommend you to use CabinJS as it// will automatically add application and worker metadata// to your log output, and also masks sensitive data for you// <https://cabinjs.com>//// NOTE: You can also pass `false` as `logger: false` to disable logging//logger:newCabin(),//// NOTE: instead of passing this Array as an option// you can create a `./jobs/index.js` file, exporting// this exact same array as `module.exports = [ ... ]`// doing so will allow you to keep your job configuration and the jobs// themselves all in the same folder and very organized//// See the "Job Options" section below in this README// for the complete list of job options and configurations//jobs:[// runs `./jobs/foo.js` on start'foo',// runs `./jobs/foo-bar.js` on start{name:'foo-bar'},// runs `./jobs/some-other-path.js` on start{name:'beep',path:path.join(__dirname,'jobs','some-other-path')},// runs `./jobs/worker-1.js` on the last day of the month{name:'worker-1',interval:'on the last day of the month'},// runs `./jobs/worker-2.js` every other day{name:'worker-2',interval:'every 2 days'},// runs `./jobs/worker-3.js` at 10:15am and 5:15pm every day except on Tuesday{name:'worker-3',interval:'at 10:15 am also at 5:15pm except on Tuesday'},// runs `./jobs/worker-4.js` at 10:15am every weekday{name:'worker-4',cron:'15 10 ? * *',cronValidate:{override:{useBlankDay:true}}},// runs `./jobs/worker-5.js` on after 10 minutes have elapsed{name:'worker-5',timeout:'10m'},// runs `./jobs/worker-6.js` after 1 minute and every 5 minutes thereafter{name:'worker-6',timeout:'1m',interval:'5m'// this is unnecessary but shows you can pass a Number (ms)// interval: ms('5m')},// runs `./jobs/worker-7.js` after 3 days and 4 hours{name:'worker-7',// this example uses `human-interval` parsingtimeout:'3 days and 4 hours'},// runs `./jobs/worker-8.js` at midnight (once){name:'worker-8',timeout:'at 12:00 am'},// runs `./jobs/worker-9.js` every day at midnight{name:'worker-9',interval:'at 12:00 am'},// runs `./jobs/worker-10.js` at midnight on the 1st of every month{name:'worker-10',cron:'0 0 1 * *'},// runs `./jobs/worker-11.js` at midnight on the last day of month{name:'worker-11',cron:'0 0 L * *',cronValidate:{useLastDayOfMonth:true}},// runs `./jobs/worker-12.js` at a specific Date (e.g. in 3 days){name:'worker-12',// <https://github.com/iamkun/dayjs>date:dayjs().add(3,'days').toDate()// you can also use momentjs// <https://momentjs.com/>// date: moment('1/1/20', 'M/D/YY').toDate()// you can pass Date instances (if it's in the past it will not get run)// date: new Date()},// runs `./jobs/worker-13.js` on start and every 2 minutes{name:'worker-13',interval:'2m'},// runs `./jobs/worker-14.js` on start with custom `new Worker` options (see below){name:'worker-14',// <https://nodejs.org/api/worker_threads.html#worker_threads_new_worker_filename_options>worker:{workerData:{foo:'bar',beep:'boop'}}},// runs `./jobs/worker-15.js` **NOT** on start, but every 2 minutes{name:'worker-15',timeout:false,// <-- specify `false` here to prevent default timeout (e.g. on start)interval:'2m'},// runs `./jobs/worker-16.js` on January 1st, 2022// and at midnight on the 1st of every month thereafter{name:'worker-16',date:dayjs('1-1-2022','M-D-YYYY').toDate(),cron:'0 0 1 * *'}]});// handle graceful reloads, pm2 support, and events like SIGHUP, SIGINT, etc.constgraceful=newGraceful({brees:[bree]});graceful.listen();// start all jobs (this is the equivalent of reloading a crontab):(async()=>{awaitbree.start();})();/*// start only a specific job:(async () => {  await bree.start('foo');})();// stop all jobsbree.stop();// stop only a specific job:bree.stop('beep');// run all jobs (this does not abide by timeout/interval/cron and spawns workers immediately)bree.run();// run a specific job (...)bree.run('beep');(async () => {  // add a job array after initialization:  const added = await bree.add(['boop']); // will return array of added jobs  // this must then be started using one of the above methods  // add a job after initialization:  await bree.add('boop');  // this must then be started using one of the above methods})();// remove a job after initialization:bree.remove('boop');*/

For more examples - including setting up bree with TypeScript, ESModules, and implementing an Email Queue, see theexamples folder.

For a more complete demo using express see:Bree Express Demo

Here is the full list of options and their defaults. Seesrc/index.js for more insight if necessary.

SeeInterval, Timeout, Date, and Cron Validate below for more insight besides this table:

These values can include Number, Object, and String variable types:

• Number values indicates the number of milliseconds for the timeout or interval
• Object values must be alater schedule object value (e.g.later.parse.cron('15 10 * * ? *')))
• String values can be either alater,human-interval, orms String values (e.g.later supports Strings such asevery 5 mins,human-interval supports Strings such as3 days and 4 hours, andms supports Strings such as4h for four hours)

Bree extends fromEventEmitter and emits two events:

• worker created with an argument ofname
• worker deleted with an argument ofname

If you'd like to know when your workers are created (or deleted), you can do so through this example:

bree.on('worker created',(name)=>{console.log('worker created',name);console.log(bree.workers.get(name));});bree.on('worker deleted',(name)=>{console.log('worker deleted',name);console.log(!bree.worker.has(name));});

If you'd like to override default behavior for worker error/message handling, provide a callback function aserrorHandler orworkerMessageHandler parameter when creating a Bree instance.

NOTE: Anyconsole.log calls, from within the worker, will not be sent tostdout/stderr until the main thread is available. Furthermore, anyconsole.log calls, from within the worker, will not be sent if the process is terminated before the message is printed. You should useparentPort.postMessage() alongsideerrorHandler orworkerMessageHandler to print tostdout/stderr during worker execution. This is a knownbug for workers.

An example use-case. If you want to call an external service to record an error (like Honeybadger, Sentry, etc.) along with logging the error internally. You can do so with:

constlogger=('../path/to/logger');consterrorService=('../path/to/error-service');newBree({jobs:[{name:'job that sometimes throws errors',path:jobFunction}],errorHandler:(error,workerMetadata)=>{// workerMetadata will be populated with extended worker information only if// Bree instance is initialized with parameter `workerMetadata: true`if(workerMetadata.threadId){logger.info(`There was an error while running a worker${workerMetadata.name} with thread ID:${workerMetadata.threadId}`)}else{logger.info(`There was an error while running a worker${workerMetadata.name}`)}logger.error(error);errorService.captureException(error);}});

We recommend that you listen for "cancel" event in your worker paths. Doing so will allow you to handle graceful cancellation of jobs. For example, you could usep-cancelable

Here's a quick example of how to do that (e.g../jobs/some-worker.js):

// <https://nodejs.org/api/worker_threads.html>const{ parentPort}=require('worker_threads');// ...functioncancel(){// do cleanup here// (if you're using @ladjs/graceful, the max time this can run by default is 5s)// send a message to the parent that we're ready to terminate// (you could do `process.exit(0)` or `process.exit(1)` instead if desired// but this is a bit of a cleaner approach for worker terminationif(parentPort)parentPort.postMessage('cancelled');elseprocess.exit(0);}if(parentPort)parentPort.once('message',message=>{if(message==='cancel')returncancel();});

If you'd like jobs to retry, simply wrap your usage of promises withp-retry.

We leave it up to you to have as much fine-grained control as you wish.

See@ladjs/graceful for more insight into how this package works.

If you need help writing cron expressions, you can referencecrontab.guru.

We supportlater,human-interval, orms String values for bothtimeout andinterval.

If you pass acron property, then it is validated againstcron-validate.

You can pass a Date as thedate property, but you cannot combine bothdate andtimeout.

If you do pass a Date, then it is only run if it is in the future.

SeeJob Interval and Timeout Values above for more insight.

If jobs are running with Node pre-v14.8.0, whichenables top-level async-await support, here is the working alternative:

const{ parentPort}=require('worker_threads');constdelay=require('delay');constms=require('ms');(async()=>{// wait for a promise to finishawaitdelay(ms('10s'));// signal to parent that the job is doneif(parentPort)parentPort.postMessage('done');elseprocess.exit(0);})();

To close out the worker and signal that it is done, you can simplyparentPort.postMessage('done'); and/orprocess.exit(0).

While writing your jobs (which will run inworker threads), you should do one of the following:

• Signal to the main thread that the process has completed by sending a "done" message (per the example above inWriting jobs with Promises and async-await)
• Exit the process if there is NOT an error with code0 (e.g.process.exit(0);)
• Throw an error if an error occurs (this will bubble up to the worker event error listener and terminate it)
• Exit the process if there IS an error with code1 (e.g.process.exit(1))

If a job is already running, a new worker thread will not be spawned, insteadlogger.error will be invoked with an error message (no error will be thrown, don't worry). This is to prevent bad practices from being used. If you need something to be run more than one time, then make the job itself run the task multiple times. This approach gives you more fine-grained control.

By default, workers run indefinitely and are not closed until they exit (e.g. viaprocess.exit(0) orprocess.exit(1), OR send to the parent port a "close" message, which will subsequently callworker.close() to close the worker thread.

If you wish to specify a maximum time (in milliseconds) that a worker can run, then passcloseWorkerAfterMs (Number) either as a default option when creating anew Bree() instance (e.g.new Bree({ closeWorkerAfterMs: ms('10s') })) or on a per-job configuration, e.g.{ name: 'beep', closeWorkerAfterMs: ms('5m') }.

As of v6.0.0 when you passcloseWorkerAfterMs, the timer will start once the worker is signaled as "online" (as opposed to previous versions which did not take this into account).

Since we uselater, you can pass an instance oflater.parse.recur,later.parse.cron, orlater.parse.text as thetimeout orinterval property values (e.g. if you need to construct something manually).

You can also usedayjs to construct dates (e.g. from now or a certain date) to millisecond differences usingdayjs().diff(new Date(), 'milliseconds'). You would then pass that returned Number value astimeout orinterval as needed.

You can pass a default worker configuration object asnew Bree({ worker: { ... } });.

These options are passed to theoptions argument when we internally invokenew Worker(path, options).

Additionally, you can pass custom worker options on a per-job basis through aworker property Object on the job definition.

Seecomplete documentation for options (but you usually don't have to modify these).

It is highly recommended to use files instead of functions. However, sometimes it is necessary to use functions.

You can pass a function to be run as a job:

newBree({jobs:[someFunction]});

(or)

newBree({jobs:[{name:'job with function',path:someFunction}]});

The function will be run as if it's in its own file, therefore no variables or dependencies will be shared from the local context by default.

You should be able to pass data viaworker.workerData (seeCustom Worker Options).

Note that you cannot pass a built-in nor bound function.

When working with a bundler or a tool that transpiles your code in some form or another, we recommend that your bundler is set up in a way that transforms both your application code and your jobs. Because your jobs are in their own files and are run in their own separate threads, they will not be part of your applications dependency graph and need to be setup as their own entry points. You need to ensure you have configured your tool to bundle your jobs into a jobs folder and keep them properly relative to your entry point folder.

We recommend setting theroot instance options topath.join(__dirname,'jobs') so that bree searches for your jobs folder relative to the file being ran. (by default it searches for jobs relative to wherenode is invoked). We recommend treating each job as an entry point and running all jobs through the same transformations as your app code.

After an example transformation - you should expect the output in yourdist folder to look like:

- dist|-jobs|-job.js|-index.js

For some example TypeScript set ups - see theexamples folder.

For another alternative also see the@breejs/ts-worker plugin.

We recommend using the following packages in your workers for handling concurrency:

• https://github.com/sindresorhus/p-all
• https://github.com/sindresorhus/p-limit
• https://github.com/sindresorhus/p-queue
• https://github.com/sindresorhus/p-map

Plugins can be added to Bree using a similar method toDay.js

To add a plugin use the following method:

Bree.extend(plugin,options);

• API
• TypeScript Worker

Plugins should be a function that recieves anoptions object and theBree class:

constplugin=(options,Bree)=>{/* plugin logic */};

More detailed examples can be found inForward Email,Lad, andGhost.

MIT ©Nick Baugh