Repository files navigation

A batching queue for items that need to be enqueued one by one, but dequeued in batches

• Works with multiple storage backends
• Queues items one by one
• Dequeues in batches (FIFO)
• Tracks the number of waiting items
• Emits an event when the number of waiting items reaches the batch size threshold

npm install batching-queue

• MemoryStore - stores items in an array (not for production use)
• AsyncMemoryStore - same as MemoryStore, but with async methods
• RedisStore - stores items in Redis

import{BatchingQueue,MemoryStore}from'batching-queue'constqueue=newBatchingQueue({store:newMemoryStore(),batchSize:12})constdrain=async(batchesWaiting)=>{constresult=awaitbatchesWaitingconsole.log(result,'batches waiting')for(leti=0;i<result;i++){constbatch=awaitqueue.dequeue()console.log(batch,'batch')}}(async()=>{// drain remaining batches immediatelydrain(queue.length)// drain batches every 12 itemsqueue.on('drain',drain)for(vari=0;i<1000;i++){awaitqueue.enqueue(i)}})()

Config:

• store (require) Storage backend for the queue (see src/stores/memory.js for a reference implementation)
• batchSize (required) Batch size

Adds an item to the queue. Returnstrue if a batch is full, otherwise returnsfalse.

Dequeues the first [batchSize] number of items. If there are not enough items to satisfy batchSize, all the items available will be returned.

Property reports the number of batches waiting in the queue.

Note: if the storage backend is not initialized, this will return null.

You may to implement your own storage backend to interface your database or cache. See the bundled MemoryStore or AsyncMemoryStore for a reference.

Initialize and set up the storage backend. Establish connections, create database collections, etc.

This will be called before callingenqueue() ordequeue() whenready === false.

Destroy the storage backend (optional).

Store an item in the queue.

The store must return theexact number of items in the store subsequent to the storage operation.

Retrieve and delete [batchSize] items from the storage backend.

Return the set of items retrieved.

Return the number of items in the store.

Returntrue if ready and setup, otherwise, returnfalse.

The RedisStore class has the following additional methods:

Config:

• redisClient (required) Instance ofnode-redis
• name Queue list key name, will be randomized if omitted

The RedisStore class has the following additional methods:

Config:

• redisClient (required) Instance ofioredis
• name Queue list key name, will be randomized if omitted

MIT License