Repository files navigation

A Node.js Buffer list collector, reader and streamer thingy.

bl is a storage object for collections of Node Buffers, exposing them with the main Buffer readable API. Also works as a duplex stream so you can collect buffers from a stream that emits them and emit buffers to a stream that consumes them!

The original buffers are kept intact and copies are only done as necessary. Any reads that require the use of a single original buffer will return a slice of that buffer only (which references the same memory as the original buffer). Reads that span buffers perform concatenation as required and return the results transparently.

const{ BufferList}=require('bl')constbl=newBufferList()bl.append(Buffer.from('abcd'))bl.append(Buffer.from('efg'))bl.append('hi')// bl will also accept & convert Stringsbl.append(Buffer.from('j'))bl.append(Buffer.from([0x3,0x4]))console.log(bl.length)// 12console.log(bl.slice(0,10).toString('ascii'))// 'abcdefghij'console.log(bl.slice(3,10).toString('ascii'))// 'defghij'console.log(bl.slice(3,6).toString('ascii'))// 'def'console.log(bl.slice(3,8).toString('ascii'))// 'defgh'console.log(bl.slice(5,10).toString('ascii'))// 'fghij'console.log(bl.indexOf('def'))// 3console.log(bl.indexOf('asdf'))// -1// or just use toString!console.log(bl.toString())// 'abcdefghij\u0003\u0004'console.log(bl.toString('ascii',3,8))// 'defgh'console.log(bl.toString('ascii',5,10))// 'fghij'// other standard Buffer readablesconsole.log(bl.readUInt16BE(10))// 0x0304console.log(bl.readUInt16LE(10))// 0x0403

Give it a callback in the constructor and use it just likeconcat-stream:

const{ BufferListStream}=require('bl')constfs=require('fs')fs.createReadStream('README.md').pipe(BufferListStream((err,data)=>{// note 'new' isn't strictly required// `data` is a complete Buffer object containing the full dataconsole.log(data.toString())}))

Note that when you use thecallback method like this, the resultingdata parameter is a concatenation of allBuffer objects in the list. If you want to avoid the overhead of this concatenation (in cases of extreme performance consciousness), then avoid thecallback method and just listen to'end' instead, like a standard Stream.

Or to fetch a URL usinghyperquest (should work withrequest and even plain Node http too!):

consthyperquest=require('hyperquest')const{ BufferListStream}=require('bl')consturl='https://raw.github.com/rvagg/bl/master/README.md'hyperquest(url).pipe(BufferListStream((err,data)=>{console.log(data.toString())}))

Or, use it as a readable stream to recompose a list of Buffers to an output source:

const{ BufferListStream}=require('bl')constfs=require('fs')varbl=newBufferListStream()bl.append(Buffer.from('abcd'))bl.append(Buffer.from('efg'))bl.append(Buffer.from('hi'))bl.append(Buffer.from('j'))bl.pipe(fs.createWriteStream('gibberish.txt'))

• new BufferList([ buf ])
• BufferList.isBufferList(obj)
• bl.length
• bl.append(buffer)
• bl.append(buffer)
• bl.get(index)
• bl.indexOf(value[, byteOffset][, encoding])
• bl.slice([ start[, end ] ])
• bl.shallowSlice([ start[, end ] ])
• bl.copy(dest, [ destStart, [ srcStart [, srcEnd ] ] ])
• bl.duplicate()
• bl.consume(bytes)
• bl.toString([encoding, [ start, [ end ]]])
• bl.readDoubleBE(),bl.readDoubleLE(),bl.readFloatBE(),bl.readFloatLE(),bl.readBigInt64BE(),bl.readBigInt64LE(),bl.readBigUInt64BE(),bl.readBigUInt64LE(),bl.readInt32BE(),bl.readInt32LE(),bl.readUInt32BE(),bl.readUInt32LE(),bl.readInt16BE(),bl.readInt16LE(),bl.readUInt16BE(),bl.readUInt16LE(),bl.readInt8(),bl.readUInt8()
• new BufferListStream([ callback ])
• bl.getBuffers()

No arguments arerequired for the constructor, but you can initialise the list by passing in a singleBuffer object or an array ofBuffer objects.

new is not strictly required, if you don't instantiate a new object, it will be done automatically for you so you can create a new instance simply with:

const{ BufferList}=require('bl')constbl=BufferList()// equivalent to:const{ BufferList}=require('bl')constbl=newBufferList()

Determines if the passed object is aBufferList. It will returntrue if the passed object is an instance ofBufferListorBufferListStream andfalse otherwise.

N.B. this won't returntrue forBufferList orBufferListStream instances created by versions of this library before this static method was added.

Get the length of the list in bytes. This is the sum of the lengths of all of the buffers contained in the list, minus any initial offset for a semi-consumed buffer at the beginning. Should accurately represent the total number of bytes that can be read from the list.

append(buffer) adds an additional buffer or BufferList to the internal list.this is returned so it can be chained.

prepend(buffer) adds an additional buffer or BufferList at the beginning of the internal list.this is returned so it can be chained.

get() will return the byte at the specified index.

get() will return the byte at the specified index.indexOf() method returns the first index at which a given element can be found in the BufferList, or -1 if it is not present.

slice() returns a newBuffer object containing the bytes within the range specified. Bothstart andend are optional and will default to the beginning and end of the list respectively.

If the requested range spans a single internal buffer then a slice of that buffer will be returned which shares the original memory range of that Buffer. If the range spans multiple buffers then copy operations will likely occur to give you a uniform Buffer.

shallowSlice() returns a newBufferList object containing the bytes within the range specified. Bothstart andend are optional and will default to the beginning and end of the list respectively.

No copies will be performed. All buffers in the result share memory with the original list.

copy() copies the content of the list in thedest buffer, starting fromdestStart and containing the bytes within the range specified withsrcStart tosrcEnd.destStart,start andend are optional and will default to the beginning of thedest buffer, and the beginning and end of the list respectively.

duplicate() performs ashallow-copy of the list. The internal Buffers remains the same, so if you change the underlying Buffers, the change will be reflected in both the original and the duplicate. This method is needed if you want to callconsume() orpipe() and still keep the original list.Example:

varbl=newBufferListStream()bl.append('hello')bl.append(' world')bl.append('\n')bl.duplicate().pipe(process.stdout,{end:false})console.log(bl.toString())

consume() will shift bytesoff the start of the list. The number of bytes consumed don't need to line up with the sizes of the internal Buffers—initial offsets will be calculated accordingly in order to give you a consistent view of the data.

toString() will return a string representation of the buffer. The optionalstart andend arguments are passed on toslice(), while theencoding is passed on totoString() of the resulting Buffer. See theBuffer#toString() documentation for more information.

All of the standard byte-reading methods of theBuffer interface are implemented and will operate across internal Buffer boundaries transparently.

See theBuffer documentation for how these work.

BufferListStream is a NodeDuplex Stream, so it can be read from and written to like a standard Node stream. You can alsopipe() to and from aBufferListStream instance.

The constructor takes an optional callback, if supplied, the callback will be called with an error argument followed by a reference to thebl instance, whenbl.end() is called (i.e. from a piped stream). This is a convenient method of collecting the entire contents of a stream, particularly when the stream ischunky, such as a network stream.

Normally, no arguments are required for the constructor, but you can initialise the list by passing in a singleBuffer object or an array ofBuffer object.

new is not strictly required, if you don't instantiate a new object, it will be done automatically for you so you can create a new instance simply with:

const{ BufferListStream}=require('bl')constbl=BufferListStream()// equivalent to:const{ BufferListStream}=require('bl')constbl=newBufferListStream()

N.B. For backwards compatibility reasons,BufferListStream is thedefault export when yourequire('bl'):

const{ BufferListStream}=require('bl')// equivalent to:constBufferListStream=require('bl')

getBuffers() returns the internal list of buffers.

bl is brought to you by the following hackers:

• Rod Vagg
• Matteo Collina
• Jarett Cruger

Copyright (c) 2013-2019 bl contributors (listed above).

bl is licensed under the MIT license. All rights not explicitly granted in the MIT license are reserved. See the included LICENSE.md file for more details.