Repository files navigation

Stream and decode Protocol Buffer arrays without memory issues. While other libraries need the entire message in memory, this one processes array elements directly from the source stream - making it perfect for large datasets.

• 💾Memory Efficient: No need to load entire message in memory
• 🌊Streaming: Process array elements directly from source (file, network, etc.)
• 🎯Specialized: Built for messages with a repeated field (array)
• ⚡Scalable: Works with arrays of any size, memory usage stays constant

npm install protobuf-array-stream

First, you need to load your Protocol Buffer definitions. The library exposes these methods directly from protobufjs for convenience:

import{RepeatedFieldStream}from"protobuf-array-stream";import{createReadStream}from"fs";// 1. Load from a .proto file (async)constroot1=awaitRepeatedFieldStream.loadRootFromFile("path/to/messages.proto",);// 2. Load from a .proto file (sync)constroot2=RepeatedFieldStream.loadRootFromFileSync("path/to/messages.proto",);// 3. Load from a proto definition stringconstroot3=RepeatedFieldStream.loadRootFromSource(`  syntax = "proto3";  message NumberList {    repeated int32 numbers = 1;  }`);// 4. Load from JSON schemaconstroot4=RepeatedFieldStream.loadRootFromJSON({nested:{NumberList:{fields:{numbers:{id:1,rule:"repeated",type:"int32",},},},},});

You're building an anti-cheat system for your massively multiplayer game. Players are generating TONS of data:

syntax="proto3";messageGameLog {repeatedPlayerActionactions=1;}messagePlayerAction {int64timestamp=1;stringplayer_id=2;stringaction=3;Positionposition=4;}messagePosition {floatx=1;floaty=2;floatz=3;}

Process large game session logs efficiently:

constsessionStream=createReadStream("game_session.bin");constdecodeStream=newRepeatedFieldStream({  root,messageName:"GameLog",});constanalyzeStream=newTransform({objectMode:true,transform(action,_,callback){const{ timestamp, playerId,action:move, position}=action;if(move==="shoot"&&position.y>500){console.log(`Detected:${playerId} at height${position.y} on${newDate(timestamp.toNumber())}`,);}callback(null,action);},});sessionStream.pipe(decodeStream).pipe(analyzeStream);

The stream emits errors in these cases:

• Message type not found in proto definitions
• Message has no fields or multiple fields
• Field is not a repeated field
• Invalid protobuf encoding in input

decodeStream.on("error",(error)=>{console.error(`${error.name} -${error.message}`);});

Creates a new transform stream for processing Protocol Buffer repeated fields.

• root: Root namespace from protobufjs containing message definitions
• messageName: Name of the message type to decode

Asynchronously loads Protocol Buffer definitions from a .proto file.

• path: Path to the .proto file
• Returns: Promise that resolves to the root namespace

Synchronously loads Protocol Buffer definitions from a .proto file.

• path: Path to the .proto file
• Returns: Root namespace

Loads Protocol Buffer definitions from a string containing proto definitions.

• source: String containing the proto definitions
• Returns: Root namespace

Loads Protocol Buffer definitions from a JSON object.

• json: JSON object containing the proto definitions
• Returns: Root namespace

• Node.js >= 16.0.0
• Protocol Buffer messages must contain exactly one repeated field

MIT License - see theLICENSE file for details