Repository files navigation

⚠️This is a fork of ShareDB published as:

npm -i @evercoder/sharedb

ShareDB is a realtime database backend based onOperational Transformation(OT) of JSONdocuments. It is the realtime backend for theDerbyJS web applicationframework.

For questions, discussion and announcements, join theShareJS mailinglist orcheck the FAQ.

Please report any bugs you find to theissuetracker.

• Realtime synchronization of any JSON document
• Concurrent multi-user collaboration
• Synchronous editing API with asynchronous eventual consistency
• Realtime query subscriptions
• Simple integration with any database -MongoDB,PostgresQL (experimental)
• Horizontally scalable withpub/sub integration
• Projections to select desired fields from documents and operations
• Middleware for implementing access control and custom extensions
• Ideal for use in browsers or on the server
• Offline change syncing upon reconnection
• In-memory implementations of database and pub/sub for unit testing

TLDR

constWebSocket=require('reconnecting-websocket');varsocket=newWebSocket('ws://'+window.location.host);varconnection=newsharedb.Connection(socket);

The native Websocket object that you feed to ShareDB'sConnection constructordoes not handle reconnections.

The easiest way is to give it a WebSocket object that does reconnect. There are plenty of example on the web. The most important thing is that the custom reconnecting websocket, must have the same API as the native rfc6455 version.

In the "textarea" example we show this off using a Reconnecting Websocket implementation fromreconnecting-websocket.

Simple app demonstrating realtime sync

Leaderboard app demonstrating live queries

In ShareDB's view of the world, every document has 3 properties:

• version - An incrementing number starting at 0
• type - An OT type. OT types are defined inshare/ottypes. Documentswhich don't exist implicitly have a type ofnull.
• data - The actual data that the document contains. This must be pureacyclic JSON. Its also type-specific. (JSON type uses raw JSON, text documentsuse a string, etc).

ShareDB implicitly has a record for every document you can access. New documentshave version 0, a null type and no data. To use a document, you must firstsubmit acreate operation, which will set the document's type and give itinitial data. Then you can submit editing operations on the document (usingOT). Finally you can delete the document with a delete operation. Bydefault, ShareDB stores all operations forever - nothing is truly deleted.

First, create a ShareDB server instance:

varShareDB=require('sharedb');varshare=newShareDB(options);

Options

• options.db(instance ofShareDB.DB)Store documents and ops with this database adapter. Defaults toShareDB.MemoryDB().
• options.pubsub(instance ofShareDB.PubSub)Notify other ShareDB processes when data changesthrough this pub/sub adapter. Defaults toShareDB.MemoryPubSub().
• options.milestoneDb(instance of ShareDB.MilestoneDB`)Store snapshots of documents at a specified interval of versions

• ShareDB.MemoryDB, backed by a non-persistent database with no queries
• ShareDBMongo, backed by a real Mongo databaseand full query support
• ShareDBMingoMemory, backed bya non-persistent database supporting most Mongo queries. Useful for fastertesting of a Mongo-based app.
• ShareDBPostgres, backed by PostgresQL. No query support.

• ShareDB.MemoryPubSub can be used with a single process
• ShareDBRedisPubSub can be usedwith multiple processes using Redis' pub/sub mechanism

Community Provided Pub/Sub Adapters

• wsbus

• sharedb-milestone-mongo, backed by Mongo

varWebSocketJSONStream=require('@teamwork/websocket-json-stream');// 'ws' is a websocket server connection, as passed into// new (require('ws').Server).on('connection', ...)varstream=newWebSocketJSONStream(ws);share.listen(stream);

For transports other than WebSockets, expose a duplexstream that writes and reads JavaScript objects. Thenpass that stream directly intoshare.listen.

Middlewares let you hook into the ShareDB server pipeline. Inmiddleware code you can read and also modify objects as theyflow through ShareDB. For example,sharedb-access uses middlewaresto implement access control.

share.use(action, fn)Register a new middleware.

• action(String)One of:
  • 'connect': A new client connected to the server.
  • 'op': An operation was loaded from the database.
  • 'readSnapshots': Snapshot(s) were loaded from the database for a fetch or subscribe of a query or document
  • 'query': A query is about to be sent to the database
  • 'submit': An operation is about to be submitted to the database
  • 'apply': An operation is about to be applied to a snapshotbefore being committed to the database
  • 'commit': An operation was applied to a snapshot; The operationand new snapshot are about to be written to the database.
  • 'afterWrite': An operation was successfully written tothe database.
  • 'receive': Received a message from a client
  • 'reply': About to send a non-error reply to a client message
• fn(Function(context, callback))Call this function at the time specified byaction.
  • context will always have the following properties:
    • action: The action this middleware is hanlding
    • agent: A reference to the server agent handling this client
    • backend: A reference to this ShareDB backend instance
  • context can also have additional properties, as relevant for the action:
    • collection: The collection name being handled
    • id: The document id being handled
    • op: The op being handled
    • req: HTTP request being handled, if provided toshare.listen (for 'connect')
    • stream: The duplex Stream provided toshare.listen (for 'connect')
    • query: The query object being handled (for 'query')
    • snapshots: Array of retrieved snapshots (for 'readSnapshots')
    • data: Received client message (for 'receive')
    • request: Client message being replied to (for 'reply')
    • reply: Reply to be sent to the client (for 'reply')

ShareDB supports exposing aprojection of a real collection, with a specified (limited) set of allowed fields. Once configured, the projected collection looks just like a real collection - except documents only have the fields you've requested. Operations (gets, queries, sets, etc) on the fake collection work, but you only see a small portion of the data.

addProjection(name, collection, fields)Configure a projection.

• name The name of the projected collection.
• collection The name of the existing collection.
• fields A map (object) of the allowed fields in documents.
  • Keys are field names.
  • Values should betrue.

For example, you could make ausers_limited projection which lets users view each other's names and profile pictures, but not password hashes. You would configure this by calling:

share.addProjection('users_limited','users',{name:true,profileUrl:true});

Note that only theJSON0 OT type is supported for projections.

By default, ShareDB logs toconsole. This can be overridden if you wish to silence logs, or to log to your own logging driver or alert service.

Methods can be overridden by passing aconsole-like object tologger.setMethods:

varShareDB=require('sharedb');ShareDB.logger.setMethods({info:()=>{},// Silence infowarn:()=>alerts.warn(arguments),// Forward warnings to alerting serviceerror:()=>alerts.critical(arguments)// Remap errors to critical alerts});

ShareDB only supports the following logger methods:

• info
• warn
• error

share.close(callback)Closes connections to the database and pub/sub adapters.

The client API can be used from either Node or a browser. First, get aShareDB.Connection object by connecting to the ShareDB server instance:

From Node:

// `share` should be a ShareDB server instancevarconnection=share.connect();

To use ShareDB from a browser, use a client bundler like Browserify or Webpack. The followingcode connects to the ShareDB server instance over WebSockets:

varShareDB=require('sharedb/lib/client');varsocket=newWebSocket('ws://localhost:8080');varconnection=newShareDB.Connection(socket);

For transports other than WebSockets, create an object implementingthe WebSocket specification and pass it into theShareDB.Connection constructor.

connection.get(collectionName, documentId)Get aShareDB.Doc instance on a given collection and document ID.

connection.createFetchQuery(collectionName, query, options, callback)connection.createSubscribeQuery(collectionName, query, options, callback)Get query results from the server.createSubscribeQuery also subscribes tochanges. Returns aShareDB.Query instance.

• query(Object)A descriptor of a database query with structure defined by the database adapter.
• callback(Function)Called with(err, results) when server responds, or on error.
• options.results(Array)Prior query results if available, such as from server rendering.
• options.*All other options are passed through to the database adapter.

connection.fetchSnapshot(collection, id, version, callback): void;Get a read-only snapshot of a document at the requested version.

• collection(String)Collection name of the snapshot

• id(String)ID of the snapshot

• version(number) [optional]The version number of the desired snapshot. Ifnull, the latest version is fetched.

• callback(Function)Called with(error, snapshot), wheresnapshot takes the following form:

  {  id:string;// ID of the snapshot  v:number;// version number of the snapshot  type:string;// the OT type of the snapshot, or null if it doesn't exist or is deleted  data:any;// the snapshot}

connection.fetchSnapshotByTimestamp(collection, id, timestamp, callback): void;Get a read-only snapshot of a document at the requested version.

• collection(String)Collection name of the snapshot

• id(String)ID of the snapshot

• timestamp(number) [optional]The timestamp of the desired snapshot. The returned snapshot will be the latest snapshot before the provided timestamp. Ifnull, the latest version is fetched.

• callback(Function)Called with(error, snapshot), wheresnapshot takes the following form:

  {  id:string;// ID of the snapshot  v:number;// version number of the snapshot  type:string;// the OT type of the snapshot, or null if it doesn't exist or is deleted  data:any;// the snapshot}

doc.type(String)TheOT type of this document

doc.id(String)Unique document ID

doc.data(Object)Document contents. Available after document is fetched or subscribed to.

doc.fetch(function(err) {...})Populate the fields ondoc with a snapshot of the document from the server.

doc.subscribe(function(err) {...})Populate the fields ondoc with a snapshot of the document from the server, andfire events on subsequent changes.

doc.unsubscribe(function (err) {...})Stop listening for document updates. The document data at the time of unsubscribing remains in memory, but no longer stays up-to-date. Resubscribe withdoc.subscribe.

doc.ingestSnapshot(snapshot, callback)Ingest snapshot data. Thesnapshot param must include the fieldsv (doc version),data, andtype (OT type). This method is generally called interally as a result of fetch or subscribe and not directly from user code. However, it may still be called directly from user code to pass data that was transferred to the client external to the client's ShareDB connection, such as snapshot data sent along with server rendering of a webpage.

doc.destroy()Unsubscribe and stop firing events.

doc.on('load', function() {...})The initial snapshot of the document was loaded from the server. Fires at thesame time as callbacks tofetch andsubscribe.

doc.on('create', function(source) {...})The document was created. Technically, this means it has a type.source will befalse for ops received from the server and defaults totrue for ops generated locally.

doc.on('before op batch'), function() {...})An operation batch is about to be applied to the data. For each partial operation a pair ofbefore op andop events will be emitted after this event.

doc.on('before op'), function(op, source) {...})An operation is about to be applied to the data.source will befalse for ops received from the server and defaults totrue for ops generated locally.

doc.on('op', function(op, source) {...})An operation was applied to the data.source will befalse for ops received from the server and defaults totrue for ops generated locally.

doc.on('after op batch'), function() {...})An operation batch was applied to the data.

doc.on('del', function(data, source) {...})The document was deleted. Document contents before deletion are passed in as an argument.source will befalse for ops received from the server and defaults totrue for ops generated locally.

doc.on('error', function(err) {...})There was an error fetching the document or applying an operation.

doc.removeListener(eventName, listener)Removes any listener you added withdoc.on.eventName should be one of'load','create','before op','op','del', or'error'.listener should be the function you passed in as the second argument toon. Note that bothon andremoveListener are inherited fromEventEmitter.

doc.create(data[, type][, options][, function(err) {...}])Create the document locally and send create operation to the server.

• data Initial document contents
• type(OT type)Defaults to'ot-json0', for whichdata is an Object
• options.source Argument passed to the'create' event locally. This is not sent to the server or other clients. Defaults totrue.

doc.submitOp(op, [, options][, function(err) {...}])Apply operation to document and send it to the server.op structure depends on the document type. See theoperations for the default'ot-json0' type.Call this after you've either fetched or subscribed to the document.

• options.source Argument passed to the'op' event locally. This is not sent to the server or other clients. Defaults totrue.

doc.del([options][, function(err) {...}])Delete the document locally and send delete operation to the server.Call this after you've either fetched or subscribed to the document.

• options.source Argument passed to the'del' event locally. This is not sent to the server or other clients. Defaults totrue.

doc.whenNothingPending(function(err) {...})Invokes the given callback function after

• all ops submitted viadoc.submitOp have been sent to the server, and
• all pending fetch, subscribe, and unsubscribe requests have been resolved.

Note thatwhenNothingPending does NOT wait for pendingmodel.query() calls.

query.ready(Boolean)True if query results are ready and available onquery.results

query.results(Array)Query results, as an array ofShareDB.Doc instances.

query.extra(Type depends on database adapter and query)Extra query results that aren't an array of documents. Available for certain database adapters and queries.

query.on('ready', function() {...}))The initial query results were loaded from the server. Fires at the same time asthe callbacks tocreateFetchQuery andcreateSubscribeQuery.

query.on('error', function(err) {...}))There was an error receiving updates to a subscription.

query.destroy()Unsubscribe and stop firing events.

query.on('changed', function(results) {...}))(Only fires on subscription queries) The query results changed. Fires only onceafter a sequence of diffs are handled.

query.on('insert', function(docs, atIndex) {...}))(Only fires on subscription queries) A contiguous sequence of documents were added to the query result array.

query.on('move', function(docs, from, to) {...}))(Only fires on subscription queries) A contiguous sequence of documents moved position in the query result array.

query.on('remove', function(docs, atIndex) {...}))(Only fires on subscription queries) A contiguous sequence of documents were removed from the query result array.

query.on('extra', function() {...}))(Only fires on subscription queries)query.extra changed.

Backend represents the server-side instance of ShareDB. It is primarily responsible for connecting to clients, and sending requests to the database adapters. It is also responsible for some configuration, such as setting upmiddleware andprojections.

varBackend=require('sharedb');varbackend=newBackend(options);

Constructs a newBackend instance, with the provided options:

• dbDB (optional): an instance of a ShareDBdatabase adapter that provides the data store for ShareDB. If omitted, a new, non-persistent, in-memory adapter will be created, which shouldnot be used in production, but may be useful for testing
• pubsubPubSub (optional): an instance of a ShareDBPub/Sub adapter that provides a channel for notifying other ShareDB instances of changes to data. If omitted, a new, in-memory adapter will be created. Unlike the database adapter, the in-memory instancemay be used in a production environment where pub/sub state need only persist across a single, stand-alone server
• milestoneDbMilestoneDB (optional): an instance of a ShareDBmilestone adapter that provides the data store for milestone snapshots, which are historical snapshots of documents stored at a specified version interval. If omitted, this functionality will not be enabled
• extraDbsObject (optional): an object whose values are extraDB instances which can bequeried. The keys are the names that can be passed into the query optionsdb field
• suppressPublishboolean (optional): if set totrue, any changes committed willnot be published onpubsub
• maxSubmitRetriesnumber (optional): the number of times to allow a submit to be retried. If omitted, the request will retry an unlimited number of times

varconnection=backend.connect();

Connects to ShareDB and returns an instance of aConnection. This is the server-side equivalent ofnew ShareDBClient.Connection(socket) in the browser.

This method also supports infrequently used optional arguments:

varconnection=backend.connect(connection,req);

• connectionConnection (optional): aConnection instance to bind to theBackend
• reqObject (optional): a connection context object that can contain information such as cookies or session data that will be available in themiddleware

Returns aConnection.

varagent=backend.listen(stream,req);

Registers aStream with the backend. This should be called when the server receives a new connection from a client.

• streamStream: aStream (orStream-like object) that will be used to communicate between the newAgent and theBackend
• reqObject (optional): a connection context object that can contain information such as cookies or session data that will be available in themiddleware

Returns anAgent, which is also available in themiddleware.

backend.close(callback);

Disconnects ShareDB and all of its underlying services (database, pubsub, etc.).

• callbackFunction: a callback with the signaturefunction (error: Error): void that will be called once the services have stopped, or with anerror if at least one of them could not be stopped

backend.use(action,middleware);

Addsmiddleware to theBackend.

• actionstring | string[]: an action, or array of action names defining when to apply the middleware
• middlewareFunction: a middleware function with the signaturefunction (context: Object, callback: Function): void;. Seemiddleware for more details

Returns theBackend instance, which allows for multiple chained calls.

backend.addProjection(name,collection,fields);

Adds aprojection.

• namestring: the name of the projection
• collectionstring: the name of the collection on which to apply the projection
• fieldsObject: a declaration of which fields to include in the projection, such as{ field1: true }. Defining sub-field projections is not supported.

backend.submit(agent,index,id,op,options,callback);

Submits an operation to theBackend.

• agentAgent: connection agent to pass to the middleware
• indexstring: the name of the target collection or projection
• idstring: the document ID
• opObject: the operation to submit
• optionsObject: these options are passed through to the database adapter'scommit method, so any options that are valid there can be used here
• callbackFunction: a callback with the signaturefunction (error: Error, ops: Object[]): void;, whereops are the ops committed by other clients between the submittedop being submitted and committed

backend.getOps(agent,index,id,from,to,options,callback);

Fetches the ops for a document between the requested version numbers, where thefrom value is inclusive, but theto value is non-inclusive.

• agentAgent: connection agent to pass to the middleware
• indexstring: the name of the target collection or projection
• idstring: the document ID
• fromnumber: the first op version to fetch. If set tonull, then ops will be fetched from the earliest version
• tonumber: The last op version. This version willnot be fetched (ieto is non-inclusive). If set tonull, then ops will be fetched up to the latest version
• options:Object (optional): options can be passed directly to the database driver'sgetOps inside theopsOptions property:{opsOptions: {metadata: true}}
• callback:Function: a callback with the signaturefunction (error: Error, ops: Object[]): void;, whereops is an array of the requested ops

backend.getOpsBulk(agent,index,fromMap,toMap,options,callback);

Fetches the ops for multiple documents in a collection between the requested version numbers, where thefrom value is inclusive, but theto value is non-inclusive.

• agentAgent: connection agent to pass to the middleware
• indexstring: the name of the target collection or projection
• idstring: the document ID
• fromMapObject: an object whose keys are the IDs of the target documents. The values are the first versions requested of each document. For example,{abc: 3} will fetch ops for document with IDabc from version3 (inclusive)
• toMapObject: an object whose keys are the IDs of the target documents. The values are the last versions requested of each document (non-inclusive). For example,{abc: 3} will fetch ops for document with IDabc up to version3 (not inclusive)
• options:Object (optional): options can be passed directly to the database driver'sgetOpsBulk inside theopsOptions property:{opsOptions: {metadata: true}}
• callback:Function: a callback with the signaturefunction (error: Error, opsMap: Object): void;, whereopsMap is an object whose keys are the IDs of the requested documents, and their values are the arrays of requested ops, eg{abc: []}

backend.fetch(agent,index,id,options,callback);

Fetch the current snapshot of a document.

• agentAgent: connection agent to pass to the middleware
• indexstring: the name of the target collection or projection
• idstring: the document ID
• options:Object (optional): options can be passed directly to the database driver'sfetch inside thesnapshotOptions property:{snapshotOptions: {metadata: true}}
• callback:Function: a callback with the signaturefunction (error: Error, snapshot: Snapshot): void;, wheresnapshot is the requested snapshot

backend.fetchBulk(agent,index,ids,options,callback);

Fetch multiple document snapshots from a collection.

• agentAgent: connection agent to pass to the middleware
• indexstring: the name of the target collection or projection
• idsstring[]: array of document IDs
• options:Object (optional): options can be passed directly to the database driver'sfetchBulk inside thesnapshotOptions property:{snapshotOptions: {metadata: true}}
• callback:Function: a callback with the signaturefunction (error: Error, snapshotMap: Object): void;, wheresnapshotMap is an object whose keys are the requested IDs, and the values are the requestedSnapshots

backend.queryFetch(agent,index,query,options,callback);

Fetch snapshots that match the provided query. In most cases, querying the backing database directly should be preferred, butqueryFetch can be used in order to apply middleware, whilst avoiding the overheads associated with using aDoc instance.

• agentAgent: connection agent to pass to the middleware
• indexstring: the name of the target collection or projection
• queryObject: a query object, whose format will depend on the database adapter being used
• optionsObject: an object that may contain adb property, which specifies which database to run the query against. These extra databases can be attached via theextraDbs option in theBackend constructor
• callbackFunction: a callback with the signaturefunction (error: Error, snapshots: Snapshot[], extra: Object): void;, wheresnapshots is an array of the snapshots matching the query, andextra is an (optional) object that the database adapter might return with more information about the results (such as counts)

AnAgent is the representation of a client'sConnection state on the server. If theConnection was created throughbackend.connect (ie the client is running on the server), then theAgent associated with aConnection can be accessed through a direct reference:connection.agent.

TheAgent will be made available in allmiddleware requests. Theagent.custom field is an object that can be used for storing arbitrary information for use in middleware. For example:

backend.useMiddleware('connect',function(request,callback){// Best practice to clone to prevent mutating the object after connection.// You may also want to consider a deep clone, depending on the shape of request.req.Object.assign(request.agent.custom,request.req);callback();});backend.useMiddleware('readSnapshots',function(request,callback){varconnectionInfo=request.agent.custom;varsnapshots=request.snapshots;// Use the information provided at connection to determine if a user can access snapshots.// This should also be checked when fetching and submitting ops.if(!userCanAccessSnapshots(connectionInfo,snapshots)){returncallback(newError('Authentication error'));}callback();});// Here you should determine what permissions a user has, probably by reading a cookie and// potentially making some database request to check which documents they can access, or which// roles they have, etc. If doing this asynchronously, make sure you call backend.connect// after the permissions have been fetched.varconnectionInfo=getUserPermissions();// Pass info in as the second argument. This will be made available as request.req in the// 'connection' middleware.varconnection=backend.connect(null,connectionInfo);

By default, ShareDB logs toconsole. This can be overridden if you wish to silence logs, or to log to your own logging driver or alert service.

Methods can be overridden by passing aconsole-like object tologger.setMethods

varShareDB=require('sharedb/lib/client');ShareDB.logger.setMethods({info:()=>{},// Silence infowarn:()=>alerts.warn(arguments),// Forward warnings to alerting serviceerror:()=>alerts.critical(arguments)// Remap errors to critical alerts});

ShareDB only supports the following logger methods:

• info
• warn
• error

ShareDB returns errors as plain JavaScript objects with the format:

{  code: 5000,  message: 'ShareDB internal error'}

Additional fields may be added to the error object for debugging context depending on the error. Common additional fields includecollection,id, andop.

• 4001 - Unknown error type
• 4002 - Database adapter does not support subscribe
• 4003 - Database adapter not found
• 4004 - Missing op
• 4005 - Op must be an array
• 4006 - Create data in op must be an object
• 4007 - Create op missing type
• 4008 - Unknown type
• 4009 - del value must be true
• 4010 - Missing op, create or del
• 4011 - Invalid src
• 4012 - Invalid seq
• 4013 - Found seq but not src
• 4014 - op.m invalid
• 4015 - Document does not exist
• 4016 - Document already exists
• 4017 - Document was deleted
• 4018 - Document was created remotely
• 4019 - Invalid protocol version
• 4020 - Invalid default type
• 4021 - Invalid client id
• 4022 - Database adapter does not support queries
• 4023 - Cannot project snapshots of this type
• 4024 - Invalid version
• 4025 - Passing options to subscribe has not been implemented

The41xx and51xx codes are reserved for use by ShareDB DB adapters, and the42xx and52xx codes are reserved for use by ShareDB PubSub adapters.

• 5001 - No new ops returned when retrying unsuccessful submit
• 5002 - Missing snapshot
• 5003 - Snapshot and op version don't match
• 5004 - Missing op
• 5005 - Missing document
• 5006 - Version mismatch
• 5007 - Invalid state transition
• 5008 - Missing version in snapshot
• 5009 - Cannot ingest snapshot with null version
• 5010 - No op to send
• 5011 - Commit DB method unimplemented
• 5012 - getSnapshot DB method unimplemented
• 5013 - getOps DB method unimplemented
• 5014 - queryPollDoc DB method unimplemented
• 5015 - _subscribe PubSub method unimplemented
• 5016 - _unsubscribe PubSub method unimplemented
• 5017 - _publish PubSub method unimplemented
• 5018 - Required QueryEmitter listener not assigned
• 5019 - getMilestoneSnapshot MilestoneDB method unimplemented
• 5020 - saveMilestoneSnapshot MilestoneDB method unimplemented
• 5021 - getMilestoneSnapshotAtOrBeforeTime MilestoneDB method unimplemented
• 5022 - getMilestoneSnapshotAtOrAfterTime MilestoneDB method unimplemented