Repository files navigation

PySyncObj is a python library for building fault-tolerant distributed systems. It provides the ability to replicate your application data between multiple servers. It has following features:

• raft protocol for leader election and log replication
• Log compaction - it use fork for copy-on-write while serializing data on disk
• Dynamic membership changes - you can do it withsyncobj_admin utility ordirectly from your code
• Zero downtime deploy - no need to stop cluster to update nodes
• In-memory and on-disk serialization - you can use in-memory mode for small data and on-disk for big one
• Encryption - you can set password and use it in external network
• Python2 and Python3 on linux, macos and windows - no dependencies required (only optional one, eg. cryptography)
• Configurable event loop - it can works in separate thread with it's own event loop - or you can call onTick function inside your own one
• Convenient interface - you can easily transform arbitrary class into a replicated one (see example below).

• Install
• Basic Usage
• "Batteries"
• API Documentation
• Performance
• Publications

PySyncObj itself:

pip install pysyncobj

Cryptography for encryption (optional):

pip install cryptography

Consider you have a class that implements counter:

classMyCounter(object):def__init__(self):self.__counter=0defincCounter(self):self.__counter+=1defgetCounter(self):returnself.__counter

So, to transform your class into a replicated one:

• Inherit it from SyncObj
• Initialize SyncObj with a self address and a list of partner addresses. Eg. if you haveserverA,serverB andserverC and want to use 4321 port, you should use self addressserverA:4321 with partners[serverB:4321, serverC:4321] for your application, running atserverA; self addressserverB:4321 with partners[serverA:4321, serverC:4321] for your application atserverB; self addressserverC:4321 with partners[serverA:4321, serverB:4321] for app atserverC.
• Mark all your methods that modifies your class fields with@replicated decorator.So your final class will looks like:

classMyCounter(SyncObj):def__init__(self):super(MyCounter,self).__init__('serverA:4321', ['serverB:4321','serverC:4321'])self.__counter=0@replicateddefincCounter(self):self.__counter+=1defgetCounter(self):returnself.__counter

And thats all! Now you can callincCounter onserverA, and check counter value onserverB - they will be synchronized.

If you just need some distributed data structures - try built-in "batteries". Few examples:

frompysyncobjimportSyncObjfrompysyncobj.batteriesimportReplCounter,ReplDictcounter1=ReplCounter()counter2=ReplCounter()dict1=ReplDict()syncObj=SyncObj('serverA:4321', ['serverB:4321','serverC:4321'],consumers=[counter1,counter2,dict1])counter1.set(42,sync=True)# set initial value to 42, 'sync' means that operation is blockingcounter1.add(10,sync=True)# add 10 to counter valuecounter2.inc(sync=True)# increment counter value by onedict1.set('testKey1','testValue1',sync=True)dict1['testKey2']='testValue2'# this is basically the same as previous, but asynchronous (non-blocking)print(counter1,counter2,dict1['testKey1'],dict1.get('testKey2'))

frompysyncobjimportSyncObjfrompysyncobj.batteriesimportReplLockManagerlockManager=ReplLockManager(autoUnlockTime=75)# Lock will be released if connection dropped for more than 75 secondssyncObj=SyncObj('serverA:4321', ['serverB:4321','serverC:4321'],consumers=[lockManager])iflockManager.tryAcquire('testLockName',sync=True):# do some actionslockManager.release('testLockName')

You can look atbatteries implementation,examples andunit-tests for more use-cases. Also there is anAPI documentation. Feel free to create proposals and/or pull requests with new batteries, features, etc. Join ourgitter chat if you have any questions.

• Adventures in fault tolerant alerting with Python
• Строим распределенную систему c PySyncObj