An implementation of synchronous**multi-master replication** based on**commit timestamps**.

Multi-master is an extension and set of patches to a Postegres database, that turns Postgres into a

synchronous shared-nothing cluster to provide OLTP scalability and high availability with automatic

disaster recovery.

1. Install`contrib/raftable` and`contrib/mmts` on each instance.

Multi-master replicates same database to all nodes in cluster and allows writes to each node. Transaction

isolation is enforced cluster-wide, so in case of concurrent updates on different nodes database will use the

same conflict resolution rules (mvcc with repeatable read isolation level) as single node uses for concurrent

backends and always stays in consistent state. Any writing transaction will write to all nodes, hence increasing

commit latency for amount of time proportional to roundtrip between nodes nedded for synchronization. Read only

transactions and queries executed locally without measurable overhead. Replication mechanism itself based on

logical decoding and earlier version of pglogical extension provided for community by 2ndQuadrant team.

Several changes was made in postgres core to implement mentioned functionality:

* Transaction manager API. (eXtensible Transaction Manager, xtm). Generic interface to plug distributed

transaction engines. More info on[postgres wiki](https://wiki.postgresql.org/wiki/DTM) and

on[the email thread](http://www.postgresql.org/message-id/flat/F2766B97-555D-424F-B29F-E0CA0F6D1D74@postgrespro.ru).

* Distributed deadlock detection API.

* Logical decoding of transactions.

Cluster consisting of N nodes can continue to work while majority of initial nodes are alive and reachable by

other nodes. This is done by using 3 phase commit protocol and heartbeats for failure discovery. Node that is

brought back to cluster can be fast-forwaded to actual state automatically in case when transactions log still

exists since the time when node was excluded from cluster (this depends on checkpoint configuration in postgres).

Read more about internals on[Architechture](/Architechture) page.

* Cluster-wide transaction isolation

* Synchronous logical replication

* DDL Replication

* Distributed sequences

* Fault tolerance

* Automatic node recovery

* Commit latency.

Current implementation of logical replication sends data to subscriber nodes only after local commit, so in case of

heavy-write transaction user will wait for transaction processing two times: on local node and al other nodes

(simultaneosly). We have plans to address this issue in future.

* DDL replication.

While data is replicated on logical level, DDL replicated by statements performing distributed commit with the same

statement. Some complex DDL scenarious including stored procedures and temp temp tables aren't working properly. We

are working right now on proving full compatibility with ordinary postgres. Currently we are passing 141 of 164

postgres regression tests.

* Isolation level.

Multimaster currently support only_repeatable__read_ isolation level. This is stricter than default_read_commited_,

but also increases probability of serialization failure during commit._Serializable_ level isn't supported yet.

* One database per cluster.

(Existing db?)

Multi-master consist of patched version of postgres and extension mmts, that provides most of functionality, but

doesn't requiere changes to postgres core. To run multimaster one need to install postgres and several extensions

to all nodes in cluster.

Ensure that following prerequisites are installed:

debian based linux:

apt-get install -y git make gcc libreadline-dev bison flex zlib1g-dev

red hat based linux:

yum groupinstall'Development Tools'

yum install git, automake, libtool, bison, flex readline-devel

After that everything is ready to install postgres along with extensions

git clone https://github.com/postgrespro/postgres_cluster.git

cd postgres_cluster

./configure&& make&& make -j 4 install

cd ./contrib/raftable&& make install

cd ../../contrib/mmts&& make install

Directort contrib/mmts also includes Dockerfile that is capable of building multi-master and starting 3 node cluster.

cd contrib/mmts

docker-compose build

docker-compose up

After things go more stable we will release prebuilt packages for major platforms.

1. Add these required options to the`postgresql.conf` of each instance in the cluster.

1. Allow replication in`pg_hba.conf`.

(link to full doc on config params)

`create extension mmts;` to gain access to these functions:

*`mtm.get_cluster_info()` -- print some debug info

*`mtm.make_table_local(relation regclass)` -- stop replication for a given table

(link to full doc on functions)

(Show TPC-C here on 3 nodes)

(Link to test/failure matrix)

Regression: 141 of 164

Isolation: n/a

To run tests:

*`make -C contrib/mmts check` to run TAP-tests.

*`make -C contrib/mmts xcheck` to run blockade tests. The blockade tests require`docker`,`blockade`, and some other packages installed, see[requirements.txt](tests2/requirements.txt) for the list. You might also want to gain superuser privileges to run these tests successfully.

docs: