Repository files navigation

See thetechnical blog for more details about background and circuit design

This project needs following third party services:

• mysql: used to storewitness,userproof,proof table;
• redis: provide distributed lock for multi provers;
• kvrocks: used to store account tree

We can use docker to run these services:

docker run -d --name zkpos-redis -p 6379:6379 redisdocker run -d --name zkpos-mysql -p 3306:3306  -v /server/docker_data/mysql_data:/var/lib/mysql -e MYSQL_USER=zkpos -e MYSQL_PASSWORD=zkpos@123 -e MYSQL_DATABASE=zkpos  -e MYSQL_ROOT_PASSWORD=zkpos@123 mysqldocker run -d --name zkpos-kvrocks -p 6666:6666 -v /server/docker_data/kvrocksdata:/kvrocksdata apache/kvrocks --dir /kvrocksdata

where/server/docker_data/ is directory in the host machine which is used to persist mysql and kvrocks docker data.

Thekeygen service is for generating zk related keys which are used to generate and verify zk proof. The updated PoR solution now supports multi-tier circuits based on the counts of asset types a user owns. TheBatchCreateUserOpsCountsTiers constant in the utils package represents the multi-tier circuit configuration that defines how many users can be created in one batch for each specific tier.

Run the following commands to startkeygen service:

// make sure the BatchCreateUserOpsCounts in utils/constants.go is expectedcd src/keygen; go run main.go

Afterkeygen service finishes running, there will be several key files generated in the current directory, like the following:

-rw-r--r--. 1 root root  524 Aug 19 09:46 zkpor350_128.vk-rw-r--r--. 1 root root  12G Aug 19 09:46 zkpor350_128.pk-rw-r--r--. 1 root root 7.5G Aug 19 09:47 zkpor350_128.r1cs-rw-r--r--. 1 root root  524 Aug 19 10:38 zkpor50_580.vk-rw-r--r--. 1 root root  12G Aug 19 10:38 zkpor50_580.pk-rw-r--r--. 1 root root  12G Aug 19 10:39 zkpor50_580.r1cs

Thewitness service is used to generate witness forprover service.

witness/config/config.json is the config filewitness service use. The sample file is as follows:

{"MysqlDataSource" :"zkpos:zkpos@123@tcp(127.0.0.1:3306)/zkpos?parseTime=true","UserDataFile":"/server/data/20230118","DbSuffix":"0","TreeDB": {"Driver":"redis","Option": {"Addr":"127.0.0.1:6666"    }  }}

Where

• MysqlDataSource: this is the mysql config;
• UserDataFile: the directory which contains all users balance sheet files;
• DbSuffix: this suffix will be appended to the ending of table name, such asproof0,witness0 table;
• TreeDB:
  • Driver:redis means account tree use kvrocks as its storage engine;
  • Option:
    • Addr:kvrocks service listen address

Run the following command to startwitness service:

cd witness; go run main.go

Thewitness service supports recovery from unexpected crash. Afterwitness service finish running, we can seewitness fromwitness table.

One witness batch contains 700 users whose assets number is less or equal than 50, and 92 users whose assets number is larger than 50.

Thedb_tool cli provide a subcommand calledpush_task_to_redis which can be used for push proof generating tasks to redis after all the witnesses data are generated. The provers will fetch the proof-generating tasks from redis, update the witness data status intoreceived, then generate the proof, and update the witness data status intofinished.

Theprover service is used to generate zk proof and supports running in parallel. It reads witness fromwitness table generated bywitness service.

prover/config/config.json is the config fileprover service uses. The sample file is as follows:

{"MysqlDataSource" :"zkpos:zkpos@123@tcp(127.0.0.1:3306)/zkpos?parseTime=true","DbSuffix":"0","Redis": {"Host":"127.0.0.1:6379",  },"ZkKeyName": ["/server/zkmerkle-proof-of-solvency/src/keygen/zkpor50_580","/server/zkmerkle-proof-of-solvency/src/keygen/zkpor350_128"],"AssetsCountTiers": [50,350]}

Where

• MysqlDataSource: this is the mysql config;
• DbSuffix: this suffix will be appended to the ending of table name, such asproof0,witness0 table;
• Redis:
  • Host:redis service listen addr;
  • Type: only supportnode type
• ZkKeyName: the list of key names generated bykeygen service
• AssetsCountTiers: The list of asset count tiers, each corresponding to a key name inZkKeyName

Run the following command to startprover service:

cd prover; go run main.go

To runprover service in parallel, just repeat executing above commands.

Note: After all prover service finishes running, We should usego run main.go -rerun command to regenerate proof for unfinished batch

After the wholeprover service finished, we can see batch zk proof inproof table.

Theuserproof service is used to generate and persist user merkle proof. It usesuserproof/config/config.json as config file, and the sample config is as follows:

{"MysqlDataSource" :"zkpos:zkpos@123@tcp(127.0.0.1:3306)/zkpos?parseTime=true","UserDataFile":"/server/data/20230118","DbSuffix":"0","TreeDB": {"Driver":"redis","Option": {"Addr":"127.0.0.1:6666"    }  }}

Where

• MysqlDataSource: this is the mysql config;
• UserDataFile: the directory which contains all users balance sheet files;
• DbSuffix: this suffix will be appended to the ending of table name, such asproof0,witness0 table;
• TreeDB:
  • Driver:redis means account tree use kvrocks as its storage engine;
  • Option:
    • Addr:kvrocks service listen address

Run the following command to runuserproof service:

cd userproof; go run main.go

Afteruserproof service finishes running, we can see every user proof fromuserproof table.

The performance: about 10k users proof generation per second in a 128GB memory and 32 core virtual machine.

Theverifier service is used to verify batch proof and single user proof.

The service useconfig.json as its config file, and the sample config is as follows:

{"ProofTable":"config/proof.csv","ZkKeyName": ["config/zkpor50_580","config/zkpor350_128"],"AssetsCountTiers": [50,350],"CexAssetsInfo": [{"TotalEquity":219971568487,"TotalDebt":9789219,"BasePrice":24620000000},{"TotalEquity":8664493444,"TotalDebt":122580,"BasePrice":1682628000000},{"TotalEquity":67463930749983,"TotalDebt":16127314913,"BasePrice":100000000},{"TotalEquity":68358645578,"TotalDebt":130187,"BasePrice":121377000000},{"TotalEquity":590353015932,"TotalDebt":0,"BasePrice":598900000},{"TotalEquity":255845425858,"TotalDebt":13839361,"BasePrice":6541000000},{"TotalEquity":0,"TotalDebt":0,"BasePrice":99991478},{"TotalEquity":267958065914051,"TotalDebt":501899265949,"BasePrice":100000000},{"TotalEquity":124934670143615,"TotalDebt":1422964747,"BasePrice":34500000}]}

Where

• ProofTable: this is proof csv file which can be exported byproof table;
• ZkKeyName: the key name generated bykeygen service;
• AssetsCountTiers: The list of asset count tiers, each corresponding to a key name inZkKeyName;
• CexAssetsInfo: this is published by CEX, it represents CEX's liability;

You can getCexAssetsInfo usingdbtool command afterwitness service run finished. Run the following command to verify batch proof:

cd verifier; go run main.go

The service useuser_config.json as its config file, and the sample config is as follows:

{"AccountIndex":9,"AccountIdHash":"000000000000000000000000000000000000000000000000000000000000006d","TotalEquity":107595993240612342000000,"TotalDebt":4812541145779934000000,"TotalCollateral":4861152676874104600000,"Assets":[{"Index":0,"Equity":14571647457,"Debt":184812783,"Loan":7285823729,"Margin":3642911864,"PortfolioMargin":1821455932},{"Index":1,"Equity":25424316291,"Debt":3323064077,"Loan":12712158145,"Margin":6356079073,"PortfolioMargin":3178039536},{"Index":2,"Equity":57834282404,"Debt":19716095367,"Loan":28917141202,"Margin":14458570601,"PortfolioMargin":7229285300},{"Index":3,"Equity":25100,"Debt":669524367015,"Loan":12550,"Margin":6275,"PortfolioMargin":3138}],"Root":"1a4940fecdbf2f7d8fe9c4f16083ceb587f69f0b9af0d02d528235757536668f","Proof":["EerEdgizAIY2QB/1c7e3vogBKXoD3Q3WN+wDO/hx/HY=","Kj6yinCJObEjIDqK3KPiXqGUetaaHdYoP6WRVWRyubE=","JxQCfxGWwtk0XQi2UkU4GUjHZKmW5uCQw0Unqf6nJUI=","CfqgZ96f0N0RUyd4IQ1sEdsm2Yq+4WRCq7awff2P1q8=","LvSjy7ocJKqWPpjRr17niwhgtmqNFMNuLS8lVLRWLbo=","EWnMgwIncL2h5IMqlObRbaa0hD2DMWcbNPuSLrb+LwY=","HiOSZAFdHtBMxcPB4yWGPY9dTuHSLMCzAOsoh4qt+VI=","EcAo/cipNXN6IeLHoQ+V9ZDX23KqZhF7lp65zJoo7MU=","EXs4mjwX3UpfJxC8Kbyey3JEyxilg8FXXUGabXvApac=","DBGelh5dqSIf/6qrFwCud4VSt77d0MMdmq41g78dmwc=","I4HkTsZT2BLXNzJCXW15MbQ8MDtzXjbbfJsZ15ZczS0=","FmFiP3/04HxlH5/MJeRG1SbSzu9OQVzJM49uQO10w1o=","C28Jko6/+VPAikr9oK2vf9jbCVhVDjWG6r699bHv+Vk=","Jm8aL9AJ54lAZRMbySEmEH8oAKiEp7jlOI2wOnu4KRo=","GdbpvwyPyQb6/HOZs17w6AMncy/htlh4R/dXIdyJv5w=","IeDs6wpjmp+DbYopQmas2GMl8CZpA+sqr9/+lzRvBZQ=","AD+yl4nRexwrClLzFeHozdiNJCcn/I1+RYKAHwWjS1k=","GdGGEDCn9AHLU8bC6QsDzqCF0F6FrVGq/DbA0jdsdHg=","JOxehD0Ia2qECvgPvgPIKzre8/cQHN9MO2g4fp1yR/k=","G/rWV3lgtPIpEYF5YPJKfzOsHOIGjaMkj+Jl5+W/Vs4=","FkShbCYN5Dmq5ePg/edPHJsgCkDWa5sbttdOntkfrs8=","Fw4CMs3uC4DLmXzr+1Z8YL5nifrZ28ERj7PJmKOAyjc=","G7sBVubhUFfBP7dPh1j5pJHUQKgIt8Urv8uKSlmDIfw=","H68ykCMipvlmIvsqggAZdTN2sVuURSsZjT62jaC7yw4=","L+fBG2iAWzztMIp9v/FVKvmA32J5y/xyK+Gkfk6/7oc=","IHcZ3Kq0U/3YtilfJZnfL3lBZZ7vF6UXDWC+YQjx9bw=","BOc4UULtPNPKmGuvrWUC5oQcEw1UoytqQRbtqRg/+E0=","F/aouPJyLNduMVnva1aMSV04QPRc0MhDhgTP2/L0ZKs="]}

Where

• AccountIndex: account index used to verify;
• AccountIdHash: account hash id which contains user info
• Root: account tree root published by cex;
• Assets: all user assets info;
• Proof: user merkle proof which usesbase64 encoding;
• TotalEquity: user total equity which is calculated by all the assets equity multipy its corresponding price
• TotalDebt: user total debt which is calculated by all the assets debt multipy its corresponding price
• Loan/Margin/PortfolioMargin: user collateral value

Run the following command to verify single user proof:

cd verifier; go run main.go -user

Run the following command to remove only kvrocks data:

cd src/dbtool; go run main.go -only_delete_kvrocks

Run the following command to delete kvrocks data and mysql:

cd src/dbtool; go run main.go -delete_all

Run the following command to get cex assets info in json format:

cd src/dbtool; go run main.go -query_cex_assets

Run the following command to query user config which is used in verifier:

cd src/dbtool; go run main.go -query_account_data 9

Run the following command to query witness data which is the input of circuit:

cd src/dbtool; go run main.go -query_witness_data 9

userproof service provides a command flag-memory_tree which can construct account tree in memoryusing user balance sheet.

Run the following command:

cd userproof; go run main.go -memory_tree

Compare the account tree root in the output log with the account tree root bywitness service, if matches, then the account tree is correctly constructed.

Note: whenuserproof service runs in the-memory_tree mode, its performance is about 75k per minute, so 3000w accounts will take about ~7 hours