Repository files navigation

This project intends to group all incremental changes made on Neo Blockchain Storage.

Data is organized in folder batches of 100k blocks, including files that group 1k blocks.Example: folderBlockStorage_100000 will include blocks from 1 to 100000.Example 2: block 5300 will be located on filedump-block-6000.json, that includes blocks from 5001 to 6000.

As a general rule,dump-block-X.json includes blocksX-1000+1 toX (inclusive), and folderBlockStorage_Y includes blocksY-100000+1 toY (inclusive).BlockStorage_0 anddump-block-0.json only contains block zero.

Each storage dump file is a valid json list, containing several block structs. Each block struct contains three fields:block (which is block height),size (number of storage actions recorded in that block), andstorage (which is the list of actions that happened in that block).

Ifstorage field is an empty list, thensize field will be zero, as in the next example (dump-block-0.json):

[   {"block":0,"size":0,"storage": []}]

In the beggining, no storage was used, so all arrays will be empty. Please take a look at folderBlockStorage_1500000, specially filedump-block-1445000.json (block 1444843), when Storage action begins ;)

Block actions can be one of the three:Added (when a new key/value is created),Changed (when a value is changed for a given existing key) orDeleted (when a key is removed together with its value).The hex format of key and value follows Neo standard for serialization of StorageKey and StorageItem (don't worry, we will explain these here in details ;) ).

Remember NeoHelloWorld example? It writes valueWorld on keyHello.

usingNeo.SmartContract.Framework.Services.Neo;namespaceNeo.SmartContract{publicclassHelloWorld:Framework.SmartContract{publicstaticvoidMain(){Storage.Put("Hello","World");}}}

This one currently has the following scripthashd741527ea66813c0c50e78bb403926b4c88a64c4 (in little-endian format).

When executing this contract, the following action will happen:

{"state":"Added","key":"d741527ea66813c0c50e78bb403926b4c88a64c448656c6c6f00000000000000000000000b","value":"0005576f726c6400"}

StorageKey isd741527ea66813c0c50e78bb403926b4c88a64c448656c6c6f00000000000000000000000b, that can be decomposed in two parts: scripthash (contract prefixd741527ea66813c0c50e78bb403926b4c88a64c4) and key48656c6c6f00000000000000000000000b (with 16-bytes padding).The key is written via helper WriteBytesWithGrouping, that writes information in blocks of 16-bytes, separated by a single zero byte, and finally padded with enough zeroes to match 16 size. The padding size is the last byte, in this case0x0b so 11 zeroes were added and real key is only48656c6c6f (meaningHello on ASCII).

StorageItem is0005576f726c6400 and consists of three parts: StateBase prefix (currently0x00), the desiredvalue (which is05576f726c64) and the storage attribute (usually0x00 meaning that data is not constant).Thevalue part is written via WriteVarBytes, which includes a prefix for the byte array size (via WriteVarInt) and content (via WriteVarBytes). So,0x05 indicates that the following 5 bytes will be stored:576f726c64 (which isWorld on ASCII).

If invocation is repeated on the same contract, the state action will be recorded asChanged (notAdded).Finally, if a key is deleted, storage action will be recorded asDeleted, and novalue will be presented, only thekey.

Just for short, these are the serialization rules to remember easily:

Forkey is:<ScriptHash 20-bytes> + <key 16-byte multiple zero padded>

Forvalue is<StateBase 0x00> + <data size> + <data contents> + <storage attribute 1-byte (usually 0x00)>

Using this raw data it's possible to easily reproduce past notifications inany given block. This is amazing!

This data can also be used to enforce newer standards regarding a storage hashing representation to make sure storage is correct in every network node.

Use the code available athttps://gist.github.com/ixje/810cb086970cec43b709f6ae8589b872.

python main.py -s Storage/ -o temp_output -b 2458, to read the audit files fromStorage/

write the new chain in./temp_output and, consequently, restore to blockheight2458

Data is freely available in MIT/Creative Commons.

NeoResearch 2018-2019