Repository files navigation

Storing and reading BLOBs to and from a cloud service. The project will consist of a library, command line tool, Apache HTTP Server module and an Nginx module. Currently, the command line tool is work in progress.

• cmake
• APR, APR Util
• cURL
• OpenSSL

SeeReleases

The command line tool is controlled via env variables and command line arguments. Command line arguments take priority and overwrite env variables settings. If the action is set toWRITE_BLOB and neitherMCS_PATH_TO_FILE nor--path_to_file is specified, the command line tool tries to read from stdin. Similarly, when then action is set toREAD_BLOB, and neitherMCS_PATH_TO_FILE nor--path_to_file is specified, the command line tool writes the blob contents to stdout.

MCS_ACTION=READ_BLOB|WRITE_BLOB|DELETE_BLOB|LIST_BLOBS|CREATE_CONTAINERMCS_AZURE_STORAGE_KEY=Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==MCS_AZURE_STORAGE_ACCOUNT=devstoreaccount1MCS_AZURE_BLOB_NAME=testMCS_AZURE_CONTAINER=your-containerMCS_PATH_TO_FILE=/tmp/mehMCS_BLOB_STORE_URL=127.0.0.1:10000MCS_TEST_REGIME=true|false

--action--azure_storage_key--azure_storage_account--blob_name--azure_container--path_to_file--blob_store_url--test_regime

• On Linux, get yourAzurite docker container up and running
• On Windows, installAzure Storage Emulator
• Use either--test_regime true parameter orMCS_TEST_REGIME=true env var as depicted in the examples below

./mod_cloud_storage \--action CREATE_CONTAINER \--azure_storage_key Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw== \--azure_storage_account devstoreaccount1 \--azure_container my-first-container \--blob_store_url 127.0.0.1:10000 \--test_regime true

Note the storage account is always called devstoreaccount1 by definition in all Azure Storage emulator implementations. The same holds for the storage key. Container and blob names are arbitrary.

• Let's create a test file, e.g.echo "Silence is golden." > /tmp/test

./mod_cloud_storage \--action WRITE_BLOB \--azure_storage_key 'Eby8vdM02xNOcqFlqUwJPLlmEtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/K1SZFPTOtr/KBHBeksoGMGw==' \--azure_storage_account 'devstoreaccount1' \--azure_container mod-cloud-storage \--blob_store_url localhost:10000 \--test_regime true \--blob_name 'testblob' < /tmp/test

Output (DEBUG):

Reading from stdinBLOB to be uploaded length: 19BLOB to be uploaded MD5sum base64: YmT6qz9/oJawL1eym/1EQQ==BLOB to be uploaded MD5sum hex:    6264faab3f7fa096b02f57b29bfd4441 libcurl said: No errorResponse Code: 201Response Body followed by \n:(null)

Note the tool is reading from stdin via pipe now...< /tmp/test. One could use eitherMCS_PATH_TO_FILE env var or--path_to_file parameter instead. If all aforementioned is specified,parameter takes precedence overenv var and env var takes precedence overstdin.

./mod_cloud_storage \--action READ_BLOB \--azure_storage_key 'Eby8vdM02xNOcqFlqUwJPLlm\EtlCDXJ1OUzFT50uSRZ6IFsuFq2UVErCz4I6tq/\K1SZFPTOtr/KBHBeksoGMGw==' \--azure_storage_account 'devstoreaccount1' \--azure_container mod-cloud-storage \--blob_store_url localhost:10000 \--test_regime true \--blob_name 'testblob'

NeitherMCS_PATH_TO_FILE nor--path_to_file was specified, so the content of the BLOB was written to stdout. Output (DEBUG):

 libcurl said: No errorResponse Code: 200Response Body followed by \n:Silence is golden.

Note that if a path to the file is specified, the content of the file isoverwritten, not appended to.

Exactly the same as reading the BLOB but for theaction parameter:

--action DELETE_BLOB

Output (DEBUG):

 libcurl said: No errorResponse Code: 202Response Body followed by \n:(null)

Note that the current implementation deletesall blob's snapshots too. Regarding leases, the current implementation does not deal with leases on blobs, so if there is a lease on the blob, the delete action fails.

• Create your Azure profile if you don't have one.
• Create a Storage account if you don't have one, Name, Resource group, all arbitrary. If you need help with this terminology, readAzure Storage manual. In this example, we use new Storage accountkarmdelete.
  
  
• Copy yourAccount name and one of the twoStorage keys.
  
  
• You are ready to use the command line tool now. First, create a container called, e.g.my-first-container.

  ./mod_cloud_storage \--action CREATE_CONTAINER \--azure_storage_key 'YourStorageKey' \--azure_storage_account karmdelete \--azure_container my-first-container

• Let's create a test file, e.g.echo "Silence is golden." > /tmp/test
• Upload the file as a BLOB called e.g.testblob to your containermy-first-container within your Storage accountkarmdelete.

  ./mod_cloud_storage \--action WRITE_BLOB \--azure_storage_key 'YourStorageKey' \--azure_storage_account karmdelete \--azure_container my-first-container \--blob_name 'testblob' \--path_to_file /tmp/test

  Output (DEBUG):

  Reading from file.BLOB to be uploaded length: 19BLOB to be uploaded MD5sum base64: YmT6qz9/oJawL1eym/1EQQ==BLOB to be uploaded MD5sum hex:    6264faab3f7fa096b02f57b29bfd4441 libcurl said: No errorResponse Code: 201Response Body followed by \n:(null)

• One may check the result on Azure Portal, click on Blobs in your Storage account:
  
  
• If you download the blob, it contains 19 bytes "Silence is golden.\n"
• Let's read the contents of the BLOB withREAD_BLOB action:

  ./mod_cloud_storage \--action READ_BLOB \--azure_storage_key 'YourStorageKey' \--azure_storage_account karmdelete \--azure_container my-first-container \--blob_name 'testblob' \--path_to_file /tmp/test_out

  Output (DEBUG):

   libcurl said: No errorResponse Code: 200Writing to file: /tmp/test_out19 bytes of response written to /tmp/test_out.

• One can delete the aforementioned blob with the same command as for reading it but for the action attribute which is supposed to change to:DELETE_BLOB.
• Do not use a storage account with valuable containers in it. This tool is just a toy.