- Notifications
You must be signed in to change notification settings - Fork30
deployment variable in a (secure) environment variable#349
New issue
Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.
By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.
Already on GitHub?Sign in to your account
base:master
Are you sure you want to change the base?
Uh oh!
There was an error while loading.Please reload this page.
Conversation
The argument name --dkenv is short for deployment key environmentvariable, still considering better names for this.
Still need to fine tune how best to print the private key to screen (newlines etc).Next need to use the environment key in deploy.
Getting the indentation to look nice (while deliberately notindenting the private key to make life simple for copy-and-paste).
I wouldn't try to test locally. I can give you push access to push this up directly and you can test it on Travis. I'm not clear if that gives you access to upload deploy keys so if it doesn't just let me know and I'll upload it. |
Looks like some useful error from TravisCI as is. Trying the deploy stuff as a shell script (locally), turned up another issue. The branch is here in the short term: https://github.com/peterjc/biopython/tree/deploy Called this file #!/bin/bash# Call ssh using our GitHub repository deploy key (set via -i)# using -F to make sure this ignores ~/.ssh/configssh -i"$HOME/.biopython_doc_deploy.key" -F /dev/null -p 22$* Called by: #!/bin/bashset -euo pipefail# Assumes being called from the Biopython repository's root folder,# (i.e. a clone of https://github.com/biopython/biopython) as part# of our continuous integration testing to save the compiled docs# to https://github.com/biopython/docs## In order to have write permissions, we put a private key into the# TravisCI settings as a secure environment variable, and put the# matching public key into the GitHub documentation repository's# settings as a deploy key with write permissions.DEST_SLUG=biopython/docsDEST_DIR=${TRAVIS_TAG:-dev}SOURCE_DIR=${TRAVIS_BUILD_DIR:-.}/Doc/api/_build/htmlWORKING_DIR=/tmp/deploy_biopython_docs# This will make git use our deployment key,# using sed to restore line breaksecho$DOC_KEY| sed -E -e's/ RSA PRIVATE /private/g'| sed -E -e's/[[:space:]]+/\n/g'| sed -E -e's/private/ RSA PRIVATE /g'>$HOME/.biopython_doc_deploy.keychmod 600$HOME/.biopython_doc_deploy.keyexport GIT_SSH=${TRAVIS_BUILD_DIR:-$PWD}/.github/ssh_via_deploy_key.sh# Clone the destination under /tmp (public URL, no key needed)rm -rf$WORKING_DIRgit clone https://github.com/$DEST_SLUG.git$WORKING_DIR# Update the files (assumes nothing removed)mkdir -p$WORKING_DIR/$DEST_DIRcp -R$SOURCE_DIR/*$WORKING_DIR/$DEST_DIR/pushd$WORKING_DIRgit checkout gh-pages# Switch the git protocol to SSH based so we can use our keygit remote set-url origin --push git@github.com:$DEST_SLUG.gitgit add$DEST_DIR/git commit -m"Automated update" --author"TravisCI <travisci@example.org>"git push origin gh-pagespopdecho"Documentation deployed!" The key finding from these experiments is a little massaging of the public key is needed to restore the line lines - done in the above with sed, but would be clearer in Python as part of |
Another important lesson from trying this on TravisCI, Most likely we need to print the private key to screen escaped suitably for TravisCI to accept it (spaces and new lines), and unescape in the script. |
Can we base64 encode it? |
Actually the private key should already be base64 encoded. There's probably an issue with the header part of the file. This is what a private key that I just generated with So we should remove the |
I forgot that we use >>>doctr.local.generate_ssh_key()(b'-----BEGIN PRIVATE KEY-----\nMIIJRAIBADANBgkqhkiG9w0BAQEFAASCCS4wggkqAgEAAoICAQC6rPXZjhn82O3+\nvWBF0mZocRTKAdOxRvdl8ezF+P2B7+tjr0uoMveTFDSLny6dIf9j+NLGAmCywtT6\n3+/KOBZXiMFIfACcepQt9CEUWJfPZSo6/u8cjGneuZHNiT9z9Jo/PbZDAeKPGbdq\nZgTtJxb01Lo51zae5Z+rqiW5HqAwK2wpV/uz8FGvcsRN4luTjrgagG8o8Eu4AMzP\nZo0NiwSn+jY9SyrJ4ynMUXdA4AyvulYPMzDZtayNyws+31M5PQYMWmuVbdJ2wSnz\nfyvqchUay0jAnXytJ87c4wKbjrMeaAuy1QfOkpw/vMXQn8xvAepIrXvbwA+6tGne\nldXnXTQgPNtaYZC5vc0PRZAAbsAHf/5qQjk1mut9v6meYUnRZgKE8qWp0Jwo6JFi\nXWQ+9+Ba/WcqqVwtAho7vydvxKM72oMVLRhg3psYWJduQfa+p6vs6DJUBBf4cTIA\nzXg+ZJ0jZJ+UX5gmitI4p2gdstzQX95SbvLCPwxElAAOnmH5rKPR32JK08RmtwBc\nPDabQs/u+DY1EC97nb2VjM4QOH6xfc7pyYI1d0x+rPqRh8TZrZCTnLa5i8F9lMVn\nLColU0KzUdRzuBHaTFeUOuVnpPJ7j/VEmA8hFP03dCLHRt36tI++xW89Gd+I7seX\nfl1u1k0KxBuI8EVyTF8sPTz/vjS3XQIDAQABAoICAQCrlyrRVHZ83aY+fzLMW284\n16EFYpKFJLdqJOvAunPECZX0ZoCD1n2N24fFQ4fkdgi8i06rJipehwqgpFFVLyMS\nSSlpiFpDe8VTTFFP03OP9uzPl+CQ/FyglzD3ng4OdyuFsCMmCsiHQ1s+WRJ0L3dB\ni3y2iCWz/1w6vka4l/ck7/UXN8GtD9z2Ced5s/T7eLev3JjRJ7hiJZIdnqVPapbY\nFP3gb4SgWMfmAIg+wPPIX96VUDe6Fu3K1HW80Ck+tuIlXsP/chiAgmQeZ6olccIG\nhA+WxeyBedMDZUPTW2M4Mul1862ea1Nmnw2yDAEtlLQXJChywWNz+jxKlq4tYpXy\n7jgg9V6wcI5FkduRIkhVPny1ZZw1kMqycl0R8VzkR0vEBioC8WVrZhkHAEswWEhS\nhtmQ2xgstlPDkHSImpZCqWOS3v0gDxv2+9Sw27nSAi9zNsZKcfp4roDvPK3+9l2m\nOR17u1Vj9OQpMsMQ7OQy9ZH4NIQEI3DV+nf2JS9b8wFhAOhgLgnMnF5Frdpmxtcm\nU0n+gXemEl+13k8FBx7Ub4beRC5Xo++AifOs+O5GSPS9fhpMmIWHIk/e0ZATABch\nOf0VhYpamj86rxhLCQwjkvMO8EqDqFBwsY4XZVT8VilNU/M58BLKgKWxODGFj6ED\n+SUrVY+N8Nz/i8+2jDrhPQKCAQEA2/jFKpZgl99J8KkjBfhXoBxF4HSPUrIQg4s1\nK0Ewg0PMNLvAbDfXV1xxJ+xLyXNaWQwELh3oIdAe8wt2ftcSS1V4xHemf7oysBB2\n4cd2lHJy/+TjOzjocmo2cMlfp0aRKFvgt0B67gNhW6hYCas8TpKQBrVaHRdBHs+m\n5+rVkX5KQzafIYiRJ9tBmKUwO3Rk2HGHYovyuDY4C+1sNfXE5MyIYaRLQLAkvDa4\nnj9Rq6z6fI5Yb2gwYFJ6/5nT9IchjwmEzGKqGeeG761Xp7v+tJTZ1ka37AVl0qTW\nHMOnAxgQm1AJdcdG4oyEjFkWbdDD4fXq7nTuQ1azVFE4J2E2zwKCAQEA2UAci5oF\n/shgyjUGBOMSzCTpi3z2W3cjajHqUyK6DREKC6Jrldjec7EIx68DdI7th+VnWlwW\n2/7RjqfKGxkfGj7IUUFMgTlBHEq4mVFRVKhqHFhdh1FBPnH+ATn8hgOp0CxGP4v/\nqO1lLXLHe/Kar2cK8zNlbHg5/u1i6PuC3K0WbM+8CdXDvn+qwl5oUQQjKs/GZoVp\nS+cKJtiT50DH2qoOx9U6vwsguNOgWyEZgQ3Miss0NdMb+D/FBDXD2WjWPQkJ+ruR\nFdWZwR/Jx7JpboDlrNJpCZsIevunYqnR+23U+ZxBTnH9dAJkuIrSFhMm6+YviSOg\nk/uBSnX/C/F6EwKCAQEAz8f37hdnnG2VeVc6tvvzQVETjEZtz25VfPv0uCv2uDdF\nYBZtV4uTxHiUhmKE4AAvOmfIVwt25uGhKnEMeBmNtU1CK0reIk5ubLLQqMpxrx1A\nlYjOP3Ws086SKA1/ZhGZMec/p7mnpMXao+qrZk6yQ4HbvAp32XzKzWDWRsEjBTCm\n00B4JgPLITvRhW+b1L1IOM9cU/Dfz7OfU1zsVzgUyQ6OULURREReHs8NqqUi7ygQ\n37DRxkJDV+jxOBlFBfjS8TrLjwgvpxJ0+lbhspY4rLjh366jMrWSjduYSEljq9+C\naEK8/NzEj2CuH6hTMF3/eaSCSsZ2/XKbKC0j/sasLwKCAQEAgidtspkpJFYp3prb\nq0vbNCCdJmtMMMn0lqem6f2xFyjxKr041UJjK06RowgP+uGyHqtqOvFW5KAKLfwK\nEif/wTqBymRjkDub7XY6l+fm4OAxCiBKkEo221FxyoxR5HwHXWdZArM+DJeE+TB9\noJ1c3N7P6ZoOFmkE3dycWFZuNQUhnTjrP70ok1VrGR10Q61F4F0wULV2uvmE1HcG\nTRI7aZ5eUoxFsLTa+sAWnuH6pJ1+wFwzQFfkttqFjxsi5Xpwd4qVxvheWIVqoxAH\nVDNoBMMGVn6MXSvbbcqconh5C7fmU1Cws22JWdohO4o3iPAablOugOuuRVn1QIXm\nseIOrwKCAQBlMAi+1WJI3aEqW3Ej3EcfP6Vp2Y1lu/p4Wj7PLoExxV9iRudU6RxQ\ndNX8JvubJUxaXGgSMgEFT5rV5q8G8Ig7tVHG0WjgTeiW9JN+i9woWygybMl7Kcyz\nAUVDQFxsC/bLAvk0bZ+PuFpmTG+rzIeEE1o8RqF9MOnffqeD0thfdt20djC4aLdp\nkI0Wl9FGpCmuLiioXC40WOLnB1sD7ykRr6VKzPxyukESgd1WQ97SlM7ol47idTKS\n+NmfeIj1zbp5cfaAq+uwh0HZ0lYJ7Jub44BJov+HLBBScTivKUIhRrkbIyzudFwZ\nmLoKXnxd/xtDY5UfWxNqpc2iVsdyYgqf\n-----END PRIVATE KEY-----\n',b'ssh-rsa AAAAB3NzaC1yc2EAAAADAQABAAACAQC6rPXZjhn82O3+vWBF0mZocRTKAdOxRvdl8ezF+P2B7+tjr0uoMveTFDSLny6dIf9j+NLGAmCywtT63+/KOBZXiMFIfACcepQt9CEUWJfPZSo6/u8cjGneuZHNiT9z9Jo/PbZDAeKPGbdqZgTtJxb01Lo51zae5Z+rqiW5HqAwK2wpV/uz8FGvcsRN4luTjrgagG8o8Eu4AMzPZo0NiwSn+jY9SyrJ4ynMUXdA4AyvulYPMzDZtayNyws+31M5PQYMWmuVbdJ2wSnzfyvqchUay0jAnXytJ87c4wKbjrMeaAuy1QfOkpw/vMXQn8xvAepIrXvbwA+6tGneldXnXTQgPNtaYZC5vc0PRZAAbsAHf/5qQjk1mut9v6meYUnRZgKE8qWp0Jwo6JFiXWQ+9+Ba/WcqqVwtAho7vydvxKM72oMVLRhg3psYWJduQfa+p6vs6DJUBBf4cTIAzXg+ZJ0jZJ+UX5gmitI4p2gdstzQX95SbvLCPwxElAAOnmH5rKPR32JK08RmtwBcPDabQs/u+DY1EC97nb2VjM4QOH6xfc7pyYI1d0x+rPqRh8TZrZCTnLa5i8F9lMVnLColU0KzUdRzuBHaTFeUOuVnpPJ7j/VEmA8hFP03dCLHRt36tI++xW89Gd+I7seXfl1u1k0KxBuI8EVyTF8sPTz/vjS3XQ==') |
Or alternately we can change the encoding argument in |
| if we do not appear to be on Travis.""") | ||
| deploy_parser_add_argument('deploy_directory',type=str,nargs='?', | ||
| help="""Directory to deploy the html documentation to on gh-pages.""") | ||
| deploy_parser_add_argument('--dkenv',type=str,metavar="ENVVAR", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
We should be able to detect this automatically.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Maybe, personally I think the interface is much simpler and more flexible if the user picks the environment variable name(s) themselves.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
We can make a flag if the user wants to change it, but there should be a default. It should be possible to just use the variable names we already use, and detect based on the content if it is pointing to a file or if it is a private key. Most users don't really care how doctr works. They just do whatever it says to do and if it pushes their stuff up to gh-pages they are happy.
| configure_parser.set_defaults(func=configure) | ||
| configure_parser.add_argument('--force',action='store_true',help="""Run the configure command even | ||
| if we appear to be on Travis.""") | ||
| configure_parser.add_argument('--dkenv',type=str,metavar="ENVVAR", |
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
We should use the existing environment variable names. If this works we should use it by default and have the old way behind a flag (or just remove it).
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
If you would be happy making the deploy key in an environment variable the default, then maybe you are right - re-using the old environment variable names would make sense.
There was a problem hiding this comment.
Choose a reason for hiding this comment
The reason will be displayed to describe this comment to others.Learn more.
Assuming there are no issues with it we should. We will have to continue to support the old way at least for deploy for existing repos. It will need a lot of testing to make sure, once we get it working.
Right now I have been using this in bash to recover the key file: # This will make git use our deployment key, using sed to restore line breaks.# On TravisCI, seems must create the variable using '\ ' and '\n'# The sed commands unescape these (but the new line one fails on macOS):echo$DOC_KEY| sed"s/\\\ / /g"| sed's/\\n/\n/g'>$HOME/.biopython_doc_deploy.keychmod 600$HOME/.biopython_doc_deploy.keyexport GIT_SSH=${TRAVIS_BUILD_DIR:-$PWD}/.github/ssh_via_deploy_key.sh Note it won't necessarily be |
asmeurer commentedJul 2, 2019 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
The SO answer says PKCS#8 uses "BEGIN PRIVATE KEY", which is what we use: Line 328 in0f19ff7
|
Yes, |
I don't know if we really need to support that, unless it is trivial to do so. |
peterjc commentedJul 10, 2019 • edited
Loading Uh oh!
There was an error while loading.Please reload this page.
edited
Uh oh!
There was an error while loading.Please reload this page.
Took a bit of poking to work this out, but while (Note - This was a temp key, created just for testing) It looks like the slashes are treated as line continuations for the export command, so by the time we come to use it while the spaces can be recovered, the newlines have become just Instead, seems must escape spaces as |
I tried finishing this over at#350. When I try to encrypt the private key, I get Traceback (most recent call last): File"/Users/aaronmeurer/anaconda3/lib/python3.6/runpy.py", line193, in_run_module_as_main"__main__", mod_spec) File"/Users/aaronmeurer/anaconda3/lib/python3.6/runpy.py", line85, in_run_codeexec(code, run_globals) File"/Users/aaronmeurer/Documents/doctr/doctr/__main__.py", line616, in<module> sys.exit(main()) File"/Users/aaronmeurer/Documents/doctr/doctr/__main__.py", line613, inmainreturn process_args(get_parser(config=config)) File"/Users/aaronmeurer/Documents/doctr/doctr/__main__.py", line270, inprocess_argsreturn args.func(args, parser) File"/Users/aaronmeurer/Documents/doctr/doctr/__main__.py", line512, inconfigure travis_token=travis_token,**login_kwargs) File"/Users/aaronmeurer/Documents/doctr/doctr/local.py", line84, inencrypt_variablereturn base64.b64encode(key.encrypt(variable, pad)) File"/Users/aaronmeurer/anaconda3/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line448, inencryptreturn _enc_dec_rsa(self._backend,self, plaintext, padding) File"/Users/aaronmeurer/anaconda3/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line68, in_enc_dec_rsareturn _enc_dec_rsa_pkey_ctx(backend, key, data, padding_enum, padding) File"/Users/aaronmeurer/anaconda3/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line122, in_enc_dec_rsa_pkey_ctx _handle_rsa_enc_dec_error(backend, key) File"/Users/aaronmeurer/anaconda3/lib/python3.6/site-packages/cryptography/hazmat/backends/openssl/rsa.py", line136, in_handle_rsa_enc_dec_error"Data too long for key size. Encrypt less data or use a"ValueError:Data too long for key size. Encrypt less data or use a larger key size. This is probably the original reason why I didn't include the private key directly in the environment variable. |
Seehttps://stackoverflow.com/questions/55155642/python-cryptography-data-too-long-for-key-size andhttps://stackoverflow.com/a/7146463/161801. I did some testing and you can only encrypt 501 bytes with the Travis public key. Any more than that and you get the error. I think if we want to do this, we need to do the double encryption like we currently do, and put the contents of what would be github_deploy_key.enc in an environment variable. This doesn't really save the user much. Instead of doing a checkout of a file in the repo they would add the contents of said file to their .travis.yml. |
This is to demonstrate that this doesn't actually work. See#349 (comment). The privatekey is too large to be encrypted with the Travis public key.Instead we have to do what we are currently doing. Create a symmetricencryption key to encrypt the actual SSH key, and then encrypt that symmetrickey into the secure environment variable (and save the encrypted SSH privatekey to a file).
@asmeurer I'm not sure why you are trying to encrypt anything - the goal of what I'm doing is to put the (slash escaped) private keydirectly into the continuous integration system's moderately secure environment variables settings (something TravisCI offers, and hopefully others too). This is working for me now using a proof of principle bash script for deployment, seebiopython/biopython#2172 |
I didn't try to use the Travis UI. I can't imagine it would work there when it doesn't work in the API, but maybe they do something different there. The encryption is what makes the key secret. It is encrypted with the public key for the repo. It's the same thing as the |
It does let me paste it in the UI. I haven't tested yet if it actually works. |
Re:https://docs.travis-ci.com/user/environment-variables/
For my use case I don't want special values in If you are willing to trust TravisCI, then you don't need to do anything extra to secure the private key. I'm OK with that, but maybe you're not? |
I guess we can usehttps://docs.travis-ci.com/api/#settings-environment-variables. I think this API didn't exist before. I still need to test that it actually works. |
Nice - if the script can set the secret TravisCI variable via the API, that's one less manual step. Note in the UI, you can't update an existing secret variable - you have to delete and re-add it with the new value. |
FYI, my standalone script version of this has been live and working for over a year now: Apologies that I have not got back to finishing this pull request for doctr. |
Unfinished attempt to implement#348, adds a new command line option
--dkenvshort for deploy key in environment variable.As far as I can tell,
doctr configure --dbenv DOC_KEY ...works nicely. Building on the existing code this generates a key, sends the public key to the destination GitHub repository, and prints the private key to the terminal with instructions to add it to the source repository's TravisCI config.However, something is breaking on the
doctr deploy --dbenv DOC_KEY ...side, possibly in how I have been attempting to test is locally, and I don't understand enough of the design to immediately see where.e.g. Something like this with
--forceand setting the$TRAVIS_REPO_SLUGvariable to attempt to run locally: