- Notifications
You must be signed in to change notification settings - Fork112
Commit328caa5
authored
Github workflows for building environments (#195)
* Run the correct installer file* Run the installer from the root directory* Try a self hosted github runner* Reduce logging for docker pull.* Adds quiet flag to docker pull command* Pull the images before expect to reduce run time* Install docker early in order to speed up install* Builds the right docker-compose file* Increase timeout for linux install expect script* Change timeout on expect script* Change the way expect watches the script* Expand the timeout when waiting for Elasticsearch* Search for more output in the expect script* Change the match for the dots in expect* Change the regex for matching dots* Change the output for catching dots* Add chrome to Dockerfile for selenium* Import selenium tests and run python tests* Activate venv when running tests* Correct path for venv in the container* Correct path for venv in the container* Running only linux tests* Adjust scripts to run as a non super user* Change the permissions on the output log to source for environment variables later* Check for output log* Make output log available to test instantiation* Change pytest cache dir to home for user* Change pytest cache dir to home for user* Change pytest cache dir permissions* Hide get-docker.sh from installs* Cleanup test files in workflow* Add the cluster workflow for github actions* Adds a cluster build* Run the test cluster in pwsh* Fail pipeline when commands fail* Catch the error from powershell* Remove duplicate run command* Set env vars explicitly* Modify the escape char for env vars* Try a different method of catching errors in pwsh script* Check failure of pwsh script* Test successful run of build_cluster* Test failure of script* Capture the output from the az commands* Continue on error condition* Simplify run command* Try catching failures in a new way.* Test failure capture* Setting error action to continue* Remove ErrorAction* Use docker-compose run instead* Capture exit code to fail step* Try propigating errors from pwsh* Capture external command exit code* Send lastexitcode* Don't exit right away* Disable immediate stop on exit* Run simple test for exit code* Cd to docker compose file* Catch exec exit code* Remove unneded flags from the command* Adds back in the build script* Adds an explicit exit for powershell script* Remove spaces after escape character* Escape the exitcode variable in the shell command* Remove extra exit from build_cluster.ps1* Add a passing command for build_cluster.ps1* Move to the install directory* Run setup testbed to get an error* Try to build a cluster with the build_cluster.ps1 script.* Check resource group variable* Set the resource group name differently* Build a cluster using the generated resource group* Make the paths relative in the build_cluster script* Move to the right directory to do an install* Destroy cluster on pipeline finish* Change the owner of the files to match the host in the development container* Su user to remove testing files* Run the docker-compose as root to clean up* Run as root to clean up containers* Build the cluster in azure* List the files in the current directory on exec* Run the files from the new path* Investigate more about the file environment* Update the envornment for building the cluster* Update the environment users before docker up* Try to start hung job* List all the files with their owners in the container* Escape the powershell commands* Check the paths and files with bash* Find the path we are on* Check powershell environment* Cd to home directory in powershell* Cd to home directory in powershell* Rebuild docker compose as the right user* Change directory to source directory for powershell* Change to proper directory for powershell* Build a full cluster in pipeline* Run the linux tests and check permissions of files* Change permissions on output file with sudo* Turn off cluster creation for speed* Comment out building cluster in steps* Only delete the resource group if it exists* Adds ability to get the public ip for fw rules* Put the tags in quotes when creating nsg rules* Output the command being run for nsg rules* Remove tags for nsg port definitions* Install lme on the cluster* Builds the full cluster install* Cleans up the useage of the environment variables in pipeline* Extract environment variables from the build script and use them in the GitHub workflow.* Do a minimal linux install* Fix the path for retrieving env vars* Check setting of github env* Source the env file and push it to github env* Print some debug information to the console* Check setting of each key in functions* Parse the output for the passwords better* Uses a unique id instead of run_id to make sure it is unique* Double quote the file name for sed in output.log* Changes the way we get passwords from output.log* Make sure key doesn't have newline* Escape dollar sign* Properly escape double quotes inside of docker-compose command* Escape all of the dollar signs in the compose command* Write the environment variables to the githut environment* Clean up debugging output* Remove more debugging output* Remove set e* Adds function to write passwords to a file for actions* List files in directory after writing passwords* Export the env vars in the github file* Fail the workflow if the environment is not set correctly* Clean up the environment vars for the container* Set the variables on run of the pwsh command* Run commands on the domain controller* Get the envrionment checker to pass* Update passing variables to remote script* Escape the powershell environment variables* Change the case of the resource group env var* Don't destroy cluster so we can manually test* Build the entire cluster to run commands against* Run a command on the linux machine* Run remote tests* Run minimal installs to debug tests* Fix escaping for test commands* Move to the correct directory for tests* Add continuation characters to the lines in the script* Remove nested double quotes* Uses the ip of LS1 to run the tests on* Put the cluster build command on one line* Destroy clusters at the end* Quote output log correctly on build* Run all api tests on cluster* Build full cluster and add verbose logging to pytest* Stop deleting the cluster in the destroy_cluster.ps1 script* Modify installer to use the new winlogbeat index pattern* Try to get the dns to resolve ls1* Add ls1 to the hosts file so it resolves always* Modify tests to pass on a working cluster* Skip the fragile test for mapping* Set up to run selenium tests on the cluster* Testing* Rerun build after rebasing to the right branch* Pass the minimal install flag to install lme* Build complete cluster and run all tests* Pull the images quietly if running without a terminal.* Run the simple tests on PR checkin and the longer ones when triggered* Build the linux docker container upon check in of a pr* Build lme container fresh before install* Runs an end to end build in docker and cluster* Print out the download feedback when pulling images* Build 1.4.0 branch* Build the cluster using the main branch of the repository* Allow passing branch to installers from the pipeline* Run tests from a different base branch* Remove the ampersand typo* Allow passing arguments to the installer scripts* Rearrange install arguments* Test passing arguments in install lme* Build lme without arguments* Install lme with no arguments* Run command as string in install_lme.ps1* Build by passing arguments* Run a complete build using arguments* Update the sources to allow for updating in the pipeline* Build the cluster using the latest branch* Set up the latest branch var* Runs an upgrade in the pipeline* Run the upgrade in the remote linux machine* Run upgrade on minimal install* Checks out the current branch to run an upgrade on linux* Capture the exit code of the upgrade script* Check the directories we are working in* Clone the git repository to run the upgrade* Checkout the proper branch from origin* Get the remote username and home dir for the remote server* Set the home directory for the az user* Use origin when checking out in the upgrade script* Revert the changes to deploy.sh* Set a dumb terminal to avoid terminal errors* Export the terminal variable correctly* Capture the output of the upgrade script to fail pipeline if it fails* Revert previous changes as they seemed to break upgrade* Use a different format for executing the pwsh script* Destroy the cluster when done* Output the upgrade information to the terminal* Try capturing the docker-compose output* Directly capture the output of the compose command* Fixes unbalanced quote* Build and run full cluster with an upgrade* Builds the current brand for the cluster* Add a unique id for the docker-compose so you can run multiple instances of the same docker-compose file* Adds upgrade.yml to gh workflows* Runs both a build and an upgrade* Adds upgrade to the gh workflows* Get gh to notice new workflow* Match build names to parent branch* Trigger gh to see the workflow* Get gh actions to trigger workflow* Update code to get gh to see the actions* Update code to use the new workflow module.* Trigger gh actions to run* Get gh to run workflows* Try to get gh to run workflows* Change upgrade branch pulling* Checking out branch for upgrade in a new way* Rename workflow for upgrade* Convert to docker compose* Run all three builds using docker compose and -p* Clean up docker containers* Build the docker containers fresh for the linux_only workflow* Adds readme and checks an upgrade where the upgrade version is the same as the current version* Fixes typo in the workflow file* Runs docker as sudo* Remove the privileged flag from the lme container* Try leaving the swarm on the host if running in non privileged environment* Leave the swarm on the host* Reset to run docker as privileged* Installs the current branch in linux only* Stop pruning system to see if elastc starts faster* Don't take down the docker containers to see why they aren't working* Removes the gh actions shell escape vulnerability* Remove the docker containers at end of run1 parentfcb7199 commit328caa5
File tree
28 files changed
+1243
-727
lines changed- .github
- workflows
- Chapter 3 Files
- testing
- configure
- lib
- development
- tests
- api_tests/winlogbeat
- test_data
28 files changed
+1243
-727
lines changedLines changed: 1 addition & 0 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
| 1 | + |
Lines changed: 253 additions & 4 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
2 | 2 |
| |
3 | 3 |
| |
4 | 4 |
| |
| 5 | + | |
| 6 | + | |
| 7 | + | |
| 8 | + | |
5 | 9 |
| |
6 |
| - | |
7 |
| - | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
8 | 18 |
| |
9 | 19 |
| |
10 |
| - | |
11 |
| - | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + | |
| 102 | + | |
| 103 | + | |
| 104 | + | |
| 105 | + | |
| 106 | + | |
| 107 | + | |
| 108 | + | |
| 109 | + | |
| 110 | + | |
| 111 | + | |
| 112 | + | |
| 113 | + | |
| 114 | + | |
| 115 | + | |
| 116 | + | |
| 117 | + | |
| 118 | + | |
| 119 | + | |
| 120 | + | |
| 121 | + | |
| 122 | + | |
| 123 | + | |
| 124 | + | |
| 125 | + | |
| 126 | + | |
| 127 | + | |
| 128 | + | |
| 129 | + | |
| 130 | + | |
| 131 | + | |
| 132 | + | |
| 133 | + | |
| 134 | + | |
| 135 | + | |
| 136 | + | |
| 137 | + | |
| 138 | + | |
| 139 | + | |
| 140 | + | |
| 141 | + | |
| 142 | + | |
| 143 | + | |
| 144 | + | |
| 145 | + | |
| 146 | + | |
| 147 | + | |
| 148 | + | |
| 149 | + | |
| 150 | + | |
| 151 | + | |
| 152 | + | |
| 153 | + | |
| 154 | + | |
| 155 | + | |
| 156 | + | |
| 157 | + | |
| 158 | + | |
| 159 | + | |
| 160 | + | |
| 161 | + | |
| 162 | + | |
| 163 | + | |
| 164 | + | |
| 165 | + | |
| 166 | + | |
| 167 | + | |
| 168 | + | |
| 169 | + | |
| 170 | + | |
| 171 | + | |
| 172 | + | |
| 173 | + | |
| 174 | + | |
| 175 | + | |
| 176 | + | |
| 177 | + | |
| 178 | + | |
| 179 | + | |
| 180 | + | |
| 181 | + | |
| 182 | + | |
| 183 | + | |
| 184 | + | |
| 185 | + | |
| 186 | + | |
| 187 | + | |
| 188 | + | |
| 189 | + | |
| 190 | + | |
| 191 | + | |
| 192 | + | |
| 193 | + | |
| 194 | + | |
| 195 | + | |
| 196 | + | |
| 197 | + | |
| 198 | + | |
| 199 | + | |
| 200 | + | |
| 201 | + | |
| 202 | + | |
| 203 | + | |
| 204 | + | |
| 205 | + | |
| 206 | + | |
| 207 | + | |
| 208 | + | |
| 209 | + | |
| 210 | + | |
| 211 | + | |
| 212 | + | |
| 213 | + | |
| 214 | + | |
| 215 | + | |
| 216 | + | |
| 217 | + | |
| 218 | + | |
| 219 | + | |
| 220 | + | |
| 221 | + | |
| 222 | + | |
| 223 | + | |
| 224 | + | |
| 225 | + | |
| 226 | + | |
| 227 | + | |
| 228 | + | |
| 229 | + | |
| 230 | + | |
| 231 | + | |
| 232 | + | |
| 233 | + | |
| 234 | + | |
| 235 | + | |
| 236 | + | |
| 237 | + | |
| 238 | + | |
| 239 | + | |
| 240 | + | |
| 241 | + | |
| 242 | + | |
| 243 | + | |
| 244 | + | |
| 245 | + | |
| 246 | + | |
| 247 | + | |
| 248 | + | |
| 249 | + | |
| 250 | + | |
| 251 | + | |
| 252 | + | |
| 253 | + | |
| 254 | + | |
| 255 | + | |
| 256 | + | |
| 257 | + | |
| 258 | + | |
| 259 | + | |
| 260 | + |
Lines changed: 96 additions & 6 deletions
Original file line number | Diff line number | Diff line change | |
---|---|---|---|
| |||
1 |
| - | |
| 1 | + | |
2 | 2 |
| |
3 | 3 |
| |
4 | 4 |
| |
5 |
| - | |
6 |
| - | |
7 |
| - | |
| 5 | + | |
| 6 | + | |
| 7 | + | |
8 | 8 |
| |
| 9 | + | |
| 10 | + | |
| 11 | + | |
| 12 | + | |
| 13 | + | |
| 14 | + | |
| 15 | + | |
| 16 | + | |
| 17 | + | |
9 | 18 |
| |
10 |
| - | |
11 |
| - | |
| 19 | + | |
| 20 | + | |
| 21 | + | |
| 22 | + | |
| 23 | + | |
| 24 | + | |
| 25 | + | |
| 26 | + | |
| 27 | + | |
| 28 | + | |
| 29 | + | |
| 30 | + | |
| 31 | + | |
| 32 | + | |
| 33 | + | |
| 34 | + | |
| 35 | + | |
| 36 | + | |
| 37 | + | |
| 38 | + | |
| 39 | + | |
| 40 | + | |
| 41 | + | |
| 42 | + | |
| 43 | + | |
| 44 | + | |
| 45 | + | |
| 46 | + | |
| 47 | + | |
| 48 | + | |
| 49 | + | |
| 50 | + | |
| 51 | + | |
| 52 | + | |
| 53 | + | |
| 54 | + | |
| 55 | + | |
| 56 | + | |
| 57 | + | |
| 58 | + | |
| 59 | + | |
| 60 | + | |
| 61 | + | |
| 62 | + | |
| 63 | + | |
| 64 | + | |
| 65 | + | |
| 66 | + | |
| 67 | + | |
| 68 | + | |
| 69 | + | |
| 70 | + | |
| 71 | + | |
| 72 | + | |
| 73 | + | |
| 74 | + | |
| 75 | + | |
| 76 | + | |
| 77 | + | |
| 78 | + | |
| 79 | + | |
| 80 | + | |
| 81 | + | |
| 82 | + | |
| 83 | + | |
| 84 | + | |
| 85 | + | |
| 86 | + | |
| 87 | + | |
| 88 | + | |
| 89 | + | |
| 90 | + | |
| 91 | + | |
| 92 | + | |
| 93 | + | |
| 94 | + | |
| 95 | + | |
| 96 | + | |
| 97 | + | |
| 98 | + | |
| 99 | + | |
| 100 | + | |
| 101 | + |
0 commit comments
Comments
(0)