- Notifications
You must be signed in to change notification settings - Fork15
Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake
License
snowplow/sql-runner
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
Run playbooks of SQL scripts in series and parallel on Snowflake DB, Amazon Redshift and PostgreSQL.
Used withSnowplow for scheduled SQL-based transformations of event stream data.
| Setup Guide | User Guide |
|---|---|
 |  |
Assuminggo,docker anddocker-compose are installed:
host> git clone https://github.com/snowplow/sql-runner host>cd sql-runner host> make setup-up# Launches Consul + Postgres for testing host> make# Builds sql-runner binaries host> maketest# Runs unit tests# DISTRO specifies which binary you want to run integration tests with host> DISTRO=darwin make integration
Note: You will need to ensure that~/go/bin is on your PATH forgox to work - the underlying tool that we use for building the binaries.
When you are done with testing simply executemake setup-down to terminate the docker-compose stack.
To reset the testing resources executemake setup-reset which will rebuild the docker containers. This can be useful if the state of these systems gets out of sync with what the tests expect.
To remove all build files:
guest> make cleanTo format the golang code in the source directory:
guest> make formatNote: Always runmake format before submitting any code.
Note: Themake test command also generates a code coverage file which can be found atbuild/coverage/coverage.html.
First either compile the binary from source using the abovemake command or download the published Binary directly from the GitHub release:
sql-runner version: 0.10.1Run playbooks of SQL scriptsin series and parallel on Redshift and PostgresUsage: -checkLock string Checks whether the lockfile already exists -consul string The address of a consul server with playbooks and SQL files storedin KV pairs -consulOnlyForLock Willread playbooks locally, but use Consulfor locking. -deleteLock string Will attempt to delete a lockfileif it exists -dryRun Runs through a playbook without executing any of the SQL -fillTemplates Will print all queries after templates are filled -fromStep string Starts from a given step definedin your playbook -help Shows this message -lock string Optional argument which checks and sets a lockfile to ensure this run is a singleton. Deletes lock on run completing successfully -playbook string Playbook of SQL scripts to execute -runQuery string Will run a single queryin the playbook -showQueryOutput Will print all output from queries -softLock string Optional argument, like'-lock' but the lockfile will be deleted evenif the run fails -sqlroot string Absolute path to SQL scripts. Use PLAYBOOK, BINARY and PLAYBOOK_CHILDfor those respective paths (default"PLAYBOOK") -var value Variables to be passed to the playbook,in the key=value format -version Shows the program version
SQL Runner is copyright 2015-2022 Snowplow Analytics Ltd.
Licensed under theApache License, Version 2.0 (the "License");you may not use this software except in compliance with the License.
Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS,WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.See the License for the specific language governing permissions andlimitations under the License.
About
Run templatable playbooks of SQL scripts in series and parallel on Redshift, PostgreSQL, BigQuery and Snowflake