- Notifications
You must be signed in to change notification settings - Fork45
Generate and Visualize Data Lineage from query history
License
tokern/data-lineage
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
Tokern Lineage Engine isfast andeasy to use application to collect, visualize and analyzecolumn-level data lineage in databases, data warehouses and data lakes in AWS and RDS.
Tokern Lineage helps you browse column-level data lineage
- visually usingkedro-viz
- analyze lineage graphs programmatically using the powerfulnetworkx graph library
- Demo of Tokern Lineage App
Checkout anexample data lineage notebook.
Check outthe post on using data lineage for cost control for anexample of how data lineage can be used in production.
Download the docker-compose file from Github repository.
# in a new directory runwget https://raw.githubusercontent.com/tokern/data-lineage/master/install-manifests/docker-compose/catalog-demo.yml# or runcurl https://raw.githubusercontent.com/tokern/data-lineage/master/install-manifests/docker-compose/tokern-lineage-engine.yml -o docker-compose.ymlRun docker-compose
docker-compose up -dCheck that the containers are running.
docker psCONTAINER ID IMAGE CREATED STATUS PORTS NAMES3f4e77845b81 tokern/data-lineage-viz:latest ... 4 hours ago Up 4 hours 0.0.0.0:8000->80/tcp tokern-data-lineage-visualizer1e1ce4efd792 tokern/data-lineage:latest ... 5 days ago Up 5 days tokern-data-lineage38be15bedd39 tokern/demodb:latest ... 2 weeks ago Up 2 weeks tokern-demodbTry out Tokern Lineage App
Head tohttp://localhost:8000/ to open the Tokern Lineage app
# in a new directory runwget https://raw.githubusercontent.com/tokern/data-lineage/master/install-manifests/docker-compose/tokern-lineage-engine.yml# or runcurl https://raw.githubusercontent.com/tokern/data-lineage/master/install-manifests/docker-compose/catalog-demo.yml -o tokern-lineage-engine.ymlRun docker-compose
docker-compose up -dIf you want to use an external Postgres database, change the following parameters intokern-lineage-engine.yml:
- CATALOG_HOST
- CATALOG_USER
- CATALOG_PASSWORD
- CATALOG_DB
You can also override default values using environement variables.
CATALOG_HOST=... CATALOG_USER=... CATALOG_PASSWORD=... CATALOG_DB=... docker-compose -f ... up -dFor more advanced usage of environment variables with docker-compose,refer to docker-compose docs
Pro-tip
If you want to connect to a database in the host machine, set
CATALOG_HOST: host.docker.internal # For mac or windows#ORCATALOG_HOST: 172.17.0.1 # Linux- Postgres
- AWS Redshift
- Snowflake
- SparkSQL
- Presto
For advanced usage, please refer todata-lineage documentation
Please take thissurvey if you are a user or considering using data-lineage. Responses will help us prioritize features better.
About
Generate and Visualize Data Lineage from query history
Topics
Resources
License
Uh oh!
There was an error while loading.Please reload this page.
Stars
Watchers
Forks
Packages0
Uh oh!
There was an error while loading.Please reload this page.
Contributors3
Uh oh!
There was an error while loading.Please reload this page.
