- Notifications
You must be signed in to change notification settings - Fork0
Open-source metadata collector based on ODD Specification
License
alek-beloff/odd-parser-aws
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
ODD Parser is a lightweight service that gathers metadata from your AWS Stack using former2 clihttps://github.com/iann0036/former2/tree/master/cli.
To learn more about collector types and ODD Platform's architecture,read the documentation.
This may help you to understand which fields you need for each adapter incollector_config.yaml and also may be helpful for a new adapter developer.
PlantUML code for above diagram:domain_classes.plantuml
To regenerate picture, you have 2 options:
- Having PlantUML installed locally, do
java -jar plantuml.jar domain_classes.plantuml
- Use PyCharm or other IDE's PlantUML plugin
docker build.librariespyodbc ,confluent-kafka andgrpcio have problem during installing and building project on Mac M1.
Possible solutions
# NOTE: be aware of versions# NOTE: easiest way is to add all export statements to your .bashrc/.zshrc file# pyodbc dependenciesbrew install unixodbc freetds opensslexport LDFLAGS="-L/opt/homebrew/lib -L/opt/homebrew/Cellar/unixodbc/2.3.11/include -L/opt/homebrew/Cellar/freetds/1.3.17/lib -L/opt/homebrew/Cellar/openssl@1.1/1.1.1t/lib"export CFLAGS="-I/opt/homebrew/Cellar/unixodbc/2.3.11/include -I/opt/homebrew/opt/freetds/include"export CPPFLAGS="-I/opt/homebrew/include -I/opt/homebrew/Cellar/unixodbc/2.3.11/include -I/opt/homebrew/opt/openssl@3/include"# cunfluent-kafkabrew install librdkafkaexport C_INCLUDE_PATH=/opt/homebrew/Cellar/librdkafka/1.9.0/includeexport LIBRARY_PATH=/opt/homebrew/Cellar/librdkafka/1.9.0/libexport PATH="/opt/homebrew/opt/openssl@3/bin:$PATH"# grpcioexport GRPC_PYTHON_BUILD_SYSTEM_OPENSSL=1export GRPC_PYTHON_BUILD_SYSTEM_ZLIB=1# mymssqlbrew install freetdsexport LDFLAGS="-L/opt/homebrew/Cellar/freetds/1.3.17/lib -L/opt/homebrew/Cellar/openssl@1.1/1.1.1t/lib"
Custom.env file for docker-compose.yaml
LOGLEVEL=DEBUGPLATFORM_HOST_URL=http://odd-platform:8080POSTGRES_PASSWORD=postgres_password_secretThere are 3 options for config field pass:
- Explicitly set it in
collector_config.yamlfile, i.edatabase: odd-platform-db - Use
.envfile or ENV variables - In situation when plugins have same field names, we can explicitly set ENV variable to
collector_config.yaml, i.e.password: !ENV ${POSTGRES_PASSWORD}
Customcollector-config.yaml
platform_host_url:http://localhost:8080default_pulling_interval:10token:""plugins: -type:postgresqlname:test_postgresql_adapterhost:"localhost"port:5432database:"some_database_name"user:"some_user_name"password:!ENV ${POSTGRES_PASSWORD} -type:mysqlname:test_mysql_adapterhost:"localhost"port:3306database:"some_database_name"user:"some_user_name"password:"some_password"
docker-compose.yaml
version:"3.8"services:# --- ODD Platform ---database:...odd-platform:...odd-collector:image:ghcr.io/opendatadiscovery/odd-collector:latestrestart:alwaysvolumes: -collector_config.yaml:/app/collector_config.yamlenvironment: -PLATFORM_HOST_URL=${PLATFORM_HOST_URL} -POSTGRES_PASSWORD=${POSTGRES_PASSWORD}depends_on: -odd-platform