- Notifications
You must be signed in to change notification settings - Fork8
Docker image for Apiary Data Lake metastore
License
NotificationsYou must be signed in to change notification settings
ExpediaGroup/apiary-metastore-docker
Folders and files
| Name | Name | Last commit message | Last commit date | |
|---|---|---|---|---|
Repository files navigation
For more information please refer to the mainApiary project page.
| Environment Variable | Required | Description |
|---|---|---|
| APIARY_S3_INVENTORY_PREFIX | No | Prefix used by S3 Inventory when creating data in the inventory bucket. Default isEntireBucketDaily. |
| APIARY_S3_INVENTORY_TABLE_FORMAT | No | Format of S3 inventory data. Valid options areORC,Parquet, orCSV. Default isORC. |
| APIARY_SYSTEM_SCHEMA | No | Name for internal system database. Default isapiary_system. |
| AWS_REGION | Yes | AWS region to configure various AWS clients. |
| AWS_WEB_IDENTITY_TOKEN_FILE | No | Path of the AWS Web Identity Token File for IRSA/OIDC AWS authentication. |
| DATANUCLEUS_CONNECTION_POOLING_TYPE | No | Type of connection pooling. Valid options areBoneCP,DBCP,DBCP2,C3P0,HikariCP. |
| DATANUCLEUS_CONNECTION_POOL_MAX_POOLSIZE | No | Maximum pool size for the connection pool. |
| DATANUCLEUS_CONNECTION_POOL_MIN_POOLSIZE | No | Minimum pool size for the connection pool. |
| DATANUCLEUS_CONNECTION_POOL_INITIAL_POOLSIZE | No | Initial pool size for the connection pool (C3P0 only). |
| DATANUCLEUS_CONNECTION_POOL_MAX_IDLE | No | Maximum idle connections for the connection pool. |
| DATANUCLEUS_CONNECTION_POOL_MIN_IDLE | No | Minimum idle connections for the connection pool. |
| DATANUCLEUS_CONNECTION_POOL_MIN_ACTIVE | No | Maximum active connections for the connection pool (DBCP/DBCP2 only). |
| DATANUCLEUS_CONNECTION_POOL_MAX_WAIT | No | Maximum wait time for the connection pool (DBCP/DBCP2 only). |
| DATANUCLEUS_CONNECTION_POOL_VALIDATION_TIMEOUT | No | Validation timeout for the connection pool (DBCP/DBCP2/HikariCP only). |
| DATANUCLEUS_CONNECTION_POOL_LEAK_DETECTION_THRESHOLD | No | Leak detection threshold for the connection pool (HikariCP only). |
| DATANUCLEUS_CONNECTION_POOL_LEAK_MAX_LIFETIME | No | Maximum lifetime for the connection pool (HikariCP only). |
| DATANUCLEUS_CONNECTION_POOL_AUTO_COMMIT | No | Auto commit for the connection pool (HikariCP only). |
| DATANUCLEUS_CONNECTION_POOL_IDLE_TIMEOUT | No | Idle timeout for the connection pool (HikariCP only). |
| DATANUCLEUS_CONNECTION_POOL_CONNECTION_WAIT_TIMEOUT | No | Connection wait timeout for the connection pool (HikariCP only). |
| DATANUCLEUS_CONNECTION_POOL_READ_ONLY | No | Read only mode for the connection pool (HikariCP only). |
| DATANUCLEUS_CONNECTION_POOL_NAME | No | Connection pool name (HikariCP only). |
| DATANUCLEUS_CONNECTION_POOL_CATALOG | No | Connection pool catalog (HikariCP only). |
| DATANUCLEUS_CONNECTION_POOL_REGISTER_MBEANS | No | Register MBeans for the connection pool (HikariCP only). |
| DISALLOW_INCOMPATIBLE_COL_TYPE_CHANGES | No | true/false value for hive.metastore.disallow.incompatible.col.type.changes, defaulttrue. |
| ENABLE_GLUESYNC | No | Option to turn on GlueSync Hive Metastore listener. |
| ENABLE_HIVE_LOCK_HOUSE_KEEPER | No | Option to turn on Hive Metastore Hive Lock House Keeper. |
| ENABLE_METRICS | No | Option to enable sending Hive Metastore and JMX metrics to Prometheus. |
| ENABLE_S3_INVENTORY | No | Option to create Hive tables on top of S3 inventory data if enabled inapiary-data-lake. Enabled if value is not null/empty. |
| ENABLE_S3_LOGS | No | Option to create Hive tables on top of S3 access logs data if enabled inapiary-data-lake. Enabled if value is not null/empty. |
| EXTERNAL_DATABASE | No | Option to enable external database mode, when specified it disables managing Hive Metastore MySQL database schema. |
| GLUE_PREFIX | No | Prefix added to Glue databases to handle database name collisions when synchronizing multiple Hive Metastores to the Glue catalog. |
| HADOOP_HEAPSIZE | No | Hive Metastore Java process heapsize. Default is1024. |
| HMS_AUTOGATHER_STATS | No | Whether or not to create basic statistics on table/partition creation. Valid values aretrue orfalse. Default istrue. |
| LIMIT_PARTITION_REQUEST_NUMBER | No | To protect the cluster, this controls how many partitions can be scanned for each partitioned table. The default value-1 means no limit. The limit on partitions does not affect metadata-only queries. |
| HIVE_METASTORE_ACCESS_MODE | No | Hive Metastore access mode, applicable values are: readwrite, readonly. |
| HIVE_DB_NAMES | No | Comma separated list of Hive database names, when specified Hive databases will be created and mapped to corresponding S3 buckets. |
| HIVE_METASTORE_LOG_LEVEL | No | Hive Metastore service Log4j log level. Default isINFO. |
| HMS_MIN_THREADS | No | Minimum size of the Hive metastore thread pool. Default is200. |
| HMS_MAX_THREADS | No | Maximum size of the Hive metastore thread pool. Default is1000. |
| INSTANCE_NAME | Yes | Apiary instance name, will be used as prefix on most AWS resources to allow multiple Apiary instance deployments. |
| KAFKA_BOOTSTRAP_SERVERS | No | Kafka Bootstrap Servers to enable Kafka Metastore listener and send Metastore events to Kafka. |
| KAFKA_CLIENT_ID | No | Kafka label you define that names the Kafka producer. |
| KAFKA_COMPRESSION_TYPE | No | Kafka Compression type, if none is specified there is no compression enabled. Values available are gzip, lz4 and snappy. Default is1048576. |
| KAFKA_MAX_REQUEST_SIZE | No | The maximum size of a request in bytes. This setting will limit the number of record batches the producer will send in a single request to avoid sending huge requests. This is also effectively a cap on the maximum uncompressed record batch size. |
| LDAP_BASE | No | LDAP base DN used to search for user groups. |
| LDAP_CA_CERT | No | Base64 encoded Certificate Authority Bundle to validate LDAP SSL connection. |
| LDAP_SECRET_ARN | No | LDAP bind DN SecretsManager secret ARN. |
| LDAP_URL | No | Active Directory URL to enable group mapping in metastore. |
| MYSQL_CONNECTION_DRIVER_NAME | No | Hive Metastore MySQL database JDBC connection Driver Name. Default iscom.mysql.jdbc.Driver. |
| MYSQL_CONNECTION_POOL_SIZE | No | MySQL Connection pool size for Hive Metastore. Default is10. Seehere for more info. |
| MYSQL_DB_HOST | Yes | Hive Metastore MySQL database hostname. |
| MYSQL_DB_NAME | Yes | Hive Metastore MySQL database name. |
| MYSQL_SECRET_ARN | Yes | Hive Metastore MySQL SecretsManager secret ARN. |
| MYSQL_SECRET_USERNAME_KEY | No | Hive Metastore MySQL SecretsManager secret username key. Default isusername. |
| MYSQL_TYPE | No | Hive Metastore MySQL database Type (mariadb, mysql). Default ismysql. |
| MYSQL_DRIVER_JAR | No | Hive Metastore MySQL connector JAR location. Default is/usr/share/java/mysql-connector-java.jar. |
| RANGER_AUDIT_DB_URL | No | Ranger audit database JDBC URL. |
| RANGER_AUDIT_SECRET_ARN | No | Ranger audit database secret ARN. |
| RANGER_AUDIT_SOLR_URL | No | Ranger Solr audit URL. |
| RANGER_POLICY_MANAGER_URL | No | Ranger admin URL from where policies will be downloaded. |
| RANGER_SERVICE_NAME | No | Ranger service name used to configure RangerAuth plugin. |
| SNS_ARN | No | The SNS topic ARN to which metadata updates will be |
|If you would like to ask any questions about or discuss Apiary please join our mailing list at
https://groups.google.com/forum/#!forum/apiary-user
This project is available under theApache 2.0 License.
Copyright 2018-2019 Expedia, Inc.
About
Docker image for Apiary Data Lake metastore
Topics
Resources
License
Code of conduct
Stars
Watchers
Forks
Packages0
No packages published