Repository files navigation

Hoplite is a Kotlin library for loading configuration files into typesafe classes in a boilerplate-free way.Define your config using Kotlin data classes, and at startup Hoplite will read from one or more config files,mapping the values in those files into your config classes. Any missing values, or values that cannot be convertedinto the required type will cause the config to fail with detailed error messages.

• Multiple formats: Write your configuration inseveral formats: Yaml, JSON, Toml, Hocon, orJava .props files or even mix and match formats in the same system.
• Property Sources: Per-system overrides are possible from JVM system properties,environment variables, JNDI or a per-user local config file.
• Batteries included: Support for manystandard types such as primitives, enums, dates, collectiontypes, inline classes, uuids, nullable types, as well as popular Kotlin third party library types suchasNonEmptyList,Option andTupleX fromArrow.
• Custom Data Types: TheDecoder interface makes it easy to add support for your custom domain types or standardlibrary types not covered out of the box.
• Cascading: Config files can bestacked. Start with a default file and then layer newconfigurations on top. When resolving config, lookup of values falls through to the first file that contains adefinition. Can be used to have a default config file and then an environment specific file.
• Beautiful errors: Fail fast at runtime, withbeautiful errors showing exactly what went wrong and where.
• Preprocessors: Support for severalpreprocessors that willreplace placeholders with values resolved from external configs, such as AWS Secrets Manager, Azure KeyVault and so on.
• Reloadable config: Trigger configreloads on a fixed interval or in response to externalevents such as consul value changes.
• Prefix Binding: Optionally, load configuration sources once, and thenbind individual prefixpaths into independent config types.

See the list of changes in each releasehere.

Add Hoplite to your build:

implementation'com.sksamuel.hoplite:hoplite-core:<version>'

You will also need to include a module for theformat(s) you to use.

Next define the data classes that are going to contain the config.You should create a top level class which can be named simply Config, or ProjectNameConfig. This class then defines a field for each config value you need. It can include nested data classes for grouping together related configs.

For example, if we had a project that needed database config, config for an embedded HTTP server, and a field which contained which environment we were running in (staging, QA, production etc), then we may define our classes like this:

data classDatabase(valhost:String,valport:Int,valuser:String,valpass:String)data classServer(valport:Int,valredirectUrl:String)data classConfig(valenv:String,valdatabase:Database,valserver:Server)

For our staging environment, we may create a YAML (or Json, etc) file calledapplication-staging.yaml.The name doesn't matter, you can use any convention you wish.

env:stagingdatabase:host:staging.wibble.comport:3306user:thebosspass:0123abcdserver:port:8080redirectUrl:/404.html

Finally, to build an instance ofConfig from this file, and assuming the config file was on the classpath, we can simply execute:

val config=ConfigLoaderBuilder.default()               .addResourceSource("/application-staging.yml")               .build()               .loadConfigOrThrow<Config>()

If the values in the config file are compatible, then an instance ofConfig will be returned.Otherwise, an exception will be thrown containing details of the errors.

As you have seen from the getting started guide,ConfigLoader is the entry point to using Hoplite. We create aninstance of this loader class through theConfigLoaderBuilder builder. To this builder we add sources, configuration,enable reports, add preprocessors and more.

To create a default builder, useConfigLoaderBuilder.default() and after adding your sources, callbuild.Here is an example:

ConfigLoaderBuilder.default()  .addResourceSource("/application-prod.yml")  .addResourceSource("/reference.json")  .build()  .loadConfigOrThrow<MyConfig>()

Thedefault method onConfigLoaderBuilder sets up recommended defaults. If you wish to start with a completely emptyconfig builder, then useConfigLoaderBuilder.empty().

There are two ways to retrieve a populated data class from config. The first is to throw an exception if the configcould not be resolved. We do this via theloadConfigOrThrow<T> function. Another is to return aConfigResult validationmonad via theloadConfig<T> function if you want to handle errors manually.

For most cases, when you are resolving config at application startup, the exception based approach is better. This isbecause you typically want any errors in config to abort application bootstrapping, dumping errors immediately to the console.

Prefixes can be used to bind selected config to independent data classes. This is useful for modular config loading.For example, independent modules or plugins load their own config from a common set of configuration sources.

For example a yaml source containing

module1:foo:barmodule2:baz:qux

can be bound to:

data classModule1Config(valfoo:String)data classModule2Config(valbaz:String)

The best way to do this is to obtain aConfigBinder fromConfigLoader, for example:

val configBinder=ConfigLoaderBuilder.default()  .addResourceSource("/application-prod.yml")  .addResourceSource("/reference.json")  .build()  .configBinder()// generally a ConfigBinder will be provided via DI, and these calls will be in their own modules!val module1Config= configBinder.bindOrThrow<Module1Config>("module1")val module2Config= configBinder.bindOrThrow<Module2Config>("module2")

With this approach, the configuration sources will only be read and parsed a single time, but can be bound to independentdata classes as many times as is necessary.

Aprefix can also be provided directly toloadConfig and its variants if only one prefix needs to be loaded.

Theprefix value does not have to refer only to root properties -- a prefix offoo.bar will access config at thefoo.bar node in the config tree thatConfigLoader creates.

When an error does occur, if you choose to throw an exception, the errors will be formatted in a human-readable wayalong with as much location information as possible. No more trying to track down aNumberFormatException in a 400line config file.

Here is an example of the error formatting for a test file used by the unit tests. Notice that the errors indicate whichfile the value was pulled from.

Error loading config because:    - Could not instantiate 'com.sksamuel.hoplite.json.Foo' because:        - 'bar': Required type Boolean could not be decoded from a Long (classpath:/error1.json:2:19)        - 'baz': Missing from config        - 'hostname': Type defined as not-null but null was loaded from config (classpath:/error1.json:6:18)        - 'season': Required a value for the Enum type com.sksamuel.hoplite.json.Season but given value was Fun (/home/user/default.json:8:18)        - 'users': Defined as a List but a Boolean cannot be converted to a collection (classpath:/error1.json:3:19)        - 'interval': Required type java.time.Duration could not be decoded from a String (classpath:/error1.json:7:26)        - 'nested': - Could not instantiate 'com.sksamuel.hoplite.json.Wibble' because:            - 'a': Required type java.time.LocalDateTime could not be decoded from a String (classpath:/error1.json:10:17)            - 'b': Unable to locate a decoder for java.time.LocalTime

Hoplite supports config files in several formats. You can mix and match formats if you really want to.For each format you wish to use, you must include the appropriate hoplite module on your classpath.The format that hoplite uses to parse a file is determined by the file extension.

If you wish to add another format you can extendParser and provide an instance of that implementation totheConfigLoaderBuilder viaaddParser.

That same function can be used to map non-default file extensions to an existing parser. For example, if you wish tohave your config in files calledapplication.data but in yaml format, then you can register .data with the Yaml parserlike this:

ConfigLoaderBuilder.default().addParser("data", YamlParser).build()

If attempting to build a "fat Jar" while using multiple file type modules, it is essential to usethe shadowJar plugin and to add the directivemergeServiceFiles() in the shadowJar Gradle task.More info

ThePropertySource interface is how Hoplite reads configuration values.

Hoplite supports several built in property source implementations, and you can write your own if required.

TheEnvironmentVariablesPropertySource,SystemPropertiesPropertySource,UserSettingsPropertySource, andXdgConfigPropertySourcesources are automatically registered, with precedence in that order. Other property sources can be passed to the config loader builderas required.

TheEnvironmentVariablesPropertySource reads config from environment variables.This property source maps environment variable names to config properties via idiomatic conventions for environment variables.Env vars are idiomatically UPPERCASE andcontain only letters (A toZ), digits (0 to9), and the underscore (_) character.

Hoplite maps env vars as follows:

• Underscores are separators for nested config. For exampleTOPIC_NAME would override a propertyname located in atopic parent.

• To bind env vars to arrays or lists, postfix with an index e.g. set env varsTOPIC_NAME_0 andTOPIC_NAME_1 to set two values for thename list property. Missing indices are ignored, which is useful for commenting out values without renumbering subsequent ones.

• To bind env vars to maps, the key is part of the nested config e.g.TOPIC_NAME_FOO andTOPIC_NAME_BAR would set the "foo" and "bar"keys for thename map property. Note that keys are one exception to the idiomatic uppercase rule -- the env var name determines thecase of the map key.

If the optional (not specified by default)prefix setting is provided, then only env vars that begin with the prefix are considered,and the prefix is stripped from the env var before processing.

As of Hoplite 3, theEnvironmentVariablesPropertySource is applied by default and may be used to override other config propertiesdirectly. There is no longer any built-in support for theconfig.override. prefix. However, the optionalprefix setting can stillbe used for the same purpose.

TheSystemPropertiesPropertySource provides config through system properties that are prefixed withconfig.override..For example, starting your JVM with-Dconfig.override.database.name would override a config key ofdatabase.name residing in a file.

TheUserSettingsPropertySource provides config through a config file defined at ~/.userconfig.[ext] where ext is one of thesupported formats.

TheInputStreamPropertySource provides config from an input stream. This source requires a parameter that indicates what the format is. For example,InputStreamPropertySource(input, "yml")

Config from files or resources are retrieved via instances ofConfigFilePropertySource. This property source is added automatically when we passstrings to theloadConfigOrThrow orloadConfig functions.

There are convenience methods onConfigLoaderBuilder to constructConfigFilePropertySources from resources on the classpath or files.

For example, the following are equivalent:

ConfigLoader().loadConfigOrThrow<MyConfig>("/config.json")

and

ConfigLoaderBuilder.default()   .addResourceSource("/config.json")   .build()   .loadConfigOrThrow<MyConfig>()

The advantage of the second approach is that we can specify a file can be optional, for example:

ConfigLoaderBuilder.default()  .addResourceSource("/missing.yml", optional=true)  .addResourceSource("/config.json")  .build()  .loadConfigOrThrow<MyConfig>()

To use a JSON string as a property source, we can use theJsonPropertySource implementation.For example,

ConfigLoaderBuilder.default()   .addSource(JsonPropertySource(""" { "database": "localhost", "port": 1234 }"""))   .build()   .loadConfigOrThrow<MyConfig>()

To use a Yaml string as a property source, we can use theYamlPropertySource implementation.

ConfigLoaderBuilder.default()   .addSource(YamlPropertySource("""        database: "localhost"        port: 1234"""))   .build()   .loadConfigOrThrow<MyConfig>()

To use a Toml string as a property source, we can use theTomlPropertySource implementation.

ConfigLoaderBuilder.default()  .addSource(TomlPropertySource("""        database = "localhost"        port = 1234"""))  .build()  .loadConfigOrThrow<MyConfig>()

To use a java.util.Properties object as property source, we can use thePropsPropertySource implementation.

ConfigLoaderBuilder.default()  .addSource(PropsPropertySource(myProps))  .build()  .loadConfigOrThrow<MyConfig>()

Hoplite has the concept of cascading or layered or fallback config.This means you can pass more than one config file to the ConfigLoader.When the config is resolved into Kotlin classes, a lookup will cascade or fall through one file to another in the order they were passed to the loader, until the first file that defines that key.

For example, if you had the following two files in yaml:

application.yaml:

elasticsearch:port:9200clusterName:product-search

application-prod.yaml:

elasticsearch:host:prd-elasticsearch.scvport:8200

And both were passed to the ConfigLoader like this:ConfigLoader().loadConfigOrThrow<Config>("/application-prod.yaml", "/application.yaml"), then lookups will be attempted in the order the files were declared.So in this case, the config would be resolved like this:

elasticsearch.port = 8200 // the value in application-prod.yaml takes priorityelasticsearch.host = prd-elasticsearch.scv // only defined in application-prod.yamlelasitcsearch.clusterName = product-search // only defined in application.yaml

Let's see a more complicated example. In JSON this time.

default.json

{"a":"alice","b": {"c":true,"d":123  },"e": [    {"x":1,"y":true    },    {"x":2,"y":false    }  ],"f":"Fall"}

prod.json

{"a":"bob","b": {"d":999  },"e": [    {"y":true    }  ]}

And we will parse the above config files into these data classes:

enumclassSeason {Fall,Winter,Spring,Summer }data classFoo(valc:Boolean,vald:Int)data classBar(valx:Int?,valy:Boolean)data classConfig(vala:String,valb:Foo,vale:List<Bar>,valf:Season)

val config=ConfigLoader.load("prod.json","default.json")println(config)

The resolution rules are as follows:

• "a" is present in both files and so is resolved from the first file - which was "prod.json"
• "b" is present in both files and therefore resolved from the file as well
• "c" is a nested value of "b" and is not present in the first file so is resolved from the second file "default.json"
• "d" is a nested value of "b" present in both files and therefore resolved from the first file
• "e" is present in both files and so the entire list is resolved from the first file. This means that the list only contains a single element, and x is null despite being present in the list in the first file. List's cannot be merged.
• "f" is only present in the second file and so is resolved from the second file.

Hoplite can be configured to throw an error if a config value is not used.This is useful to detect stale configs.

To enable this setting, use.strict() on the config builder. For example:

ConfigLoaderBuilder.default()  .addResourceSource("/config-prd.yml",true)  .addResourceSource("/config.yml")  .strict()  .build()  .loadConfig<MyConfig>()

An example of this output is:

Error loading config because:    Config value 'drop_drop' at (classpath:/snake_case.yml:0:10) was unused    Config value 'double_trouble' at (/home/sam/.userconfig.yml:2:16) was unused

If you wish to refactor your config classes and rename a field, but you don't want to have to update all your config files,you can add a migration path by allowing a field to use more than one name. To do this we use the @ConfigAlias annotation.

For example, with this config file:

database:host:String

We can marshall this into the following data classes.

data classDatabase(valhost:String)data classMyConfig(valdatabase:Database)

or

data classDatabase(@ConfigAlias("host")valhostname:String)data classMyConfig(valdatabase:Database)

Hoplite provides an interfaceParameterMapper which allows the parameter name to be modified before it is looked upinside a config source. This allows hoplite to find config keys which don't match the exact name. The main use case forthis is to allowsnake_case orkebab-case names to be used as config keys.

For example, given the following config class:

data classDatabase(valinstanceHostName:String)

Then we can of course define our config file (using YML as an example):

database:instanceHostName:server1.prd

But because Hoplite registersKebabCaseParamMapper andSnakeCaseParamMapper automatically, we can just as easily use:

database:instance-host-name:server1.prd

or

database:instance_host_name:server1.prd

Hoplite converts the raw value in config files to JDK types using instances of theDecoder interface.There are built in decoders for all the standard day to day types, such as primitives, dates, lists, sets, maps, enums, arrow types and so on. The full list is below:

Duration types support unit strings in the following format (lower case only), with an optional space between the unit value and the unit type.

• ns,nano,nanos,nanosecond,nanoseconds
• us,micro,micros,microsecond,microseconds
• ms,milli,millis,millisecond,milliseconds
• s,second,seconds
• m,minute,minutes
• h,hour,hours
• d,day,days

For example,10s,3 days, or12 hours.

Hoplite supports what it calls preprocessors. These are just functions that are applied to every value as they are read from the underlying config file.The preprocessor is able to transform the value (or return the input - aka identity function) depending on the logic of that preprocessor.

For example, a preprocessor may choose to perform environment variable substitution, configure default values,perform database lookups, or whatever other custom action you need when the config is being resolved.

You can add custom pre-processors in addition to the built in ones, by using the functionwithPreprocessor on theConfigLoader class, and passing in an instance of thePreprocessor interface.A typical use case of a custom preprocessor is to lookup some values in a database, or from a third party secrets store such asVault orAmazon Parameter Store.

One way this can be implemented is to have a prefix, and then use a preprocessor to look for the prefix in strings, and if the prefix is present, use the rest of the string as a key to the service. ThePrefixProcessor abstract class implements this by handling the node traversal, while leaving the specific processing as an exercise for the reader.

For example

database:user:rootpassword:vault:/my/key/path

Note: You can repeatedly apply preprocessors by setting the propertywithPreprocessingIterations on theConfigLoaderBuilder to a value greater than 1.This causes looped application of all preprocessors. This can be useful if you wish to have one preprocessor resolve a value that then needs to be resolved by another preprocessor.

These built-in preprocessors are registered automatically.

These preprocessors must be added to theConfigBuilder before they take effect, and require extra modules to be added to the build.

The random preprocessor replaces placeholder strings with random values.

For example:

my.number=${random.int}my.bignumber=${random.long}my.uuid=${random.uuid}my.number.less.than.ten=${random.int(10)}my.number.in.range=${random.int[1024,65536]}

It is quite common to output the resolved config at startup for reference when debugging. In this case, the defaulttoString generated by Kotlin's data classes is very useful.However configuration typically includes sensitive information such as passwords or keys which normally you would not want to appear in logs.

To avoid sensitive fields appearing in the log output, Hoplite provides a built in type calledMasked which is a wrapper around a String.By declaring a field to have this type, the value will still be loaded from configuration files, but will not be included in the generatedtoString.

For example, you may define a config class like this:

data class Database(val host: String, val user: String, val password: Masked)

And corresponding json config:

{"host":"localhost","user":"root","password":"letmein"}

And then the output of the Database config class viatoString would beDatabase(host=localhost, user=root, password=****)

Note: The masking effect only happens if you usetoString.If you marshall your config to a String using a reflection based tool like Jackson, it will still be able to see the underlying value.In these cases, you would need to register a custom serializer.For the Jackson project, aHopliteModule object is available in thehoplite-json module.Register this with your Jackson mapper, likemapper.registerModule(HopliteModule) and thenMasked values will be ouputted into Json as "****"

Some developers, this writer included, like to have strong types wrapping simple values. For example, aPort object rather than an Int.This helps to alleviate Stringy typed development.Kotlin has support for what it calls inline classes which fulfil this need.

Hoplite directly supports inline classes.When using inline classes, you don't need to nest config keys.

For example, given the following config classes:

inlineclassPort(valvalue:Int)inlineclassHostname(valvalue:String)data classDatabase(valport:Port,valhost:Hostname)

And then this config file:

port:9200host:localhost

We can parse directly:

val config=ConfigLoader().loadConfigOrThrow<Database>("config.file")println(config.port)// Port(9200)println(config.host)// Hostname("localhost")

Hoplite will support sealed classes where it is able to match up the available config keys with the parametersof one of the implementations. For example, lets create a config hierarchy as implementations of a sealed class.

sealedclassDatabase {data classElasticsearch(valhost:String,valport:Int,valindex:String) : Database()data classPostgres(valhost:String,valport:Int,valschema:String,valtable:String) : Database()}data classTestConfig(valdatabases:List<Database>)

For the above definition, if hoplite encountered ahost,port, andindex then it would be clearthat it should instantiate anElasticsearch instance. Similarly, if the config keys werehost,port,schema, andtable, then thePostgres implementation should be used. If the keys don't match an implementation,the config loader would fail. If keys match multiple implementations then the first match is taken.

For example, the following yaml config file could be used:

databases:  -host:localhostport:9200index:foo  -host:localhostport:9300index:bar  -host:localhostport:5234schema:publictable:faz

And the output would be:

TestConfig(  databases=[    Elasticsearch(host=localhost, port=9200, index=foo),    Elasticsearch(host=localhost, port=9300, index=bar),    Postgres(host=localhost, port=5234, schema=public, table=faz)  ])

Hoplite additionally supports using objects in sealed classes.For example, lets expand the database definition to include anEmbedded object subclass:

sealedclassDatabase {data classElasticsearch(valhost:String,valport:Int,valindex:String) : Database()data classPostgres(valhost:String,valport:Int,valschema:String,valtable:String) : Database()object Embedded : Database()}data classTestConfig(valdatabases:List<Database>)

We can indicate to Hoplite to use theEmbedded option in two ways. The first is by referencing the type name:

For example, in yaml:

database:Embedded

Or in Json:

{"database":"Embedded"}

This also works for lists, and we can mix and match:

Yaml:

databases:  -"Embedded"  -host:localhostport:9300index:bar

Json:

{"databases": ["Embedded", {"host":"localhost","port":9200,"index":"foo" }]}

The second method is only for Json by specifying an empty object:

{"database": { }}

When using the second option, there must be only a single object instance in the hierarchy, otherwise a disambiguation error is thrown.If you want to support multiple object instances, then refer to the type by name.

Hoplite embraces immutable config, but if you require that config is dynamic, then Hoplite provides aReloadableConfig wrapper.This functionality is available by adding the modulehoplite-watch to your build. The reloader requires aConfigLoaderand then accepts one or moreWatchables which cause the config to be reloaded when whatever they are watching is triggered.

To be clear, once Hoplite has parsed a config object, it won't mutate that object. This reloadable wrapper will, in the background,reload the config once a watcher is triggered. Then you can obtain the latest parsed config whenever you wish by using the methodgetLatest.

If you wish to be notified whenever the config is reloaded, you can callsubscribe on the reloader.

A simple example would be to referesh config every 10 seconds:

// create a watchable that will trigger every 10 secondsval watcher=FixedIntervalWatchable(10.seconds)// create our config loader which will parse config when invokedval loader=ConfigLoaderBuilder.default()  .addSource(PropertySource.resource("/application.yml"))  .build()// create the reloader, adding the watcher, the config loader, and specifying the target config classval reloader=ReloadableConfig(configLoader,TestConfig::class)  .addWatcher(watcher)// obtain the latest config whenever we wantreloader.getLatest()// or subscribe for notifications: (TestConfig) -> Unitreloader.subscribe {println("New config!:$it") }

You can implement theWatchable interface directly, with whatever triggering logic you wish, or use one of thepredefined implementations:

Hoplite makes available several other modules that add functionality outside of the main core module. They are inseperate modules because they bring in dependencies from those projects and so the modules are optional.

GraalVM native image example can be found inside anexample-native subdirectory.

This software is licensed under the Apache 2 license, quoted below.Copyright 2019 Stephen SamuelLicensed under the Apache License, Version 2.0 (the "License"); you may notuse this file except in compliance with the License. You may obtain a copy ofthe License at    http://www.apache.org/licenses/LICENSE-2.0Unless required by applicable law or agreed to in writing, softwaredistributed under the License is distributed on an "AS IS" BASIS, WITHOUTWARRANTIES OR CONDITIONS OF ANY KIND, either express or implied. See theLicense for the specific language governing permissions and limitations underthe License.