Repository files navigation

This is a replacement for, and rewrite oftravis-yaml.

This library is used for parsing, normalizing, and validating Travis CI buildconfiguration (.travis.yml).

It serves these main functions:

• Defining and producing a specification for our build configuration format in the form of JSON Schema.
• Merging actual build configuration parts (such as.travis.yml)
• Applying the spec to a merged build configuration in order to normalize and validate it.
• Expanding a build config to job matrix configs.
• Generating reference documentation from the spec.

The specification (schema) produced by the library is being used for bothnormalizing and validating build configs, and for generating referencedocumentation. I.e. these two parts of the code base are clients to theschema.

Applying the specification to a build configuration produces structuredmessages on the levelsinfo,warn, anderror. Such messages include amessage code (e.g.unknown_key,deprecated_key etc.) and arguments(e.g. the given key, value) so they can be translated to human readablemessages by clients. These can be used in the UI, and links to thedocumentation, suggesting fixes.

For example:

yaml='rvm: 2.3'config=Travis::Yml.load(yaml)config.serialize# {#   language: 'ruby',#   os: ['linux'],#   rvm: ['2.3']# }config.msgs# [#   [:info, :language, :default, key: :language, default: 'ruby'],#   [:info, :language, :default, key: :os, default: 'linux'],# ]config.msgs.map{ |msg|Travis::Yml.msg(msg)}# [#   '[info] on root: missing :language, using the default: "ruby"',#   '[info] on root: missing :os, using the default: "linux"',# ]

The library uses a custom YAML parser based on Psych, using its safe YAMLparsing features.

Diverging from the YAML specification this library does not:

• convert the stringsyes oron to the booleantrue, or the stringsnooroff to the booleanfalse
• convert YAML integers or floats to Ruby integers or floats
• convert symbol keys to symbols

For the following reasons:

• Deployment config uses the keyon. This being converted totrue as a hashkey does not make sense for us.
• Version numbers like1.10 used to be truncated to1.1 by the YAML parser,causing confusion, and requiring users to specify these as strings ("1.10")instead.
• Keys need to be strings so they can carry additional meta information (suchas the source name of the build config, and the line number for messages).

The parser uses the classesKey,Map, andSeq for representing Ruby stringsthat are hash keys, hashes, and arrays, in order to be able to carry additionalmeta information. This is then used while merging config parts, and normalizingand validating the resulting config.

Integers and floats are not converted by the YAML parser, but they are castedlater by the build config normalization, so they can be casted only if therespective node wants an integer or float.

Borrowing from YAML's terminology the library uses:

• map for hashes (key value mappings)
• seq for arrays (sequence)

It also uses the following scalar types:

• str for strings
• num for numbers
• bool for booleans
• secure for secure strings

Scalars can be enums, as defined by JSON Schema, i.e. they can have a number ofknown and allowed values. Examples for keys that hold enums are:language,os,dist etc.

Amap can be strict or not strict. Maps are strict by default. A strict mapdisallows keys that are not known.

Asecure resembles our build config format for encrypted key/value pairs.Secures also can be strict or not strict, and they are strict by default. A notstrict secure would accept a plain string without producing a warning (e.g. onkeys likeusername oremail).

The spec is included to the repository asschema.json.

The spec is defined inSchema::Def, and can be produced by:

Travis::Yml.schema

Classes inSchema::Def use classes inSchema::Type to build a tree of nodesthat define allowed keys, types, and various options in a readable and succinctway (using a DSL for describing the schema).

Nodes on this tree that match certain well-known patterns are then transformedaccording to these patterns using classes inSchema::Type::Form. E.g. asequence of strings always also accepts a single string, which will thenautomatically be wrapped into a sequence during the config normalizationprocess. Therefore the JSON Schema needs to accept both forms.

The resulting tree is then serialized by classes inSchema::Json to a largeHash which serves as a specification in the form of a JSON Schema.

A good starting point for exploring the schema definition is theroot node.

Examples for various nodes on this specification can be found in the tests, e.g.for thegit,heroku, oros nodes.

Most nodes can be sufficiently specified by mapping known keys (e.g.language) to types (str,bool,map,seq etc.) with certain options,such as:

• values: known values on a scalar type
• default: default value
• required: the node has to have a value
• edge: the node represents an experimental feature
• only: the node is valid only if the given value matches the currentlanguage oros
• except: the node is valid only if the given value does not match the currentlanguage oros
• expand: the node represents a matrix key for matrix expansion

In order to keep the JSON payload reasonably small the library uses JSONSchema's mechanism for defining and referencing shared sub schemas. All nodesthat have a registered definition class are exported as such a defined subschema, and then referenced on the respective nodes that use them.

TBD: mention JSON Schema limitations, and how travis-yml interprets particulartypes (all, any) in a specific way that is not defined by JSON Schema.

Before the tree representing the schema can be applied to an actual buildconfiguration it will be turned into another object oriented representationoptimized for this purpose, so non-parametrized methods can be memoized forbetter performance.

The methodTravis::Yml.expand returns this object oriented tree, usingclasses inDoc::Schema.

This representation of the schema can be applied to a build configuration by:

# given a YAML stringTravis::Yml.load(yaml)# given a Ruby hashTravis::Yml.apply(config)

Both methods also accept an optional options hash. Please seeherefor a list of known options.

When the schema is applied to a build configuration three things happen:

• The config is turned into an object oriented representation as well by theway of callingDoc::Value.build. This method uses classes inDoc::Valuein order to build a tree of nodes that maps to the given build config hash.

• The config structure is normalized by the way of callingDoc::Change.apply.This method applies various strategies in order to attempt to fix potentialproblems with the given structure, such as typos, misplaced keys, wrongsection types. In some cases it will store messages on the resulting tree.Change strategies are determined based on the type of the given node. Somestrategies can be required by the schema for certain sections that need veryspecific normalizations, such ascache,env_vars,enable.

• The resulting config is validated by the way of callingDoc::Validate.apply.This method applies various validations, and sets default values. It alsostores (most of the) messages on the resulting tree. Sections also canrequire specific validations. The only section specific validation currentlyistemplate (which validates used var names in notification templates).

Examples of type specific change strategies:

• downcase: downcase a string if required by the spec
• keys: add required keys, and attempt to fix an unknown key by removing special chars and finding typos (uses a dictionary, as well levenshtein and similar simple strategies)
• prefix: turn the given value into a hash with a prefix key (e.g. turningenv: ["FOO=bar"] intoenv: { matrix: ["FOO=foo"] })
• pick: pick the first value of a given sequence for a scalar node
• value: de-alias fixed node values, and try fixing typos in unknown values
• wrap: wrap the given node into a sequence if required by the spec (e.g.os needs to result in an array)

Section specific change strategies:

• env_vars: normalize env vars according to our build config format, parse env vars in order to validate them
• enable: normalizeenabled anddisabled values, setenabled if missing (used by, for example,notifications)
• inherit: inherit certain keys from the parent node if present (used by, for example,notifications)

Examples of the validations:

• alert: add analert level message if a node that expects a secure string accepts a plain string
• default: use a default value as required by the spec if the node does not have a value
• empty: warn about an empty section
• format: unset the value and if the given value does not conform with the format as required by the spec
• invalid_type: unset the value if the given value's type is not compatible with the spec's node type
• required: add anerror level message if the given map is missing a required key
• template: drop the value if the given template string uses unknown variables
• unknown_keys: add anerror level message for an unknown key
• unknown_value: add anerror level message for an unknown value

Env vars given as strings will be parsed into hashes using the librarysh_vars.Conditions will be parsed using the librarytravis-conditions.

There are three sets of classes that are used to build trees:

• Travis::Yml::Spec builds the static, unexpanded Ruby hash that can be served as JSON.
• Travis::Yml::Doc::Spec is used to build an object oriented tree of nodes from the expanded spec.
• Travis::Yml::Doc::Value is used to build an object oriented tree of nodes that represent the given build config.

Only the last one,Doc::Value, is re-used at runtime, i.e. only for the givenbuild configs we build new trees. TheDoc::Spec representation is kept inmemory, is static, and remains unchanged.

For each build config we then apply all relevant changes (Doc::Change) andall relevant validations (Doc::Validate) to each node.

A given build configuration can be expanded to job matrix configurations by:

Travis::Yml.matrix(config)

E.g. a build config like:

{language:'ruby',ruby:['2.2','2.3'],env:['FOO=foo','BAR=bar']}

will be expanded to:

[{language:'ruby',ruby:'2.2',env:{FOO:'foo'}},{language:'ruby',ruby:'2.2',env:{BAR:'bar'}},{language:'ruby',ruby:'2.3',env:{FOO:'foo'}},{language:'ruby',ruby:'2.3',env:{BAR:'bar'}},]

This gem also contains a web API for parsing and expanding a build config,which is used by travis-gatekeeper when processing a build request.

To start the server:

$ bundle exec rackup

The API contains three endpoints:

Home

$ curl -X GET /v1 | jq .

{"version":"v1"}

Parse

The body of the request should be raw YAML. The response contains parsing messages and the validated and normalised config.

$ curl -X POST --data-binary @travis.yml /v1/parse | jq .

{"version":"v1","messages": [    {"level":"info","key":"language","code":"default","args": {"key":"language","default":"ruby"      }    }  ],"full_messages": ["[info] on language: missing :language, defaulting to: ruby"  ],"config": {"language":"ruby","os": ["linux"    ],"dist":"trusty","sudo":false  }}

Expand

The body of the request should be the JSON config found in the response from/v1/parse. The response will contain the expanded matrix of jobs.

$ curl -X POST --data-binary @config.json /v1/expand | jq .

{"version":"v1","matrix": [    {"os":"linux","language":"ruby","dist":"trusty","sudo":false    }  ]}