Repository files navigation

OVE is gathering git repositories and the knowledge how to build and test them.Well sort of, it is up to you to feed this information to OVE. However, OVEprovides a well-defined structure for using and sharing this information withothers. OVE also provides a number of commands for common tasks, flexible waysof including all sorts of projects as well as the ability to expand OVE on thego! OVE is not a one-entry-point tool, but rather a shell enhancer: All partsof the OVE workflow can be done manually from prompt. We like to view OVE as away of removing not-updated-lately wikis, and instead share ready-to-usefunctionality.

To have a localized, yet versioned, top project source view to enable fastmodify-build-test loops in parallel development. For developers, for anyonethat prefers a see-the-big-picture approach and for those who just want to takea quick peek.

OVE is built with the developer in focus. We embrace the fact that whilecomputers (e.g. CI/CD hosts) generally do not get easily frustrated, developersdo.

Eager to get going? We have a tutorialhere. Try OVE out with a pre-madetutorial project and get up to speed on OVE in just a few minutes.

OVE provides a top project, and on this top level OVE therefore needs to handlefour major functionality areas:

• Versioning

• Build chain

• System tests

• Project specific tasks

To do this, OVE uses a top git repository (OWEL) containing information relatedto these tasks. Before we dig into details, let us just elaborate on a fewsubjects:

Versioning is handled entirely through git. The top repo and whatever sub reposare added are all git repos.

For OVE, a project is something that produces output (e.g. an executable, alibrary or anything else machine-made). Even though projects are normallycontained within a corresponding git repo, OVE treats projects and reposindependently. Multiple projects can be configured using code and build systemsfrom the same repo, and one project can use code and build systems frommultiple repos.

In order for OVE to build at the top level, independently of any toolchain usedby sub-projects, a contract must be set up between OVE and any includedproject. This is a one-sided contract. Nothing needs to (nor should) go into asub-project on OVE's account. To set up this contract, some typical build steps(bootstrap, configure, build, install) are specified for added sub projects.

System tests tend to be quite tricky to generalize, so we simply do not. Whatis provided is a way of keeping track of entry points and groups of entrypoints to system tests. This creates a template for keeping track of tests anda way to pass information that OVE holds down to test suites.

Regardless of how much features go into tools or frameworks for softwareprojects, they are never complete. There are always per-project specific needs.OVE is made with a less-is-more approach. Rather than trying to implement asmany feature requests as possible, we wanted to provide a solid functional basetogether with a simple, intuitive way of adding project-specific features. Itis therefore possible to expose customized OVE commands from an OWEL, aworkspace or from any git that OVE knows about. These commands are calledplugins. They are basically just a bunch of executables (most often small bashscripts) that can leverage on the project information held by OVE.

Enough said, let us dig into details! We start with versioning:

To make it transparent and intuitive for the developer to quickly grasp whatrevision state a certain workspace or project is in, OVE tries to be as shortand clear as possible about it. Therefore, the baseline for a project isdefined by a plain, line-by-line, text file in the OWEL. It is called 'revtab'and only contains four fields:

• name: Unique identifier of the git repository.
• fetch URL: The fetch URL.
• push URL: The push URL.
• revision: The git revision. This is passed on to 'git checkout'.

Example:

$ cat revtab# name        fetch URL          push URL           revisionrepoX         ssh://xyz/repoX    ssh://xyz/repoX    maindeps/repoY    https://xyz/repoY  https://xyz/repoY  stable

That's it! This is how OVE keeps track of git revisions. There is nointermediate representation for revisioning in OVE. What you put in the'revision' column travels untouched to git, which means you can safely putanything there that git understands. Now, let's move on to top-view builds:

How does OVE keep track of dependencies? Well, to start with there are (atleast) two types of dependencies: First, there are prerequisites for mostprojects to build, usually installed using a package manager. Secondly, withina top project handled by OVE the sub-projects almost always have dependenciesto each other. To specify these two types, you use a YAML file in the OWEL,'projs', that contains a list of projects with the following syntax:

name:  deps:    ...  needs:    ...  path:    ...  tags:    ...  version:    ...

Project name. Characters allowed: a-z, A-Z, 0-9 and underscore. 'common' isa reserved word.

A list of OVE projects that need to be built before this project can be built.

A list of OS packages that need to be installed before this project can bebuilt. Shell command substitution is allowed.

Can be extended with specific distro requirements, syntax:

needs[_ID][_VER]]

ID is a string and is matched vs. one string within: "${OVE_OS_ID//-/}${OVE_OS_ID_LIKE//-/}". Examples: "ubuntu", "debian", "centos", "rhel","fedora", "opensuse_tumbleweed", "suse". VER is a string and is matched vs."${OVE_OS_VER//[.-]/_}". Examples: "18_04", "3_12_0", "20200923".

The path to project work directory. Relative to OVE_BASE_DIR or an absolutepath. Variables are allowed.

Mark a project with one or many tags. Tags will allow you to refererence groupsof projects (e.g. for builds).

Optional. Passed on as a bash variable to all steps for this project.

$ cat projs---projA:  deps:    projB  needs:    autoconf    automake    g++  path:    repoX  tags:    small    uiprojB:  deps:    projC  needs:    build-essential    linux-headers-$(uname -r)  path:    repoY  tags:    backend    mediumprojC:  needs:    build-essential  needs_ubuntu:    pkgA  needs_ubuntu_20_04:    pkgB  needs_debian:    pkgC  needs_rhel:    pkgE  path:    /tmp/projC  tags:    large    ui  version:    2.0.0

That's how OVE resolves external and internal dependencies for builds. As youjust read above, the 'version:' keyword creates an environment variable that ispassed to all build steps. What are those steps exactly? We cover that in thenext section:

OVE is agnostic when it comes to build systems. Well, not entirely true. Youneed to be in a UNIX-like environment. That said, there are still a multitudeof ways to build and install software that need to be taken care of. OVEhandles this by providing a way of defining, for each sub project, how thatparticular project is built. In the OWEL, there is a directory called'projects'. Within this projects directory, sub directories need to be presentfor each sub project containing executables (normally tiny shell scripts) foreach build step. The projects structure typically look like this (output fromtree):

├── projects/│   ├── projA/│   │   ├── bootstrap│   │   ├── build│   │   ├── configure│   │   └── install│   ├── projB/│   │   ├── bootstrap│   │   ├── build│   │   ├── configure│   │   └── install│   ├── projC/│   │   ├── bootstrap│   │   ├── build│   │   ├── common│   │   ├── configure│   │   └── install│   └── common/│   │   ├── bootstrap│   │   ├── bootstrap.post│   │   ├── bootstrap.pre│   │   └── build.pre

When OVE builds the top project the following happens: First, OVE sorts out thebuild order as explained in the previous section. Secondly, each projects'build steps are executed (bootstrap, build, configure, install). When done, youshould be able to find the final output of the build in the staging area. Inmost cases, these are then picked up by an OVE plugin that creates deliverablepackages of some kind (.rpm, .deb or similar).

Particularly interesting here are the "configure" and "install" steps. In orderfor OVE to get intermediate build results into the staging area, this kind ofconstruct is typically used from within the 'configure' script:

./configure --prefix=${OVE_STAGE_DIR}${OVE_PREFIX}

This way, the install step will install any built items into'${OVE_STAGE_DIR}${OVE_PREFIX}'. Of course the way to do this depends on whatbuild system is used, but the same goes for any project you put into an OVEproject: You need to be able to get the build results into the staging area.

The 'common' directory is special. In the example above, before each individualproject's 'projX/bootstrap' file is executed, the 'common/bootstrap' file issourced. This will allow you to put common environment flags, checks etc. intothat 'common/{bootstrap,configure,build,install,...}' shell script. Thepre/post files are sourced before/after the first/last bootstrap command.

Example:

$ ove bootstrap projA projBprojects/common/bootstrap.preA: projects/common/bootstrapA: projects/projA/bootstrapB: projects/common/bootstrapB: projects/projB/bootstrapprojects/common/bootstrap.post

Each OVE project may also have a 'common' file within the project directory.This 'common' file is sourced before the project command file is executed.

Example:

$ ove build projCprojects/common/build.preC: projects/projC/commonC: projects/projC/build

For convenience, each project command has access to a few OVE_ACTIVE_PROJECT_variables:

You now know how to build sub projects together, but what about testing from asystem perspective? We cover that in the next section:

We have already covered how OVE keeps track of repos, how sub-project buildmethods can be included and how they can all form a larger, top view project.We also showed how these parts are built together using OVE's staging area. Onthe same note, it also makes sense to provide a way to execute system tests,tests that need more than one sub-project or repo to execute. As stipulatedearlier, OVE takes a rather defensive approach here. Quite often, test systemsalready exist for most functionality you want to develop, at least partly. Andyou want to re-use them. OVE is able to launch any tests as long as they canexecute from prompt. Two files, 'systests' and 'systests-groups' give OVEinformation about what tests are available and how to execute them:

'systests' is a text file that contains a list of tests. One row is one test:

• name: unique identifier for the test
• timeout: time in seconds when the test should finish. 0 = no timeout.
• type:
  • 0 = 00b = run in fg
  • 1 = 01b = run in fg and abort test suite on errors
  • 2 = 10b = run in bg
  • 3 = 11b = run in bg and abort test suite on errors
• path: where to execute the test (either relative to OVE_BASE_DIR or an absolute path)
• command: command(s) to execute

Example:

$ cat systests# name       timeout (s)   type   path   command# ----------------------------------------------t1              5          0      repoX  sleep 4t2              1          0      .      sleep 2t3           3600          0      repoY  ./long-duration-testt4              3          0      $HOME  echo hellu $LOGNAME; ls -l; whoamit5              3          0      /tmp   pwd

'systests-groups' is a YAML file that contains groups/sets of tests. Example:

$ cat systests-groupsall:  - t1  - t2  - t3sanity:  - t1

Using the above structure, you would be able to execute either one test (t1, t2or t3), a series of them (t1 t2) or a test group ("all" or "sanity"). Askingove what test are available in this case would look like this:

$ ove list-systestsallsanityt1t2t3

That's it for system tests! Now lets go ahead and look at plugins:

As discussed in the Overview, in most larger projects there is a strong needfor flexibility when it comes to what a developer or CI/CD machinery wants tobe able to do with it. To accommodate these needs, OVE provides a way ofextending the OVE command list with customized commands. We call them plugins,and they can be exposed to your OVE project in three ways: From your workspace,from your OWEL (top repo) or from any repo included in the revtab. What arethey really then? They are executables, optionally accompanied with a help textand/or a tab completion script. OVE looks for plugins at the followinglocations:

$OVE_OWEL_DIR/scripts/<all repositories>/.ove/scripts/

Any executable found in any of these locations will become an OVE command. Andprovided that tab completion scripts and help texts exist at the same location,they will also be part of the OVE help and support tab completion for theirarguments.

If you are using a plugin that reads from stdin AND you need this plugin withina pipe, please use this construct:

echo foo | ove-bar

We now covered the four main functionality areas of OVE. Next we will gothrough how to make life easy for developers or CI/CD machines when it comes tosetting up an OVE project:

An existing OVE project is typically setup (or downloaded if you will) by thedeveloper or CI/CD machine using the following oneliner:

curl -sSL https://raw.githubusercontent.com/Ericsson/ove/master/setup | bash -s <name> <OWEL>

• name: Path to the OVE workspace.
• OWEL: URL of the top git repository

The setup script will do two things:

• create the 'name' directory at your current location
• clone the OVE (ove itself) and OWEL (top repo) git repos

The 'setup' script will then urge the developer to enter the OVE workspacedirectory and run

source ove

Doing this, OVE will check that you have the required programs installed onyour machine and prompt for installation otherwise. This is the current list:

• bash (>=4.3)
• bzip2
• column
• file
• flock
• git (>=1.8.5)
• gzip
• ld
• less
• pgrep
• script
• tar
• tsort

OVE is also dependent on 'sed/grep/tail/awk/...' but they are not checked forsince it is quite uncommon to lack these. To unlock all OVE features you alsoneed:

• ag
• convert
• dig
• dot
• ffmpeg
• fzf
• graph-easy
• inotifywait
• jp2a
• incus
• locate
• lxc
• make
• md5deep
• rg
• shellcheck
• sshpass
• strace
• task
• tmux
• tsp
• whois
• yamllint
• xdotool
• xpra

Run 'ove list-externals' or checkthis page for acomplete list of commands that OVE is dependent on.

After successfully sourcing OVE, further instructions are given to enter theOVE workspace and fetch the rest of the repos. When the fetch is completed,everything is ready in order for man or machine to start working with theproject! For the sake of clarity, lets look at an example:

$ curl -sSL https://raw.githubusercontent.com/Ericsson/ove/master/setup | bash -s abc ssh://github.com/Ericsson/xyzCloning into '.ove'...Cloning into 'xyz'......$ cd abc$ source oveOVE [SHA-1: ... @ Ubuntu 19.10]$ ove fetchCloning into 'repoX'...Cloning into 'repoY'......repoX ## main..origin/mainrepoY ## main..origin/stable.ove  ## master..origin/master

Done! As simple as that. Lets give a final example of what an OVE project filestructure can look like when ready:

$ tree├── ove -> .ove/ove├── .ove/│   ├── .git/│   ├── LICENSE│   ├── ove│   ├── ove.png│   ├── README.md│   ├── scripts/│   ├── setup│   ├── tests/│   └── yex├── .owel -> xyz/├── repoX/│   ├── .git/│   └── README├── repoY/│   ├── .git/│   └── README└── xyz/    ├── .git/    ├── projects/    │   ├── projA/    │   │   ├── bootstrap    │   │   ├── build    │   │   ├── configure    │   │   └── install    │   ├── projB/    │   │   ├── bootstrap    │   │   ├── build    │   │   ├── configure    │   │   └── install    │   ├── projC/    │   │   ├── bootstrap    │   │   ├── build    │   │   ├── configure    │   │   └── install    │   └── common/    │       └── build    ├── projs    ├── revtab    ├── systests    └── systests-groups

Oneliner:

git clone https://github.com/Ericsson/ove.git .ove && source .ove/ove

In this example you will end up with a complete OVE workspace.

Example:

# '$HOME/src' has four git repositories$ cd $HOME/src$ git clone https://github.com/Ericsson/ove.git .ove && source .ove/ove...Directory to scan for git repositories? Leave blank to search in '$HOME/src': [ENTER]OWEL name? Leave blank to name it 'top': [ENTER]Scanning '$HOME/src'. #repos: 5Initialized empty Git repository in $HOME/src/top/.git/Create example/skeleton files? (y/N) [ENTER]...# you now have a OVE workspace in '$HOME/src' that contains six repos: four repos + OVE + OWEL# try 'ove status'$ ove status...

Oneliner:

source <(curl -sSL https://raw.githubusercontent.com/Ericsson/ove/master/ove)

In this example you will create an OWEL within a git repository.

Example:

$ cd a-git-repo$ source <(curl -sSL https://raw.githubusercontent.com/Ericsson/ove/master/ove)# the commands below will:#   - create a git commit of the OWEL specific files#   - publish the OWEL commit upstream#   - print OVE oneliner#   - try the oneliner _after_ the push#   - remember to run the oneliner outside of the 'foo' repo# create a git commitgit add projects projs revtab scripts SETUP systests systests-groups && git commit -m "initial commit"# publish the commitgit push# OVE onelinercurl -sSL https://raw.githubusercontent.com/Ericsson/ove/master/setup | bash -s foo https://...

Oneliner:

ove init

Same as above except that OVE is already available on the host.

# an existing OVE workspace$ source ove# move to a git repository$ cd a-git-repo$ ove init...# create a git commitgit add projects projs revtab scripts SETUP systests systests-groups && git commit -m "initial commit"# publish the commitgit push# OVE onelinercurl -sSL https://raw.githubusercontent.com/Ericsson/ove/master/setup | bash -s bar https://...

Up until now we covered everything you need to know to get to the point wheredevelopers (or machines) can start working with your OVE project. Going throughthese steps, You might have noticed us mention OVE commands several times. Itis time to have a closer look at how they work:

OVE will enhance your bash shell with commands to manage your OVE basedproject. We divide them into the following categories:

OVE implements a subset of the standard git commands as "high level" gitcommands. These commands executes the corresponding git command on all (orselective)revtab repositories.

• add
• apply
• blame
• branch
• checkout
• commit
• describe
• diff
• fetch
• grep
• pull
• show
• status
• stash
• worktree

This is a list of build related commands:

• buildme / buildme-parallel
• make
• mrproper

The above list will be dynamically populated with project commands found underthe "projects/proj/" directories. So, for a "normal" OVE project, thesecommands are usually also present:

• bootstrap
• configure
• build
• install

Note: For each project command there is a "command-parallel" version of thatcommand.

Here's a list (not complete) of a few utility commands:

Please find the full command referencehere

Each OVE command can be invoked using three different methods: normal, quick orqueue. The table below tries to explain the differencies on a few aspects.

Example:

# method: normal## ls-files command can take a while$ time ove ls-files...real 0m2,011s# the output (and input) of the ls-files command is saved and you can use LOG commands to view or replay the output$ ove list-commands LOG...# run strace in background and filter on all execve calls# ove loglevel 3$ ove ls-files...# method: quick## ls-files using the quick invocation method$ time ove-ls-files...real 0m0,949s# method: queue## queue the ls-files command, the command will silently be run in background$ time OVE_BATCH_IT=1 ove ls-files0real 0m0,193 s# check task spooler status$ ove ts...

Configurable OVE commands can be foundhere

A list of OVE environment variables that will remain stable across OVE versionscan be foundhere.

OVE has been tested for the following Linux distributions:

Want to know more about OVE? Please check out the OVEtutorial or ask OVE:

ove help