Repository files navigation

The easiest way to run microservices, big data, and containers in production.

Like traditional operating systems, DC/OS is system software that manages computer hardware and software resources and provides common services for computer programs.

Unlike traditional operating systems, DC/OS spans multiple machines within a network, aggregating their resources to maximize utilization by distributed applications.

To learn more, see theDC/OS Overview.

• Learn More -https://dcos.io/
• Find the Docs -https://dcos.io/docs/
• Install -https://dcos.io/install/
• Get Started -https://dcos.io/get-started/
• Get Help -http://chat.dcos.io/
• Join the Discussion -https://groups.google.com/a/dcos.io/d/forum/users
• Report an Issue -https://jira.dcos.io
• Contribute -https://dcos.io/contribute/

DC/OS releases are publicly available onhttp://dcos.io/releases/

Release artifacts are managed by Mesosphere on Amazon S3, using a CloudFront cache.

To find the git SHA of any given release, check the latest commit in the versioned branches on GitHub:https://github.com/dcos/dcos/branches/

Linux is required for building and testing DC/OS.

1. Linux distribution:
   • Docker doesn't have all the features needed on OS X or Windows
   • tar needs to be GNU tar for the set of flags used
   • unzip needs to be installed
2. tox
3. git 1.8.5+
4. Docker 1.11+
   • Install Instructions for various distributions. Docker needs to be configured so your user can run docker containers. The commanddocker run alpine /bin/echo 'Hello, World!' when run at a new terminal as your user should just print"Hello, World!". If it says something like "Unable to find image 'alpine:latest' locally" then re-run and the message should go away.
5. Python 3.6
   • Arch Linux:sudo pacman -S python
   • Fedora 23 Workstation: Already installed by default / no steps
   • Ubuntu 16.04 LTS:
     • pyenv-installer
     • Python dependencies:sudo apt-get install make build-essential libssl-dev zlib1g-dev libbz2-dev libreadline-dev libsqlite3-dev wget curl llvm libncurses5-dev libncursesw5-dev xz-utils liblzma-dev python3-venv
     • Install Python 3.6.3:pyenv install 3.6.3
     • Create DC/OS virtualenv:pyenv virtualenv 3.6.3 dcos
     • Activate environment:pyenv activate dcos
6. Over 10GB of free disk space and 8GB of RAM
   • The build makes use of hard links, so if you're using VirtualBox the disk space cannot be a synced folder.
7. Optional pxz (speeds up package and bootstrap compression)
   • ArchLinux:pxz-git in the AUR. The pxz package corrupts tarballs fairly frequently.
   • Fedora 23:sudo dnf install pxz

Unit tests can be run locally but require thedevelopment environment specified above.

tox

Tox is used to run the codebase unit tests, as well as coding standard checks. The config is intox.ini.

Integration tests can be run on any deployed DC/OS cluster. For installation instructions, seehttps://dcos.io/install/.

Integration tests are installed via thedcos-integration-test Pkgpanda package.

Integration test files are stored on the DC/OS master node at/opt/mesosphere/active/dcos-integration-test.Therefore, in order to test changes to test files, move files frompackages/dcos-integration-test/extra/ in your checkout to/opt/mesosphere/active/dcos-integration-test on the master node.

The canonical source of the test suite's results is the continuous integration system.There may be differences between the results of running the integration tests as described in this document and the results given by the continuous integration system.In particular, some tests may pass on the continuous integration system and fail locally or vice versa.

• 1 master node
• 2 private agent nodes
• 1 public agent node
• Task resource allocation is currently insignificantly small
• DC/OS itself requires at least 2 (virtual) cpu cores on each node

1. SSH into a master nodeThe tests can be run via Pytest while SSH'd as root into a master node of the cluster to be tested.

2. Switch to root

   sudo su -

3. Add the test user

   dcos-shell python /opt/mesosphere/active/dcos-oauth/bin/dcos_add_user.py albert@bekstil.net

   Running the above mentioned command will result in an output

   User albert@bekstil.net successfully added

   This test user has a known login token with far future expiration. DO NOT USE IN PRODUCTION.After the test, remember to delete the test user.

   For more information, seeUser Management.

4. Run the tests using pytest in the cluster.

   cd /opt/mesosphere/active/dcos-integration-testdcos-shell pytest

One way to run the integration tests is to useDC/OS Docker.

1. Setup DC/OS in containers usingDC/OS Docker.

One way to do this is to use theDC/OS Docker Quick Start tool.

2. Runmake test.

DC/OS can be built locally but requires thedevelopment environment specified above.

DC/OS builds are packaged as a self-extracting Docker image wrapped in a bash script calleddcos_generate_config.sh.

WARNING: Building a release from scratch the first time on a modern dev machine (4 cores / 8 hyper threads, SSD, reasonable internet bandwidth) takes about 1 hour.

./build_local.sh

That will run a simple local build, and output the resulting DC/OS installers to $HOME/dcos-artifacts. You can run the created `dcos_generate_config.sh like so:

$ $HOME/dcos-artifacts/testing/`whoami`/dcos_generate_config.sh

If you look inside of the bash scriptbuild_local.sh there are the commands with descriptions of each.

The general flow is to:

1. Check the environment is reasonable
2. Write arelease tool configuration if one doesn't exist
3. Setup a python virtualenv where we can install the DC/OS python tools to in order to run them
4. Install the DC/OS python tools to the virtualenv
5. Build the release using therelease tool

These steps can all be done by hand and customized / tweaked like standard python projects. You can hand create a virtualenvironment, and then do an editable pip install (pip install -e) to have a "live" working environment (as you change code you can run the tool and see the results).

This release tool always loads the config indcos-release.config.yaml in the current directory.

The config isYAML. Inside it has two main sections.storage which contains a dictionary of different storage providers which the built artifacts should be sent to, andoptions which sets general DC/OS build configuration options.

Config values can either be specified directly, or you may use $ prefixed environment variables (the env variable must set the whole value).

All the available storage providers are inrelease/storage. The configuration is a dictionary of a reference name for the storage provider (local, aws, my_azure), to the configuration.

Each storage provider (ex: aws.py) is an available kind prefix. The dictionaryfactories defines the suffix for a particular kind. For instancekind: aws_s3 would map to the S3StorageProvider.

The configuration options for a storage provider are the storage provider's constructor parameters.

Sample config storage that will save to my home directory (/home/cmaloney):

storage:local:kind:local_pathpath:/home/cmaloney/dcos-artifacts

Sample config that will store to a local archive path as well as AWS S3. To authenticate with AWS S3, reference theboto3 docs to learn how to configure access.

storage:aws:kind:aws_s3bucket:downloads.dcos.ioobject_prefix:dcosdownload_url:https://downloads.dcos.io/dcos/local:kind:local_pathpath:/mnt/big_artifact_store/dcos/

DC/OS itself is composed of many individual components precisely configured to work together in concert.

This repo contains the release and package building tools necessary to produce installers for various on-premises and cloud platforms.

Pull requests automatically trigger a new DC/OS build and run several tests. These are the details on the various status checks against a DC/OS Pull Request.