Prerequisites

Option 1: use Helm chart

Option 2: you have Minikube

$ minikube start --cpus=4 --memory=8G --disk-size=20G --driver kvm2

$ minikube start --cpus=4 --memory=8G --disk-size=20G --driver=hyperkit

$ brew install qemu$ brew install socket_vmnet$ brew tap homebrew/services$ HOMEBREW=$(which brew) && sudo ${HOMEBREW} services start socket_vmnet$ minikube start --cpus=4 --memory=8G --disk-size=20G --driver qemu --network socket_vmnet

$ DISM /Online /Enable-Feature /All /FeatureName:Microsoft-Hyper-V # Enable Hyper-V$ minikube start --cpus=4 --memory=8G --disk-size=20G --driver=hyperv

1. Prerequisites:

   • RunningOpenshift 4.x cluster

   • oc client installed and pointing to the cluster. Installation was tested whenoc has administrator permissions.

2. Create a project (the same as Kubernetes namespace) for Moon:

   $ oc new-project moon

   In the next steps we assume that Openshift project for Moon is calledmoon.

3. Add Aerokubecharts repository:

   $ helm repo add aerokube https://charts.aerokube.com/$ helm repo update

4. To install or upgrade Moon type:

   $ helm upgrade --install --set ingress.openshift=true -n moon moon aerokube/moon2

   Here-n moon points to the project created on the previous step.

5. Edit user and group id inconfiguration object to match values allowed by Openshift policies (e.g. set to1000650000, exact value depends on Openshift configuration):

   $ oc edit config.moon.aerokube.com default -n moon

Helm values files

Configuring Moon Custom Resources

quota:# Stores a list of quotasquota1:# Quota1 specificationquota2:# Quota2 specificationconfigs:# Stores a list of configuration objectsconfig1:# Config1 specificationconfig2:# Config2 specificationbrowsers:# Stores a list of browsers setsbrowsersset1:# Browsersset1 specificationbrowsersset2:# Browsersset2 specificationlicense:# Stores Moon license key (currently only one license key is supported in Helm chart)

browsers:my-browserset:selenium:firefox:repository:quay.io/browser/firefoxchrome:repository:quay.io/browser/chrome

apiVersion:moon.aerokube.com/v1kind:BrowserSetmetadata:name:my-browsersetnamespace:moon# Your namespace name for Moon will be herespec:selenium:firefox:repository:quay.io/browser/firefoxchrome:repository:quay.io/browser/chrome

Moon cluster consists of several components:

1. One or moreMoon application instances. Their main purpose is to start and stop browser containers. These replicas are usually exposed asKubernetes service available on standard Selenium port4444. You should run all the tests against this service. Also, this application provides an API to get information about running browser instances (in Moon 1.x this was a separate application calledMoon API).

2. One or moreMoon Conf application instances. This application is restarting Moon pods when you update a license key.

3. One or moreMoon UI application instances.Moon UI collects information from Moon and visualizes it. It is usually available on HTTP port8080.

4. Running browser pods.

Moon 2.x has two different operation modes:single namespace mode andmultiple namespaces mode.

In single namespace mode Moon itself and all launched browser pods are running in the one Kubernetes namespace. Moon 1.x was only able to work like this. This is still suitable if only one team is using Moon or you don’t need to limit browser consumption of different Moon users. By default, Moon is launched in this mode.

In multiple namespaces mode Moon is running in one namespace and browsers are launched in separate namespaces. The total number of such namespaces is unlimited. This mode is mainly needed when you want to control computing resources, browsers or network access rules (network policies) available for every team. How to enable this mode is describedhere.

In addition to container with browser every pod created by Moon contains one or more service images.

The following table summarizes what needs to be accessible for Moon in single namespace mode:

When running browsers in multiple namespaces required permissions differ. The following table shows Moon permissions in the namespace where it is running:

For every user namespace Moon needs the following permissions:

Moon comes with a powerful user interface allowing to list and filter running browser sessions, view browser screen, launch browser sessions for manual testing and so on.

The main screen of the UI is showing the list of browser sessions that are currently running or being started. For every session you can see browser automation tool, browser name, version, test name, labels, status, duration, enabled features and so on. To delete a running session you haveto click twice on the button with a trash can icon (two clicks are needed to prevent accidentally removing a running session). To create a browser session formanual testing - click on the button with browser name at the top of UI screen.

One Moon cluster can run dozens, hundreds or even thousands of browser sessions in parallel. To find sessions that belong to your project or exact build number -use filters. These controls allow to filter browser sessions byid,name andlabels:

• Sessionid is a unique value automatically assigned to every browser session by Moon. You can’t change it. Full session id looks like this:chrome-73-0-ac15ffaa-e641-4c7f-a54c-f25b5be1f135. In the UI we are printing only a few symbols of this long value.

• Sessionname is a free-form value allowing to describe purpose of the session in the UI. Usually this value contains test case name. To change session name - usename capability for Selenium orname parameter forPlaywright,Cypress ordeveloper tools, e.g.:

  wss://moon.example.com/playwright/chromium?name=MyTestCaseName

• Sessionlabels are free-form key-value pairs allowing to put additional metadata on every browser session. This can be for example build number, project name, release information and so on. To add labels - uselabels capability for Selenium or configurelabels in browsers set. Every label requested to be set by Moon is converted to aKubernetes label added to respective browser pod. User interface is using exactly the same syntax oflabel selectors that is supported by Kubernetes. For example, havingproject andbuildNumber labels set to some browser session you can use expressions like this:

  How to filter by labels

  project=MyCoolProject # Exact match of one labelproject=MyCoolProject,buildNumber=42 # Match of project AND buildNumber valuesproject in (MyCoolProject, AnotherProject),buildNumber!=42 # Select project from the listproject notin (MyCoolProject, AnotherProject),!buildNumber # Select project not from the list, build number not setproject!=AnotherProject,buildNumber # Any project except AnotherProject, build number should be set

When you click on a row in the sessions list, browser screen is shown automatically if possible. For example for browser sessions with invisible browser window (so-calledheadless browsers) - nothing happens when you click on a row. By default, browser screen is in view-only mode. To interact with the browser - click on the button with the lock icon (🔒). Click on this button again - to switch back to view-only mode. Browser screen allows to follow automated test execution and intercept it when needed. When you launch a manual testing session - use this screen to manually execute your scenario step-by-step. All browser features like opening developer toolbar are working exactly the same as on your personal computer. To copy and paste values from your computer to Moon browser session - useCtrl+C\Ctrl+V orCmd+C\Cmd+V shortcuts on your keyboard.

Console is a command-line interface available on Moon UI screen and giving more power to manual testing. The following features are available:

• Listing available browsers and mobile emulation devices

• Starting, listing and deleting browser sessions with desktop browsers and with mobile emulation enabled

• Easily providing arbitrary additional browser capabilities

• Opening VNC sessions for manual testing

This is how console looks like:

To open console:

• Option 1. Click on the button shown on the picture above.

• Option 2. Type~ (tilde) symbol on the keyboard.

Moon console works similarly to standard Unix terminal. To list all available commands type:

To show command syntax add-h or-help to command:

To view previous commands use arrow up and arrow down buttons. To clear previous commands output type:

To exit from console just close console window with mouse or type:

To list available browsers:

To list last 5 versions ofchrome browser:

To list mobile emulation devices:

To launch a Chrome browser:

To launch mobile emulation:

To view browser screen using VNC:

To list available browser sessions:

To delete a running browser session:

https://moon.example.com/wd/hub

fromseleniumimportwebdrivercapabilities = {"browserName":"chrome"}driver = webdriver.Remote(    command_executor='https://moon.example.com/wd/hub',    desired_capabilities=capabilities)

capabilities = {"browserName":"chrome","browserVersion":"96.0"}

Moon supports a set of extension capabilities. You can pass them in your code to enable or disable some features. All these capabilities should be passed undermoon:options key:

In statically-typed languages like Java or C# you should use a Map (Dictionary) to pass Moon capabilities, e.g.:

Custom Screen Resolution: screenResolution

screenResolution: "1280x1024"

screenResolution: "1280x1024x24"

Custom Test Name: name

name: "myCoolTestName"

Video Recording: enableVideo, videoName, videoScreenSize, videoFrameRate, videoCodec, pattern

enableVideo: true

Capturing web page network activity: enableHAR

enableHAR: true

Per-session Environment Variables: env

env: ["LANG=ru_RU.UTF-8", "LANGUAGE=ru:en", "LC_ALL=ru_RU.UTF-8"]

capabilities.setCapability("moon:options",Map.of("env",Arrays.asList("LANG=ru_RU.UTF-8","LANGUAGE=ru:en","LC_ALL=ru_RU.UTF-8")));

Hosts Aliases: hosts

hosts: ["example.com:192.168.0.1", "test.com:192.168.0.2"]

Custom DNS Servers: nameservers

nameservers: ["192.168.0.1", "192.168.0.2"]

Custom Session Timeout: sessionTimeout

sessionTimeout: "1m30s"

Mobile Emulation: mobileDevice

"mobileDevice": {    "deviceName": "Apple iPhone XR",    "orientation": "landscape"}

deviceName: "Apple iPhone XR"

orientation: "landscape"

capabilities.setCapability("moon:options",Map.of("mobileDevice",Map.of("deviceName":"Apple iPhone XR","orientation":"landscape"    )));

Pod Labels: labels

labels: {"project": "MyCoolProject", "build-number": "14353"}

Browser Log Level: logLevel

logLevel: "INFO"

Enable Additional Fonts: additionalFonts

additionalFonts: true

Additional Browser Data: context

context: "https://example.com/browser-data.tar.gz"

Disable Window Manager: windowManager

windowManager: false # Default is true

Automatically Select Mutual TLS Client Certificate: autoSelectClientCerts

autoSelectClientCerts: true

By default, all browsers in Moon are started with visible browser window. The majority of browsers nowadays support so-called "headless" mode, when browser is opening pages in the background and no window is visible to the user. Usually such mode is enabled by passing--headless flag to browser startup command in Selenium capabilities.

Moon automatically detects when browser is started in headless mode. Headless browsers do not require any graphical components like X-server or window manager, so Moon does not start such components when not needed. Because of this feature, there is no need to additionally passenableVNC capability to show browser screen in Moon user interface.

When video recording is configured, recording a video of Selenium session is as easy as adding one capability to your test:

You can optionally add othercapabilities to change recorded video name, screen size, frame rate and so on.

Running automated tests in mobile platforms is nowadays very important. Using a set of real devices connected to server via USB requires too much work to deploy and maintain. Running Android Emulators requires hardware server or virtual machines with nested virtualization enabled. Running iOS Simulators requires to have Apple hardware. Even with correct computing resources tests are slower than on desktop platforms and consume slightly more CPUs and memory per browser.

Your goal however is catching bugs and not deploying complicated browser automation infrastructure. There are a lot of cases when a bug related to mobile version of tested web application can be reproduced simply by having exactly the same screen size andUser-Agent HTTP header being sent by browser. This feature is already available in Chromium-based browsers and is calledMobile Emulation.

Example capabilities to enable this functionality are shown below:

Moon comes with a preconfigured list of supported devices stored indevices set. Full list of available devices is available inSupported Mobile Devices section. In order to add your own mobile devices definitions - simply update this list.

Sometimes you may need to interact with the clipboard to check that your application copy-paste feature works. Moon has a dedicated API to interact with the clipboard:

1. Start a new session, for example with IDfirefox-95-0-f2bcd32b-d932-4cdc-a639-687ab8e4f840.

2. To get clipboard value send the following HTTP request:

   $ curl -H 'Accept: application/json' https://moon.example.com/wd/hub/session/firefox-95-0-f2bcd32b-d932-4cdc-a639-687ab8e4f840/aerokube/clipboard{"value": "some-clipboard-value", "media": ""}

   If clipboard contains an image, then response will containBase64-encoded image bytes:

   {"value":"iVBORw0KGgoAAAAN....","media":"image/png"}

3. To update clipboard with text value:

   $ curl -X POST -H 'Content-Type: application/json' --data '{"value": "some-clipboard-value"}' https://moon.example.com/wd/hub/session/firefox-95-0-f2bcd32b-d932-4cdc-a639-687ab8e4f840/aerokube/clipboard

4. To update clipboard with image value, sendBase64-encoded image bytes and :

   $ curl -X POST -H 'Content-Type: application/json' --data '{"value": "iVBORw0KGgoAAAAN....", "media": "image/png"}' https://moon.example.com/wd/hub/session/firefox-95-0-f2bcd32b-d932-4cdc-a639-687ab8e4f840/aerokube/clipboard

Uploading files to browser is a built-in Selenium feature supported in the majority of Selenium clients. How to do this in different programming languages is shown below:

Often when working with a browser you may need to use additional data: browser extensions, test files to be uploaded, browser settings files (also known asbrowser profile) and so on. In standard Selenium every such additional file is uploaded with different code snippet. The most important thing is that anyway all such data is being uploaded as HTTP request body. Every time you send file like this your Selenium implementation needs to read all the bytes into memory and this dramatically increases memory consumption. A lot more efficient way of delivering the same functionality is packing all required files to a single archive (e.g. on your CI server). An URL to this archive is then sent as a Selenium capability, thus allowing every browser session to download it before actually launching the browser. We call an archive like this abrowser context and respective capability is named justcontext:

When you providecontext capability, Moon will download an archive and unpack it to/home/<user> directory, where<user> is name of the user configured inconfiguration object. Default username is justuser, so default directory is/home/user/.

To create an archive with browser context:

Possible use cases of this feature include:

1. Uploading files to browser. You pack any test files and just set their path to file input fields or open them in the browser.

   Capabilities to upload test files to browser

   {"browserName":"chrome","moon:options":{"context":"https://example.com/browser-data.tar.gz"}}

   The same as HTTP request:

   An HTTP request to upload test files to browser

   $ curl https://moon.example.com/wd/hub/session -d'{"capabilities":{"alwaysMatch":{"browserName":"chrome", "moon:options":{"context":"https://example.com/browser-data.tar.gz"}}}}'

   Now just use unpacked files in your Selenium code:

   Using uploaded files from context

   // Find file input elementWebElement input = driver.findElement(By.cssSelector("input[type='file']"));// Specify path of the file from context directoryinput.sendKeys("/home/user/some-directory/one-more-file.png");// You can also open files from context directory in browserdriver.get("file:///home/user/some-file.txt");

2. Using browser extensions. You repack your extension (extension.crx) to the archive (extension.tar.gz) and then load it using browser command-line flags. To repack extension:

   How to repack extension to*.tar.gz

   $ unzip extension.crx -d extension # The same works for *.xpi as both are zip archives$ tar cvzf extension.tar.gz extension

   Respective capabilities can look like this:

   Capabilities to use a browser extension

   {"browserName":"chrome","goog:chromeOptions":{"args":["--disable-extensions-except=/home/user/extension","--load-extension=/home/user/extension"    ]  },"moon:options":{"context":"https://example.com/extension.tar.gz"}}

   The same as HTTP request:

   An HTTP request to use browser extension

   $ curl https://moon.example.com/wd/hub/session -d'{"capabilities":{"alwaysMatch":{"browserName":"chrome", "goog:chromeOptions":{"args":["--disable-extensions-except=/home/user/extensions","--load-extension=/home/user/extensions"]}, "moon:options":{"context":"https://example.com/extensions.tar.gz"}}}}'

3. Overriding browser profile. You pack a directory with browser profile (profile) to the archive (profile.tar.gz) and then load it using browser command-line flags. Respective capabilities for Chrome can look like this:

   Capabilities to override Chrome profile

   {"browserName":"chrome","goog:chromeOptions":{"args":["--user-data-dir=/home/user/profile"]  },"moon:options":{"context":"https://example.com/profile.tar.gz"}}

   The same as HTTP request:

   An HTTP request to override Chrome profile

   $ curl https://moon.example.com/wd/hub/session -d'{"capabilities":{"alwaysMatch":{"browserName":"chrome", "goog:chromeOptions":{"args":["--user-data-dir=/home/user/profile"]}, "moon:options":{"context":"https://example.com/profile.tar.gz"}}}}'

   For Firefox approach remains the same but command-line flags differ:

   Capabilities to override Firefox profile

   {"browserName":"firefox","moz:firefoxOptions":{"args":["-profile","/home/user/profile"]  },"moon:options":{"context":"https://example.com/profile.tar.gz"}}

   The same as HTTP request:

   An HTTP request to override Firefox profile

   $ curl https://moon.example.com/wd/hub/session -H'Content-Type: application/json' -d'{"capabilities":{"alwaysMatch":{"browserName":"firefox", "moz:firefoxOptions":{"args":["-profile","/home/user/profile"]}, "moon:options":{"context":"https://example.com/profile.tar.gz"}}}}'

4. Overriding various user settings. Previously we understood that browser context archive is unpacked to user home directory in browser pod. You can use this to override various operating system configuration files (~/.bashrc,~/.gtkrc-3.0 to e.g. turn off cursor blinking) and directories (~/.ssh,~/.gpg and so on).

5. Emulating web camera video. You upload a video file to browser pod and then use it as fake web camera video. You start by preparing a fake video:

   Converting an *.mp4 video to *.y4m

   $ mkdir webcam-video$ ffmpeg -i my-video.mp4 -vf hflip -pix_fmt yuv420p -s 1280x720 webcam-video/webcam-video.y4m

   Then you add resulting video to an archive:

   Creating an archive with video

   $ tar cvzf webcam-video.tar.gz webcam-video

   Having an archive you can now create a Chrome session with the following capabilities:

   Capabilities for Chrome web camera emulation

   {"browserName":"chrome","goog:chromeOptions":{"args":["--disable-gpu","--use-fake-ui-for-media-stream","--use-fake-device-for-media-stream","--use-file-for-fake-video-capture=/home/user/webcam-video/webcam-video.y4m"    ]  },"moon:options":{"context":"https://example.com/webcam-video.tar.gz"}}

   The same as HTTP request:

   An HTTP request to emulate web camera in Chrome

   $ curl https://moon.example.com/wd/hub/session -H'Content-Type: application/json' -d'{"capabilities":{"alwaysMatch":{"browserName":"chrome", "goog:chromeOptions":{"args":["--disable-gpu", "--use-fake-ui-for-media-stream", "--use-fake-device-for-media-stream", "--use-file-for-fake-video-capture=/home/user/webcam-video/webcam-video.y4m"]},"moon:options":{"context":"https://example.com/webcam-video.tar.gz"}}}'

Your tests may need to download files with browsers. To analyze these files a common requirement is then to somehow extract downloaded files from browser containers. Moon provides an API to work with such files:

1. Start a new session, for example with IDfirefox-95-0-f2bcd32b-d932-4cdc-a639-687ab8e4f840.

2. In tests code save all files to/home/<user>/Downloads directory, where<user> is name of the user configured inconfiguration object. Default username is justuser, so default directory is/home/user/Downloads.

3. To list available files:

   curl -H 'Accept: application/json' https://moon.example.com/wd/hub/session/firefox-95-0-f2bcd32b-d932-4cdc-a639-687ab8e4f840/aerokube/download/{"value": ["myfile.txt", "another-file.png"]}

4. Access any file contents using the following URL:

   curl https://moon.example.com/wd/hub/session/firefox-95-0-f2bcd32b-d932-4cdc-a639-687ab8e4f840/aerokube/download/myfile.txtfile-contents-go-here

5. To delete a file:

   curl -X DELETE https://moon.example.com/wd/hub/session/firefox-95-0-f2bcd32b-d932-4cdc-a639-687ab8e4f840/aerokube/download/myfile.txt

6. Close the session

Selenium 4 and above hasbidirectional functionality allowing to access advanced browser features. This works just out of the box. An example project demonstrating how to use it storedhere.

Moon 1.x and Selenoidhave custom/devtools/ API allowing direct access to browser usingChrome Developer Tools Protocol. For backwards compatibility this is also supported in Moon 2.x. InW3C WebDriver standard Selenium extension commands should be located under vendor prefix, so having a Selenium session ID to access this API in Moon 2 you have to use URL like this:

In some test cases you may need to override preferred browser locale. You can do this with standard Selenium capabilities. How to override locale depends on browser.

Firefox

FirefoxOptions options =new FirefoxOptions();options.setCapability("browserVersion","75.0");options.addPreference("intl.accept_languages","de");WebDriver driver =new RemoteWebDriver(newURL("https://moon.example.com/wd/hub"), options);

Chromium-based Browsers

ChromeOptions options =new ChromeOptions();options.setCapability("browserVersion","81.0");options.setCapability("moon:options",Map.of("env",Arrays.asList("LANG=de_AT.UTF-8","LANGUAGE=at:de","LC_ALL=de_AT.UTF-8")));WebDriver driver =new RemoteWebDriver(newURL("https://moon.example.com/wd/hub"), options);

A common testing task is checking that your web application behaves as expected in differenttime zones. Depending on tested web application one of the following approaches can help.

Option 1: Setting TZ environment variable

ChromeOptions options =new ChromeOptions();capabilities.setCapability("moon:options",Map.of("env",Arrays.asList("TZ=America/New_York")// This is where you set TZ variable with values like "America/New_York" or "Europe/London"));WebDriver driver =new RemoteWebDriver(newURL("https://moon.example.com/wd/hub"), options);driver.get("https://dateful.com/time-zone-converter");// An example web site that respects TZ setting

Option 2: Overriding Browser Geolocation

ChromeOptions options =new ChromeOptions();WebDriver driver =new RemoteWebDriver(newURL("https://moon.example.com/wd/hub"), options);driver =new Augmenter().augment(driver);DevTools devTools = ((HasDevTools) driver).getDevTools();devTools.createSession();// Location of London (change this to 40.715502419712244, -74.00597334074466 for New York)devTools.send(Emulation.setGeolocationOverride(Optional.of(51.495930861102245),        Optional.of(0.010205721644136127),        Optional.of(1)));driver.get("https://google.com");WebElement element = driver.findElement(By.name("q"));Actions actionProvider =new Actions(driver);Action select = actionProvider        .sendKeys("what is my time zone\n")        .build();select.perform();

Moon expects to run the majority of browsers in pods inside Kubernetes or Openshift cluster. However sometimes you may need to run Selenium tests on some external hosts: hardware servers or virtual machines. Mainly this could be needed in two situations:

1. Running Selenium tests on complicated platforms such as MacOS or iOS. According to license agreement these platforms require Apple hardware devices, and it is complicated to run Kubernetes on top of these devices.

2. Using Selenium online platforms for some browsers. In that case you can run the majority of browsers (e.g. Firefox, Chrome, Opera) in Moon and complicated browsers (don’t work on standard virtual machines) such as Chrome Mobile or real devices in external Selenium platform.

To use external hosts you should have the following:

1. A set of hosts with Selenium-compatible solution (Selenoid, Appium, Selenium Grid, etc.):host1.example.com:4444,host1.example.com:4444 and so on.

2. Optionally aVNC server listening on every such host on standard port5900. Every VNC server should be password protected with the same password having 8+ characters.

For every browser type you need to add the following tobrowsers set:

With such configuration Selenium session requests with be randomly load-balanced across the hosts specified inURLS environment variable. VNC feature should also work - you should be seeing remote host screen in Moon UI.

In some cases you may need to configure launched browser to go through some proxy server. Selenium WebDriver protocolsupports standard capabilities to configure proxy server for any browser. For example:

For some programming languages there is a wrapper object calledProxy allowing to set the same value in more type-safe way:

Very often proxy servers require to provide username and password for authentication. While the majority of Selenium implementations don’t work with such proxy servers, Moon allows to configure authentication out of the box, using the same capabilities - simply add username and password to proxy host value (username:password@host:port):

You can request one of browsers supported by Cypress (chrome,chromium,edge,electron orfirefox) by specifying its name in URL. By default, Moon will usebrowsers/cypress-<browser-name>:latest public image.

Cypress API can change from version to version. Because of that it is recommended to make sure that Cypress version being used in your project corresponds to Cypress version in browser image. To use an image compatible with exact Cypress version - add this version as follows:

To enablevideo recording - simply addenableVideo parameter to URL:

You can optionally add otherparameters to change recorded video name, screen size, frame rate and so on.

In addition to selecting Cypress version - you can enable additional features like changing screen resolution, passing custom test name and so on. All these optional features are set by adding parameters to URL:

Full list of supported parameters and their meaning is shown below.

Cypress providesCypress Dashboard - an online service for storing test runs information. To send information about executed tests to this service you have to send your access key usingCYPRESS_RECORD_KEY environment variable:

const { firefox } = require('playwright');(async () => {  const browser = await firefox.connect({timeout:0,wsEndpoint:'wss://moon.example.com/playwright/firefox/playwright-1.23.3' });  const page = await browser.newPage();  await page.goto('https://aerokube.com/moon/');  await page.screenshot({path:`screenshot.png` });  await browser.close();})();

You can request one of browsers supported by Playwright (chrome,chromium,firefox orwebkit) by specifying its name in URL. By default, Moon will usequay.io/playwright-<browser-name> repository to download images. Currently, Playwright API can change from version to version. Because of that it is recommended to make sure that Playwright client version being used in your code corresponds to Playwright server version in browser image. To use an image compatible with exact Playwright version - add this version as follows:

To enablevideo recording - simply addenableVideo parameter to URL:

You can optionally add otherparameters to change recorded video name, screen size, frame rate and so on.

Similarly to Selenium, you can make browser pod automatically download arbitrary files as a single archive and unpack them to user directory. This feature is described in detailhere. The main particularity in Playwright is that archive URL is being passed as Playwrightcontext URL parameter and thus needs to beURL encoded.

For example to upload a browser extension:

In addition to selecting browser and its version - you can enable additional features like using headless browser versions, passing environment variables and so on. All these optional features are set by adding parameters to URL:

Full list of supported parameters and their meaning is shown below.

wss://moon.example.com/devtools/chrome

const puppeteer = require('puppeteer-core');const host ='moon.example.com';(async () => {    const devtools = await puppeteer.connect(        {timeout:0,browserWSEndpoint:`wss://${host}/devtools/chrome` }    );// For every call of this method a new browser is started    const page = await devtools.newPage();    await page.goto('https://aerokube.com');    await page.screenshot({path:'screenshot.png'});    const title = await page.title();    console.log(title);    await devtools.close();})();

You can choose desired browser version by changing connection URL:

To enablevideo recording - simply addenableVideo parameter to URL:

You can optionally add otherparameters to change recorded video name, screen size, frame rate and so on.

You can enable additional features by changing connection URL:

$ cat license.keyMG1RSVdpc2Z6YjdQQVZjd2lpei9KMkd1T3dzMTFuL1dlRjVSc3NOMUcxZk9QaUxWa3Q5SnBIakIxa09wWm0vVFJqQ0tsa21xVG1OODVRZnlQbjBjVmRHVWFLampTOFF1a3VLRXRPcEUwbnEySG16QWFQWHRDYTVjMm9jZzZFaUJqeFd5ODE4UFBHZzNCNWpCYXlha3oweFBscFl1RnB0V0U1Q3FwOGl5VDdKTk9abG5aSmlPdnRmZDFvSG1nNnVwVXBLV2E4RmYwWHcreERIR29ZTE1XTldPb1hvT2ZCUnZpcDhPWW05a1FqN0hBWWVOYUtLT1lPWlVJa1dsb1gxdjNOT1htTFpZalhsQ3h1Q3V6NWhiQjIwSjVIY0JTYnZybm9zYm14RXFkSFpQWVBKWUlKTzZvVlBnODhQeFErZ1EyTk5sWG82TC9XeXU3aisrNU0rSEdPcXlOSEdlNGx4Zm1nNVhjMWlnNkN1OCtNSVVYRzNqUllqOUY4ZHdReWpSbFNMNmFpL2dRQnc3TzY0U0lwdVF2d29jYi9kVzFSYWFRVkd3ZXYrOVdING8zRWRrYkVONUhRTmQ2MUxsUnFNdmtKeWVHV21tVlVUZ2dsMDRsTFFLTmZNVG81L2JVakNBMGhNeER5VHNJdmVRRGFMMklvTWpvcFk4VERlK1U2bUJvUDVxNVYrcCtDQVhjbjYxQlRaUVp0bmNqL0JBVkdNOEZ4NW9rWHRYSVAxUkY0a1VCckZVTDFyTWF1VkZqSk5xU1pLT293dUpMTTg2SEZ0Sld0eUlRK3ZZZm1pZU0xM292MnVleDBoRlhRdFkvMkt1dUhhN3dKV2pFT0pqaEVzTjhXSy82ZlFFbi9EQzcrNkw3NzhlbmVVZ2lLZ3VFbjlMMXZMYVZ5VWtQaWc9O2V5SnNhV05sYm5ObFpTSTZJa1JsWm1GMWJIUWlMQ0p3Y205a2RXTjBJam9pVFc5dmJpSXNJbTFoZUZObGMzTnBiMjV6SWpvMGZRPT0=

To list available licenses:

The output above is shown when the free license key is used. Columns meaning is as follows:

• Name. License key object name.

• Licensee. License key owner name. Usually equals to company name, e.g.Acme LLC. For free license key with 4 parallel sessions equals toDefault.

• Sessions. Maximum number of browser sessions available in this license key.

• Expires. The number of days this license key expires in. Equals toAlready when license key already expired and equals toNever if license key never expires.

• Status. License key status. Can be one of:Ok - license key is active,Expired - license key has expired,Broken - invalid license key data was provided.

• Namespace. Name of Kubernetes namespace where this license key is used.

You may have multiple custom Kubernetes resources namedlicense. In that case in order to work with Moon licenses - simply use fully qualified resource name:

To view a license key in YAML format:

To update an existing license key - simply updatedata field in respective license object:

When you update a license key - all changes are applied immediately. This usually also leads to graceful Moon pods restart (does not interrupt running browser sessions).

Moon 2.xsupports sharing the same license key among several Kubernetes namespaces and in the majority of cases a single license key should be enough. However, in some cases you may want to use a separate Moon instance and a separate license key for some teams. To achieve this:

1. Deploy two independent Moon clusters to namespacens1 andns2

2. Create two license objects withnamespace field set tons1 andns2 and save them to file (e.g.license-keys.yaml):

   License keys to be created

   $ cat license-keys.yamlapiVersion: moon.aerokube.com/v1kind: Licensemetadata:  name: license-key-ns1spec:  data: <license-key-1>  namespace: ns1---apiVersion: moon.aerokube.com/v1kind: Licensemetadata:  name: license-key-ns2spec:  data: <license-key-2>  namespace: ns2

3. Apply resulting file:

   $ kubectl apply -f license-keys.yaml

   If you try to create two license keys with the samedata field value, then one of them will be considered as a duplicate and automatically deleted. If you have two different license keys with the samenamespace field, then Moon will always choose the most recently created one.

4. License keys will be applied automatically, and you will see the following in licenses list:

   Two license keys are applied

   $ kubectl get licensesNAME              LICENSEE    SESSIONS   EXPIRES   STATUS   NAMESPACElicense-key-ns1   Acme Inc.   10         32d       Ok       ns1license-key-ns2   Acme Inc.   20         27d       Ok       ns2

To delete an existing license key - simply delete respective license object:

When you delete the last license key withnamespace field set to some Moon namespace, Moon will automatically fall back to the free license key with 4 parallel sessions included.

There are several ways to always have active Moon license keys:

Option 1: Check Expiring License Keys with kubectl

$ kubectl get licensesNAME              LICENSEE    SESSIONS   EXPIRES   STATUS   NAMESPACElicense-key-ns1   Acme Inc.   10         32d       Ok       ns1license-key-ns2   Acme Inc.   20         today     Ok       ns2

$ kubectl get licensesNAME              LICENSEE    SESSIONS   EXPIRES   STATUS    NAMESPACElicense-key-ns1   Acme Inc.   10         32d       Ok        ns1license-key-ns2   Acme Inc.   20         Already   Expired   ns2

Option 2: Use Prometheus License Key Expiration Metric

$ curl -s https://moon.example.com/metrics | grep license_expire# HELP moon_license_expire Moon license expiration time.# TYPE moon_license_expire gaugemoon_license_expire 1.6444512e+09

In some cases you still may want to store license key in aKubernetes secret and allow Moon to automatically read license key from this secret. To achieve this we provide a dedicated component calledlicense-ops. This component is aKubernetes job that reads license key contents from configured secret and automatically updates custom Kubernetes resource being used by Moon. To enabledlicense-ops you need to install one more Helm chart:

1. Having Mooninstalled, create a regular Kubernetes secret in Moon namespace with license key contents:

   Example secret with license key

   apiVersion:v1kind:Secretmetadata:name:licensekeynamespace:moonstringData:license.key:MG1RSVdpc2Z6....# Insert license key contents here

2. Now install one more Helm chart (source code can be foundhere):

   Installing license-ops Helm chart

   $ helm upgrade --install -n moon license-ops aerokube/license-ops

   To change secret name, job schedule and other parameters - use Helm values:

   Updating license-ops parameters

   $ helm upgrade --install --set secretName=mysecret --set schedule="0 * * * *" -n moon license-ops aerokube/license-ops

Moon 2.x compared to Moon 1.x hasno built-in authentication mechanism. This is because the recommended way to deliver authentication in Kubernetes is using available Ingress authentication features or sidecar containers delivering authentication. Moon reads username fromX-Moon-Quota HTTP header being set by Ingress or sidecar container. The following sections describe possible authentication configurations.

Basic HTTP Authentication

Authorization: Basic base64("username:password")

https://username:password@example.com/

OpenID Connect Support

The main configuration object in Moon is calledquota. This object contains all configuration specific to one Moon user. To list available quotas:

As you can see every quota is a native Kubernetes object containing the following information:

• Namespace. Kubernetesnamespace name where Moon starts browsers. By default, this is the same namespace where Moon is running. More details on what you can do with namespaces is describedhere.

• Config. Name of Moonconfiguration object allowing to adjust resources consumption for Moon system images, user and group identifiers and other features.

• Browsers. Name of Moonbrowsers set to use for this quota.

• Devices. Names of Moondevices set to use for this quota.

Using object names instead of their contents allows to easily reuse the same browsers set or devices set for different quotas. To view the same list in YAML format:

To edit a quota object:

Configuration object stores various configuration options: computing resources being assigned to system Moon images, group and user identifiers to run browser pods and so on. This object corresponds toservice.json configuration file in Moon 1.x. To list available configuration objects:

To view the same list in YAML format:

To edit a configuration object:

The browsers set stores browsers startup configuration. This object corresponds tobrowsers.json file in Moon 1.x. To list available browser sets:

To view the same list in YAML format:

To edit a browser set object:

If you were previously using Moon 1.x, you could notice that browsers set in Moon 2.x is slightly different compared tobrowsers.json from Moon 1.x. This is how a typicalbrowsers.json file looks like:

For every browser type and version you had to provide an exact browser image (e.g.browsers/chrome:97.0) and this file contained configuration only for Selenium browsers. The main problem with this approach is that it requires a manual update from Moon cluster administrator every time a new browser image appears. Container images for different versions of the same browser are usually stored in the same repository with different tags, e.g.:

In Moon 2.x instead of copy-pasting the same image specification you only need to provide repository name in browsers set object:

This new configuration format means that all images forchrome browser used in Selenium tests will be downloaded fromquay.io/browser/google-chrome-stable repository. Concrete image tag is determined when you request to start a new browser. For example, you pass the followingSeleniumcapabilities:

In that case Moon will usequay.io/browser/google-chrome-stable:96.0 image. Similarly, inCypress,Playwright andChrome Developer Tools you are using URL path and parameters to pass the same information.

Browser Versions

selenium:chrome:repository:quay.io/browser/google-chrome-stableversions:["96.0", "97.0"](1)default:"96.0"(2)port:4444(3)path:"/"(4)

Computing Resources

selenium:chrome:repository:quay.io/browser/google-chrome-stableresources:(1)limits:(2)cpu:"1.0"(3)memory:"1Gi"(4)requests:(5)cpu:"1.0"(6)memory:"1Gi"(7)

Environment Variables

selenium:chrome:repository:quay.io/browser/google-chrome-stableenv:(1)      -name: TZ(2)value:"Europe/Paris"      -name: LANGvalue:"fr_FR.UTF-8"

selenium:chrome:repository:quay.io/browser/google-chrome-stableenv:        -name: MY_NODE_NAMEvalueFrom:fieldRef:fieldPath:spec.nodeName# Pod spec field value        -name: SECRET_USERNAMEvalueFrom:secretKeyRef:name:some-secret# Secret namekey:username# Key name        -name: MEM_LIMITvalueFrom:resourceFieldRef:containerName:some-container# Container nameresource:limits.memory# Resource parameterdivisor:1Mi        -name: SOME_KEYvalueFrom:configMapKeyRef:name:some-map# ConfigMap namekey:some-key# Key name

Custom Annotations

apiVersion:moon.aerokube.com/v1  kind: BrowserSet  metadata:    name: default    namespace: moon    # Other Kubernetes metadata  spec:    annotations:(1)      key1: "value1"(2)      key2: "value2"

selenium:chrome:repository:quay.io/browser/google-chrome-stableannotations:(1)key1:"value1"(2)key2:"value2"

Custom Labels

apiVersion:moon.aerokube.com/v1  kind: BrowserSet  metadata:    name: default    namespace: moon    # Other Kubernetes metadata  spec:    labels:(1)      key1: "value1"(2)      key2: "value2"

selenium:chrome:repository:quay.io/browser/google-chrome-stablelabels:(1)key1:"value1"(2)key2:"value2"

Node Selectors

apiVersion:moon.aerokube.com/v1  kind: BrowserSet  metadata:    name: default    namespace: moon    # Other Kubernetes metadata  spec:    nodeSelector:(1)      node-label-1: "label1-value"(2)      node-label-2: "label2-value"

selenium:chrome:repository:quay.io/browser/google-chrome-stablenodeSelector:(1)node-label-1:"label1-value"(2)node-label-2:"label2-value"

Affinity

apiVersion:moon.aerokube.com/v1  kind: BrowserSet  metadata:    name: default    namespace: moon    # Other Kubernetes metadata  spec:    affinity:(1)      nodeAffinity:        requiredDuringSchedulingIgnoredDuringExecution:          nodeSelectorTerms:          - matchExpressions:            - key: kubernetes.io/e2e-az-name              operator: In              values:              - e2e-az1              - e2e-az2

selenium:chrome:repository:quay.io/browser/google-chrome-stableaffinity:(1)nodeAffinity:requiredDuringSchedulingIgnoredDuringExecution:nodeSelectorTerms:          -matchExpressions:            -key: kubernetes.io/e2e-az-nameoperator:Invalues:              -e2e-az1              -e2e-az2

Tolerations

apiVersion:moon.aerokube.com/v1  kind: BrowserSet  metadata:    name: default    namespace: moon    # Other Kubernetes metadata  spec:    tolerations:(1)    - key: "key1"      operator: "Equal"      value: "value1"      effect: "NoSchedule"

selenium:chrome:repository:quay.io/browser/google-chrome-stabletolerations:(1)    -key: "key1"operator:"Equal"value:"value1"effect:"NoSchedule"

Networking

selenium:chrome:repository:quay.io/browser/google-chrome-stablednsConfig:(1)nameservers:        -1.2.3.4searches:        -ns1.svc.cluster-domain.example        -my.dns.search.suffixoptions:        -name: ndotsvalue:"2"        -name: edns0hostAliases:(2)      -ip: "127.0.0.1"hostnames:        -"foo.local"        -"bar.local"      -ip: "10.1.2.3"hostnames:        -"foo.remote"        -"bar.remote"

Privileged Mode

selenium:chrome:# For example Android emulators may require privileged moderepository:browsers/androidprivileged:true(1)

Moon loads information about available mobile devices forMobile Emulation fromdevices set object. This object corresponds todevices.json file in Moon 1.x. To list available device sets:

To view the same list in YAML format:

To edit a device set object:

1. Create an S3 bucket. In this example bucket name ismoon-test. You can create an S3-compatible bucket in the majority of public cloud platforms. How to configure Moon with these platforms in shown in the table below:

   Table 12. S3 settings for popular cloud platforms

   Platform Name	Service Name	Endpoint	Signature Version
   AWS	AWS S3	Depends on region, e.g.https://s3.us-east-2.amazonaws.com. See AWSdocumentation for detailed list of endpoints.	S3v4
   DigitalOcean	DigitalOcean Spaces	Depends on region, e.g.https://nyc3.digitaloceanspaces.com. Seedocumentation for more details.	S3v4
   Google Cloud	Google Cloud Storage	https://storage.googleapis.com	S3v2
   Microsoft Azure	Azure Blob Storage	No built-in S3 support. Need to deploy additional software likeMinio.	S3v4

2. Access to S3 bucket can be provided either witha pair of static credentials (anaccess key and asecret key) or byadding cloud platform roles. This section shows how to configure static credentials. How to configure role-based access to S3 bucket is shownbelow. How to load S3 credentials from Kubernetes secret is describedhere.

3. Update storage settings inconfiguration object:

   apiVersion:moon.aerokube.com/v1kind:Configmetadata:name:defaultnamespace:moon# Other Kubernetes metadataspec:# Some fields beforestorage:accessKey:"AKIAXXXXXXXXXXXXXXXX"# Set if a pair of credentials is usedbucket:"moon-test"filename:""# Recorded video file name, e.g. myvideo.mp4endpoint:"https://s3.us-east-2.amazonaws.com"pattern:""# See belowsecretKey:"okUa0XXXXXXXXXXXXXXXXXXXX"# Set if a pair of credentials is used# Other fields

How to enable video recording depends on browser automation technology you are using:

• Selenium

• Cypress

• Playwright

• Chrome Developer Tools

Custom S3 Layout

\---my-bucket    \---- <session-id>        |---- video.mp4

$quota/$browserName/$browserVersion/$sessionId

my-bucket/chrome-71-0-686efb96-eabe-4435-af31-21a33c8a4c8b/video.mp4

apiVersion:moon.aerokube.com/v1kind:Configmetadata:name:defaultnamespace:moon# Other Kubernetes metadataspec:# Some fields beforestorage:# Other S3 storage settingspattern:"$quota/$browserName/$browserVersion/$sessionId"# Other fields

Getting S3 credentials from Kubernetes secret

Role-based Access to S3

Uploading Videos Through Proxy

apiVersion:moon.aerokube.com/v1kind:Configmetadata:name:defaultnamespace:moon# Other Kubernetes metadataspec:# Some fields beforestorage:# Other S3 storage settingsnoProxy:"*.example.com"# Don't use proxy for these hostshttpProxy:"proxy.example.com:3128"# Proxy host and port for HTTP traffichttpsProxy:"proxy.example.com:3128"# Proxy host and port for HTTPS traffic# Other fields

Disabling instance metadata support

apiVersion:moon.aerokube.com/v1kind:Configmetadata:name:defaultnamespace:moon# Other Kubernetes metadataspec:# Some fields beforestorage:# Other S3 storage settingsmetadata:false# This disables instance metadata (default is true)# Other fields

To activate this solution you need to install one moreHelm chart:

1. Add Aerokubecharts repository if you don’t have it already:

   $ helm repo add aerokube https://charts.aerokube.com/$ helm repo update

2. Installbrowser-ops chart:

   $ helm upgrade --install -n moon browser-ops aerokube/browser-ops

3. Run update manually for the first time (otherwise you will have to wait for the time schedule configured in chart values):

   $ kubectl create job -n moon --from=cronjob/browser-ops update-browsersjob.batch/update-browsers created$ kubectl get jobs -n moonNAME              COMPLETIONS   DURATION   AGEupdate-browsers   0/1           4s         4s

Like for any other Helm chart,browser-ops configuration options are stored invalues.yaml file and applied like this:

• By default, browser versions will be updated every night. To change version update schedule:

  schedule:"0 */2 * * *"# Run every two hours

• By default,browser-ops will configure Moon to use the latest available browser version. You can easily change this behavior to configure all available browser versions or fixed number of the latest available browser versions:

  browserImageVersions:all# Use all available browser versions

  browserImageVersions:5# Use 5 available browser versions

• By default,browser-ops will use long browser versions like102.0.1245.30 instead of102.0. To use short versions:

  browserImageTagFormat:short

  When using short browser versions the latest available browser version will not be present in the list. This is because for one major browser version there could be several minor updates. Such behavior allows to not cache a minor browser version update that can be later updated one more time by browser developers.

• By default,browser-ops only updates browsers set object nameddefault. If you have several browsers set objects, provide all required names of such objects:

  browsersets:-default-alpha-beta

Sometimes things go wrong: user can unexpectedly disconnect or browser session starts longer than needed. This can lead to overall cluster degradation because of broken browser pods occupying all available hardware. To prevent such cases Moon automatically detects and closes idle browser sessions. A session is considered idle when the delay between separate HTTP requests corresponding to a running session is bigger than configured timeout. Idle timeout may need to be increased when tested application pages are loading too slowly. To view or change idle timeout setting - take a look atconfiguration object:

Several rarely needed Mooncommand-line flags are responsible for advanced timeout settings:

Not only Moon timeout settings can cause your tests to break. A typical Moon installation looks like the following:

In addition to Moon timeouts other possible sources of timeouts exist:

1. Client-side Timeout. Every Selenium library is internally using an HTTP client having default request timeout settings. If you are frequently seeingclient disconnected messages (meaning that client disconnected before request handling completed) in Moon log - this could be a sign to increase HTTP client timeouts in your code.

2. Load Balancer Timeout. Usually Moon is running behind load balancer (LoadBalancer,Ingress orRouter), and it also has a default request proxy timeout. A frequent value is60 seconds, so if you are often seeing test fails with502 Bad Gateway or504 Gateway Timeout errors - this could be a sign to increase load balancer timeout. How to do this depends on your cloud platform and load balancer type being used. So refer to their documentation for more details. An example of doing this for AWS cloud is shown inConnection was closed unexpectedly section.

3. Cluster Capacity Reached. If you are seeing a lot ofunexpected status messages in the log that can signalize that you used all available computing resources (CPUs and memory) assigned to Moon namespace.

4. Cluster Fragmentation. Similarly to the previous one, in some cases you can have sufficient number of cores and not all browsers are exhausted. However, sometimes for example you can have 4 CPUs available distributed among 4 Kubernetes nodes (1 available CPU per node) and a new browser pod requiring at least 2 CPUs to start (all pod containers always run on the same node). In that case although total number of available CPUs is sufficient to start a pod, there is no node where pod will be able to start. If you are seeing too many browser pods inPending state - check withkubectl command why these pods are not starting.

Moon has reasonable defaults for resources consumed by every browser pod. Sometimes you may need to override these settings. To override resource settings globally for every browser image - useconfiguration object:

To update resource settings - simply edit configuration object, save and exit:

You can also override the same values for every browser type inbrowsers set. An example snippet can be foundhere.

To check service images resources requirements - simply show configuration object for your quota in YAML format:

To adjust CPU and memory consumption for each service image - simply updateconfiguration object accordingly.

Browser automation stability and speed highly depends on how many computing resources are actually available to browser pods. Kubernetes has so-calledQuality of Service (QoS) defining how many resources are assigned to pods being started. For stable browser automation we recommend always settingGuaranteed QoS class to Moon browser pods. To deliver this you have to make sure thatrequests andlimits values for CPU and memory have equal values:

1. Moon by default setsrequests equal tolimits for service images likedefender,logger andvideoRecorder. But in recent releases you can override them independently if you wish.

2. For browser containers you can overriderequests andlimits independently. Anyway we also recommend setting them to equal values. Only this way you will be sure that browsers are always getting the same computing resources. Otherwise, you may encounter randomly failing browser tests caused by insufficient computing resources assigned to some browser pods.

apiVersion:v1kind:Podmetadata:name:my-appannotations:key1:"value1"key2:"value2"labels:key1:"value1"key2:"value2"spec:containers:  -name: appimage:my-company/my-app:1.0.0resources:requests:memory:"64Mi"cpu:"250m"limits:memory:"128Mi"cpu:"500m"

apiVersion:moon.aerokube.com/v1kind:BrowserSetmetadata:name:defaultnamespace:moon# Other Kubernetes metadataspec:annotations:key1:"value1"key2:"value2"labels:key1:"value1"key2:"value2"

This is configured globally or for concrete browser types inbrowsers set. How to do this is describedhere.

This is configured globally or for concrete browser types inbrowsers set. How to do this is describedhere. Also, you can override labels usinglabels capability.

Network policies are dedicated Kubernetes objects allowing to control network firewall rules. Using them with Moon is straightforward:

1. Create aNetworkPolicy object. This is how it can look like:

   apiVersion:networking.k8s.io/v1kind:NetworkPolicymetadata:name:my-network-policynamespace:moonspec:podSelector:matchLabels:# This rule will apply pods matching labels belowrole:browseringress:  -from:    -podSelector:        matchLabels:          role: my-appports:    -protocol: TCPport:6379

2. Usecustom labels to add respective label to browser pods:

   selenium:chrome:repository:quay.io/browser/google-chrome-stablelabels:role:browser# Every Chrome pod will have this label set

This is configured globally or for concrete browser types inbrowsers set. How to do this is describedhere.

This is configured globally or for concrete browser types inbrowsers set. How to do this is describedhere.

This is configured globally or for concrete browser types inbrowsers set. How to do this is describedhere.

This is configured for concrete browser types inbrowsers set. How to do this is describedhere.