About
Getting Started
Client Testing Steps
Server and CLI Testing Steps

VS Code has an extensive release process, calledEndgame, that contains multiple testing processes to ensure that we deliver a product of the highest quality to our users. Sanity testing is the final verification process a build must pass before being released.

NOTE: This testing is done for each platform build we ship.

Sanity testing is a manual process and is the opportunity to have a final set of eyes on a build before it is released to users. This testing ensures that the VS Code stable release candidate (Stable RC) installs and runs as expected.

We are looking for errors, failures, or anything else that is not desired behavior. Examples are things not loading or errors when trying to open the build in the first place.

• On the most basic level: install, launch, then open a file or the about dialog
• Other options to test a bit more:
  • Click through all the items on the sidebar
  • Try some command from the command palette
  • Open / click around in the terminal
  • Install and use an extension

During sanity testing, sections are assigned by operating system. Some operating systems along with the sections that they can test are as follows:

In addition to those operating systems, any web browser on devbox.microsoft.com can test Windows x64, Linux x64, Linux Server, and Linux CLI sections.

Each section on the endgame plan lists specific builds to test. Some of the builds are as follows:

• Client: the following group of builds
  • Installer: on Windows, the system and user executables are installers that install VS Code to a system or user directory, respectively. With these executables, a user can launch VS Code after installing it.
  • Archive: a zip or tar.gz archive. On Windows and Linux, these builds do not have an installer and can be run automatically after extracting them. On macOS, the builds can be run after extracting them and moving the application file to the/Applications folder.
  • Universal Archive: an archive specifically for macOS that supports both Intel and Apple Silicon chips by bundling two binaries into a single product.
  • Debian, RPM, and Snap packages: Linux packages that require differing steps per package to install. Steps for each package are listed in the following sections.
• Server: theVS Code server. Steps for each platform are listed in the following sections.
• CLI: the VS Code command line interface that comes bundled with VS Code. During sanity testing, we use the CLI to testremote tunnels. Steps for each platform are listed in the following sections.

Once sections are assigned, follow the steps for your assigned section and/or builds below, then check off your assigned section and/or builds on the endgame plan.

• On macOS you can get a Parallels license from the team admin to spin up Windows and Linux VMs.
• On Windows (virtualized or not), you can use thesanity testing WSL images created by @sbatten.
• Dev Box, thanks to @lszomoru, allows us to spin up and access Windows x64 VMs from any browser.
• AnUbuntu Desktop VM can run the Debian, Archive, and Snap builds. AFedora Workstation VM can run the RPM and Archive builds.

We now have a command line utility to aid in sanity testing (forx64 andarm64). It builds on top ofvscode-bisect:

You can run it as so:

npx @vscode/vscode-bisect@latest --sanity --commit<commit SHA of build to test>

It will guide you through the process and download, install and execute each build artefact that needs to be tested.

Supported builds and flavors (all):

The Windows system and user executables are installers. After installing VS Code, it can be launched from the start menu.
The Windows archive can be run by extracting the archive and double-clicking on the extracted executable.
The macOS archives can be run by extracting the archive, moving the app to the/Applications folder, and launching it.
The Linux archive can be run by extracting it, opening a terminal to that directory, and running./code.
The Linux packages require differing commands to install and remove depending on what package manager you have available:

After installing a Linux package, run VS Code by runningcode in the terminal.If both the Debian and snap packages are installed, you can run the snap package by runningsnap run code in the terminal.

You can sanity test the CLI and the server at the same time, by using the CLI to start a tunnel and connecting to it from a browser, specifying the right commit hash for the server version under test.

1. Download and extract the CLI
2. Run it with./code tunnel
3. The CLI will print avscode.dev link. Open that link, but add?vscode-version=<commit hash> to the end of the URL, where<commit hash> is the hash of the build that is being tested. You can find this on the builds page.
4. Navigate to that URL. This will cause the CLI to download the specified version of the server and open a remote session in your browser over the tunnel.

Note: You can run this test from macOS or Windows too by installing and runningDocker Desktop.

1. Look up the build's commit id and replace<commit> with that value below.
2. Run the following commands one-by-one. They register the QEMU hardware emulator and start 5 Linux builds of the CLI:

export COMMIT="<commit>"# Bash$env:COMMIT='<commit>'# PowerShelldocker run --privileged --rm tonistiigi/binfmt --uninstall'*'docker run --pull always --privileged --rm tonistiigi/binfmt --install alldocker run -e COMMIT -it --rm --pull always --platform linux/amd64 mcr.microsoft.com/devcontainers/base:latest /bin/sh -c'apt update && DEBIAN_FRONTEND=noninteractive apt install -y wget libatomic1 ca-certificates python3-minimal && wget "https://update.code.visualstudio.com/commit:$COMMIT/cli-linux-x64/stable" -O- | tar -xz && ./code tunnel'docker run -e COMMIT -it --rm --pull always --platform linux/arm64 mcr.microsoft.com/devcontainers/base:latest /bin/sh -c'apt update && DEBIAN_FRONTEND=noninteractive apt install -y wget libatomic1 ca-certificates python3-minimal && wget "https://update.code.visualstudio.com/commit:$COMMIT/cli-linux-arm64/stable" -O- | tar -xz && ./code tunnel'docker run -e COMMIT -it --rm --pull always --platform linux/arm/v7 arm32v7/ubuntu /bin/sh -c'apt update && DEBIAN_FRONTEND=noninteractive apt install -y wget libatomic1 ca-certificates python3-minimal && wget "https://update.code.visualstudio.com/commit:$COMMIT/cli-linux-armhf/stable" -O- | tar -xz && ./code tunnel'docker run -e COMMIT -it --rm --pull always --platform linux/amd64 amd64/alpine /bin/sh -c'apk update && apk add musl libgcc libstdc++ && wget "https://update.code.visualstudio.com/commit:$COMMIT/cli-alpine-x64/stable" -O- | tar -xz && ./code tunnel'docker run -e COMMIT -it --rm --pull always --platform linux/arm64 arm64v8/alpine /bin/sh -c'apk update && apk add musl libgcc libstdc++ && wget "https://update.code.visualstudio.com/commit:$COMMIT/cli-alpine-arm64/stable" -O- | tar -xz && ./code tunnel'

For each CLI test:

1. Make sure the program starts and you see and can agree to the license
2. Connect to the tunnel viahttps://vscode.dev/tunnel/<name>?vscode-version=COMMIT
3. Close the tab, then hit Ctrl+C on the server.

Use the Remote-SSH extension to connect from any client platform to a Windows remote. You can connect to localhost on your own Windows machine, a parallels VM, or another machine. You'll need toset up and start OpenSSH services, such as the OpenSSH SSH Server service and the OpenSSH Authentication Agent service.

Note: if you are anAAD user on your Windows machine, you may have to apply anSSH workaround.

Here is an example localhost config that can be added to your SSHconfig file:

Host localhost    HostName localhost    User <username>@microsoft.com

Where<username> can be found by runningwhoami /user on cmd.exe.When connecting, the password is the same as the user password (which might be the Active Directory password, but definitely not the login pin).

Use the Remote-SSH extension to connect from any client platform to a macOS remote (connecting tolocalhost is ok). To start the SSH server, you just have toenable Remote Login.