Prerequisites

For running all compiler tests you may want to optionally install a fewdependencies (evmone,z3,Eldarica,cvc5 <https://github.com/cvc5/cvc5>).

On macOS systems, some of the testing scripts expect GNU coreutils to be installed.This can be easiest accomplished using Homebrew:brewinstallcoreutils.

On Windows systems, make sure that you have a privilege to create symlinks,otherwise several tests may fail.Administrators should have that privilege, but you may alsogrant it to other usersorenable Developer Mode.

Running the Tests

Solidity includes different types of tests, most of them bundled into theBoost C++ Test Framework applicationsoltest.Runningbuild/test/soltest or its wrapperscripts/soltest.sh is sufficient for most changes.

The./scripts/tests.sh script executes most Solidity tests automatically,including those bundled into theBoost C++ Test Frameworkapplicationsoltest (or its wrapperscripts/soltest.sh), as well as command-line tests andcompilation tests.

The test system automatically tries to discover the location oftheevmone for running the semantic tests.

Theevmone library must be located in thedeps ordeps/lib directory relative to thecurrent working directory, to its parent or its parent’s parent. Alternatively, an explicit locationfor theevmone shared object can be specified via theETH_EVMONE environment variable.

evmone is needed mainly for running semantic and gas tests.If you do not have it installed, you can skip these tests by passing the--no-semantic-testsflag toscripts/soltest.sh.

Theevmone library should end with the file nameextension.so on Linux,.dll on Windows systems and.dylib on macOS.

For running SMT tests, thez3 executable must be present inPATH.A few SMT tests useEldarica instead ofz3.These require its executable (eld) to be present inPATH for the tests to pass.However, ifEldarica is not found, these tests will be automatically skipped.

Ifz3 is not present on your system, you should disable theSMT tests by exportingSMT_FLAGS=--no-smt before running./scripts/tests.sh orrunning./scripts/soltest.sh--no-smt.These tests arelibsolidity/smtCheckerTests.

For quicker results you can run a subset of, or specific tests.

To run a subset of tests, you can use filters:./scripts/soltest.sh-tTestSuite/TestName,whereTestName can be a wildcard*.

Or, for example, to run all the tests for the yul disambiguator:./scripts/soltest.sh-t"yulOptimizerTests/disambiguator/*"--no-smt.

./build/test/soltest--help has extensive help on all of the options available.

See especially:

• show_progress (-p) to show test completion,

• run_test (-t) to run specific tests cases, and

• report-level (-r) give a more detailed report.

If you want to debug using GDB, make sure you build differently than the “usual”.For example, you could run the following command in yourbuild folder:

cmake-DCMAKE_BUILD_TYPE=Debug..make

This creates symbols so that when you debug a test using the--debug flag,you have access to functions and variables in which you can break or print with.

The CI runs additional tests (includingsolc-js and testing third party Solidityframeworks) that require compiling the Emscripten target.

Writing and Running Syntax Tests

Syntax tests check that the compiler generates the correct error messages for invalid codeand properly accepts valid code.They are stored in individual files inside thetests/libsolidity/syntaxTests folder.These files must contain annotations, stating the expected result(s) of the respective test.The test suite compiles and checks them against the given expectations.

For example:./test/libsolidity/syntaxTests/double_stateVariable_declaration.sol

open in Remix

contracttest{uint256variable;uint128variable;}// ----// DeclarationError: (36-52): Identifier already declared.

A syntax test must contain at least the contract under test itself, followed by the separator//----. The comments that follow the separator are used to describe theexpected compiler errors or warnings. The number range denotes the location in the source where the error occurred.If you want the contract to compile without any errors or warning you can leaveout the separator and the comments that follow it.

In the above example, the state variablevariable was declared twice, which is not allowed. This results in aDeclarationError stating that the identifier was already declared.

Theisoltest tool is used for these tests and you can find it under./build/test/tools/. It is an interactive tool which allowsediting of failing contracts using your preferred text editor. Let’s try to break this test by removing the second declaration ofvariable:

open in Remix

contracttest{uint256variable;}// ----// DeclarationError: (36-52): Identifier already declared.

Running./build/test/tools/isoltest again results in a test failure:

syntaxTests/double_stateVariable_declaration.sol: FAIL    Contract:        contract test {            uint256 variable;        }    Expected result:        DeclarationError: (36-52): Identifier already declared.    Obtained result:        Success

isoltest prints the expected result next to the obtained result, and alsoprovides a way to edit, update or skip the current contract file, or quit the application.

It offers several options for failing tests:

• edit:isoltest tries to open the contract in an editor so you can adjust it. It either uses the editor given on the command-line (asisoltest--editor/path/to/editor), in the environment variableEDITOR or just/usr/bin/editor (in that order).

• update: Updates the expectations for contract under test. This updates the annotations by removing unmet expectations and adding missing expectations. The test is then run again.

• skip: Skips the execution of this particular test.

• quit: Quitsisoltest.

All of these options apply to the current contract, exceptquit which stops the entire testing process.

Automatically updating the test above changes it to

open in Remix

contracttest{uint256variable;}// ----

and re-run the test. It now passes again:

Re-running test case...syntaxTests/double_stateVariable_declaration.sol: OK

Command-line Tests

Our suite of end-to-end command-line tests checks the behaviour of the compiler binary as a wholein various scenarios.These tests are located intest/cmdlineTests/,one per subdirectory, and can be executed using thecmdlineTests.sh script.

By default the script runs all available tests.You can also provide one or morefile name patterns,in which case only the tests matching at least one pattern will be executed.It is also possible to exclude files matching a specific pattern by prefixing it with--exclude.

By default the script assumes that asolc binary is available inside thebuild/ subdirectoryinside the working copy.If you build the compiler outside of the source tree, you can use theSOLIDITY_BUILD_DIR environmentvariable to specify a different location for the build directory.

Example:

exportSOLIDITY_BUILD_DIR=~/solidity/build/test/cmdlineTests.sh"standard_*""*_yul_*"--exclude"standard_yul_*"

The commands above will run tests from directories starting withtest/cmdlineTests/standard_ andsubdirectories oftest/cmdlineTests/ that have_yul_ somewhere in the name,but no test whose name starts withstandard_yul_ will be executed.It will also assume that the filesolidity/build/solc/solc inside your home directory is thecompiler binary (unless you are on Windows – thensolidity/build/solc/Release/solc.exe).

There are several kinds of command-line tests:

• Standard JSON test: contains at least aninput.json file.In general may contain:

  • input.json: input file to be passed to the--standard-json option on the command line.

  • output.json: expected Standard JSON output.

  • args: extra command-line arguments passed tosolc.

• CLI test: contains at least aninput.* file (other thaninput.json).In general may contain:

  • input.*: a single input file, whose name will be supplied tosolc on the command line.Usuallyinput.sol orinput.yul.

  • args: extra command-line arguments passed tosolc.

  • stdin: content to be passed tosolc via standard input.

  • output: expected content of the standard output.

  • err: expected content of the standard error output.

  • exit: expected exit code. If not provided, zero is expected.

• Script test: contains atest.* file.In general may contain:

  • test.*: a single script to run, usuallytest.sh ortest.py.The script must be executable.

English Language

Use International English, unless using project or brand names. Try to reduce the usage oflocal slang and references, making your language as clear to all readers as possible.Below are some references to help:

• Simplified technical English

• International English

Title Case for Headings

Usetitle case for headings. This means capitalise all principal words intitles, but not articles, conjunctions, and prepositions unless they start thetitle.

For example, the following are all correct:

• Title Case for Headings.

• For Headings Use Title Case.

• Local and State Variable Names.

• Order of Layout.

Expand Contractions

Use expanded contractions for words, for example:

• “Do not” instead of “Don’t”.

• “Can not” instead of “Can’t”.

Active and Passive Voice

Active voice is typically recommended for tutorial style documentation as ithelps the reader understand who or what is performing a task. However, as theSolidity documentation is a mixture of tutorials and reference content, passivevoice is sometimes more applicable.

As a summary:

• Use passive voice for technical reference, for example language definition and internals of the Ethereum VM.

• Use active voice when describing recommendations on how to apply an aspect of Solidity.

For example, the below is in passive voice as it specifies an aspect of Solidity:

Functions can be declaredpure in which case they promise not to readfrom or modify the state.

For example, the below is in active voice as it discusses an application of Solidity:

When invoking the compiler, you can specify how to discover the first elementof a path, and also path prefix remappings.

Common Terms

• “Function parameters” and “return variables”, not input and output parameters.

Code Examples

A CI process tests all code block formatted code examples that begin withpragmasolidity,contract,libraryorinterface using the./test/cmdlineTests.sh script when you create a PR. If you are adding new code examples,ensure they work and pass tests before creating the PR.

Ensure that all code examples begin with apragma version that spans the largest where the contract code is valid.For examplepragmasolidity>=0.4.0<0.9.0;.

Running Documentation Tests

Make sure your contributions pass our documentation tests by running./docs/docs.sh that installs dependenciesneeded for documentation and checks for any problems such as broken links or syntax issues.