<dependency>    <groupId>de.tum.in.ase</groupId>    <artifactId>artemis-java-test-sandbox</artifactId>    <version>1.13.0</version>    <scope>test</scope></dependency>

testImplementation'de.tum.in.ase:artemis-java-test-sandbox:1.13.0'

java.lang.SecurityException: do not use System.exit(int)/// potential problem location: Penguin.getName(Penguin.java:12) ///

2.3.1. What you need to do outside Ares

Sadly, due to the way classes are loaded and the class path works whentesting student code with maven, there are still vulnerabilities ifstudents manage to load classes that would be in trusted packages. Thisis especially problematic if they shadow library classes, such asJUnit’sAssertions.

To prevent that, you have to use the Maven Enforcer Plugin to make sureno student content lands in trusted packages:

Maven:

Gradle:

1. Important: you want to enforce the non-existence of classes aftertheir generation but before testing.

2. This is where all folders/packages go that we don’t want to existin student code. You will always find the most recent recommendationfor Ares here. If you use additional third-party libraries that needto be configured using@AddTrustedPackage, you should add thosepackages here as well. Ares will check that all entries are present.
   If you don’t want Ares to do so, set theares.maven.ignore orares.gradle.ignore system property totrue. In case you want Aresto look into a different file, you can set theares.maven.pom orares.gradle.build to a path other than the defaultpom.xml orbuild.gradle.

2.4.1. Path Access and Class Loading

You can use@WhitelistPath and@BlacklistPath to control access topaths. By default, no access is granted, and so you need to use@WhitelistPath to give student code the permission to read and writefiles explicitly. You can specify exceptions using@BlacklistPathwhich will overpower the whitelisted paths.

The following examples will make use ofcourse1920xyz as placeholdervalue for the real Artemis exercise name/id. Replace it with the realone when borrowing code snippets, or nothing will work as expected.

Most importantly, this does not only apply to explicit file IO, but alsoto the.class files that the class loader reads, as needed. Thisalready happens if one student class requires another one, that has notbeen loaded after that. You can recognize that in the standard erroroutput:

This usually means the class loader could not load the class. Theparentheses show, that the problem is the missing whitelisting.Therefore, all test setups should have some whitelisting.

A number of examples how you can whitelist paths in Ares:

• @WhitelistPath("") will grant read access to the paths in thedirectory of execution, which is usually where thepom.xml is.

• @WhitelistPath("pom.xml") will allow students to read thepom.xml.

• @WhitelistPath("..") will allow read access to the level above themaven project. In Eclipse, that is the level of your workspace.

• @WhitelistPath(value = "../course1920xyz**", type = PathType.GLOB)grants read access to projects beginning with the exercise "id" usedby Artemis. Should you use the Eclipse feature "Referenced Projects"(or the analog to that in your IDE) to link the student/solution projectto the tests, you will need a setting like this.

• @WhitelistPath(value = "data", level = PathActionLevel.DELETE) willallow students to read, write and delete files in thedata directoryand subdirectories.

• @WhitelistPath("target") allows reading files in target (Maven output folder)

• @BlacklistPath(value = "*Test.{java,class}", type = PathType.GLOB)prevents access to classes in source code or compiled form that containTest. If you leave away the* afterTest, nested classes are notblacklisted. Student classes should not be called something with"Test" then.

That was not everything but already quite a lot. Take a look at theJavadoc of the annotations and enums used, if you want to know more.Before you give up, here is my recommendation how to start:

Add a@BlacklistPath for other important classes, like your referenceimplementations of the solution to test against should you use thatapproach.

Note: the Artemis project starts withcourse1920xyz, but the build inBamboo (by Artemis) will happen in a directory named after the buildplan, which is in upper case and therefore, begins withCOURSE1920XYZ.Make sure that you do not build multiple student solutions in the samedirectory on the same machine using the git clone (lower case) approach.Otherwise, adjust the whitelisting to your needs.

2.4.2. Timeouts

JUnit already provides means of applying timeouts to tests. However,those arenot strict in the sense of "enforced in the strongestpossible way". What is meant by that?

There are three different ways how the timeouts can work:

• likeorg.junit.jupiter.api.Timeout
  This timeout is not preemptive, and the test itself runs in the samethread executing the tests. It will only try to stop the test via aninterrupt. If that fails like it does for an endless loop, the testwill definitively fail. After it is finished. Which might neverhappen and the main reason not to use this when it comes to testingunknown code.

• likeorg.junit.jupiter.api.Assertions.assertTimeoutPreemptively
  This will fail the test preemptively by executing theExecutableargument itself in a different thread than the thread executing alltests. It will only try to stop the test via an interrupt, but ifthat fails it will simply carry on. The test thread might still run,though.

• likede.tum.in.test.api.StrictTimeout
  This uses a mechanism similar toassertTimeoutPreemptively, butwill resort to harder means if necessary.It will in the following order:

  1. wait the given duration

  2. interrupt the thread executing the test and wait no longer (likeassertTimeoutPreemptively)

  3. block the creation of new threads

  4. interrupt all threads created during the test and try to join thethreads

  5. if that fails, useThread.stop() on all remaining threadsand try to join again

  6. repeat step 5 multiple times, if required

  7. Should that fail, report a special SecurityException that not allthreads could be stopped. (see the standard error output for a detailedreport then)If that happens, no more tests can be properly executedbecause the security cannot be guaranteed and the test cases cannot beexecuted "in isolation". All following tests will fail.

Rule 1: When testing with Ares, always use@StrictTimeout fortimeouts, the others will not work reliably, especially in conjunctionwith the Ares security.

Rule 2: When writing tests for Artemis, always use@StrictTimeout.There is no reason to omit the timeout, since you do not know the codestudents will write. (And they will write code spawning millions ofthreads in endless loops, which in turn will do the same recursively.)

2.4.3. Showing Standard Output

By default, Ares will record standard and error output of each testinternally and not print it to the console. The recorded output can thenbe obtained and tested, seeIOTester The reason for this is onthe one hand to keep the console and logs short and clean and on theother hand prevent students from accidentally messing up the logs withmillions of lines. Ares also has a hard limit on the total number ofprinted chars at around 10 million.

To mirror the output recorded by Ares to the console, use the@MirrorOutput annotation on the test class or method.

It is also worth noting that Ares enforces valid UTF-8being printed and throws an appropriate exception otherwise.

2.4.4. Testing the Exercise before Release

Hidden tests will be executed by Ares only after the deadline. Thisposes the problem, how the exercise creators should work on the tasks,tests and the sample solution. One possible solution would be to use analternative deadline annotation or change the deadline temporarily. Theproblem is that it is quite likely one might forget to change it backagain, and protecting the hidden tests would fail.

Use@ActivateHiddenBefore just like@Deadline to state theLocalDateTime before which hidden tests should be executed. This dateshould, of course, be before the release of the exercise on Artemis.

2.4.5. Extending a Deadline and Disability Compensation

You can use@ExtendedDeadline together with a duration like1d or2d 12h 30m to extend the deadline by the given amount.@ExtendedDeadline("1d"), for example, extends the deadline by one day.If you use the annotation on different levels (e.g. class and method)without stating a new deadline (e.g. deadline only on class level), theextensions will be added together.

2.4.6. Threads and Concurrency

By default, Ares will not allow non-whitelisted code to use threads atall. That includes thread pools, but excludes the common pool and itsusers, like parallel streams. To allow the use of Threads, use theannotation@AllowThreads. The number of active threads is alsolimited, the default value of that is 1000, but can be changed in theannotation. Please keep in mind that this limit should not be largerthan 1000 to prevent performance and timeout chaos.

New threads are for security reasons not directly whitelisted by Aresand will not be allowed to do anything security critical. If you trust athread (at least its entry point), you can explicitly request the threadto be whitelisted usingArtemisSecurityManager.requestThreadWhitelisting(Thread). The threadcalling the method and its stack must be whitelisted, of course.

2.4.7. Testing Console Interaction

One example showing some possibilities here:

1. DeclareIOTester as parameter.

2. Provide input lines before calling the student code.This content will be used for reading lines fromSystem.in.

3. Call the student code to process the input and produce output.

4. Assert that nothing was printed toSystem.err.

5. Assert that the standard output (in this case excluding the finalline break) is equal to the given text block (if you use textblocks, be aware of their newline handling).

Note that Ares normalizes the line breaks to\n, andOutputTesteroffers many different approaches to checking output (e.g. single string, list of strings, …​).

If students read more lines than provided, they get the following feedback:…​java.lang.IllegalStateException: no further console input request after the last(number 1: "5") expected.…​

See alsoIOTester and for more examples, theInputOutputUsertest.

Showing Standard Output covers how the student output ismanaged and shown in the test logs.

2.4.8. Networking

Ares allows for local network connections by using the@AllowLocalPortannotation.

There are plenty of configuration options, and the code can get complicatedquickly due to the threads required to test network connections.One issue can be that waiting network connections block threads in sucha way that they cannot be stopped (waiting in native code), so werecommend using timeouts for connections at least on one end consistently.

For examples, have a look at the testNetworkUser.

2.4.9. Locale

You can set a locale for Ares (and the rest of Java) by adding the@UseLocale JUnit extension to classes/methods, which will set the Javadefault localethat is also used by Ares. The locale is changed only for the scopewhere the annotation is applied.

Ares is currently localized in German (de_DE) and English (en_US),whereen_US is the fallback for any other locale.

See also theLocaleUsertest for more examples.