Conventional mutation testing tries to identifygaps in test coverage, whereas Necessist tries to identifybugs in existing tests.

Conventional mutation testing tools (such asuniversalmutator) randomly inject faults into source code, and see whether the code's tests still pass. If they do, it could mean the code's tests are inadequate.

Notably, conventional mutation testing is about finding deficiencies in the set of tests as a whole, not in individual tests. That is, for any given test, randomly injecting faults into the code is not especially likely to reveal bugs in that test. This is unfortunate since some tests are more important than others, e.g., because ensuring the correctness of some parts of the code is more important than others.

By comparison, Necessist's approach of iteratively removing statements and method calls does target individual tests, and thus can reveal bugs in individual tests.

Of course, there is overlap in the sets of problems the two approaches can uncover, e.g., a failure to find an injected fault could indicate a bug in a test. Nonetheless, for the reasons just given, we see the two approaches as complementary, not competing.

The following criterion (*) comes close to describing the statements that Necessist aims to remove:

• (*) StatementS'sweakest preconditionP has the same context (e.g., variables in scope) asS's postconditionQ, andP does not implyQ.

The notion that (*) tries to capture is: a statement that affects a subsequent assertion. In this section, we explain and motivate this choice. For concision, we focus on statements, but the remarks in this section apply to method calls as well.

Recall the two kinds ofpredicate transformer semantics: weakest precondition and strongest postcondition. With the former, one reasons about the weakest precondition that could hold prior to a statement, given a postcondition that holds after the statement. With the latter, one reasons about the strongest postcondition that could hold after a statement, given a precondition that holds prior to the statement. Generally speaking, the former is more common (seeAldrich 2013 for an explanation), and it is the one we use here.

Consider a test through this lens. A test is a function with no inputs or outputs. Thus, an alternative procedure for determining whether a test passes is the following. Starting withTrue, iteratively work backwards through the test's statements, computing the weakest precondition of each. If the precondition arrived at for the test's first statement isTrue, then the test passes. If the precondition isFalse, the test fails.

Now, imagine we were to apply this procedure, and consider a statementS that violates (*). We argue that it might not make sense to removeS:

Case 1:S adds or removes variables from the scope (e.g.,S is a declaration), orS changes a variable's type. Then removingS would likely result in a compilation failure. (On top of that, sinceS's precondition and postcondition have different contexts, it's not clear how to compare them.)

Case 2:S's precondition is stronger than its postcondition (e.g.,S is an assertion). ThenS imposes constraints on the environments in which it executes. Put another way,Stests something. Thus, removingS would likely detract from the test's overarching purpose.

Conversely, consider a statementS that satisfies (*). Here is why it might make sense to removeS. Think ofS asshifting the set of valid environments, rather than constraining them. More precisely, ifS's weakest preconditionP does not implyQ, and ifQ is satisfiable, then there is an assignment toP andQ's free variables that satisfies bothP andQ. If such an assignment results from each environment in whichS is actually executed, then the necessity ofS is called into question.

The main utility of (*) is in helping to select the functions, macros, and method calls that Necessist ignores. Necessist ignores certain of these by default. Suppose that, for one of the frameworks, we are considering whether Necessist should ignore some functionfoo. If we imagine a predicate transformer semantics for the framework's testing language, we can ask: if statementS were a call tofoo, wouldS satisfy (*)? If the answer is "no," then Necessist should likely ignorefoo.

Consider Rust'sclone method, for example. A call toclone can be unnecessary. However, if we imagine a predicate transformer semantics for Rust, a call toclone is unlikely to satisfy (*). For this reason, Necessist does not attempt to removeclone calls.

In addition to helping to select the functions, etc. that Necessist ignores, (*) has other nice consequences. For example, the rule that the last statement in a test should be ignored follows from (*). To see this, note that such a statement's postconditionQ is alwaysTrue. Thus, if the statement doesn't change the context, then its weakest precondition necessarily impliesQ.

Having said all this, (*) doesn't quite capture what Necessist actuallydoes. Consider a statement likex -= 1;. Necessist will remove such a statement unconditionally, but (*) says maybe Necessist shouldn't. Assumingoverflow checks are enabled, computing this statement's weakest precondition would look something like the following:

{ Q[(x - 1)/x] ^ x >= 1 }x -= 1;{ Q }

Note thatx -= 1; does not change the context, and thatQ[(x - 1)/x] ^ x >= 1 could implyQ. For example, ifQ does not containx, thenQ[(x - 1)/x] = Q andQ ^ x >= 1 impliesQ.

Given the discrepancy between (*) and Necessist's current behavior, one can ask: which of the two should be adjusted? Put another way, should Necessist remove a statement likex -= 1; unconditionally?

One way to look at this question is: which statements are worth removing, i.e., which statements are "interesting?" As implied above, (*) considers a statement "interesting" if its removal could affect a subsequent assertion. But there are other possible, useful definitions of an "interesting" statement. For example, one could consider strongest postconditions (mentioned above), orframeworks besides Hoare logic entirely.

To be clear, Necessist does not apply (*) formally, e.g., Necessist does not actually compute weakest preconditions. The current role of (*) is to help guide which statements Necessist should ignore, and (*) seems to do well in that role. As such, we leave resolving the aforementioned discrepancy to future work.

In addition to the below, the Anchor backend ignores:

• throw statements

• assert
• Anything beginning withassert. (e.g.,assert.equal)
• Anything beginning withconsole. (e.g.,console.log)
• expect

• toNumber
• toString

In addition to the below, the Foundry backend ignores:

• a statement immediately following a use ofvm.prank or any form ofvm.expect (e.g.,vm.expectRevert)
• anemit statement

• Anything beginning withassert (e.g.,assertEq)
• Anything beginning withvm.expect (e.g.,vm.expectCall)
• Anything beginning withconsole.log (e.g.,console.log,console.logInt)
• Anything beginning withconsole2.log (e.g.,console2.log,console2.logInt)
• vm.getLabel
• vm.label
• vm.startSnapshotGas
• vm.stopSnapshotGas

In addition to the below, the Go backend ignores:

• defer statements

• Anything beginning withassert. (e.g.,assert.Equal)
• Anything beginning withrequire. (e.g.,require.Equal)
• panic

• Close
• Error
• Errorf
• Fail
• FailNow
• Fatal
• Fatalf
• Helper
• Log
• Logf
• Parallel
• Skip
• Skipf
• SkipNow

* This list is based primarily ontesting.T's methods. However, some methods with commonplace names are omitted to avoid colliding with other types' methods.

The ignored functions and methods are the same as for Anchor above.

• assert
• assert_eq
• assert_matches
• assert_ne
• debug
• eprint
• eprintln
• error
• info
• panic
• print
• println
• trace
• unimplemented
• unreachable
• warn

• as_bytes
• as_encoded_bytes
• as_mut
• as_mut_os_str
• as_mut_os_string
• as_mut_slice
• as_mut_str
• as_os_str
• as_path
• as_ref
• as_slice
• as_str
• borrow
• borrow_mut
• clone
• cloned
• copied
• deref
• deref_mut
• expect
• expect_err
• into_boxed_bytes
• into_boxed_os_str
• into_boxed_path
• into_boxed_slice
• into_boxed_str
• into_bytes
• into_encoded_bytes
• into_os_string
• into_owned
• into_path_buf
• into_string
• into_vec
• iter
• iter_mut
• success
• to_os_string
• to_owned
• to_path_buf
• to_string
• to_vec
• unwrap
• unwrap_err

* This list is essentially the watched trait and inherent methods of Dylint'sunnecessary_conversion_for_trait lint, with the following additions:

• clone (e.g.std::clone::Clone::clone)
• cloned (e.g.std::iter::Iterator::cloned)
• copied (e.g.std::iter::Iterator::copied)
• expect (e.g.std::option::Option::expect)
• expect_err (e.g.std::result::Result::expect_err)
• into_owned (e.g.std::borrow::Cow::into_owned)
• success (e.g.assert_cmd::assert::Assert::success)
• unwrap (e.g.std::option::Option::unwrap)
• unwrap_err (e.g.std::result::Result::unwrap_err)

The ignored functions and methods are the same as for Anchor above.