Repository files navigation

A Go time testing library for writing deterministic unit tests

Our high level goal is to write unit tests that

1. execute quickly
2. don't flake
3. are straightforward to write and understand

For tests to execute quickly without flakes, we want to focus ondeterminism: the test should runthe same each time, and it should be easy to force the system into a known state (no races) beforeexecuting test assertions.time.Sleep,runtime.Gosched(), andpolling/Eventually are allsymptoms of an inability to do this easily.

In your application code, maintain a reference to aquartz.Clock instance to start timers andtickers, instead of the baretime standard library.

import"github.com/coder/quartz"typeComponentstruct {...// for testingclock quartz.Clock}

Whenever you would call intotime to start a timer or ticker, callComponent'sclock instead.

In production, set this clock toquartz.NewReal() to create a clock that just transparently passesthrough to the standardtime library.

In your tests, you can use a*Mock to control the tickers and timers your code under test gets.

import ("testing""github.com/coder/quartz")funcTestComponent(t*testing.T) {mClock:=quartz.NewMock(t)comp:=&Component{...clock:mClock,}}

The*Mock clock starts at Jan 1, 2024, 00:00 UTC by default, but you can set any start time you'd like prior to your test.

mClock:=quartz.NewMock(t)mClock.Set(time.Date(2021,6,18,12,0,0,0,time.UTC))// June 18, 2021 @ 12pm UTC

Once you begin setting timers or tickers, you cannot change the time backward, only advance itforward. You may continue to useSet(), but it is often easier and clearer to useAdvance().

For example, with a timer:

fired:=falsetmr:=mClock.AfterFunc(time.Second,func() {fired=true})mClock.Advance(time.Second)

When you callAdvance() it immediately moves the clock forward the given amount, and triggers anytickers or timers that are scheduled to happen at that time. Any triggered events happen on separategoroutines, sodo not immediately assert the results:

fired:=falsetmr:=mClock.AfterFunc(time.Second,func() {fired=true})mClock.Advance(time.Second)// RACE CONDITION, DO NOT DO THIS!if!fired {t.Fatal("didn't fire")}

Advance() (andSet() for that matter) return anAdvanceWaiter object you can use to wait forall triggered events to complete.

fired:=false// set a test timeout so we don't wait the default `go test` timeout for a failurectx,cancel:=context.WithTimeout(context.Background(),10*time.Second)tmr:=mClock.AfterFunc(time.Second,func() {fired=true})w:=mClock.Advance(time.Second)err:=w.Wait(ctx)iferr!=nil {t.Fatal("AfterFunc f never completed")}if!fired {t.Fatal("didn't fire")}

The construction of waiting for the triggered events and failing the test if they don't complete isvery common, so there is a shorthand:

w:=mClock.Advance(time.Second)err:=w.Wait(ctx)iferr!=nil {t.Fatal("AfterFunc f never completed")}

is equivalent to:

w:=mClock.Advance(time.Second)w.MustWait(ctx)

or even more briefly:

mClock.Advance(time.Second).MustWait(ctx)

One important restriction on advancing the clock is that you may only advance forward to the nexttimer or ticker event and no further. The following will result in a test failure:

funcTestAdvanceTooFar(t*testing.T) {ctx,cancel:=context.WithTimeout(10*time.Second)defercancel()mClock:=quartz.NewMock(t)varfiredAt time.TimemClock.AfterFunc(time.Second,func() {firedAt:=mClock.Now()})mClock.Advance(2*time.Second).MustWait(ctx)}

This is a deliberate design decision to allowAdvance() to immediately and synchronously move theclock forward (even without callingWait() on returned waiter). This helps meet Quartz's designgoals of writing deterministic and easy to understand unit tests. It also allows the clock to beadvanced, deterministicallyduring the execution of a tick or timer function, as explained in thenext sections on Traps.

Advancing multiple events can be accomplished via looping. E.g. if you have a 1-second ticker

fori:=0;i<10;i++ {mClock.Advance(time.Second).MustWait(ctx)}

will advance 10 ticks.

If you don't know or don't want to compute the time to the next event, you can useAdvanceNext().

d,w:=mClock.AdvanceNext()w.MustWait(ctx)// d contains the duration we advanced

d, ok := Peek() returns the duration until the next event, if any (ok istrue). You can usethis to advance a specific time, regardless of the tickers and timer events:

desired:=time.Minute// time to advancefordesired>0 {p,ok:=mClock.Peek()if!ok||p>desired {mClock.Advance(desired).MustWait(ctx)break}mClock.Advance(p).MustWait(ctx)desired-=p}

A trap allows you to match specific calls into the library while mocking, block their return,inspect their arguments, then release them to allow them to return. They help you writedeterministic unit tests even when the code under test executes asynchronously from the test.

You set your traps prior to executing code under test, and then wait for them to be triggered.

funcTestTrap(t*testing.T) {ctx,cancel:=context.WithTimeout(10*time.Second)defercancel()mClock:=quartz.NewMock(t)trap:=mClock.Trap().AfterFunc()defertrap.Close()// stop trapping AfterFunc callscount:=0gomClock.AfterFunc(time.Hour,func(){count++})call:=trap.MustWait(ctx)call.MustRelease(ctx)ifcall.Duration!=time.Hour {t.Fatal("wrong duration")}// Now that the async call to AfterFunc has occurred, we can advance the clock to trigger itmClock.Advance(call.Duration).MustWait(ctx)ifcount!=1 {t.Fatal("wrong count")}}

In this test, the trap serves 2 purposes. Firstly, it allows us to capture and assert the durationpassed to theAfterFunc call. Secondly, it prevents a race between setting the timer and advancingit. Since these things happen on different goroutines, ifAdvance() completes beforeAfterFunc() is called, then the timer never pops in this test.

Any untrapped calls immediately complete using the current time, and callingClose() on a trapcauses the mock clock to stop trapping those calls.

You may alsoAdvance() the clock between trapping a call and releasing it. The call uses thecurrent (mocked) time at the moment it is released.

funcTestTrap2(t*testing.T) {ctx,cancel:=context.WithTimeout(10*time.Second)defercancel()mClock:=quartz.NewMock(t)trap:=mClock.Trap().Now()defertrap.Close()// stop trapping AfterFunc callsvarlogs []stringdone:=make(chanstruct{})gofunc(clk quartz.Clock){deferclose(done)start:=clk.Now()phase1()p1end:=clk.Now()logs=append(fmt.Sprintf("Phase 1 took %s",p1end.Sub(start).String()))phase2()p2end:=clk.Now()logs=append(fmt.Sprintf("Phase 2 took %s",p2end.Sub(p1end).String()))}(mClock)// starttrap.MustWait(ctx).MustRelease(ctx)// phase 1call:=trap.MustWait(ctx)mClock.Advance(3*time.Second).MustWait(ctx)call.MustRelease(ctx)// phase 2call=trap.MustWait(ctx)mClock.Advance(5*time.Second).MustWait(ctx)call.MustRelease(ctx)<-done// Now logs contains []string{"Phase 1 took 3s", "Phase 2 took 5s"}}

When multiple goroutines in the code under test call into the Clock, you can usetags todistinguish them in your traps.

trap:=mClock.Trap.Now("foo")// traps any calls that contain "foo"defertrap.Close()foo:=make(chan time.Time)gofunc(){foo<-mClock.Now("foo","bar")}()baz:=make(chan time.Time)gofunc(){baz<-mClock.Now("baz")}()call:=trap.MustWait(ctx)mClock.Advance(time.Second).MustWait(ctx)call.MustRelease(ctx)// call.Tags contains []string{"foo", "bar"}gotFoo:=<-foo// 1s after startgotBaz:=<-baz// ?? never trapped, so races with Advance()

Tags appear as an optional suffix on allClock methods (type...string) and are ignored entirelyby the real clock. They also appear on all methods on returned timers and tickers.

We use the Option pattern to inject the mock clock for testing, keeping the call signature inproduction clean. The option pattern is compatible with other optional fields as well.

typeOptionfunc(*Thing)// WithTestClock is used in tests to inject a mock ClockfuncWithTestClock(clk quartz.Clock)Option {returnfunc(t*Thing) {t.clock=clk}}funcNewThing(<requiredargs>,opts...Option)*Thing {t:=&Thing{...clock:quartz.NewReal()}for_,o:=rangeopts {o(t)}returnt}

In tests, this becomes

funcTestThing(t*testing.T) {mClock:=quartz.NewMock(t)thing:=NewThing(<requiredargs>,WithTestClock(mClock))...}

Tag yourClock method calls as:

func (c*Component)Method() {now:=c.clock.Now("Component","Method")}

or

func (c*Component)Method() {start:=c.clock.Now("Component","Method","start")...end:=c.clock.Now("Component","Method","end")}

This makes it much less likely that code changes that introduce new components or methods will spoilexisting unit tests.

Writing good unit tests for components and functions that use thetime package is difficult, eventhough several open source libraries exist. In building Quartz, we took some inspiration from

• github.com/benbjohnson/clock
• Tailscale'ststest.Clock
• github.com/aspenmesh/tock

Quartz shares the high level design of aClock interface that closely resembles the functions inthetime standard library, and a "real" clock passes thru to the standard library in production,while a mock clock gives precise control in testing.

As mentioned in our introduction, our high level goal is to write unit tests that

1. execute quickly
2. don't flake
3. are straightforward to write and understand

For several reasons, this is a tall order when it comes to code that depends on time, and we foundthe existing libraries insufficient for our goals.

The following example comes from the README from benbjohnson/clock:

mock:=clock.NewMock()count:=0// Kick off a timer to increment every 1 mock second.gofunc() {ticker:=mock.Ticker(1*time.Second)for {<-ticker.Ccount++}}()runtime.Gosched()// Move the clock forward 10 seconds.mock.Add(10*time.Second)// This prints 10.fmt.Println(count)

The first race condition is fairly obvious: moving the clock forward 10 seconds may generate 10ticks on theticker.C channel, but there is no guarantee thatcount++ executes beforefmt.Println(count).

The second race condition is more subtle, butruntime.Gosched() is the tell. Since the tickeris started on a separate goroutine, there is no guarantee thatmock.Ticker() executes beforemock.Add().runtime.Gosched() is an attempt to get this to happen, but it makes no hardpromises. On a busy system, especially when running tests in parallel, this can flake, advance thetime 10 seconds first, then start the ticker and never generate a tick.

Let's talk about how Quartz tackles these problems.

In our experience, an extremely common use case is creating a ticker then doing a 2-armselectwith ticks in one and context expiring in another, i.e.

t:=time.NewTicker(duration)for {select {case<-ctx.Done():returnctx.Err()case<-t.C:err:=do()iferr!=nil {returnerr}}}

In Quartz, we refactor this to be more compact and testing friendly:

t:=clock.TickerFunc(ctx,duration,do)returnt.Wait()

This affords the mockClock the ability to explicitly know when processing of a tick is finishedbecause it's wrapped in the function passed toTickerFunc (do() in this example).

In Quartz, when you advance the clock, you are returned an object you canWait() on to ensure allticks and timers triggered are finished. This solves the first race condition in the example.

(As an aside, we still support a traditional standard library-styleTicker. You may find it usefulif you want to keep your code as close as possible to the standard library, or if you need to usethe channel in a largerselect block. In that case, you'll have to find some other mechanism tosync tick processing to your test code.)

To prevent race conditions related to the starting of the ticker, Quartz allows you to set "traps"for calls that access the clock.

funcTestTicker(t*testing.T) {mClock:=quartz.NewMock(t)trap:=mClock.Trap().TickerFunc()defertrap.Close()// stop trapping at endgorunMyTicker(mClock)// async calls TickerFunc()call:=trap.MustWait(context.Background())// waits for a call and blocks its returncall.MustRelease(ctx)// allow the TickerFunc() call to return// optionally check the duration using call.Duration// Move the clock forward 1 tickmClock.Advance(time.Second).MustWait(context.Background())// assert results of the tick}

Trapping and then releasing the call toTickerFunc() ensures the ticker is started at adeterministic time, so our calls toAdvance() will have a predictable effect.

Take a look atTestExampleTickerFunc inexample_test.go for a complete worked example.

Another difficult issue to handle when unit testing is when some code under test makes multiplecalls that depend on the time, and you want to simulate some time passing between them.

A very basic example is measuring how long something took:

varmeasurement time.Durationgofunc(clock quartz.Clock) {start:=clock.Now()doSomething()measurement=clock.Since(start)}(mClock)// how to get measurement to be, say, 5 seconds?

The two calls into the clock happen asynchronously, so we need to be able to advance the clock afterthe first call toNow() but before the call toSince(). Doing this with the libraries wementioned above means that you have to be able to mock out or otherwise block the completion ofdoSomething().

But, with the trap functionality we mentioned in the previous section, you can deterministicallycontrol the time each call sees.

trap:=mClock.Trap().Since()varmeasurement time.Durationgofunc(clock quartz.Clock) {start:=clock.Now()doSomething()measurement=clock.Since(start)}(mClock)c:=trap.MustWait(ctx)mClock.Advance(5*time.Second)c.MustRelease(ctx)

We wait until we trap theclock.Since() call, which implies thatclock.Now() has completed, thenadvance the mock clock 5 seconds. Finally, we release theclock.Since() call. Any changes to theclock that happenbefore we release the call will be included in the time used for theclock.Since() call.

As a more involved example, consider an inactivity timeout: we want something to happen if there isno activity recorded for some period, say 10 minutes in the following example:

typeInactivityTimerstruct {mu sync.Mutexactivity time.Timeclock quartz.Clock}func (i*InactivityTimer)Start() {i.mu.Lock()deferi.mu.Unlock()next:=i.clock.Until(i.activity.Add(10*time.Minute))t:=i.clock.AfterFunc(next,func() {i.mu.Lock()deferi.mu.Unlock()next:=i.clock.Until(i.activity.Add(10*time.Minute))ifnext==0 {i.timeoutLocked()return}t.Reset(next)})}

The actual contents oftimeoutLocked() doesn't matter for this example, and assume there are otherfunctions that record the latestactivity.

We found that some time testing libraries hold a lock on the mock clock while calling the functionpassed toAfterFunc, resulting in a deadlock if you made clock calls from within.

Others allow this sort of thing, but don't have the flexibility to test edge cases. There is asubtle bug in ourStart() function. The timer may pop a little late, and/or some measurable realtime may elapse beforeUntil() gets called inside theAfterFunc. If there hasn't been activity,next might be negative.

To test this in Quartz, we'll use a trap. We only want to trap the innerUntil() call, not theinitial one, so to make testing easier we can "tag" the call we want. Like this:

func (i*InactivityTimer)Start() {i.mu.Lock()deferi.mu.Unlock()next:=i.clock.Until(i.activity.Add(10*time.Minute))t:=i.clock.AfterFunc(next,func() {i.mu.Lock()deferi.mu.Unlock()next:=i.clock.Until(i.activity.Add(10*time.Minute),"inner")ifnext==0 {i.timeoutLocked()return}t.Reset(next)})}

All QuartzClock functions, and functions on returned timers and tickers support zero or morestring tags that allow traps to match on them.

funcTestInactivityTimer_Late(t*testing.T) {// set a timeout on the test itself, so that if Wait functions get blocked, we don't have to// wait for the default test timeout of 10 minutes.ctx,cancel:=context.WithTimeout(10*time.Second)defercancel()mClock:=quartz.NewMock(t)trap:=mClock.Trap.Until("inner")defertrap.Close()it:=&InactivityTimer{activity:mClock.Now(),clock:mClock,}it.Start()// Trigger the AfterFuncw:=mClock.Advance(10*time.Minute)c:=trap.MustWait(ctx)// Advance the clock a few ms to simulate a busy systemmClock.Advance(3*time.Millisecond)c.MustRelease(ctx)// Until() returnsw.MustWait(ctx)// Wait for the AfterFunc to wrap up// Assert that the timeoutLocked() function was called}

This test case will fail with our bugged implementation, since the triggered AfterFunc won't calltimeoutLocked() and instead will reset the timer with a negative number. The fix is easy, usenext <= 0 as the comparison.