Repository files navigation

bkt (pronounced "bucket") is a subprocess caching utility written in Rust,inspired bybash-cache.Wrapping expensive process invocations withbkt allows callers to reuse recentinvocations without complicating their application logic. This can be useful inshell prompts, interactive applications such asfzf, and long-runningprograms that poll other processes.

bkt is available as a standalone binary as well as aRust library. Seehttps://docs.rs/bkt/ forlibrary documentation. This README covers thebkt binary.

Runcargo install bkt to compile and installbkt locally. You will need toinstallcargoif it's not already on your system.

Pre-compiled binaries for common platforms are attached to eachrelease (starting with 0.5). Pleaseopen an issue or send a PR if you would like releases to include binaries foradditional platforms.

Package manager support is being trackedhere; volunteers are welcome.

bkt --ttl=DURATION [--stale=DURATION] [--cwd] [--env=ENV ...] [--modtime=FILE ...] [--scope=SCOPE] [--discard-failures] [--warm|--force] -- <command>...

bkt is easy to start using - simply prefix the command you intend to cachewithbkt --ttl=[some duration] --, for example:

# Execute and cache an invocation of 'date +%s.%N'$ bkt --ttl=1m -- date +%s.%N1631992417.080884000# A subsequent invocation reuses the same cached output$ bkt --ttl=1m -- date +%s.%N1631992417.080884000

Whenbkt is passed a command it hasn't seen before (or recently) it executesthe command synchronously and caches its stdout, stderr, and exit code. Callingbkt again with the same command reads the data from the cache and outputs itas if the command had been run again.

Two flags,--ttl and--stale, configure how long cached data is preserved.The TTL (Time to Live) specifies how long cached data will be used. Once theTTL expires the cached data will be discarded and the backing command re-run.A TTL can also be configured by setting aBKT_TTL environment variable.

When the data expiresbkt has to re-execute the command synchronously, whichcan introduce unexpected slowness. To avoid this, pass--stale with a shorterduration than the TTL. When the cached data is older than the stale thresholdthis causesbkt to refresh the cache in the background while still promptlyreturning the cached data.

Both flags (andBKT_TTL) accept duration strings such as10s or1hour 30min. The exact syntax is defined in thehumantimelibrary.

Some commands' behavior depends on more than just the command line arguments.It's possible to adjust howbkt caches such commands so that unrelatedinvocations are cached separately.

For example, attempting to cachepwd will not work as expected by default:

$ $ bkt --ttl=1m --pwd/tmp/foo$cd ../bar# Cached output for 'pwd' is reused even though the directory has changed$ bkt --ttl=1m --pwd/tmp/foo

To havebkt key off the current working directory in addition to the commandline arguments pass--cwd:

$ bkt --cwd --ttl=1m --pwd/tmp/foo$cd ../bar$ bkt --cwd --ttl=1m --pwd/tmp/bar

Similarly, to specify one or more environment variables as relevant for thecommand being cached use--env, such as--env=LANG. This flag can beprovided multiple times to key off additional variables. Invocations withdifferent values for any of the given variables will be cached separately.

bkt can also check the last-modified time of one or more files and includethis in the cache key using--modtime. For instance passing--modtime=/etc/passwd would cause the backing command to be re-executed anytime/etc/passwd is modified even if the TTL has not expired.

It's also possible to trigger refreshes manually using--force or--warm.The former behaves exactly as if the cached data was not found, executing theprocess and caching the result. This is useful if you know the cached datais no longer up-to-date, e.g. because something external changed.

Alternatively, it can be useful to refresh the cache asynchronously, which--warm provides. This triggers a refresh in the background but immediatelyends the current process with no output. This is useful if you expectadditional invocations in the near future and want to ensure they get a cachehit. Note that until the warming process completes concurrent calls may stillsee a cache miss and trigger their own invocation.

Cached data is persisted to disk (but seebelow), and isavailable to any process that invokesbkt. Generally this is desirable, butcertain usages may want to isolate their invocations from other potentialconcurrent calls.

To do so pass--scope=... with a sufficiently unique argument, such as a fixedlabel for the calling program, the current process ID, or a timestamp.

$ bkt --ttl=1m -- date +%s.%N1631992417.080884000# Changing the scope causes the command to be cached separately$ bkt --scope=foo --ttl=1m -- date +%s.%N1631992418.010562000

Alternatively, define aBKT_SCOPE environment variable to configure aconsistent scope across invocations. This can be useful within a script toensure all commands share a scope.

#!/bin/bash# Set a unique scope for this script invocation using the PID and current timeexport BKT_SCOPE="my_script_$$_$(date -Ins)"

By default, all invocations are cached regardless of their output or exit code.In situations where failures should not be cached pass--discard-failures toonly persist successful invocations (those that return a0 exit code).

WARNING: Passing this flag can cause the backing command to be invoked morefrequently than the--ttl would suggest, which in turn can create unexpectedload. If the backing command is failing due to an outage or bug (such as anoverloaded website) triggering additional calls can exacerbate the issue andeffectively DDoS the hampered system. It is generally safernot to set thisflag and instead make the client robust to occasional failures.

By default, cached data is stored under your system's temporary directory(typically/tmp on Linux).

You may want to use a different location for certain commands, for instance tobe able to easily delete the cached data as soon as it's no longer needed. Youcan specify a custom cache directory via the--cache-dir flag or by definingaBKT_CACHE_DIR environment variable.

Note that the choice of directory can affectbkt's performance: if the cachedirectory is on atmpfs or solid-statepartition it will be significantly faster than one using a spinning disk.

If your system's temporary directory is not a good choice for the default cachelocation (e.g. it is not atmpfs) you can specify a different location bydefining aBKT_TMPDIR environment variable (for example in your.bashrc).These two environment variables,BKT_TMPDIR andBKT_CACHE_DIR, have similareffects butBKT_TMPDIR should be used to configure the system-wide default,and--cache-dir/BKT_CACHE_DIR used to override it.

bkt periodically prunes stale data from its cache, but it also assumes theoperating system will empty its temporary storage from time to time (for/tmpthis typically happens on reboot). If you opt to use a directory that thesystem does not maintain, such as~/.cache, you may want to manually deletethe cache directory on occasion, such as when upgradingbkt.

The default cache directory is potentially world-readable. On Unix the cachedirectory is created with700 permissions, meaning only the current user canaccess it, but this is not foolproof.

You can customize the cache directory (seeabove) to a locationyou trust such as~/.cache, but note that your home directory may be slower thanthe temporary directory selected by default.

In general, if you are not the only user of your system it's wise to configureyourTMPDIR to a location only you can access. If that is not possible useBKT_TMPDIR to configure a custom temporary directory specifically forbkt.

Please share how you're usingbkt on theDiscussion Board!

bkt works well with interactive tools likefzf that execute other commands. Becausefzf executes the--preview command every time an element is selected it canbe slow and tedious to browse when the command takes a long time to run. Usingbkt allows each selection's preview to be cached. Compare:

$printf'%s\n' 1 0.2 3 0.1 5| \  fzf --preview="bash -c 'sleep {}; echo {}'"$printf'%s\n' 1 0.2 3 0.1 5| \  fzf --preview="bkt --ttl=10m --stale=10s -- bash -c 'sleep {}; echo {}'"

You'll generally want to use a long TTL and a short stale duration so thateven if you leavefzf running for a while the cache remains warm and isrefreshed in the background. You may also want to set a--scope if it'simportant to invalidate the cache on subsequent invocations.

Seethis discussion for a morecomplete example of usingbkt withfzf, including warming the commands beforethe user starts navigating the selector.

You may want to distribute shell scripts that utilizebkt without requiringevery user also install it. By falling back to a no-op shell function whenbktis not available your script can take advantage of it opportunistically withoutcomplicating your users' workflow. Of course if they choose to installbktthey'll get a faster script as a result!

# Cache commands using bkt if installedif!command -v bkt>&/dev/null;then# If bkt isn't installed skip its arguments and just execute directly.bkt() {while [["$1"== --* ]];doshift;done"$@"  }# Optionally, write a msg to stderr suggesting users install bkt.echo"Tip: install https://github.com/dimo414/bkt for faster performance">&2fi

It is sometimes helpful to cacheall invocations of a command in a shellscript or in your shell environment. You can use a decorator function patternsimilar to what bash-cache does to enable caching transparently, like so:

# This is Bash syntax, but other shells support similar syntaxexpensive_cmd() {  bkt [bkt args ...] -- expensive_cmd"$@"}

Calls toexpensive_cmd in your shell will now go throughbkt behind thescenes. This can be useful for brevity and consistency but obviously changingbehavior like this is a double-edged-sword, so use with caution. Should youneed to bypass the cache for a single invocation Bash provides thecommand builtin,socommand expensive_cmd ... will invokeexpensive_cmd directly. Othershells provide similar features.