Repository files navigation

These small utilities allow creating very lightweight job queuesystems which require no setup, maintenance, supervision, or anylong-running processes.

nq should run on any POSIX.1-2008 compliant system which alsoprovides a working flock(2). Tested on Linux 2.6.37, Linux 4.1,OpenBSD 5.7, FreeBSD 10.1, NetBSD 7.0.2, Mac OS X 10.3 andSmartOS joyent_20160304T005100Z.

The intended purpose is ad-hoc queuing of command lines (e.g., forbuilding several targets of a Makefile, downloading multiple files oneat a time, running benchmarks in several configurations, or simply asa glorifiednohup). But as any good Unix tool, it can be abused forwhatever you like.

Job order is enforced by a timestampnq gets immediately whenstarted. Synchronization happens on file-system level. Timerresolution is milliseconds. No sub-second file system time stamps arerequired. Polling is not used. Exclusive execution is maintainedstrictly.

Enforcing job order works like this:

• every job has a flock(2)ed output file, ala,TIMESTAMP.PID
• every job starts only after all earlier flock(2)ed files are unlocked
• Why flock(2)? Because it locks the file handle, which is sharedacross exec(2) with the child process (the actual job), and it willunlock when the file is closed (usually when the job terminates).

You enqueue (get it?) new jobs usingnq CMDLINE.... The job ID isoutput (unless suppressed using-q) andnq detaches immediately,running the job in the background. STDOUT and STDERR are redirectedinto the log file.

nq tries hard (but does not guarantee) to ensure the log file of thecurrently running job has+x bit set. Thus you can usels -F to geta quick overview of the state of your queue.

The "file extension" of the log file is actually the PID, so you cankill jobs easily. Before the job is started, it is the PID ofnq,so you can cancel a queued job by killing it as well.

Due to the initialexec line in the log files, you can resubmit ajob by executing it as a shell command file (i.e. runningsh $jobid).

You can wait for jobs to finish usingnq -w, possibly listing jobIDs you want to wait for; the default is all of them. Likewise, youcan test if there are jobs which need to be waited upon using-t.

By default, job IDs are per-directory, but you can set$NQDIR to putthem elsewhere. Creatingnq wrappers setting$NQDIR to providedifferent queues for different purposes is encouraged.

All these operations take worst-case quadratic time in the amount oflock files produced, so you should clean them regularly.

Build targetsclean,depends,all, without occupying the terminal:

% nq make clean% nq make depends% nq make all% nqtail... look at output, can interrupt with C-c any timewithout stopping the build ...

Simple download queue, accessible from multiple terminals:

% mkdir -p /tmp/downloads% alias qget='NQDIR=/tmp/downloads nq wget'% alias qwait='NQDIR=/tmp/downloads nqtail -q'window1% qget http://mymirror/big1.isowindow2% qget http://mymirror/big2.isowindow3% qget http://mymirror/big3.iso% qwait... wait for all downloads to finish ...

Asnohup replacement (The benchmark will run in background, every rungets a different output file, and the command line you ran is logged,too!):

% ssh remoteremote% nq ./run-benchmark,14f6f3034f8.17035remote% ^D% ssh remoteremote% nqtail... see output, nqtail exits when job finished ...

nq will only work correctly when:

• $NQDIR (respectively.) is writable.
• flock(2) works in$NQDIR (respectively.).
• gettimeofday behaves monotonic (usingCLOCK_MONOTONIC wouldcreate confusing file names). Else job order can be confused andmultiple tasks can run at once due to race conditions.
• No other programs put files matching,* into$NQDIR (respectively.).

Two helper programs are provided:

nqtail outputs the log of the currently running jobs, exitingwhen the jobs are done. If no job is running, the output of the lastjob is shown.nqtail -a shows the output of all jobs,nqtail -qonly shows one line per job.nqtail usesinotify on Linux andfalls back to polling for size change else. (nqtail.sh is a similartool, not quite as robust, implemented as shell-script callingtail.)

nqterm wrapsnq and displays thenqtail output in a newtmux or screen window.

(A pure shell implementation ofnq is provided asnq.sh. It needsflock from util-linux, and only has a timer resolution of 1s.Lock files fromnq andnq.sh should not be mixed.)

Usemake all to build,make install to install relative toPREFIX(/usr/local by default). TheDESTDIR convention is respected.You can also just copy the binaries into yourPATH.

You can usemake check to run a simple test suite, if you havePerl'sprove installed.

• at runs jobs at a given time.batch runs jobs "when system load levels permit".nq andtask-spoolerrun jobs in sequence with no regard to the system's load average.

• at andbatch have 52 built-in queues: a-z and A-Z.Any directory can be a queue fornq.task-spooler can have different queues for different terminals.

• You can follow the output of annq queue tail-style withnqtail.

• The syntax is different:at andbatch take whole scripts fromthe standard input or a file;nq takes a single command as itscommand line arguments.

• nq doesn't rely on a daemon, and uses a directory to manage the queue.task-spooler automatically launches a daemon to manage a queue.

• task-spooler can set a maximum number of simultaneous jobs.

nq is in the public domain.

To the extent possible under law,Leah Neukirchenleah@vuxu.orghas waived all copyright and related orneighboring rights to this work.

http://creativecommons.org/publicdomain/zero/1.0/