Repository files navigation

How can you make shell scripts portableand run faster? That are the questionsthese test cases aim to answer.

Table of Contents

• 1.0 SHELL SCRIPT PERFORMANCE AND PORTABILITY
  • 1.1 THE PROJECT STRUCTURE
  • 1.2 THE PROJECT DETAILS
• 3.0 ABOUT PERFORMANCE
  • 3.1 GENERAL PERFORMANCE ADVICE
  • 3.2 SHELLS AND PERFORMANCE.
  • 3.3 MAJOR PERFORMANCE GAINS
  • 3.4 MODERATE PERFORMANCE GAINS
  • 3.5 MINOR PERFORMANCE GAINS
  • 3.6 NO PERFORMANCE GAINS
• 4.0 PORTABILITY
  • 4.1 LEGACY SHELL SCRIPTING
  • 4.2 REQUIREMENTS AND SHELL SCRIPTS
  • 4.3 WRITING POSIX COMPLIANT SHELL SCRIPS
  • 4.4 SHEBANG LINE IN SCRIPTS
    • 4.4.1 About Bash and Shebang
    • 4.4.2 About Python and Shebang
  • 4.5 PORTABILITY OF UTILITIES
    • 4.5.1 Case Study: sed
    • 4.5.2 Case Study: awk
  • 4.6 MISCELLANEUS NOTES
• 5.0 RANDOM NOTES
• 6.0 FURTHER READING
  • 6.1 BASH PROGRAMMING
  • 6.2 PORTABILITY AND UTILITIES
  • 6.3 STANDARDS
  • 6.4 MISCELLANEOUS LINKS
• COPYRIGHT
• LICENSE

The tests reflect results under Linuxusing GNU utilities. The focus is on thefeatures found inBashandPOSIX.1-2024compliantsh shells. The term compliantis used here as "most POSIX compliant",as there is no, and has never been,shell that is fully POSIX compliant.POSIX is useful if you are looking formore portable scripts. See also POSIX inWikipedia.

Please note thatsh here refers tomodern, best-of-effort POSIX-compatible,minimal shells likedashandposh.See sectionPORTABILITY, SHELLS AND POSIX.

In Linux like systems, for all roundedshell scripting, Bash is the sensiblechoice for data manipulation in memorywitharrays,associative arrays, and strings with anextended set of parameter expansions,regular expressions, including extractingregex matches and utilizing functions.

In other operating systems, for exampleBSD, the obvious choice for shellscripting would be fastKsh(ksh93,mksh,etc.).

Shell scripting is about combiningredirections, pipes, calling externalutilities, and gluing them all together.Shell scripts are also quite portable bydefault, requiring no additionalinstallation.PerlorPythonexcel in their respective fields, wherethe requirements differ from those of theshell.

Certain features in Bash are slow, butknowing the cold spots and usingalternatives helps. On the other hand,small POSIXsh, for exampledash,scrips are much faster at callingexternal processes and functions. Moreabout this in sectionSHELLS AND PERFORMANCE.

The results presented in this READMEprovide only some highlighs from the testcases listed in RESULTS. Consider the rawtimeresults only as guidance, as they reflectonly the system used at the time oftesting. Instead, compare the relativeorder in which each test case producedthe fastest results.

• RESULTS
• RESULTS-BRIEF
• RESULTS-PORTABILITY
• The test cases and code inbin/
• USAGE
• CONTRIBUTING

    bin/            The tests    doc/            Results by "make doc"    COPYING         License (GNU GPL)    INSTALL         Install instructions    USAGE.md        How to run the tests    CONTRIBUTING.md Writing test cases

• Homepage:https://github.com/jaalto/project--shell-script-performance-and-portability

• To report bugs:see homepage.

• Source repository:see homepage.

• Depends:Bash, GNU coreutils, file/usr/share/dict/words(Debian package: wamerican).

• Optional depends:GNU make.For some tests: GNU parallel.

Regardless of the shell you use forscripting(sh,ksh,bash),consider these factors.

• If you run scripts on many small files,set up a RAM disk and copy the files toit. This can lead to massive speedgains. In Linux, seetmpfs,which allows you to set a size limit,unlike the memory-hoggingramfs,which can fill all available memory andpotentially halt your server.

• If you know the files beforehand,preload them into memory. This can alsolead to massive speed gains.In Linux, seevmtouch.

• If you have tasks that can be run concurrently,usePerlbasedGNU parallelfor massive gains in performance.See also how to usesemaphores(tutorial)to wait for all concurrent tasks tofinish before continuing with the restof the tasks in the pipeline. In somecases, even parallelizing work with GNUxargs --max-procs=0can help.

• Use GNU utilities. According tobenchmarks, likeStackOverflow,the GNUgrep is considerably fasterand more optimized than the operatingsystem's default. For shells, the GNUutilities consist mainly ofcoreutils,grep andawk.If needed, arrangePATH to preferGNU utilities (for example, onmacOS).

• Minimize extra processes as much aspossible. In most cases, a singleawkcan handle all ofsed,cut,grepetc.chains. Theawk binary program isveryfast and more efficient thanPerlorPythonscripts where startup time and highermemory consumption is a factor.Note: If you need to process largefiles, use a lot of regularexpressions, manipulate or work ondata extensively, there is probablynothing that can replace the speed ofPerlunless you go even lower-levellanguages likeC. But then again,we assume that you know how to chooseyour tools in those cases.

    cmd| awk'{...}'# ... could probably# replace all of these    cmd| head ...| cut ...    cmd| grep ...| sed ...    cmd| grep ...| grep -v ...| cut ...

• Note: if you have hordes of RAM,no shortage of cores, and largefiles, then utilize pipelines<cmd> | ... as much as possible becausethe Linux Kernel will optimize thingsin memory better. In more powerfulsystems, many latency and performanceissues are not as relevant.

• Use Shell built-ins(seeBash)and not binaries:

echo# not /usr/bin/echoprintf# not /usr/bin/printf    [ ... ]# not /usr/bin/test

TODO

• In Bash, It is at least 60 times fasterto perform regular expression stringmatching using the binary operator=~rather than to calling externalPOSIX utilitiesexprorgrep.

  NOTE: In POSIXsh, likedash, calling utilities isextremely fast. Compared to Bash's[[]],theexpr indash is only 5xslower, which is negligible becausethe time differences are measured inmere few milliseconds.Seecode

    str="abcdef"    re="b.*e"# Bash, Ksh    [[$str=~$re ]]# In Bash, at least 60x slower    expr match"$str"".*$re"# In Bash, at least 100x slowerecho"$str"| grep -E"$re"# --------------------------------# Different shells compared.# --------------------------------    ./run.sh --shell dash,ksh93,bash t-string-match-regexp.sh    Run shell: dash# t1     <skip># t2     real 0.010s  expr# t3     real 0.010s  grep    Run shell: ksh93# t1     real 0.001s [[ =~ ]]# t2     real 0.139s expr# t3     real 0.262s grep    Run shell: bash# t1     real 0.003s [[ =~ ]]# t2     real 0.200s expr# t3     real 0.348s grep

• In Bash, it is about 50 timesfaster to do string manipulation inmemory, than calling externalutilities. Seeing the measurementsjust how expensive it is, reminds usto utilize the possibilities of POSIX#,##,% and%%parameter expansions.See more inBash.Seecode.

    str="/tmp/filename.txt.gz"# (1) Almost instantaneous# Delete up till first "."    ext=${str#*.}# (2) In Bash, over 50x slower## NOTE: identical in speed# and execution to:# cut -d "." -f 2,3 <<< "$str"    ext=$(echo"$str"| cut -d"." -f 2,3)# (3) In Bash, over 70x slower    ext=$(echo"$str"| sed's/^[^.]\+//')# --------------------------------# Different shells compared.# --------------------------------    ./run.sh --shell dash,ksh93,bash t-string-file-path-components.sh    Run shell: dash# t3aExt real 0.009s (1)# t3cExt real 0.008s (2)# t3eExt real 0.009s (3)    Run shell: ksh93# t3aExt real 0.001s# t3cExt real 0.193s# t3eExt real 0.288s    Run shell: bash# t3aExt real 0.004s# t3cExt real 0.358s# t3eExt real 0.431s

• In Bash, it is about 10 times fasterto read a file into memory as astring and usepattern matchingor regular expressions binaryoperator=~on string. In-memory handling is muchmore efficient than calling thegrep command in Bash on a file,especially if multiple matches areneeded.Seecode.

# Bash, Ksh    str=$(< file)if [[$str=~$regexp1 ]];then        ...elif [[$str=~$regexp2 ]];then        ...fi# --------------------------------# Different shells compared.# --------------------------------    (1)read once + case..end    (2) loop do.. grep file ..done    (3) loop do.. case..end ..done    ./run.sh --shell dash,ksh93,bash t-file-grep-vs-match-in-memory.sh    Run shell: dash# t1b    real 0.023s (1) once# t2     real 0.018s (2) grep# t3     real 0.021s (3) case    Run shell: ksh93# t1b    real 0.333s (1) once# t2     real 0.208s (2) grep# t3     real 0.453s (3) case    Run shell: bash# t1b    real 0.048s (1) once# t2     real 0.277s (2) grep# t3     real 0.415s (3) case

• In Bash, it is about 8 times faster,to usenamerefto return a value. In Bash, theret=$(fn) is inefficient to callfunctions. On the other hand, inPOSIXsh shells, likedash,thereis practically no overhead in using$(fn).Seecode.

# An exmaple only. Not needed in# POSIX sh shells as ret=$(fn)# is already fast.fnNamerefPosix()    {# NOTE: uses non-POSIX# 'local' but it is widely# supported in POSIX-compliant# shells: dash, posh, mksh,# ksh93 etc.local retref=$1shiftlocal arg=$1eval"$retref=\$arg"    }fnNamerefBash()    {local -n retref=$1shiftlocal arg=$1        retref=$arg    }# Return value returned to# variable 'ret'    fnNamerefPosix ret"arg"    fnNamerefBash ret"arg"# --------------------------------# Different shells compared.# --------------------------------    ./run.sh --shell dash,ksh93,bash t-function-return-value-nameref.sh    Run shell: dash# t1     <skip># t2     real 0.006s fnNamerefPosix# t3     real 0.005s ret=$(fn)    Run shell: ksh93# t1     <skip># t2     real 0.004s fnNamerefPosix# t3     real 0.005s ret=$(fn)    Run shell: bash# t1     real 0.006s fnNamerefBash# t2     real 0.006s fnNamerefPosix# t3     real 0.094s ret=$(fn)

• In Bash, it is about 2 times fasterfor line-by-line handling to readthe file into an array and then loopthrough the array. The built-inreadarrayis synonym formapfile,Seecode.

# Bash    readarray< fileforlinein"${MAPFILE[@]}"do        ...done# POSIX. In bash, slowerwhileread -r linedo        ...done< file# --------------------------------# Different shells compared.# --------------------------------    ./run.sh --shell dash,ksh93,bash t-file-read-content-loop.sh    Run shell: dash# t1     <skip># t2     real 0.085  POSIX    Run shell: ksh93# t1     <skip># t2     real 0.021  POSIX    Run shell: bash# t1     real 0.045  readarray# t2     real 0.108  POSIX

• In Bash, it is about 2 times faster toprefilter withgrepto process onlycertain lines instead of reading thewhole file into a loop and thenselecting lines. Theprocess substitutionis more general because variablespersist after the loop. Thedashis very fast compared to Bash.Seecode.

# Bashwhileread -r ...do        ...done<<(grep"$re" file)# POSIX# Problem: while runs in# a separate environment    grep"$re" file)|whileread -r ...do        ...done# POSIX# NOTE: extra calls# required for tmpfile    grep"$re" file)> tmpfilewhileread -r ...do        ...done< tmpfile    rm tmpfile# Bash, Slowest,# in-loop prefilterwhileread -r linedo       [[$line=~$re ]]||continue       ...done< file# --------------------------------# Different shells compared.# --------------------------------    ./run.sh --shell dash,ksh93,bash t-file-read-match-lines-loop-vs-grep.sh    Run shell: dash# t1a    real 0.015s grep prefilter# t2a    real 0.012s loop: case...esac    Run shell: ksh93# t1a    real 2.940s# t2a    real 1.504s    Run shell: bash# t1a    real 4.567s# t2a    real 10.88s

• It is about 10 times faster to splita string into an array using listrather than using Bash here-string.This is becauseHERE STRING<<< uses a pipe or temporary file,whereas Bash list operates entirelyin memory. The pipe buffer behavorwas introduced inBash 5.1 section c.Warning: Please note that using the(list) statement will undergopathname expansion so globbingcharacters like*,?, etc. instring would be a problem. Thepathname expansion can be disabled.Seecode.

    str="1:2:3"# Bash, Ksh. Fastest.    IFS=":"eval'array=($str)'fn()# Bash    {local str=$1# Make 'set' locallocal -# Disable pathname# expansionset -o nogloblocal -a array        IFS=":"eval'array=($str)'        ...    }# Bash. Slower than 'eval'.    IFS=":"read -ra array<<<"$str"# In Linux, see what Bash uses# for HERE STRING: pipe or# temporary file    bash -c'ls -l --dereference /proc/self/fd/0 <<< hello'

• It is about 2 times faster to readfile into a string using Bash commandsubstitution$(< file).NOTE: In POSIXsh, likedash, the$(cat file) isextremely fast.Seecode.

# Bash    str=$(< file)# In Bash: 1.8x slower# Read max 100 KiBread -r -N$((100*1024)) str< file# In Bash: POSIX, 2.3x slower    str=$(cat file)# --------------------------------# Different shells compared.# --------------------------------    ./run.sh --shell dash,ksh93,bash t-file-read-into-string.sh    Run shell: dash# t1     <skip># t2     <skip># t3     real 0.013s $(cat ...)    Run shell: ksh93# t1     real 0.088s $(< ...)# t2     real 0.095s read -N# t3     real 0.267s $(cat ...)    Run shell: bash# t1     real 0.139s $(< ...)# t2     real 0.254s read -N# t3     real 0.312s $(cat ...)

According to the results, none ofthese offer practical benefits.

• The Bashbrace expansion{N..M} might offer aneglible advantage. However it may beimpractical becauseN..M cannot beparameterized. Surprisingly, thesimple and elegant$(seq N M) isfast, even thoughcommand substitutionuses a subshell. The last POSIXwhile loop example was slightlyslower in all subsequent tests.Seecode.

    N=1    M=100# Bashforiin {1..100}do        ...done# POSIX, fastforiin$(seq$N$M)do        ...done# Bash, slowfor((i=$N; i<=$M; i++))do        ...done# POSIX, slowest    i=$Nwhile ["$i"-le"$M" ]do        i=$((i+1))done

• One might think that choosingoptimizedgrep options would make adifference. In practice, for typicalfile sizes (below few Megabytes),performance is nearly identical evenwith the ignore case option included.Nonetheless, there may be cases whereselectingLANG=C, using--fixed-strings, and avoiding--ignore-case might improveperformance, at least according toStackOverflow discussions with largefiles.Seecode.

# The same performance. Regexp# engine does not seem to be# the bottleneck    LANG=C grep --fixed-strings ...    LANG=C grep --extended-regexp ...    LANG=C grep --perl-regexp ...    LANG=C grep --ignore-case ...

None of these offer any advantages to speed up shell scripts.

• The Bash-specific expression[[]]might offer a minuscule advantage butonly in loops of 10,000 iterations.Unless the safeguards provided byBash[[ ]] are important, the POSIXtests will do fine.Seecode.

    ["$var"="1" ]# POSIX    [[$var= 1 ]]# Bash    [!"$var" ]# POSIX    [[!$var ]]# Bash    [-z"$var" ]# archaic

• There are no practical differencesbetween these. The POSIXarithmetic expansion$(())compound command will do fine. Notethat the null command:utilizes the command's side effect to"do nothing, but evaluate elements"and therefore may not be the mostreadable option.Seecode.

    i=$((i+1))# POSIX, preferred:$((i++))# POSIX, Uhm:$((i= i+1))# POSIX, Uhm((i++))# Bash, Kshlet i++# Bash, Ksh

• There is no performancedifference between aBash-specific expression[[]]for pattern matching compared toPOSIXcase..esac. Interestinglypattern matching is 4x slower underdashcompared to Bash. However,that means nothing because the timedifferences are measured in minusculemilliseconds (0.002s).Seecode.

    string="abcdef"    pattern="*cd*"# Bashif [[$string==$pattern ]];then    ...fi# POSIXcase$stringin$pattern)return 0# in function, true            ;;*)return 1# in function, false            ;;esac# --------------------------------# Different shells compared.# --------------------------------    ./run.sh --shell dash,ksh93,bash t-string-match-regexp.shRun shell: dash# t1     <skip># t2      real 0.011 POSIXRun shell: ksh93# t1     real 0.004  [[ == ]]# t2     real 0.002  POSIXRun shell: bash# t1     real 0.003  [[ == ]]# t2     real 0.002  POSIX

• There is no performance differencebetween a regular while loop and aprocess substitutionloop. However, the latter is moregeneral, as any variable set duringthe loop will persist afterandthere is no need to clean uptemporary files like in POSIX (1)solution. The POSIX (1) loop ismarginally faster but the speed gainis lost by the extrarm command.Seecode.

# Bash, Kshwhileread -r ...do        ...done<<(command)# POSIX (1)# Same, but with# temporary filecommand> filewhileread -r ...do        ...done< file    rm file# POSIX (2)# while is being run in# separate environment# due to pipe(|)command|whileread -r ...do        ...done

• Withgrep, the use ofGNU parallel,aperl program, makes thingsnotably slower for typical filesizes. The idea of splitting a fileinto chunks of lines and running thesearch in parallel is intriguing, butthe overhead of starting Perlinterpreter withparallel is orders of magnitudemore expensive compared to runningalready optimizedgrep only once.Usually the limiting factor whengrepping a file is the disk's I/Ospeed. Otherwise, GNUparallel isexcellent for making full use ofmultiple cores. Based onStackOverflow discussions, if filesizes are in the several hundreds ofmegabytes or larger, GNUparallelcan help speed things up.Seecode.

# Possibly add: --block -1    parallel --pipepart --arg-file"$largefile" grep"$re"

In typical cases, the legacysh(Bourne Shell)is not a relevant target for shellscripting. The Linux and and modernUNIX operating systems have longprovided ansh that isPOSIX-compliant enough. Nowadaysshis usually a symbolic link todash(on Linux since 2006),ksh(on some BSDs), or it may point toBash(on macOS).

Examples of pre-2000 shell scriptingpractises:

if [ x$a= y ] ...# test if variable's lenght is non-zeroif [-n"$a" ] ...# Vtest of variable's lenght is zeroif [-z"$a" ] ...# Deprecated in next POSIX# version. Operands are# not portable.# -o (OR)# -a (AND)if ["$a"="y"-o"$b"="y" ] ...# POSIX allows leading# opening "(" parencase$varin        (a*)true             ;;        (*)false             ;;esac

Modern equivalents:

# Equalityif ["$a"="y" ] ..# Variable has somethingif ["$a" ] ...# Variable does not have something,# that is: variable is emptyif [!"$a" ] ...# Logical OR between statementsif ["$a"="y" ]|| ["$b"="y" ] ...# Without leading "(" parencase$varin         a*):# The ":" is same as built-in or external command "true"             ;;*)false             ;;esac

Writing shell scripts inherentlyinvolves considering several factors.

• Personal scripts. When writingscripts for personal oradministrative tasks, the choice ofshell is unimportant. On Linux, theobvious choice is Bash. On BSDsystems, it would be Ksh. On macOS,Zsh might be handy.

• Portable scripts. If you intend touse the scripts across some operatingsystems — from Linux to Windows(Git Bash,Cygwin,MSYS2 [*][**]) —the obvious choice would be Bash.Between macOS and Linux, writingscripts in Bash is generally moreportable than writing them in Zshbecause Linux doesn't have Zshinstalled by default. With macOShowever, the choice of Bash is a bitmore involved (see next).

• POSIX-compliant scripts. If youintend to use the scripts across avariety of operating systems — fromLinux, BSD, and macOS to variousWindows Linux-like environments — theissues become quite complex. You areprobably better off writingshPOSIX-compliant scripts and testingthem withdash,since relying on Bash can lead tounexpected issues — different systemshave different Bash versions, andthere’s no guarantee that a scriptwritten on Linux will run withoutproblems on older Bash versions, suchas the outdated 3.2 version in/bin/bash on macOS. Requiring usersto install a newer version on macOSis not trivial because/bin/bash isnot replaceable.

[*] "Git Bash" is available with thepopular native Windows installation ofGit for Windows.Under the hood, the installation is based onMSYS2, which in turn is based onCygwin. The common denominator ofall native Windows Linux-likeenvironments is theCygwinbase which, in allpractical terms, provides the usualcommand-line utilities,including Bash. For curious readers,Windows softwareMobaXterm,offers X server, terminalsand other connectivity features, but alsosomes with Cygwin-basedBash shell with its ownDebian-styleapt package manager which allows installingadditional Linux utilities.

[**] In Windows, there is alsothe Windows Subsystem for Linux(WSL),where you can install (Seewsl --list --onlline)Linux distributions likeDebian,Ubuntu,OpenSUSE andOracle Linux.Bash is the obvious choice for shellscripts in this environment.

As this document is more focused onLinux, macOS, and BSD compatibility,and less on legacy UNIX operatingsystems, for all practical purposes,there is no need to attempt to writepure POSIX shell scripts. Strictermeasures are required only if youtarget legacy UNIX operating systemswhosesh may not have changed in 30years. your best guide probably isthe wealth of knowledge collected bythe GNU autoconf project; see"11 Portable Shell Programming".For more discussion see4.6 MISCELLANEUS NOTES.

Let's first consider the typicalshshells in order of their strictness toPOSIX:

• posh.Minimalsh, Policy-compliantOrdinary SHell. Very close to POSIX.Stricter thandash.Supportslocal keyword to define localvariables in functions. The keywordis not defined in POSIX.

• dash.Minimalsh, Debian Almquish Shell.Close to POSIX. Supportslocalkeyword. The shell aims to meet therequirements of the Debian Linuxdistribution.

• Busybox ashis based ondashwith some morefeatures added. Supportslocalkeyword. See ServerFault"What's the Busybox default shell?"

Let's also consider what the/bin/shmight be in different OperatingSystems. For more about the history ofthesh shell, see the well-roundeddiscussion on StackExchange.What does it mean to be "sh compatible"?

Picture"Bourne Family Shells" bytangentsoft.com

• On Linux, most distributions alreadyuse, or are moving towards using,sh as a symlink todash.Older Linux versions (Red Hat,Fedora, CentOS) used to havesh tobe a symlink tobash.

• On the most conservative NetBSD,it isash, the oldAlmquist shell.On FreeBSD,sh is alsoash.OnOpenBSD, sh isksh93from theKsh family.

• On many commercial, conservativeUNIX systemssh is nowadays quitecapableksh93.

• On macOS,sh points tobash --posix, where the Bash version isindefinitely stuck at version 3.2.x(GNU Bash from 2006) due to Appleavoiding the GPL-3 license in laterBash versions. If you write/bin/shscripts in macOS, it is good idea tocheck them for portability with:

# Check better /bin/sh# compliance    dash -nx script.sh    posh -nx script.sh

In practical terms, if you plan to aimfor POSIX-compliant shell scripts, thebest shells for testing would beposhanddash.You can also extend testingwith BSD Ksh shells and other shells.SeeFURTHER READINGfor external utilities to check andimprove shell scripts even more.

# Save in shell startup file# like ~/.bashrcshelltest(){    local script shell    for script # Implicit "$@"    do        for shell in \            posh \            dash \            "busybox ash" \            mksh \            ksh \            bash \            zsh        do            if command -v "$shell" > /dev/null; then                echo "-- shell: $shell"                $shell -nx "$script"            fi        done    done}# Use is like:shelltest script.sh# External utilityshellcheck script.sh# External utilitycheckbashisms script.sh

Note that POSIX does not define theshebang— the traditional first line thatindicates which interpreter to use. SeePOSIX C language's section "exec familyof functions" andRATIONALE

(...) Another way that somehistorical implementations handleshell scripts is by recognizing thefirst two bytes of the file as thecharacter string "#!" and using theremainder of the first line of thefile as the name of the commandinterpreter to execute.

The first bytes of a script typicallycontain two special ASCII codes, aspecial comment#! if you wish, whichis read by the kernel. Note that thisis a de facto convention, universallysupported even though it is not definedby POSIX.

#! <interpreter> [word]## 1. whitespace is allowed in#    "#!" for readability.## 2. The <interpreter> must be#     full path name. Not like:##    #! sh## 3. ONE word can be added#    after the <interpreter>.#    Any more than that may not#    be portable accross Linux#    and some BSD Kernels.##    #! /bin/sh -eu#    #! /usr/bin/awk -f#    #! /usr/bin/env bash#    #! /usr/bin/env python3

Note that on Apple macOS,/bin/bash ishard-coded to Bash version 3.2.x (from 2006)where lastest Bash is5.x.You cannot uninstall it, even with rootaccess, without disabling SystemIntegrity Protection. If you install anewer Bash version withbrew install bash, it will be located in/usr/local/bin/bash.

On macOS, to use the latest Bash, theuser must arrange/usr/local/binfirst inPATH.If the script starts with#! /bin/bash, the user cannot arrange itto run under different Bash versionwithout modifying the script itself, orafter modifyingPATH, run itinconveniently withbash <script>.

... portable#! /usr/bin/env bash... traditional#! /bin/bash

There was a disruptive change fromPython 2.x to Python 3.x in 2008. Theolder programs did not run withoutchanges with the new version. In Pythonprograms, the shebang should specifythe Python version explicitly, eitherwithpython (2.x) orpython3.

... The de facto interpreters#! /usr/bin/python#! /usr/bin/python3.... not supported#! /usr/bin/python2#! /usr/bin/python3.13.2

But this is not all. Python is oneof those languages which might requiremultiple virtual environments based onprojects. It is typical to manage theseenvironments with tools likeuvor oldervirtualenv,pyenvetc. For even better portability, thefollowing would allow user to use hisactive Python environment:

... portable#! /usr/bin/env python3

The fine print here is thatenvis a standard POSIX utility, but itspath is not mandated by POSIX. However,in 99.9% of cases, the de factoportable location is/usr/bin/env.

In the end, the actual implementationof the shell you use (dash, bash,ksh...) is less important than whatutilities you use and how you use them.

It's not just about choosing to writein POSIXsh; the utilitiescalled from the script also has to beconsidered. Those ofecho,cut,tail make big part of of the scripts.If you want to ensure portability,check options defined inPOSIX.See top left menu "Shell & Utilities"followed by bottom left menu"4. Utilities"

Notable observations:

• Use POSIXcommand -vto check if command exists.Note that POSIX also definestype,as intype <command>without any options. POSIX alsodefines utilityhash,as inhash <command>. Problem withtype is that the semantics, returncodes, support or output are notnecessarily uniform. Problem withhash are similar. Neithertypenorhash is supported byposh;see tableRESULTS-PORTABILITY.Note: Thewhich <command> isneither in POSIX nor portable. Formore information aboutwhich, seeshellcheckSC2230,BashFAQ081,StackOverflow discussion"How can I check if a program exists from a Bash script?",and Debian project plan aboutdeprecating the command in LWNarticle"Debian's which hunt".

    REQUIRE="sqlite curl"RequireFeatures ()    {local cmdfor cmd# Implicit "$@"doif!command -v"$cmd"> /dev/null;thenecho"ERROR: not in PATH:$cmd">&2return 1fidone    }# Before program starts    RequireFeatures$REQUIRE||exit$?    ...

• Use plainechowithout any options. Useprintfwhen more functionality is needed.Relying solely onprintf may not beideal. In POSIX-compliantsh shells,printf is not always a built-incommand (e.g., inposhormksh)which can lead to performance overheaddue to the need to invoke an externalprocess.

# POSIXecho"line"# (1)echo"line"printf"no newline"# (2)# Not POSIXecho -e"line\nline"# (1)echo -n"no newline"# (2)

• Usegrep -E.In 2001 POSIX removedegrep.

• read.POSIX requires a VARIABLE, so alwayssupply one. In Bash, the command woulddefault to variableREPLY if omitted.You should also always use option-rwhich is eplained inshellcheckSC2162,BashFAQ001,POSIXIFSandBashWikiIFS.See in depth detailshow thereadcommand does not reads characters andnot lines in StackExchange discussionUnderstanding "IFS= read -r line".

# POSIX   REPLY=$(cat file)# Bash, Ksh# Read max 100 KiB to $REPLYread -rN$((100*1024))< filecase$REPLYin*pattern*)# match            ;;esac

• Useshift Nalways with shell special parameter$#

set -- 1# POSIX# shift all positional argsshift$## Any greater number terminates# the whole program in:# dash, posh, mksh, ksh93 etc.shift 2

As a case study, the Linux GNUsed(1)and its options differ orare incompatible. The Linux GNUsed--in-place option for replacing filecontent cannot be used in macOS andBSD. Additionally, in macOS and BSD,you will find GNU programs under ag-prefix, such asgsed(1), etc. SeeStackOverflow"sed command with -i option failing on Mac, but works on Linux". For morediscussions about the topic, seeStackOverflow1,StackOverflow2,StackOverflow3.

# Linux (works)## GNU sed(1). Replace 'this'# with 'that'sed -i 's/this/that/g' file# macOS (does not work)## This does not work. The '-i'# option has different syntax# and semantics. There is no# workaround to make the '-i'# option work across all# operating systems.sed -i 's/this/that/g' file# Maybe portable## In many cases Perl might be# available although it is not# part of the POSIX utilities.perl -i -pe 's/this/that/g' file# Portable## Avoid -i option.tmp=$(mktemp)sed 's/this/that/g' file > "$tmp" &&mv "$tmp" file &&rm -f "$tmp"

POSIXawk,does not support the-v option todefine variables. You can useassignments after the program instead.

# POSIXawk '{print var}' var=1 file# GNU awkawk -v var=1 '{print var}' file

However, don't forget that suchassignments are not evaluated untilthey are encountered, that is, afteranyBEGIN action. To use awk foroperands without any files:

# POSIXvar=1 awk 'BEGIN {print ENVIRON["var"] + 1}' < /dev/null# GNU awkawk -v var=1 'BEGIN {print var + 1; exit}'

• The shell's null command:might be slightly preferrable thanutlitytrueaccording toGNU autoconf's manual"11.14 Limitations of Shell Builtins"which states that: might not bealways builtin and "(...) theportable shell community tends toprefer using :".

while:dobreakdone# Create an empty file:> file

• Prefer POSIX$(cmd)

  command substitution instead ofleagacy POSIX backtics as in `cmd`.For more information, seeBashFaq098and shellcheckSC2006.For 20 years all the modernshshells have supported$().Including UNIX like AIX, HP-UX andconservative Oracle Solaris 10 (2005)whose support ends in2026(see Solarisversion history).

# Easily nested        lastdir=$(basename$(pwd))# Readabilty problems        lastdir=`basename\`pwd\``

See the Bash manual how to usetimereserved word withTIMEFORMATvariable to display results indifferent formats. The use of time as areserved word permits the timing ofshell builtins, shell functions, andpipelines.

TIMEFORMAT='real: %R'  # '%R %U %S'

You could also drop kernel cache beforetesting:

echo 3 > /proc/sys/vm/drop_caches

• BashManual
• Greg's Bash Wiki and FAQhttps://mywiki.wooledge.org/BashGuide
• List of which features were added tospecific releases of Bashhttps://mywiki.wooledge.org/BashFAQ/061

• GNU autoconf's manual section"11 Portable Shell Programming"Note: This presents informationintended to overcome operating systemportability issues dating back to the1970s. Consider some tips with a grainof salt, given the capabilities of moremodern POSIX-compliant shells.
• For cross platform operating systemdetection, see useful files to check:http://linuxmafia.com/faq/Admin/release-files.html
• shellcheck (Haskell)can help to improve and write portablePOSIX scripts. It can statically Lintscripts for potential mistakes. Thereis also a web interface where you canupload the script athttps://www.shellcheck.net. In Debian,see package "shellcheck". The manualpage is athttps://manpages.debian.org/testing/shellcheck/shellcheck.1.en.html
• checkbashisms can help to improve andwrite portable POSIX scripts. InDebian, the command is available inpackage "devscripts". The manual pageis athttps://manpages.debian.org/testing/devscripts/checkbashisms.1.en.html

Relevant POSIX links from 2000 onward:

• https://en.wikipedia.org/wiki/POSIX
• POSIX.1-2024IEEE Std 1003.1-2024https://pubs.opengroup.org/onlinepubs/9799919799
• POSIX.1-2018IEEE Std 1003.1-2018https://pubs.opengroup.org/onlinepubs/9699919799https://pubs.opengroup.org/onlinepubs/9699919799.2018edition/
• POSIX.1-2008IEEE Std 1003.1-2008https://pubs.opengroup.org/onlinepubs/9699919799.2008edition/
• POSIX.1-2004 (2001-2004)IEEE Std 1003.1-2004https://pubs.opengroup.org/onlinepubs/009695399

Relevant UNIX standardization links.Single UNIX Specification(SUSv4) documents are derived from POSIXstandards. For an operating system tobecome UNIX certified, it must meet allspecified requirements—a process that isboth costly and arduous. The only"Linux-based" system that has undergonefull certification is Apple'smacOS10.5 Leopard in 2007. Read the storyshared by Apple’s project lead,Terry Lambert, in a Quora's discussionforum"What goes into making an OS to be Unix compliant certified?"

• The Single UNIX Specification,Version 4https://unix.org/version4/
• See discussion at StackExchnage about"Difference between POSIX, Single UNIX Specification, and Open Group Base Specifications?".

• A comprehensive history ofash."Ash (Almquist Shell) Variants" bySven Mascheckhttps://www.in-ulm.de/~mascheck/various/ash/
• Late Jörg Shillings'sschilitoolscontainspbosh shell that can be usedfor POSIX-sh-like testing.See discussion of preservingthe project and some history atReddit.
• Super simples command interpreterto write shell-like scripts (securityoriented):https://github.com/rain-1/s

Copyright (C) 2024-2025 Jari Aalto

These programs are free software; you can redistribute it and/or modifythem under the terms of the GNU General Public License as published bythe Free Software Foundation; either version 2 of the License, or(at your option) any later version.

These programs are distributed in the hope that it will be useful,but WITHOUT ANY WARRANTY; without even the implied warranty ofMERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE. See theGNU General Public License for more details.

You should have received a copy of the GNU General Public Licensealong with these programs. If not, seehttp://www.gnu.org/licenses/.

License-tag: GPL-2.0-or-later
Seehttps://spdx.org/licenses

Keywords: shell, sh, POSIX, bash,ksh, ksh93, programming,optimizing, performance, profiling,portability