Repository files navigation

All the broken
Too many broken
Shells
In our shellmounds
—Grayceon, Shellmounds

This is the material for a series of workshops I ran at my workplaceon how to write a Unix shell.

The focus is slightly more on building an interactive shell than ascripting-oriented shell, only because I think this is moregratifying, even if it's less useful.

Be warned that some of the suggestions and discussion make opinionatedchoices without discussing equally-valid alternatives.

This is a work in progress and there may remain many infelicities.Patches Thoughtfully Considered. Feel free to report issues viaGithub.

The shell is at the heart of Unix. It's the glue that makes all thelittle Unix tools work together so well. Understanding it sheds lighton many of Unix's important ideas, and writing our own is the bestpath to that understanding.

This workshop has three goals:

• to give you a better understanding of how Unix processes work;
  • this will make you better at designing and understanding softwarethat runs on Unix;
• to clarify some common misunderstandings of POSIX shells;
  • this will make you more effective at using and scriptingubiquitous shells like bash;
• to help you build a working implementation of a shell you can beexcited about working on.
  • there are endless personal customizations you can make to yourown shell, and can help you think about how you interact withyour computer and how it might be different.

(some of this rationale is expanded on in my blog post,Buildingshells with a grain of salt)

I've tried to break this up into progressive stages that cover mostlyorthogonal topics. Each stage contains a description of thefacilities that will be discussed, a list of manpages to consult, anda set of tests. I've tried to also hint at some functionality that isfun but not necessary for the tests to pass.

In the root of this repository, there is a script calledvalidate;you can run all the tests against your shell-in-progress by specifyingthe path to your shell's executable, like this:

$ ./validate ../mysh/mysh

It should tell you what stage you need to implement next. You canalso run a stage by itself, or an individual test:

$ ./validate ../mysh/mysh stage_2$ ./validate ../mysh/mysh stage_3 03

To run the tests, you will needexpect, which is usually in apackage calledexpect, and a C compiler. The way the tests areimplemented is less robust than one might hope, but should suffice forour pedagogical goals. They are unfortunately somewhat timingsensitive, such that some tests will be flaky. If you encounter aspecifically flaky test, please let me know.

The tests assume you will be implementing a vanilla Bourne-flavoredshell with some ksh influences. Feel free to experiment withalternate syntax, but if so, you may need to adjust the tests. Exceptwhere specifically noted,bash (andksh) should pass all thetests, so you can "test the tests" that way. (Try./validate /bin/bash.) Likewise,cat should fail all the tests.

Originally, I targeted plain/bin/sh, but I decided the material instage 5 was too important. Still,dash will pass everything butstage 5. There are also some other minor compatibility differenceswith some existing shells; you may run into them if you try them out.Any failure (of a supposedly POSIX shell) that isn't documented in thecomments of a test should be reported as a bug.

In which we discuss the basics of Unix processes, write the simplestpossible shell, and then lay the foundations for the rest of thesteps.

In which we add pipes and fd redirection to our shell.

In which we discuss signals and add support for ever-helpful chordslike^C,^\, and^Z.

In which we discuss environments, variables, globbing, and otheroft-misunderstood concepts of the shell.

In which we apply some polish to our shell to make it usable forinteractive work.

In which I prompt you to go further.

I'll link to some of the shells that were written as a result of thisworkshop here shortly, including a couple I wrote to serve as examplesof different approaches.

• The [shtepper] is a great resource for understanding shellexecution. You can input an expression and see in excruciatingdetail how it should be evaluated.

shtepper

• Advanced Programming in the Unix Environment by Stevens coversall this stuff and is a must-read. I call thisAPUE throughoutthis tutorial.
• Chet Ramey describesthe Bourne-Again Shell inthe Architectureof Open Source Applications; this is probably the best thing toread to understand the structure of a real shell.
• Michael Kerrisk'sthe Linux Programming Interface, though fairlyLinux-specific, has some great coverage of many of the topics we'lltouch on. I call thisLPI throughout this tutorial.
• Unix system programming in OCaml shows the development of a simple shell.
• Advanced Unix Programming by Rochkind; chapter 5 has a simple shell.
• thetour of the Almquist shell is outdated but may help you findwhere some things are implemented indash and otherashdescendants.

I wrote this workshop partially because I felt other tutorials don'tgo far enough, but all of these are worth reading, especially ifyou're having trouble with a stage they cover:

• Stephen Brennan'sWrite a Shell in C is a more detailed look atwhat isstage 1 here.
• Jesse Storimer'sA Unix Shell in Ruby gets as far as pipes;
• Kamal Marhubi'sLet's Build a Shell also goes about that far;
• glibc'sImplementing a Job Control Shell shows specifically howto implement job control;
• Nelson Elhage'sSignalling and Job Control covers someofstage 3's material.

• thePOSIX standard explains the expectations for the shell andits utilities in reasonable detail.
• there arePOSIX conformance test suites but they don't seem to beavailable in convenient, non-restricted forms.
• yash's posix-shell-tests are only runnable with yash, but thetests themselves are full of useful ideas.

• busybox: C; contains both ash and hush, and test suites.
• mksh: C; non-interactive tests.
• rc: C; fairly minimal.
• zsh: C; extremely maximal.
• bash: C.
• fish: C++11; has expect-based interactive tests.
• Thompson shell: C; the original Unix shell; very minimal.
• scsh: Scheme and C; intended for scripting.
• cash: OCaml; based on scsh.
• eshell: Emacs Lisp.
• oil: Python and C++; has an extensive test suite.
• xonsh: Python.
• oh: Go.
• yash-rs: Rust.

Although there is an elegant relationship between C and Unix whichmakes it attractive to write a shell in the former, to minimizefrustration I suggest trying a higher-level language first. Ideallythe language will have good support for:

• making POSIX syscalls
• string manipulation
• hash tables

Languages that provide a lot of their own infrastructure with regardssignals or threads may be much more difficult to use.

http://basepath.com/aup/ex/group__Ux.html

The most convenient library would beiolib, which you can getthroughQuicklisp. You'll need to installlibfixposix first.There's alsosb-posix insbcl for the daring.

• usethe unix package
• Hell might be a starting point

You will probably run into issues related to the JVM, particularlywith signals and forking, but as a starting point, you could do worsethan loading libc with JNA.

There's alsojtux.

There are a variety of approaches, butljsyscall looks promising.luaposix might be sufficient.

• Unix module
• lambda-term
• parsing with OCamllex and menhir

See alsoUnix system programming in OCaml,cash.

Seeperlfunc(3perl); all the functions we want are at hand, usuallywith the same name.

Although Python provides higher-level abstractions likesubprocess, for the purposes of this workshop you probably want touse the functions inos.

Please note an important gotcha for stage 2! SincePython 3.4, fdshave defaulted to non-inheritable, which means you'll need toexplicitlyos.set_inheritable(fd, True) any file descriptor youintend to pass down to a child.

The implementation seems a little too heavy to do this conveniently,but see the Scheme section below for alternatives.

Process has most of what you need. You can useShellwords but youdecide if it's cheating or not.

Although we use few enough calls that you could just create bindingsdirectly, eitherto libc with the FFI or bydirectly making syscalls, for justgetting something working, thenix-rust library should provide allthe necessary facilities.

Guile already has all the calls you need; seethe POSIX section of the Guile manual. Another approach would be touse something likeChibi Scheme with bindings to libc calls.

Although core Tcl doesn't provide what's necessary,expect probablydoes. For example, Tcl doesn't have a way toexec, but expectprovidesoverlay to do this.