Chapter 2

The Shell Game

Programming languages come in all shapes and sizes and some of them hardly seem like programming languages at all. Of course, that depends on what you count as a programming language; as far as I’m concerned, a programming language is a language for specifying computations. But that’s pretty broad and maybe we should narrow our definition to include only languages used for specifying computations to machines, that is, languages for talking with computers. Remember, though, that programmers often communicate with one another by sharing code and the programming language used to write that code can significantly influence what can or can’t be easily communicated.

C, Java and Scheme are so-called general-purpose, high-level programming languages.¹ Plenty of other programming languages were designed to suit particular purposes, among them the languages built into mathematical programming packages like Maple, Matlab and Mathematica. There are also special-purpose languages called scripting languages built into most word-processing and desktop-publishing programs that make it easier to perform repetitious tasks like personalizing invitations or making formatting changes throughout a set of documents.

Lots of computer users find themselves constantly doing routine house-cleaning tasks like identifying and removing old files and searching for documents containing specific pieces of information. Modern operating systems generally provide nice graphical user interfaces to make such house-cleaning easier, but many repetitive tasks are easy to specify but tedious to carry out with these fancy interfaces. Sophisticated users and the programmers who manage large computer systems use programs called shellsthat let them talk more directly with the operating system.

The core programs making up an operating system are located in what is commonly called the kernelof the operating system. The programs in the kernel are responsible for juggling multiple programs, allocating memory and dealing with everything from detecting keyboard and mouse input to reading from and writing to disks. A shell is a program surrounding the kernel that makes it easy for the user to do useful work while at the same time protecting the user (and all the other users in a multiple-user operating system) from doing anything stupid or malicious. Applications are built on top of the shell but you can access the shell directly.

Most operating systems support multiple shells and programmers often have strong opinions on which ones they think are best. Almost by definition, a shell requires a programming language, sometimes called a command language, in order to make the operating system perform useful work. These languages are general-purpose languages and are often used to write sizable programs. However, shell programs also allow an interactive or conversational programming style that makes it quite easy to write short programs for specialized tasks.

When most people think about programming a computer, they think about writing large programs consisting of many lines of code, but other styles of communication are possible. Writing a sizable program is like sending a letter to a friend: the message has to be largely self-contained and you have to be careful to make sure you’re not misunderstood. You may not have a chance to clarify and if your friend ends up confused (or worse misled) it may be some time before you have a chance to set her straight.

When you interact with a shell, on the other hand, you talk directly with the operating system. Here again you don’t want to be misunderstood, but shells are designed to allow opportunity for feedback. Often the commands you submit to the shell have options that make them query you before doing something that can’t be easily undone (other options turn off all prompts and warnings and let you accept the risks of, say, deleting everything in your home directory if you so choose). The shell acts as an interpreter — the analog of a human interpreter seems particularly appropriate here — intercepting your commands, prompting you for clarification if need be and performing any necessary adjustments and translations before sending them along to the operating system.

In addition to providing a general-purpose programming language, most shells include a set of powerful commands that are themselves programs of considerable complexity. The most common of these commands are used for such tasks as searching and modifying files and are available in most shells. Some of these commands are so powerful that one must use command-specific mini-programming languages in order to exploit them fully. If all this sounds frighteningly complex, it is, but don’t imagine that everyone who makes good use of a shell knows everything about that shell — that’s not true, not by a long shot. Luckily, you can learn to be quite effective programming a shell without knowing all the ugly details of several programming languages.

2.1  Shell programming

There are lots of shells to choose from, bash, csh, ksh, shand tcshbeing the most popular and readily available. The bash shell owes its name to Steven Bourneat AT&T Bell Laboratories who developed one of the first shells for the Unix operating system, variants of which are still popular today. bash, the “Bourne again shell,” is intended as the shell for the GNU operating system. The csh shell or C-shell, written by Bill Joyat the University of California at Berkeley, is the choice of many C programmers. The tcsh shell is roughly csh with various niceties such as file-name completion² and command-line editing.³ ksh is a public-domain version of the Korn shell developed by David Kornat AT&T Bell Laboratories. The Korn shell is upward-compatible with the Bourne shell and includes many of the features of csh.

These shells run under most modern operating systems including many variants of Unix, Linux and the Apple and Microsoft operating systems. On some systems sh is a version of the original Bourne shell but on others it is a default or generic shell; it is likely to correspond to different shells on different machines. Sampling machines around my department, I found sh corresponding to bash on one machine, ksh on another and to various hybrids on other machines. Here I’ll use tcsh but stick to a command subset common to most shells.

A program written to run in a shell is called a shell script. We already met a couple of such scripts in Chapter 1; now we’ll look shells and shell scripts in more detail. Shell scripts typically deal with objects like files and running programs called processes and operations like starting and stopping processes and creating, deleting and modifying files.

To stretch the language metaphor somewhat, you can think of programs and commands as verbs and file and directory names as nouns. We’ll also encounter analogs for adverbs and adjectives, conjunctions, and, of course, punctuation (in the form of seemingly arbitrary, annoying and easily mistyped syntactic conventions). My objective here is not to teach you shell programming but rather to tempt you to learn it on your own and to give you some idea of what shell programming languages are good for.

Before we look at some specific instances of shell programs, let’s think about what you generally need in specifying computations. Programs are typically broken down into smaller components called, variously, procedures, functions, programs, methods or commands. Each component is designed to carry out some specific computation; thus a sorting component puts lists in some specified order, say, alphabetic or numeric. Programming languages provide the means of orchestrating the various computations that result from invoking components by stringing them together and enabling them to pass information among themselves.

The term “flow of control”refers to various ways of stringing computations together, for example, as sequences — do this and then do that, conditionals — if such and such is true then do this otherwise do that, and loops — do this again and again until such and such is true. Information can be passed between components by allowing one computation to take as input the result (or output) of another computation, or, more generally, by allowing one component to store information in a convenient place so that it can be used by some subsequent computation.

Some of the most common mechanisms for information passing and flow of control in shell programming languages are rare in other programming languages. The methods of orchestrating computations in shells suit a particular type of computing and a particular class of computations. Understanding these methods is useful in itself and also gives some insight into the advantages and disadvantages of other methods and other programming languages that we’ll explore in subsequent chapters.

Files are one of the most convenient ways of keeping track of information. They provide a very useful way of thinking about computer memory that connects directly to almost everything you’re likely to do involving computers — files for important documents, web pages, and email messages, as well as digital music, photos and movies. Not surprisingly, shells provide lots of operations for manipulating files and the directories that contain them.

We’ll start with one of the most useful commands, ls, for listing the contents (the names of files and nested directories) of the specified directories. In these exchanges with the shell, the prompt is the % character followed by a space; I typed everything to the right of the prompt. The intervening lines until the next prompt appears are the shell’s responses, the output resulting from a command. The command ls appearing by itself is a request to list the contents of the current working directory.

% ls
bolts.txt nails.txt nuts.html 

To find out what the current working directory is you can use the command pwd, which returns the absolute pathname of the current working directory. We talked about absolute and relative pathnames in Chapter 1 and we’ll talk about navigating in a hierarchy of directories in Chapter 13, but for the remainder of this chapter the current working directory remains fixed.

% pwd
/u/tld

Shell command optionsare typically specified by a dash, -, followed by one or more characters. In the next exchange, ls is invoked with the “long-format” option:

% ls -l
total 0
-rw-r--r--  1 tld  staff  0 Dec  3 07:42 bolts.txt
-rw-r--r--  1 tld  staff  0 Dec  3 07:42 nails.txt
-rw-r--r--  1 tld  staff  0 Dec  3 07:42 nuts.html

The long-format option makes ls list information that’s handy for lots of purposes, from sharing files with your friends (or protecting your files from the prying eyes of those who are not your friends) to remembering when you created a file. Other options allow you to list files sorted by time or size, distinguish directories from files and extract all sorts of useful information.

Commands like ls take arguments in addition to options. Options are like adverbs: they modify a command and tell the shell how something is to be done. Arguments are like the objects of imperative sentences: they indicate what or whom the command applies to. Commands can take any number of arguments, which are entered as sequences (or strings) of characters and typically separated by whitespace(spaces, tabs and, in some cases, carriage-return and new-line characters).⁴ Here are two invocations of ls with different arguments:

% ls bolts.txt
bolts.txt
% ls bolts.txt tacks.html
ls: tacks.html: No such file or directory
bolts.txt

In the first case, the single argument bolts.txt simply makes ls list the specified file if it’s in the current directory and complain otherwise. In the second case, two arguments are supplied and we see ls complain that it can’t find a file named tacks.html in the current directory; it does, however, indicate that bolts.txt is present by listing the file name.

Distinctions between options and arguments and the analogies to different parts of speech should be taken lightly. The shell isn’t going to wince the way your grade-school teacher did if you use an adjective instead of an adverb. However, shells, like most programming languages, are syntactically less forgiving than even the strictest grammarian. Generally speaking, options modify the behavior of a command (often by specifying the format of the output) and arguments determine the primary input to the computation performed by a given command — what the computation computes on, as it were.

The computations performed by a shell correspond to more than the computations performed by commands. Recall that the shell is itself a program that reads lines typed by the user and interprets them. When a shell reads the arguments following a command, it is free to perform computations involving those arguments before passing them (or its own interpretation of them) to the program corresponding to the command. Consider these two invocations of ls:

% ls *.html
nuts.html
% ls *.txt
bolts.txt nails.txt

In the first invocation, we are asking ls to list any files that end with .html. As we saw in Chapter 1, the * is called a wildcard and matches any sequence of characters.⁵ Most shells employ a pattern-matching language that allows a great deal of control over matching. For example, the pattern *.??? matches any file consisting of any sequence of characters followed by a period followed by exactly three characters. The pattern [nt]*.txt matches any file that begins with an “n” or a “t” followed by zero or more characters followed by .txt. The shell looks at each argument in turn, matches the pattern (think of nuts.html as a particularly simple pattern that matches only one possible file name) against the names of the files in the directory, and then passes those names to the command.

This pattern matching on file names is just one example of the sort of computations performed by the shell before passing information on to commands. If you’re curious about what gets passed to a command after the shell is finished with its interpretation (or preprocessing as it’s called) of the arguments, the command echosimply prints out what the shell would send on to a command. echo is useful for all sorts of purposes and we’ll exploit it often in the rest of this chapter.

% echo *s.txt tacks.html
bolts.txt nails.txt tacks.html

Playing around with echo can give some additional insights into the shells computations in preprocessing arguments. Arguments are specified as strings that you type to the shell but not all the arguments correspond to the names of files or patterns specifying the names of files. Consider this invocation of echo with two arguments:

% echo '*.* expands to' *.*
*.* expands to bolts.txt nails.txt nuts.html

In the first argument corresponding to the string ’*.* expands to ’, the part of the string *.* is treated literally; the single-quote marks (which don’t appear in the output) tell the shell not to do any preprocessing on the string enclosed by the single quotes. The single quotes also tell the shell to treat spaces as part of the enclosed string and, in particular, not to treat them as whitespace separating arguments. You can think of the single quotes as telling the shell to turn off all preprocessing.

2.2  Shell variables

It’s often useful to set aside information about the current state of a computation in order to use it later. While you can store information in files for later use, it’s often convenient to use a more readily available shell variableto keep track of a name, a number or a list of items.

For our purposes, shell variables correspond to typed sequences of alphanumeric characters, for instance, tmp, VAR or files, that can be assigned values and whose assigned values can be read in shell scripts. A special case of shell variables is the so-called environment variablethat operating systems use to track your preferences for printers, user names, and the like. For example, USER, HOME, PWD and SHELL correspond, respectively, to my user name, home directory, current working directory (the value returned by the pwd command) and the absolute path of the shell program I’m using:

% echo $SHELL
/bin/tcsh

In preprocessing arguments, the dollar sign ($) tells the shell to interpret the characters immediately following as a variable name and substitute the value of the variable in the expression. As you might expect, ls $PWD does the same as ls, whereas ls $HOME lists the files in your home directory no matter what your current working directory. If the characters following the dollar sign don’t correspond to a variable with an assigned value, the shell protests:

% echo $NOT
NOT: Undefined variable.

This is the shell’s attempt to be helpful — a pretty feeble attempt, but as you gain experience programming, you’ll come to appreciate the challenges of getting a shell or any other programming environment to do the right thing in response to an ambiguous or garbled command. For example, you might imagine that if you had a variable NOTE it might make sense for the shell to infer that you’d inadvertently left off the E and to go ahead and just make the substitution. Sounds obvious in this particular case, but until shell programs become more intelligent, it’s probably better for them to play dumb.

If we embed a dollar sign in a string set off by single quotes, its special meaning is ignored and the shell doesn’t treat the subsequent characters differently from any other characters appearing in the string:

% echo 'my shell is $SHELL'
my shell is $SHELL

If, however, we refer to a variable in a string set off with double quotes, the shell looks up the value of the variable and inserts it into the string before passing it on:

% echo "my shell is $SHELL"
my shell is /bin/tcsh

The double quotes tell the shell to interpret a string selectively, treating some characters, such as spaces, literally, but preprocessing others as though the quotes were not there. As usual, the full story concerning single and double quotes, and the many other special characters recognized by the shell, is complicated, but plenty of quite able shell programmers get along fine without understanding the story completely.

To read a variable as an argument you append the dollar sign to the name of the variable. To assign a variable you use set and =. Here we set the variable tmp to have a value corresponding to a simple string:

% set tmp = "a simple string"
% echo $tmp
a simple string

So far the strings we’ve typed as arguments to commands are largely passive; they specify fodder for computation but not the computations themselves. Wouldn’t it be interesting to create programs that construct strings out of bits and pieces of other strings and then, when a computation is fully constructed, cause that computation to be carried out, perhaps just once but possibly many times?

To construct a string encoding a computation, you have to control how characters are interpreted, suppressing interpretation in some cases and forcing it in others. This example uses both single and double quotes to create a simple computation that is assigned to be the value of a variable and is later invoked using the evalcommand, which treats its arguments as though they were typed directly to the shell:

% set myshell = "echo 'my shell is $SHELL'"
% echo $myshell
echo 'my shell is /bin/tcsh'
% eval $myshell
my shell is /bin/tcsh

In the first line assigning the variable myshell, the shell reads the entire line. The single quotes are interpreted literally, ignoring their special meaning, but the dollar sign is recognized and the value of SHELL is inserted into the string. If we reverse the use of quotes, substituting single quotes for double quotes and vice versa, we end up specifying a different computation even though the result, in this case, turns out to be the same:

% set nowshell = 'echo "my shell is $SHELL"'
% echo $nowshell
echo "my shell is $SHELL"
% eval $nowshell
my shell is /bin/tcsh

If we change the value of SHELL, the difference between the two programs becomes apparent:

% set SHELL = /bin/bash
% eval $myshell 
my shell is /bin/tcsh
% eval $nowshell 
my shell is /bin/bash

These simple examples illustrate the interchangeability of programs and data. You can encode a program in a string, carry it around in a variable, change the program by altering the string and evaluate it whenever the mood strikes. And as we saw in Chapter 1, even very simple programs can write and run other programs.

A computer virus is a program that disguises itself as data, insinuates itself into your file system by embedding itself in an email message or web page, and then tricks the operating system into believing it’s one of your programs; the operating system then runs the virus and allows it to wreak havoc on your computer. Viruses often manifest both their program and data personae simultaneously by rewriting themselves to avoid detection and escape being harvested by antivirus programs.

Often invoking a program can involve multiple levels of interpretation and code generation. Figuring out what happens when, as in determining when the variable SHELL is evaluated, can get complicated, but having the flexibility to control when computations take place (and indeed where, once we consider computations on networked computers) is essential in working with modern computer systems.

As you learn more about programming, you’ll meet lots of programs that take other programs as input and generate still other programs as output. Variously called compilers, interpreters, assemblers, disassemblers, virtual machines, preprocessors, postprocessing back ends, bytecode compilers, cross compilers and macro-expansion packages, all these programs transform a specification for a computation in one form into a specification for a different but related computation in another form. These program-transforming programs can optimize the performance of programs, specialize programs to run on particular machines, generalize programs to run on a range of machines, or customize programs to run in conjunction with complementary programs.

2.3  Information passing

Let’s return to orchestrating computations in shell scripts. So far we’ve seen only a few component computations in the form of commands like ls and echo. More often than not we’ll perform a computation as part of a more complicated computation in order to obtain an intermediate result for use in a subsequent computation. To create a personalized phone book, say, we might first list the names of all our friends, including their nicknames. We might then look up their names in an online telephone directory, producing a new list that includes phone numbers in addition to names and nicknames. Finally, we might sort the list by nicknames and then print it out.

Suppose you are the system administrator for a large network of computers and suppose you want to find out how many people are using each of the different shell programs we mentioned earlier: bash, csh, ksh, sh and tcsh. In most operating systems that support shells, you can specify a default shell by setting the SHELL environment variable in the .login file in your home directory. For example, if you want tcsh to be your default shell program, you can include the line set SHELL = /bin/tcsh in your .login file.

In a computer network for a business or a university, users typically visualize the files and directories of all users on the network as a single file system. The fact that files reside on many different hard drives connected to one or several networked computers is hidden from the user. The executable binary files for commonly used programs are often found in the directory /bin/ and the home directories for individual users are likely to be in a directory called /u/ (or /users/ or /home/). Since my login name is tld, my home directory is /u/tld/. I could use the ls command to list all users’ .login files:⁶

% ls /u/*/.login 
/u/dhl/.login     /u/jfh/.login     /u/jsb/.login 
/u/jwc/.login     /u/lpk/.login     /u/tld/.login

In order to figure out what these users prefer for their default shells, however, we’ll have to look inside these files. The grep command and its many variants are perfect for this job. We’ll take a closer look at grep in Chapter 3, which is devoted to searching through files. For our immediate purposes, grep takes a pattern and a file specification and returns all lines that contain a string matching the pattern in all files indicated by the file specification. If the file specification indicates multiple files, then the matching lines are prefixed with the name of the file in which the line appeared. Here’s how we would extract all the lines in all the .login files that set the SHELL environment variable:⁷

% grep "set SHELL" /u/*/.login 
/u/dhl/.login:set SHELL = /bin/ksh
/u/jfh/.login:set SHELL = /bin/tcsh
/u/jsb/.login:set SHELL = /bin/bash
/u/jwc/.login:set SHELL = /bin/bash
/u/lpk/.login:set SHELL = /bin/tcsh
/u/tld/.login:set SHELL = /bin/tcsh

This is the first intermediate result in a multistage computation. To use this result in subsequent stages, we’re going to store it in a file. Files are an obvious choice for storing information, and text files have a simple structure that can be used to organize data. A file consists of zero or more lines of text and each line of text can be further subdivided into words, numbers or other distinct data items separated by whitespace or other forms of punctuation. In our examples so far, the output of commands like ls has simply been printed to the shell (or, as it’s often called, the standard output). Alternatively, we can force the output of a computation to be sent to a file using the redirect output operator, >:

% ls > ls.txt
% cat ls.txt
bolts.txt
ls.txt
nails.txt
nuts.html

Here we’ve redirected the output of the ls command to be sent to a new file ls.txt.⁸ Listing the contents of ls.txt using the cat command, we see the names of all the files in the current working directory at the time we called ls, including ls.txt. If you experiment with ls in directories containing lots of files, you’ll find that the output sent to the standard output and displayed by the shell is formatted in accord with your display window. In redirecting output to a file, individual items such as file names are listed one item to a line; this turns out to be handy in using such output files as input to other computations. We can redirect the output of grep to create a file — which we’ll call login.txt — in the same way as we did with ls:

% grep "set SHELL" /u/*/.login > login.txt 

The file login.txt contains the information we need but not in a form we can readily use in order to tabulate how many users prefer each shell. As our next step, we’ll get rid of all the extraneous information in the file login.txt by using another program called sed. The name of this program derives from “stream editor” but here “text filter” is a more appropriate description. We’ll use sed to filter the file, eliminating the parts we don’t want and leaving only those parts we do. In general, sed takes strings corresponding to lines of text from files and returns versions of those strings with various additions, deletions and substitutions. sed is a complex program with its own commands and its own specialized programming language. We’ll use just one of sed’s commands to filter login.txt. You can do a lot with this one command, and indeed many shell programmers get by fine knowing only this command. (If this seems strange, think about how few people know how to use all the options on their television remotes, VCRs, automotive cruise controls, kitchen appliances or even Swiss Army knives.)

A sed command of the form s"pattern"replacement"g substitutes the string specified by replacement for all strings matching pattern.⁹ If you leave off the g (for “global”) at the end of the command, it replaces only the first string matching the pattern. So, for example, sed ’s"java"scheme"g’ old.txt > new.txt would create a file new.txt that looks like old.txt except that all occurrences of the string java in old.txt are replaced by the string scheme in new.txt. Usually it’s important to establish some context for substitutions; in our case, we’re looking for the string that follows /bin/. We’ll also have to do some gymnastics to keep track of the parts of the string that we want to keep. Take a look at sed in action and then we’ll consider more carefully how it did what it did.

% sed 's".*/bin/\([a-z]*\)"\1"' < login.txt
ksh
tcsh
bash
bash
tcsh
tcsh

Let’s look at a single line of input from login.txt. Given the string /u/jsb/.login:set SHELL = /bin/bash, the .* part of the pattern matches /u/jsb/.login:set SHELL = , the /bin/ part matches /bin/ to establish the context for extracting the shell name, and the [a-z]* part matches bash. The \( and \) operators save whatever is matched by the pattern between them. In general, the first pair of \(\) saves its matched string into \1, the second pair into \2, and so on. In the present use of sed, the string bash is saved into \1, and so the entire line set SHELL = /bin/bash is replaced simply by bash.

So far we’ve also specified the inputs to computations as arguments typed to the shell or, in this case, the standard input. Inputs can also be redirected using the redirect input operator, <. For the next step in our multistage computation, we take the input from the last step in login.txt and redirect the output to a new file called shell.txt:

% sed 's".*/bin/\([a-z]*\)"\1"' < login.txt > shells.txt

Now if we could just count the number of occurrences of each shell program we’d be done. Unfortunately, there isn’t a standard program for doing this, so we’ll use two additional commands in a two-step trick that you’ll probably see often if you look at a lot of shell scripts. First we sort the lines in shells.txt using the sortcommand:

% sort < shells.txt 
bash
bash
ksh
tcsh
tcsh
tcsh

We’ll need to save that result and so we redirect sorted lines into the file sorted.txt:

% sort < shells.txt > sorted.txt 

The next command, uniq, eliminates identical adjacent lines in the input. Repeated lines in the input that are not adjacent are not detected — this is why we sorted the list before submitting it to uniq. This command unadorned won’t do us much good, but uniq with the “count-identical-adjacent-lines” option, specified with -c, is exactly what we need to get the answer we wanted:

% uniq -c < sorted.txt
   2 bash
   1 ksh
   3 tcsh

The trick of using sort followed by uniq -c is definitely a hack, albeit a hack known to lots of shell programmers. We saw another, more elegant trick in Chapter 1 for passing information between commands, and now we’re going to use it to simplify computing user shell preferences.

We have the answer we’re looking for, but in the process we generated three files, login.txt, shells.txt and sorted.txt, to keep track of intermediate results that we now have no use for. The folks at AT&T who developed the Unix operating system created a very elegant abstraction that lets us dispense with the creation and subsequent elimination of files for storing intermediate results. It’s an abstraction because it hides the details of how information is passed from one computational component to another. Indeed, we no longer have to even think about files. Unix pipes, denoted by vertical bars (|), allow us to take the output produced by one command and “pipe” it into the input for another command.

Here’s a shell script that uses the Unix plumbing abstraction to perform our example task of listing user shell preferences:

% grep "set SHELL" /u/*/.login | sed 's".*/bin/\([a-z]*\)"\1"' | sort | uniq -c
   2 bash
   1 ksh
   3 tcsh

But there’s more to pipes than merely eliminating temporary files. In the temporary-file approach, we created the entire file containing all the intermediate results, one line at a time, before invoking the next command that used those results. Why not send along each line of text as soon as it’s ready? Why not have all the commands running simultaneously and working on lines of text as soon as they become available? That’s exactly what the abstraction of pipes allows.

The pipe abstraction supports the idea of lines of text flowing through pipes being processed by various plumbing fixtures (commands) along the way (see Figure 3 for a sketch from this plumber’s perspective). As soon as the first line of text is processed by the first command, the second command can start working. Once a line is written to a file, it’s not going to change, so why not start working on it? The abstraction doesn’t require pipes to be implemented in this way, but it allows it, and that’s why it’s such a powerful way of thinking about computation.

2.4  Asynchronous processes

In order to make the most of the plumbing abstraction, however, we need another abstraction, that of asynchronous processes. Executing a command in a shell could easily require thousands if not millions of more primitive instructions. Modern operating systems allow multiple programs to run asynchronously by interleaving the execution of their instructions — for example, first run a couple of instructions from program one, then a couple from program two, and then go back and give program one another chance. This interleaving happens so quickly that the different programs appear to run in parallel. Modern operating systems typically run dozens of programs quietly in the background, where you’re not even aware of them, to do all sorts of tasks (much as your body has all sorts of subsystems that manage breathing, heart rate, digestion and the like).

Your operating system often runs a program by starting a new process. A processis just a computation that can be run simultaneously with other computations. A running program that initiates a new process is itself running as a process, called the parent processof the newly initiated process. A parent process can initiate multiple child processes and its children can initiate their own child processes. The shell is also a program running in a process. Every time the shell reads a command you’ve typed, it starts up a new child process to run the program corresponding to that command. It’s the operating system’s job to manage all these processes.

The best way to learn about processes is to create a bunch of them and watch what they do. The Unix sleepcommand takes a single integer argument specifying how many seconds the process in which the command is running should sleep for. To illustrate, we’ll introduce another flow-of-control mechanism for sequencing commands. A shell script of the form one ; two, where one and two are commands, first executes one and then two — the commands are executed in sequence. What do you expect to happen in the this example?

% sleep 10 ; echo done

When you run this script, there is a pause of about ten seconds and then the shell prints “done”. In the next example, the shell starts two separate processes, one for each of the two commands. What do you think happens next?

% sleep 10 | echo done 

The shell almost immediately prints out “done” and then there is a pause of about ten seconds at the end of which the prompt reappears.

How can we determine what’s going on? One of the commands the shell provides for managing processes is the process status command ps, but there’s a problem in using this command to inspect processes such as those generated by this shell script. To inspect running processes, I have to be able to type commands to the shell. The problem is that a running shell script takes over the standard input and output, preventing me from interacting with the shell. To get around this, I can use the & operator, which allows me to run a script in the background and returns control over input and output to the shell.¹⁰ Note what happens when I run this simple script terminated by an &:

% sleep 10 | sleep 20 &
[1] 1453 1454

The 1 in [1] is the number of the job associated with the command. You can see the jobs you have running in the background by using the jobscommand:

% jobs
[1]  + Running                       sleep 10 | sleep 20

The other two numbers, 1453 and 1454, are the process identifiersor PIDsof the two processes created by the shell and used to run the two commands sleep 10 and sleep 20. We can examine these processes with the process status command:

% ps -o "pid command"
  PID  COMMAND
  893  -bin/tcsh 
 1453  sleep 10
 1454  sleep 20

The -o option lets me tell ps exactly what status information to list. The unadorned ps command lists several pieces of status information in addition to the process identifier and command. You can learn more about ps by invoking man ps or info ps from the shell (although, some of the documentation may be incomprehensible without a course on operating systems).

By the way, if you ever create a process and then decide you don’t want it, you can get rid of it by using the kill command. I could get rid of the second sleep command by typing kill 1454. Processes in their final death throes can hang around, however, and kill 1454 with no options is not as strong (or as terminal) as kill -KILL 1454.

The ability of an operating system to handle multiple processes enables a word-processing program to keep up with your typing at the same time it’s checking your spelling, or an office delivery robot to look for nameplates on office doors at the same time it’s avoiding obstacles in the hallways. In Chapter 10 we’ll talk about processes that really make your computer work hard and thus better show off the advantages of running multiple processes.

Unix processes are called asynchronous communicating processes. The ‘asynchronous’ part refers to the fact that the individual instructions comprising the programs associated with the different processes can be executed in any order. But even asynchronous processes occasionally need to coordinate their efforts — that’s the ‘communicating’ part. This is especially true when one process depends on input from another.

For processes to communicate with one another, they need some way to synchronize their behavior. The operating system provides synchronization machinery so that processes that must send information to another process can do so and get on with whatever else they’re doing without waiting for the other process to accept or acknowledge receipt. The same machinery allows processes that need information from other processes to go to sleep and ask to be awakened when it becomes available. In Chapter 10 we’ll see how this machinery keeps robots from getting confused and banks from giving you more (or less) money than you’re entitled to.

Pipes support unidirectional flow of information. Other interprocess communication and synchronization mechanisms support bidirectional exchanges of information between processes. One such mechanism, sockets, is used to enable processes on different machines to exchange information and supports the client-server model that in turn supports the World Wide Web. We’ll look at the client-server model in some detail in Chapter 11.

You can learn a lot about shell programming by reading the online documentation available on your system or on the web. Sometimes, however, it’s comforting to have a nice fat book full of examples to guide you through the maze of arcane syntax and technical terms. Ellie Quigley01’s Unix Shells by Example provides a good general introduction to shell programming with side-by-side comparisons of the most popular shells. I also recommend KochanandWood89’s Unix Shell Programming as an introduction and general reference.

Computer scientists are notorious for mixing metaphors and seeing computation in all sorts of strange places. In some respects, shell programming languages are pretty tame stuff, not nearly as exotic as programming models like object-oriented programming. Still, orchestrating computations using files, pipes and processes is a powerful basis for computation and we’ve barely scratched the surface of what you can do within a shell. In the next chapter, we’ll see some more simple shell scripts that are useful for everyday computing tasks.