| sed | |
|---|---|
 An excerpt from GNU sed'sman page | |
| Paradigm | scripting |
| Designed by | Lee E. McMahon |
| First appeared | 1974; 52 years ago (1974) |
| Implementation language | C |
| Influenced by | |
| ed | |
| Influenced | |
| Perl,AWK | |
sed (short forstreameditor) is a utility that transformstext via a script written in a relatively simple and compactprogramming language. It was developed from 1973 to 1974 byLee E. McMahon ofBell Labs,[1]and is available today for most operating systems.[2] The functionality of sed is based on the scripting features of the interactive editored ("editor", 1971) and the earlierqed ("quick editor", 1965–66). It was one of the earliest tools to supportregular expressions, and remains in use for text processing, most notably with the substitution command. Popular alternative tools for text manipulation and stream editing includeAWK andPerl. Theshellcommand thatruns the utility has the same name:sed.
First appearing inVersion 7 Unix,[3] sed is one of the early Unix utilities built for command line processing of data files. It evolved as the natural successor to the populargrep command.[4] The original motivation was an analogue of grep (g/re/p) for substitution, hence "g/re/s".[3] Foreseeing that further special-purpose programs for each command would also arise, such as g/re/d, McMahon wrote a general-purpose line-oriented stream editor, which became sed.[4] The syntax for sed, notably the use of/ forpattern matching, ands/// for substitution, originated withed, the precursor to sed, which was in common use at the time,[4] and the regular expression syntax has influenced other languages, notablyECMAScript andPerl. Later, the more powerful languageAWK developed, and these functioned as cousins, allowing powerful text processing to be done byshell scripts. sed and AWK are often cited as progenitors and inspiration for Perl, and influenced Perl's syntax and semantics, notably in the matching and substitution operators.
GNU sed added several new features, includingin-place editing of files.Super-sed is an extended version of sed that includes regular expressions compatible withPerl. Another variant of sed isminised, originally reverse-engineered from 4.1BSD sed byEric S. Raymond and currently maintained byRené Rebe. minised was used by theGNU Project until the GNU Project wrote a new version of sed based on the new GNU regular expression library. The current minised contains some extensions to BSD sed but is not asfeature-rich as GNU sed. Its advantage is that it is very fast and uses little memory. It is used on embedded systems and is the version of sed provided withMinix.[5]
sed reads text, line by line, from aninput stream (such as a file) into an internal buffer called thepattern space. As specified via a script, sed applies commands (calledactions in sed documentation) to the pattern space. Unless directed otherwise, sed then outputs the pattern space (the modified line) and begins the cycle again with the next line. Other end-of-script behaviors are available via command-line options and via script commands such asd to delete the pattern space,q to quit,N to add the next line to the pattern space immediately. Thus, a sed script corresponds to the body of a loop that iterates through the lines of a stream, where the loop itself and the loop variable (the current line number) are implicit and maintained by sed.
Because theiteration over input lines, variables (pattern space and hold space), input and output streams, and default actions (copy line to pattern space, print pattern space) are implicit, it is possible to write terseone-liner programs. For example, the script10q prints the first 10 lines of input, then stops.
Commands accept an optionaladdress argument in terms as a line number orregular expression. The address determines when the command applies. For example,2d would run thed (delete) command on the second input line, while/^ /d would delete all lines beginning with a space. A separate special buffer, thehold space, may be used by a few commands to hold and accumulate text between cycles. The language provides only two variables (the "hold space" and the "pattern space") andGOTO-like branching functionality; nevertheless, the language isTuring-complete,[6][7] andesoteric sed scripts exist for games such assokoban,arkanoid,[8]chess,[9] andtetris.[10]
sed supports regular expression syntax for matching input text to a pattern. For example, the script'/^ *$/d' uses thed command to filter out lines that only contain spaces, or only contain the end of line character.
Supported regular expressionmetacharacters include:
^)$)*)+)?).)The original motivation for sed was substitution.[4] The following command line uses a substitution command.
sed's/regexp/replacement/g'inputFileName>outputFileNameThes stands for substitute, and theg stands for global, which means that all matching occurrences in the line would be replaced. The regular expression pattern to be searched is placed after the first delimiting symbol (slash here) and the replacement follows the second symbol. Slash (/) is the conventional symbol, originating in the character for "search" in ed, but any other could be used to make syntax more readable if it does not occur in the pattern or replacement; this is useful to avoid "leaning toothpick syndrome".
The substitution command, which originates in search-and-replace in ed, implements simple parsing andtemplating. Theregexp provides both pattern matching and saving text via sub-expressions, while thereplacement can be either literal text, or a format string containing the characters& for "entire match" or the specialescape sequences\1 to\9 for thenth saved sub-expression. For example,sed -r "s/(cat|dog)s?/\1s/g" replaces all occurrences of "cat" or "dog" with "cats" or "dogs", without duplicating an existing "s":(cat|dog) is the 1st (and only) saved sub-expression in the regexp, and\1 in the format string substitutes this into the output.
Flow of control can be managed via alabel (a colon followed by a string) and the branchb or conditional brancht instruction. The commandb FOO moves control to the command following the label ":FOO". Thet instruction only does so if there was a successful substitution since the previoust (or the start of the program, in case of the firstt encountered).
The{ instruction starts ablock of commands up to the matching}. In most cases, the block is conditioned by an address pattern.
sed is often used as afilter in apipeline. In the following command line, theecho command writes "xyz"" to standard output which is piped to sed which replaces "x" with "y" as shown on the second line.
$echoxyz|sed's/x/y/g'yyz
The script can either be specified on thecommand line (-e option) or read from a file (-f option).
When in the command line, quotes around the expression are only necessary if the shell would otherwise not interpret the expression as a single token. However, quotes are usually included for clarity, and are often necessary, notably for whitespace (e.g.,'s/x x/y y/'). Most often single quotes are used, to avoid having the shell interpret$ as a shell variable. Double quotes are used, such as"s/$1/$2/g", to allow the shell to substitute for a command line argument or other shell variable.
A script can be stored as a file; one command per line. Using a script file avoids problems with shell escaping and substitution. The following command line uses the-f option to select use of commands from filesubst.sed.
sed-fsubst.sedinputFileName>outputFileName
A script file may be made directly executable via the command line by including ashebang prefix followed by commands. For example:
#!/bin/sed -fs/x/y/g
The file (subst.sed) may be made executable via a command like:
chmod+xsubst.sed
The file may then be executed from the command line as:
subst.sedinputFileName>outputFileName
The in-place option (-i), introduced in GNU sed, allows for modifying the input file. A temporary output file is created and then the original file is replaced with the temporary file.
To replace every occurrence of "yourpassword" in a file with "REDACTED":
s/yourpassword/REDACTED/g
To delete any line containing the word "yourword":
/yourword/d
To delete all instances of the word "yourword":
s/yourword//g
To delete two words from a file:
s/firstword//gs/secondword//g
To express the previous example on one line, such as when entering at the command line, one may join two commands via the semicolon:
s/firstword//g;s/secondword//g
The scriptN;s/\n//;P;D removes newlines from sentences where the second line starts with one space. This demonstrates line merging which is notable since sed usually operates on each line in isolation. It can be explained as:
N) add the next line to the pattern space;s/\n / /) find a new line followed by a space, replace with one space;P) print the top line of the pattern space;D) delete the top line from the pattern space and run the script again.For input text:
This is my dog, whose name is Frank.This is my fish,whose name is George.This is my goat, whose name is Adam.
The output is:
This is my dog, whose name is Frank.This is my fish,whose name is George.This is my goat, whose name is Adam.
While simple and limited, sed is sufficiently powerful for a large number of purposes. For more sophisticated processing, more powerful languages such asAWK orPerl are used instead. These are particularly used if transforming a line in a way more complicated than a regex extracting and template replacement, though arbitrarily complicated transforms are in principle possible by using the hold buffer.
Conversely, for simpler operations, specialized Unix utilities such asgrep (print lines matching a pattern),head (print the first part of a file),tail (print the last part of a file), andtr (translate or delete characters) are often preferable. For the specific tasks they are designed to carry out, such specialized utilities are usually simpler, clearer, and faster than a more general solution such as sed.
The ed/sed commands and syntax continue to be used in descendent programs, such as the text editorsvi andvim. An analog to ed/sed issam/ssam, where sam is thePlan 9 editor, and ssam is a stream interface to it, yielding functionality similar to sed.
A while later a demand arose for another special-purpose program, gres, for substitution: g/re/s. Lee McMahon undertook to write it, and soon foresaw that there would be no end to the family: g/re/d, g/re/a, etc. As his concept developed it became sed…
sed – Shell and Utilities Reference,The Single UNIX Specification, Version 5 fromThe Open Groupsed(1) – Plan 9 Programmer's Manual, Volume 1| File system | |
|---|---|
| Processes | |
| User environment | |
| Text processing | |
| Shell builtins | |
| Searching | |
| Documentation | |
| Software development | |
| Miscellaneous | |
| File system | |
|---|---|
| Processes | |
| User environment | |
| Text processing | |
| Shell builtins | |
| Networking | |
| Searching | |
| Software development | |
| Miscellaneous | |
| International | |
|---|---|
| National | |
