Insoftware engineering, apipeline consists of a chain of processing elements (processes,threads,coroutines,functions,etc.), arranged so that the output of each element is the input of the next. The concept is analogous to a physicalpipeline. Usually some amount ofbuffering is provided between consecutive elements. The information that flows in these pipelines is often astream ofrecords,bytes, orbits, and the elements of a pipeline may be calledfilters. This is also called thepipe(s) and filtersdesign pattern which ismonolithic. Its advantages are simplicity and low cost while its disadvantages are lack ofelasticity,fault tolerance andscalability.[1] Connecting elements into a pipeline is analogous tofunction composition.
Narrowly speaking, a pipeline is linear and one-directional, though sometimes the term is applied to more general flows. For example, a primarily one-directional pipeline may have some communication in the other direction, known as areturn channel orbackchannel, as inthe lexer hack, or a pipeline may be fully bi-directional. Flows with one-directional trees anddirected acyclic graph topologies behave similarly to linear pipelines. The lack of cycles in such flows makes them simple, and thus they may be loosely referred to as "pipelines".
Pipelines are often implemented in amultitaskingOS, by launching all elements at the same time as processes, and automatically servicing the data read requests by each process with the data written by the upstream process. This can be called amultiprocessed pipeline. In this way, thescheduler will naturally switch the CPU among the processes so as to minimize its idle time. In other common models, elements are implemented as lightweight threads or as coroutines to reduce the OS overhead often involved with processes. Depending on the OS, threads may be scheduled directly by the OS or by a thread manager. Coroutines are always scheduled by a coroutine manager of some form.
Read and write requests are usually blocking operations. This means that the execution of the source process, upon writing, is suspended until all data can be written to the destination process. Likewise, the execution of the destination process, upon reading, is suspended until at least some of the requested data can be obtained from the source process. This cannot lead to adeadlock, where both processes would wait indefinitely for each other to respond, since at least one of the processes will soon have its request serviced by the operating system, and continue to run.
For performance, most operating systems implementing pipes use pipebuffers, which allow the source process to provide more data than the destination process is currently able or willing to receive. Under most Unixes and Unix-like operating systems, a special command is also available, typically called "buffer", that implements a pipe buffer of potentially much larger and configurable size. This command can be useful if the destination process is significantly slower than the source process, but it is desired that the source process complete its task as soon as possible. E.g., if the source process consists of a command which reads an audio track from aCD and the destination process consists of a command which compresses thewaveform audio data to a format likeMP3. In this case, buffering the entire track in a pipe buffer would allow the CD drive to spin down more quickly, and enable the user to remove the CD from the drive before the encoding process has finished.
Such a buffer command can be implemented usingsystem calls for reading and writing data. Wastefulbusy waiting can be avoided by using facilities such aspoll orselect ormultithreading.
Some notable examples of pipeline software systems include:
CMS Pipelines is a port of the pipeline idea toVM/CMS andz/OS systems. It supports much more complex pipeline structures than Unix shells, with steps taking multiple input streams and producing multiple output streams. (Such functionality is supported by the Unix kernel, but few programs use it as it makes for complicated syntax and blocking modes, although some shells do support it via arbitraryfile descriptor assignment).
Traditional application programs on IBM mainframe operating systems have no standard input and output streams to allow redirection or piping. Instead of spawning processes with external programs, CMS Pipelines features a lightweight dispatcher to concurrently execute instances of more than 200 built-in programs that implement typical UNIX utilities and interface to devices and operating system services. In addition to the built-in programs, CMS Pipelines defines a framework to allow user-writtenREXX programs with input and output streams that can be used in the pipeline.
Data on IBM mainframes typically resides in arecord-oriented filesystem and connected I/O devices operate in record mode rather than stream mode. As a consequence, data in CMS Pipelines is handled in record mode. For text files, a record holds one line of text. In general, CMS Pipelines does not buffer the data but passes records of data in a lock-step fashion from one program to the next. This ensures a deterministic flow of data through a network of interconnected pipelines.
Beside byte stream-based pipelines, there are also object pipelines. In an object pipeline, processing elements output objects instead of text.PowerShell includes an internal object pipeline that transfers.NET objects between functions within the PowerShell runtime.Channels, found in theLimbo programming language, are other examples of this metaphor.
Graphical environments such asRISC OS andROX Desktop also use pipelines. Rather than providing a savedialog box containing afile manager to let the user specify where aprogram should write data, RISC OS and ROX provide a save dialog box containing anicon (and a field to specify the name). The destination is specified bydragging and dropping the icon. The user can drop the icon anywhere an already-saved file could be dropped, including onto icons of other programs. If the icon is dropped onto a program's icon, it is loaded and the contents that would otherwise have been saved are passed in on the new program's standard input stream.
For instance, a user browsing theworld-wide web might come across a .gz compressed image which they want to edit and re-upload. Using GUI pipelines, they could drag the link to their de-archiving program, drag the icon representing the extracted contents to theirimage editor, edit it, open the save as dialog, and drag its icon to their uploading software.
Conceptually, this method could be used with a conventional save dialog box, but this would require the user's programs to have an obvious and easily accessible location in the filesystem. As this is often not the case, GUI pipelines are rare.
The name "pipeline" comes from a rough analogy with physical plumbing in that a pipeline usually[2] allows information to flow in only one direction, like water often flows in a pipe.
Pipes andfilters can be viewed as a form offunctional programming, using byte streams as data objects. More specifically, they can be seen as a particular form ofmonad forI/O.[3]
The concept of pipeline is also central to theCocoon web developmentframework or to anyXProc (the W3C Standards) implementations, where it allows a source stream to be modified before eventual display.
This pattern encourages the use of text streams as the input and output of programs. This reliance on text has to be accounted when creatinggraphic shells to text programs.