Incomputing,batch processing is the running of asoftwarejob in an automated and unattended way. A user schedules a job to run and then waits for a processing system to run it. Typically, a job is scheduled to run at a configuredtime of day or when anevent occurs or whencomputer resources are available.
The term "batch processing" originates in the traditional classification ofmethods of production asjob production (one-off production),batch production (production of a "batch" of multiple items at once, one stage at a time), andflow production (mass production, all stages in process at once).
Early computers were capable of running only one program at a time. Each user had sole control of the machine for a scheduled period of time. They would arrive at the computer with program and data, often onpunched paper cards and magnetic or paper tape, and would load their program, run and debug it, and carry off their output when done.
As computers became faster the setup and takedown time became a larger percentage of available computer time. Programs calledmonitors, the forerunners ofoperating systems, were developed which could process a series, or "batch", of programs, often frommagnetic tape prepared offline. The monitor would be loaded into the computer and run the first job of the batch. At the end of the job it would regain control and load and run the next until the batch was complete. Often the output of the batch would be written to magnetic tape and printed or punched offline. Examples of monitors were IBM'sFortran Monitor System, SOS (Share Operating System), and finallyIBSYS for IBM's709x systems in 1960.[1][2]
Third-generation computers[clarification needed][3] capable ofmultiprogramming began to appear in the 1960s. Instead of running one batch job at a time, these systems can have multiple batch programs running at the same time in order to keep the system as busy as possible. One or more programs might be awaiting input, one actively running on the CPU, and others generating output. Instead of offline input and output, programs calledspoolers read jobs from cards, disk, or remote terminals and place them in ajob queue to be run. In order to preventdeadlocks thejob scheduler needs to know each job's resource requirements—memory, magnetic tapes, mountabledisks, etc., so various scripting languages were developed to supply this information in a structured way. Probably the most well-known is IBM'sJob Control Language (JCL). Job schedulers select jobs to run according to a variety of criteria, including priority, memory size, etc.Remote batch is a procedure for submitting batch jobs from remote terminals, often equipped with apunch card reader and aline printer.[4] Sometimesasymmetric multiprocessing is used to spool batch input and output for one or more large computers using an attached smaller and less-expensive system, as in the IBM System/360Attached Support Processor.[a]
The first general purpose time sharing system,Compatible Time-Sharing System (CTSS), was compatible with batch processing. This facilitated transitioning from batch processing tointeractive computing.[5]
From the late 1960s onwards, interactive computing such as via text-basedcomputer terminal interfaces (as inUnix shells orread-eval-print loops), and latergraphical user interfaces became common. Non-interactive computation, both one-off jobs such as compilation, and processing of multiple items in batches, became retrospectively referred to asbatch processing, and the termbatch job (in early use often "batchof jobs") became common. Early use is particularly found at theUniversity of Michigan, around theMichigan Terminal System (MTS).[6]
Although timesharing did exist, its use was not robust enough for corporate data processing; none of this was related to the earlierunit record equipment, which was human-operated.
Non-interactive computation remains pervasive in computing, both for general data processing and for system "housekeeping" tasks (usingsystem software). A high-level program (executing multiple programs, with some additional "glue" logic) is today most often called ascript, and written inscripting languages, particularlyshell scripts for system tasks; inIBM PC DOS andMS-DOS this is instead known as abatch file. That includesUNIX-based computers,Microsoft Windows,macOS (whose foundation is theBSD Unix kernel), and evensmartphones. A running script, particularly one executed from an interactivelogin session, is often known as ajob, but that term is used very ambiguously.
"There is no direct counterpart to z/OS batch processing in PC or UNIX systems. Batch jobs are typically executed at a scheduled time or on an as-needed basis. Perhaps the closest comparison is with processes run by anat orcron command in UNIX, although the differences are significant."[7]
Batch applications are still critical in most organizations in large part because many common business processes are amenable to batch processing. While online systems can also function when manual intervention is not desired, they are not typically optimized to perform high-volume, repetitive tasks. Therefore, even new systems usually contain one or more batch applications for updating information at the end of the day, generating reports, printing documents, and other non-interactive tasks that must complete reliably within certain business deadlines.
Some applications are amenable to flow processing, namely those that only need data from a single input at once (not totals, for instance): start the next step for each input as it completes the previous step. In this case flow processing lowerslatency for individual inputs, allowing them to be completed without waiting for the entire batch to finish. However, many applications require data from all records, notably computations such as totals. In this case the entire batch must be completed before one has a usable result: partial results are not usable.
Modern batch applications make use of modern batch frameworks such as Jem The Bee,Spring Batch[8] or implementations ofJSR 352[9] written forJava, and other frameworks for other programming languages, to provide thefault tolerance andscalability required for high-volume processing. In order to ensure high-speed processing, batch applications are often integrated withgrid computing solutions topartition a batch job over a large number of processors, although there are significant programming challenges in doing so. High volume batch processing places particularly heavy demands on system and application architectures as well. Architectures that feature stronginput/output performance and verticalscalability, including modernmainframe computers, tend to provide better batch performance than alternatives.
Scripting languages became popular as they evolved along with batch processing.[10]
Abatch window is "a period of less-intensive online activity",[11] when the computer system is able to run batch jobs without interference from, or with, interactive online systems.
A bank'send-of-day (EOD) jobs require the concept ofcutover, where transaction and data are cut off for a particular day's batch activity ("deposits after 3 PM will be processed the next day").
As requirements for online systems uptime expanded to supportglobalization, theInternet, and other business needs, the batch window shrank[12][13] and increasing emphasis was placed on techniques that would require online data to be available for a maximum amount of time.
Thebatch size refers to the number of work units to be processed within one batch operation. Some examples are:
TheIBM mainframez/OSoperating system or platform has arguably the most highly refined and evolved set of batch processing facilities owing to its origins, long history, and continuing evolution. Today such systems commonly support hundreds or even thousands of concurrent online and batch tasks within a singleoperating system image. Technologies that aid concurrent batch and online processing includeJob Control Language (JCL), scripting languages such asREXX, Job Entry Subsystem (JES2 andJES3),Workload Manager (WLM), Automatic Restart Manager (ARM), Resource Recovery Services (RRS),IBM Db2 data sharing,Parallel Sysplex, unique performance optimizations such asHiperDispatch,I/O channel architecture, and several others.
The Unix programscron,at, andbatch (todaybatch is a variant ofat) allow for complex scheduling of jobs. Windows has ajob scheduler. Mosthigh-performance computingclusters use batch processing to maximize cluster usage.[15]
IBSYS was an operating system for the 7090 that evolved from SOS (SHARE Operating System)
CTSS was called "compatible" in the sense thatFMS could be run in B-core as a "back-ground" user, nearly as efficiently as on a bare machine, and also because programs compiled for FMS batch could be loaded and executed in the "foreground" time-sharing environment (with some limitations). ... This feature allowed the Computation Center to make the transition from batch to timesharing gradually
JSR 352, the open standard specification for Java batch processing. ... The programming languages used evolved over time based on what was available
a multi-user, shared and smart batch processing system improves the scale ..... MostHPC clusters are in Linux