US20170264564A1 - Systems, methods, and devices for dynamic resource monitoring and allocation in a cluster system - Google Patents

Abstract

In an embodiment, the systems, methods, and devices disclosed herein comprise a computer resource monitoring and allocation system. In an embodiment, the resource monitoring and allocation system can be configured to allocate computer resources that are available on various nodes of a cluster to specific jobs and/or sub-jobs and/or tasks and/or processes.

Description

CROSS-REFERENCE TO RELATED APPLICATIONS
• The present application is a continuation application of U.S. patent application Ser. No. 14/053,089, filed Oct. 14, 2013 and titled SYSTEMS, METHODS, AND DEVICES FOR DYNAMIC RESOURCE MONITORING AND ALLOCATION IN A CLUSTER SYSTEM, which claims the benefit under 35 U.S.C. 119(c) to U.S. Provisional Application No. 61/841,007, filed Jun. 28, 2013 and titled SYSTEMS, METHODS, AND DEVICES FOR DYNAMIC RESOURCE MONITORING AND ALLOCATION IN A CLUSTER SYSTEM, U.S. Provisional Application No. 61/841,074, filed Jun. 28, 2013 and titled SYSTEMS, METHODS, AND DEVICES FOR DYNAMIC RESOURCE MONITORING AND ALLOCATION IN A CLUSTER SYSTEM, U.S. Provisional Application No. 61/841,127, filed Jun. 28, 2013 and titled SYSTEMS, METHODS, AND DEVICES FOR DYNAMIC RESOURCE MONITORING AND ALLOCATION IN A CLUSTER SYSTEM, U.S. Provisional Application No. 61/841,025, filed Jun. 28, 2013 and titled SYSTEMS, METHODS, AND DEVICES FOR DYNAMIC RESOURCE MONITORING AND ALLOCATION IN A CLUSTER SYSTEM, U.S. Provisional Application No. 61/841,106, filed Jun. 28, 2013 and titled SYSTEMS, METHODS, AND DEVICES FOR DYNAMIC RESOURCE MONITORING AND ALLOCATION IN A CLUSTER SYSTEM, and U.S. Provisional Application No. 61/841,061, filed Jun. 28, 2013 and titled SYSTEMS, METHODS, AND DEVICES FOR DYNAMIC RESOURCE MONITORING AND ALLOCATION IN A CLUSTER SYSTEM. The foregoing applications are hereby incorporated herein by reference in their entirety, including specifically but not limited to the systems and methods relating to dynamic resource monitoring and allocation in a cluster computer system.
BACKGROUND
• Field
• The embodiments of the disclosure generally relate to computer clusters, and more particularly to systems, methods, and devices for the efficient management of resources of computer clusters.
• Description of the Related Art
• In general, a computer cluster comprises a set of connected computers that communicate and work together in order to act as a single system. A computer cluster can comprise several types of components, including a fast local area network, a plurality of computers referred to generally as nodes, and operating systems running on each node. An advantage of computer clusters is the ability to utilize low cost computer servers in order to achieve high performance distributed computing that was only previously available through the use of highly expensive main frame computers. A disadvantage of computer clusters is the increased operational challenges that arise when adding more and more nodes to the computer cluster. Generally, in order to manage the operational complexities of vast numbers of nodes in a computer cluster, a software layer can be employed to manage the activities of the various computing nodes in order to allow the users to treat the computer cluster as a single computing unit.
• Typically, the software layer for organizing the nodes and orchestrating the activities on the nodes can be responsible for receiving jobs to be processed by the computer cluster. In many instances, the software layer will divide the job into several tasks or sub-jobs or processes or job processes to be processed by various nodes in the computer cluster. Generally, the software layer is responsible for distributing these tasks and or sub-jobs or processes or job processes to the available nodes in the computer cluster. This distribution of tasks or sub-jobs or processes or job processes to the various available nodes in a computer cluster can lead to performance degradations and/or resource underutilization.
SUMMARY
• Various embodiments of the present invention relate to the utilization of computer cluster technology, which generally refers to a plurality of computer servers connected to each other through a fast network connection. In an embodiment, the systems, methods, and devices disclosed herein comprise a computer resource monitoring and allocation system. In an embodiment, the resource monitoring and allocation system can be configured to allocate computer resources that are available on various nodes of a cluster to specific jobs and/or sub-jobs and/or tasks and/or processes. For example, the system can be configured to control network utilization across two or more nodes wherein the system can reduce network utilization of a first job that is being performed on a first node in order to allocate additional network capacity to a second job or sub-job that is being performed on a second node. In another example, the system can be configured to reduce the amount of CPU usage on a single particular node that a first job or sub-job is using on the node in order to allocate additional CPU capacity to a second job or sub-job or process or job process operating on the node.
• Generally, the systems and methods herein are configured to process large amounts of data received from the various nodes in a cluster in order to generate, in real time or in substantially real time or on a periodic basis, instructions for allocating computer resources on the nodes in the cluster. In an embodiment, the system is configured to dynamically tune or adjust up or down access to or availability of the computer resources provided for on particular nodes in order to ensure that user-defined goals are satisfied and/or to ensure that the cluster is operating efficiently. In general, the system is configured to continuously and/or periodically receive data relating to resource allocation and/or usage at particular nodes. Additionally, the system can be configured to continuously and/or periodically generate instructions for allocating computer resources at particular nodes for specific jobs and/or sub-jobs being performed on the nodes of the cluster. The continuous and dynamic changing of resource allocations on a computer cluster in combination with the continuous and/or periodic monitoring of the resource allocations and/or usage on particular nodes of a cluster results in thousands of transactions over a short period of time, and makes it impossible for a human being to perform such tasks entirely in a person's mind or by a person using a writing instrument and paper.
• Through the continuous monitoring of the nodes in the cluster and through the dynamic allocation of computer resources on particular nodes, the system can be configured to ensure that jobs and/or sub-jobs that have high prioritization are completed as soon as possible and/or by a user-defined time period. The systems, methods, and devices disclosed herein can be utilized in conjunction with specific computer cluster types, such as hadoop clusters, or can be configured to operate with other distributed systems.
• In an embodiment, a hadoop computer cluster comprises a master node computing device comprising a management controller and a supervisor controller, the management controller configured to coordinate parallel processing of data across a plurality of computer system nodes, the supervisor controller configured to coordinate allocation of system resources at particular computer system nodes to particular tasks. The plurality of computer system nodes can be configured to communicate with the supervisor controller and to perform processing of received tasks. In an embodiment, the computer system nodes each comprise: one or more processors configured to perform computing processes on received tasks and an agent controller. In an embodiment, the agent controller is configured to monitor utilization by tasks of system resources of the computer system node, the system resources comprising CPU, disk input/output, network, and memory by the computer system node. In an embodiment, the agent controller is configured to report the monitored system resource utilization to the supervisor in substantially real-time. In an embodiment, the agent controller is configured to generate instructions for controlling utilization by tasks of system resources of the computer system node, the instructions based on data received from the supervisor controller. The supervisor controller can be configured to, based on goals specified by an operator of the hadoop computer cluster and the substantially real-time reporting from a plurality of agent controllers, determine assignment of tasks to respective computer system nodes and/or resource allocations for each task on those nodes such that the operator-specified goals are satisfied and processing capabilities of the hadoop computer cluster are maximized. In an embodiment, the management controller comprises a job tracker. In an embodiment, the management controller comprises a yarn system or yarn resource manager.
• In an embodiment, a supervisor controller is configured to manage system resource allocation for a hadoop computer cluster. The supervisor controller can comprise a management controller interface configured to communicate with a management controller to access data representing an assignment of a plurality of job processes across a plurality of computer system nodes in the hadoop computer cluster, the management controller configured to coordinate parallel processing of data across a plurality of computer system nodes, an agent controller interface configured to communicate with an agent controller, the agent controller configured to transmit to the agent controller interface data representing utilization of system resources by the plurality of job processes operating on a particular computer system node, a system resource allocation engine configured to dynamically determine system resource allocations for particular job processes operating on particular computer system nodes, the dynamic determination generated by the system resource engine based on the data representing utilization of system resource by the plurality of job processes operating on the particular system node; and the agent controller interface configured to generate data for transmission to the agent controller of a particular computer system node based on the dynamic determination generated by the system resource engine, the data configured to instruct the agent controller to allocate a level of system resources to a particular job process operating on the particular computer system node.
• In an embodiment, a computer cluster comprises a management computing device comprising a supervisor controller configured to coordinate processing of a plurality of sub-jobs for a plurality of overall jobs; a plurality of computer system nodes configured to communicate with the supervisor controller and to perform processing of received sub-jobs, the computer system nodes each comprising: one or more processors configured to perform computing processes on received sub-jobs; an agent controller configured to: monitor utilization of system resources of the computer system node, the system resources comprising CPU, electronic storage input/output, network, and memory by the computer system node; report the monitored system resource utilization to the supervisor controller in substantially real-time; wherein the supervisor controller is configured to, based on goals specified by an operator of the computer cluster and the substantially real-time reporting from a plurality of agent controllers, determine assignment of system resource allocations for each sub-job on those nodes such that the operator-specified goals are satisfied and processing capabilities of the computer cluster are substantially utilized, wherein the management computing device and the plurality of computer system nodes comprise a computer processor and an electronic storage medium.
• In an embodiment, a supervisor controller is configured to manage system resource allocation for a computer cluster. The supervisor controller comprises an agent controller interface configured to communicate with an agent controller, the agent controller configured to transmit to the agent controller interface data representing utilization of system resources by a plurality of sub-jobs operating on a particular computer system node, a system resource allocation engine configured to dynamically determine system resource allocations for particular sub-jobs operating on particular computer system nodes, the dynamic determination generated by the system resource engine based on the data representing utilization of system resource by the plurality of sub-jobs operating on the particular system node; and the agent controller interface configured to generate data for transmission to the agent controller of a particular computer system node based on the dynamic determination generated by the system resource engine, the data configured to instruct the agent controller to allocate a level of system resources to a particular sub-job operating on the particular computer system node.
• For purposes of this summary, certain aspects, advantages, and novel features of the invention are described herein. It is to be understood that not necessarily all such advantages may be achieved in accordance with any particular embodiment of the invention. Thus, for example, those skilled in the art will recognize that the invention may be embodied or carried out in a manner that achieves one advantage or group of advantages as taught herein without necessarily achieving other advantages as may be taught or suggested herein.
BRIEF DESCRIPTION OF THE DRAWINGS
• The foregoing and other features, aspects and advantages of the embodiments of the invention are described in detail below with reference to the drawings of various embodiments, which are intended to illustrate and not to limit the invention. The drawings comprise the following figures in which:
• FIG. 1 is an embodiment of a schematic diagram illustrating a computer cluster.
• FIG. 2 is an embodiment of a schematic diagram illustrating a computer cluster comprising an embodiment of a dynamic monitoring and/or resource allocation system.
• FIG. 2A is an embodiment of a schematic diagram illustrating a computer cluster comprising an embodiment of a dynamic monitoring and/or resource allocation system.
• FIG. 2B is an embodiment of a schematic diagram illustrating a computer cluster comprising an embodiment of a dynamic monitoring and/or resource allocation system.
• FIG. 3 is a flowchart depicting an embodiment of a process for dynamically monitoring and/or allocating resources across a computer cluster.
• FIG. 3A is a flowchart depicting an embodiment of a process for dynamically monitoring and/or allocating resources across a computer cluster.
• FIG. 4 is an embodiment of a schematic diagram illustrating a computer cluster comprising an embodiment of a dynamic monitoring and/or resource allocation system.
• FIG. 5 is a flowchart depicting an embodiment of a process for monitoring and/or allocating cluster resources, such as RAM, network usage, CPU usage, and disk I/O usage.
• FIG. 6 is a block diagram depicting a high level overview of an embodiment of a distributor system.
• FIG. 7 is a flow chart depicting an embodiment of a process for a distributor as illustrated inFIG. 6.
• FIG. 8A is a block diagram depicting a high level overview of an embodiment of virtual clusters.
• FIG. 8B is a block diagram depicting a high level overview of an embodiment of virtual clusters.
• FIG. 8C is a block diagram depicting a high level overview of an embodiment of virtual clusters.
• FIG. 8D is a block diagram depicting a high level overview of an embodiment of virtual clusters.
• FIG. 8E is a block diagram depicting a high level overview of an embodiment of virtual clusters.
• FIG. 9 is a flowchart depicting an embodiment of a process for processing jobs using a virtual cluster.
• FIG. 10 is a flowchart depicting an embodiment of a process for processing jobs using a virtual cluster.
• FIG. 11 is a flowchart depicting an embodiment of a process for processing jobs using job groups.
• FIG. 12 is a flowchart depicting an embodiment of a process for monetizing and/or budget accounting for resources on a computer cluster.
• FIG. 13 is a block diagram depicting a high level overview of an embodiment of a computer cluster comprising heterogeneous nodes.
• FIG. 14 is a flowchart depicting an embodiment of a process for processing jobs utilizing a heterogeneous computer cluster.
• FIG. 15 is a schematic diagram illustrating an embodiment of utilizing job histories for improving resource allocation of a computer cluster.
• FIG. 16 is a flowchart depicting an embodiment of a process for generating reports relating to hardware modifications and/or additions to a computer cluster.
• FIG. 17 is a flowchart depicting an embodiment of a process for generating reports relating to resource reallocation on a computer cluster.
• FIG. 17A is a flowchart depicting an embodiment of a process for determining resource reallocation levels for application to jobs or sub-jobs.
• FIG. 18 is a block diagram depicting a high level overview of an embodiment of a computer cluster comprising a dynamic monitoring and/or resource allocation system.
• FIG. 19 is a block diagram depicting an embodiment of a computer hardware system configured to run software for implementing one or more embodiments of the dynamic monitoring and/or resource allocation systems disclosed herein.
DETAILED DESCRIPTION OF THE EMBODIMENTS
• Although several embodiments, examples and illustrations are disclosed below, it will be understood by those of ordinary skill in the art that the inventions described herein extend beyond the specifically disclosed embodiments, examples, and illustrations, and include other uses of the inventions and obvious modifications and equivalents thereof. Embodiments of the inventions are described with reference to the accompanying figures, wherein like numerals refer to like elements throughout. The terminology used in the description presented herein is not intended to be interpreted in any limiting or restrictive manner simply because it is being used in conjunction with a detailed description of certain specific embodiments of the inventions. In addition, embodiments of the inventions can comprise several novel features and no single feature is solely responsible for its desirable attributes or is essential to practicing the inventions herein described.
• In general, computer clusters comprise a plurality of computer servers that are connected to each other through a network connection. In many instances, the network connection is a fast network connector such that all of the computer servers in the cluster can communicate with each other quickly and efficiently. For example, a computer cluster can comprise a number of low cost commercially available off-the-shelf computers connected through a fast local area network (LAN). In general, a computer cluster can comprise a master node and a plurality of slave nodes. The master node can be configured to coordinate the activities of the slave nodes. In an embodiment, the computer hardware for a master node and for slave nodes are the same or are substantially the same, and are only distinguishable by the assigned roles each computer server receives when the cluster has been created. In an embodiment, a cluster can comprise one or more master nodes that coordinate the activities of various slave nodes.
• To implement the coordination between the master node(s) and the various slave nodes, a computer cluster can comprise middleware software that operates on each node and that allows communication and coordination between the nodes in order for the computer cluster to act like a single cohesive computing unit. In general, a master node can be configured to divide jobs and/or processes into smaller jobs and/or processes to be executed or processed on one or more slave nodes in order to efficiently and quickly complete the job. After transmitting a sub-job to a slave node, a master node generally does not monitor the performance of the processing of the sub-job. In some cases, the master node will only determine whether a sub-job has been completed by a designated slave node.
• Accordingly, there are several disadvantages for typical cluster configurations. For example, by not verifying or monitoring the status of a sub-job that is being processed by a slave node, a computer cluster system may not be able to process a particular job within a time frame desired by the user. Further, by not monitoring and verifying the progress of a sub-job, the cluster system runs the risk of slowing down high priority jobs when the master node adds additional jobs to a particular slave node. For example, a computer cluster can be configured to run a job for generating reports on a daily basis. In an embodiment, the computer cluster can be configured to receive additional jobs during the period in which the cluster is working on the job for generating the periodic reports. In such an example, the master node can be configured to divide the additional job into sub-jobs for further processing by various nodes in the cluster. These additional sub-jobs to be processed by the slave nodes can in some instances slow down the completion of the job for generating the periodic reports.
• Without monitoring the progress and/or completion of the job and/or a plurality of jobs for generating the reports, the computer cluster cannot determine whether the addition of such ad hoc jobs that are added to a node are slowing down the time sensitive periodic report generation job. Accordingly, it can be advantageous for a cluster system to monitor the completion progress of a particular job and/or a plurality of jobs in order to ensure that such jobs are completed on a timely basis pursuant to the specified goals of a user.
• Typical computer clusters cannot efficiently handle the addition of ad hoc jobs without affecting the performance of jobs that are regularly scheduled for processing by the cluster. Additionally, typical clusters cannot determine whether a particular node is being overloaded by jobs assigned to the slave node. The overutilization of resources on a slave node can cause the slave node to experience performance degradations.
• For example, if the sub-jobs assigned to a slave node required the use of RAM that exceeds the amount of physical RAM on the node, the slave node can start to utilize the hard drive to compensation for the lack of RAM. Writing to a hard drive in order to compensate for the lack of RAM can cause the slave node to experience significant performance delays because writing to a hard drive is slower than writing to physical RAM or flash memory. The writing to and reading from a hard drive in lieu of RAM or flash memory can cause severe performance degradations, which can cause “thrashing” of the computer server, requiring the computer server to be rebooted.
• As an example, if multiple sub-jobs assigned to a slave node requested more disk I/O accesses per unit time than the node can support, one or more of the tasks can be slowed down dramatically waiting for disk I/O access. In some cases, the task(s) that may be slowed down could be the high-priority regularly scheduled task(s), being slowed down by the ad hoc jobs.
• Without the active and dynamic monitoring of the resources on a slave node with respect to the jobs and/or sub-jobs assigned to the slave node, the computer cluster cannot account for resource overloads on a particular slave node.
• Similarly, without monitoring the resource utilization on the slave nodes within a cluster, the system cannot determine which slave nodes are being underutilized. For example, certain sub-jobs may not require significant amounts of RAM in order to be processed. In certain circumstances, it can be advantageous for the cluster to assign additional sub-jobs to the slave node in order to utilize the available RAM on the slave node. The additional assignment of jobs and/or sub-jobs for the slave node can ensure that the resources of the slave node are being fully utilized.
• Typical clusters also do not have the ability to determine which jobs, sub-jobs, processes and/or users are utilizing the cluster to a greater extent than other jobs and/or users. For example, typical cluster systems cannot determine whether a human resource group is responsible for a greater utilization of the cluster relative to a legal department of an organization. By not monitoring the resource utilization of sub-jobs on slave nodes, the cluster system cannot determine how much of the resources of the cluster are being utilized by particular jobs and/or users and/or groups of users. It can be advantageous to determine the percent usage of the cluster by a particular job and/or user and/or groups of users in order to bill such utilization to a particular job and/or user and/or group of users and/or company department or the like. For example, if the system is configured to determine that a human resource department utilizes 50% of the resources of the cluster, the system can be configured to bill or perform a budgetary accounting that causes the human resources department of a company to be responsible for 50% of the costs for maintaining the cluster for the company.
• Another drawback to typical computer clusters is the system cannot generally determine what additional hardware should be added to the cluster in order to efficiently process the jobs and/or sub-jobs being sent to the cluster for processing. Without monitoring the performance of jobs and/or sub-jobs being processed by specific slave nodes, the computer cluster cannot determine whether bottlenecks exist in the computer cluster, wherein the bottlenecks prevent the completion of a job and/or sub-job in a timely manner. For example, a system that can be configured to monitor and determine the resource utilization at particular slave nodes by particular sub-jobs, can be configured to identify overutilization of RAM in the cluster. Based on the determination that RAM utilization across the cluster is consistently above the available RAM capacity of the cluster, the system can be configured to output a message to the user or operator of the cluster to add additional slave nodes with increased RAM capacity.
• In general, typical computer clusters require that the computer servers making up the cluster be of the same or similar type of machines. Accordingly, in many instances computer clusters cannot generally comprise heterogeneous machine types. For example, many computer clusters cannot efficiently operate in an environment where some of the computer servers have faster CPU processors than other computers in the cluster. For example, without monitoring the available resources on particular slave nodes, the cluster system cannot determine that certain slave nodes with faster CPU processors can be configured to take on additional sub-jobs as compared to other slave nodes in the cluster that have slower CPU processors that can take on only a limited number of sub-jobs. Therefore, it can be advantageous for a computer cluster to dynamically monitor and allocate resources on a particular slave node in order to allow a cluster system to fully utilize heterogeneous computer servers in a cluster.
• The foregoing shortcomings and disadvantages of typical computer clusters can be addressed by the resource monitoring and allocation systems disclosed herein. In an embodiment, the system can be configured to monitor, track, and dynamically control system resources at a per-task/per-process level, an overall per-node level, and an overall per-cluster level in order to maximize the efficiency and/or utilization of the resources provided for by the nodes in the cluster. The system resources include but are not limited to CPU usage, RAM usage (both actual usage and current max limits as set via the virtual machine or kernel), network bandwidth usage, and disk I/O usage (read bandwidth, write bandwidth, and number of disk operations/seeks). In an embodiment, the system can be configured to monitor, track, and dynamically control at a per-task/per-process level, an overall per-node level, and an overall per-cluster level several fine-grained resources including but not limited to:
• Disk I/O on a per-device basis; for example, a node with multiple physical disk drives will generally have read/write bandwidth, seeks, and operations monitored/controlled for each of the physical disk drives as well as overall.
• Network bandwidth broken down by type of access; for example, bandwidth may be monitored/controlled separately for local rack network access (to the other nodes sharing the same top-of-rack switch), remote rack access (to other nodes in the same cluster but on a different rack, which can mean using central switch/network bandwidth), and off-cluster access (to network locations outside the cluster, such as an external database or service).
• Distributed filesystem (for example, HDFS) access, which can include a combination of local disk I/O, local rack network, and remote rack network. Depending on the kind of access, distributed filesystem usage can actually take up resources from one or more of the local disk, local rack network, and/or remote rack network. Accordingly, in an embodiment this distributed filesystem resource needs to be monitored and controlled along with direct access to these underlying resources.
• Usage of other cluster resources, such as access to the hadoop name node, and the like.
• Usage of off-cluster resources, such as load on an external database, ETL tool, web service, and the like.
• In an embodiment, the resource monitoring and allocation systems can be configured to work in conjunction with the software middleware of a computer cluster. For example, the software middleware of the computer cluster can be configured to operate normally by receiving jobs from a user, analyzing the received job, dividing the received job into sub-jobs, and distributing the sub-jobs across various slave nodes in the cluster for processing. The resource monitoring and allocation system can complement the activities of the software middleware by monitoring the jobs and/or sub-jobs being processed on various slave nodes in the cluster.
• By monitoring the resource utilization of particular jobs and sub-jobs on various slave nodes, the resource monitoring and allocation system can be configured to dynamically reallocate resources on particular slave nodes to particular sub-jobs being processed. The reallocation of resources to particular sub-jobs being processed on particular nodes can allow the computer cluster to operate more efficiently. For example, the resource monitoring and allocation system can be configured to reallocate additional network capacity to high priority sub-jobs in order for the high priority job to be completed on time. By reallocating network capacity to high priority sub-jobs, the resource monitoring and allocation system can be configured to slow down the processing of non-high priority sub-jobs by reducing the amount of network capacity dedicated to the non-priority sub-jobs.
• In an embodiment, the resource monitoring and allocation system can comprise a supervisor controller system that is configured to monitor the overall jobs and/or sub-jobs that were initially processed by the software middleware for assignment and processing by the various slave nodes. For example, the supervisor controller can be configured to determine what resources are being utilized by particular sub-jobs operating on particular slave nodes. Further, the supervisor controller can be configured to determine the overall progress in completing an overall job that has been divided into a plurality of sub-jobs being processed by a plurality of slave nodes. By determining the overall progress for completing a particular job, the supervisor controller can ensure that the overall job is completed to the specifications and/or requirements set forth by a client and/or user. In order to determine the particular resource utilization of certain sub-jobs, the resource monitoring and allocation system can comprise an agent system.
• In an embodiment, the agent system is configured to operate on one or more of the slave nodes in the computer cluster. In an embodiment, the agent system is configured to operate on each of the slave nodes in a computer cluster. In an embodiment, the agent system is configured to operate on a master node. The agent system can be configured to determine the specific resource utilization at a particular node for each of the particular sub-jobs. After determining the resource utilization of a particular sub-job on a particular node, the agent controller system can be configured to transmit the resource utilization data to a supervisor controller system. In an embodiment, the supervisor controller system can be configured to aggregate resource utilization data from a plurality of agent controller systems operating on various nodes in the cluster. The supervisor controller system can be configured to analyze the resource utilization data to determine the status of the cluster and/or how efficiently the cluster is operating. Further, the supervisor controller system can be configured to analyze the resource utilization data to determine whether an overall job is likely to be completed by the specified time goals set forth by a user of the cluster and/or client.
• If the supervisor determines that resources should be reallocated for particular jobs being processed on particular nodes, the supervisor controller system can be configured to generate instructions for transmission to the agent controller system. The agent controller system can be configured to analyze the instructions received from the supervisor controller system in order to generate specific instructions for implementing the resource reallocation on the particular node that the agent controller system has control over. Accordingly, the agent controller system can serve various roles.
• In an embodiment, the agent controller system is configured to monitor resource utilization on a particular node and to determine how each sub-job being processed on the particular node is utilizing resources of the particular node. The agent controller system is also responsible for transmitting and/or reporting the resource utilization data to the supervisor controller system. In an embodiment, the agent controller system is also responsible for implementing or enforcing the resource reallocation instructions received from a supervisor controller system. The agent controller system can also be configured to control the allocation of resources to particular jobs and/or sub-jobs that are being processed on a particular node. Further, the agent controller system can be configured to independently decide whether to reallocate resources of the particular computer node without receiving instructions from the supervisor controller system.
• The resources of the node that are being utilized by the system to complete the jobs and/or sub-jobs include but are not limited to RAM, CPU capacity, network capacity, and disk I/O capacity. For example, an agent system can be configured to operate on a particular slave node that is processing a particular sub-job. The agent system can be configured to determine the amount of CPU capacity, RAM capacity, network capacity, and/or disk I/O capacity that is being utilized by the particular sub-job that is being processed on the particular slave node.
• In an embodiment, the system can be configured to obtain the current resource utilization differently depending on the type of resource. For example, the system can be configured to determine CPU capacity by measuring actual CPU time used via a call to the kernel and/or reading files written by the kernel. In an embodiment, the system can be configured to determine RAM capacity by measuring virtual machine statistics and/or kernel statistics. In an embodiment, the system can be configured to determine network capacity by creating a “wrapper” around the code that actually accesses the network, wherein the “wrapper” is configured to report statistics of network usage. Alternatively, the system can be configured to determine network capacity by using a virtual network interface to intermediate requests to the network, and/or using a “traffic control” command of the kernel or similar kernel-level mechanism to adjust network usage. In an embodiment, the system can be configured to determine disk I/O by measuring one or more of the following: creating a “wrapper” around the code that actually accesses the disk I/O capacity in order to report statistics, and/or using kernel-level controls to adjust disk I/O usage.
• In an embodiment, the agent system can be configured to transmit the resource utilization data for the particular slave node to the supervisor system. In an embodiment, the supervisor system and/or the agent system can be configured to determine whether a reallocation of resources should occur at the particular slave node in order to delay or accelerate the processing of the particular sub-job that is being processed by the particular slave node. For example, the supervisor system can be configured to analyze the resource utilization of the particular sub-job that is being processed by a particular slave node and compare the processing performance to other sub-jobs of the same overall job being processed by other slave nodes operating within the computer cluster.
• In an embodiment, the supervisor system can be configured to reallocate additional CPU capacity to the selected sub-job in order to allow the particular sub-job to be completed within about the same timeframe as other sub-jobs that are being processed by other slave nodes in the computer cluster. By adding the additional CPU capacity to the particular sub-job, the computer cluster can be configured to prevent the particular sub-job from being a bottleneck in the completion of the overall job. By removing the bottleneck, the computer cluster system can be configured to complete the overall job within a user specified time period.
• In an embodiment, the agent controller system can be configured to determine independently from the supervisor controller whether to reallocate resources to a particular sub-job without receiving input from the supervisor controller system. For example, the agent system can be configured to reallocate additional CPU capacity to a particular sub-job being processed on the particular slave node based on determining that the particular sub-job has a higher priority than other sub-jobs being processed by the particular slave node. By adding additional resource capacity to completing the particular sub-job, the particular slave node can decrease the amount of processing time necessary to complete the high priority sub-job. The foregoing examples can also be applied to other resource types, such as but not limited to RAM capacity, network capacity, disk I/O capacity, and the like.
• The supervisor controller system and/or the agent controller system can be configured to control the allocation of resources on a particular node through a variety of methods. For example, the agent controller system can be configured to control the amount of RAM usage by a particular sub-job on a particular node by invoking the kill command in an operating system. The kill command is a function that is provided for in a number of commercially available operating systems. The kill command can be configured to send signals to a running process or processes to request the termination of the process. In an embodiment, the agent controller system can be configured to reduce the amount of RAM utilized by a particular sub-job by sending a kill command to the sub-job thereby eliminating the sub-job's use of any RAM resources in the node.
• Alternatively, the agent controller system can be configured to invoke the JVM (Java Virtual Machine) garbage collection command or other garbage collection command in order to control the RAM usage for a particular sub-job. The JVM garbage collection command or other garbage collection command are generally a form of automatic memory management that can be provided for in computer languages, such as Java, C, C++, and the like. In general, garbage collection commands operate by finding data objects in a program that are no longer in use and by reclaiming the resources used by the data objects no longer in use, the garbage collection commands can reduce the amount of RAM usage on a node. In an embodiment, the agent controller system can be configured to control RAM utilization by a particular sub-job by using the garbage collection command to reduce the amount of RAM and/or to recover RAM resources not utilized by the particular sub-job.
• In an embodiment, the agent controller system can be configured to control RAM usage for a particular sub-job by adjusting a maximum RAM usage limit function in a virtual machine and/or kernel, and/or by adjusting the number of tasks/processes allowed to run on the node through the virtual machine or kernel. In an embodiment, the agent controller system can set the maximum RAM limit for a particular sub-job based on the history of similar sub-jobs. For example, if similar sub-jobs have used no more than 500 megabytes of RAM in past runs, the maximum RAM limit for a sub-job can be set to 500 megabytes, instead of a higher default maximum that is used for sub-jobs in general.
• The ability to control RAM is different from the ability to control CPU usage, network usage, and disk I/O usage. For example, an agent controller system can be configured to slow down or delay a job and/or process in order to reduce or increase the use of network utilization, CPU utilization, and/or disk I/O utilization. However, with respect to RAM, if a program and/or process requires a certain amount of RAM in order to process a sub-job, the agent controller cannot generally negotiate with the process in order to reduce the RAM utilization because the required RAM resources are either provided to the sub-job or the sub-job dies. Accordingly, the agent controller system can be configured to either kill a particular sub-job in order to eliminate the RAM utilization by a particular sub-job, or the agent controller system can be configured to use the garbage collection functionality in order to recapture unused RAM by the process or the sub-job.
• In an embodiment, the agent controller system can be configured to control the amount of network usage utilized by a particular sub-job on a particular node. The agent controller system can be configured to utilize the sleep command in order to reduce the network utilization by a particular sub-job. The sleep command is provided for in operating systems that are commercially available. The sleep command enables a process or program to be suspended or delayed for a specific period of time before the process or program is allowed to execute on the computer node and/or utilize specific resources on the node.
• In an embodiment, the agent controller system can be configured to control the network utilization by invoking the sleep command. The sleep command will force the sub-job to suspend operations and/or processing, which will in turn suspend and/or delay the network utilization by the particular sub-job. In an embodiment, the supervisor controller and/or the agent controller can be configured to generate and/or insert code into a sub-job and/or job wherein the code can invoke a sleep call based on instructions from the supervisor controller and/or the agent controller. Alternatively, the agent controller system can be configured to reduce the network utilization of a sub-job by controlling and/or reducing the bandwidth usage or the amount of bandwidth made available to a sub-job. In an embodiment, the agent controller system can be configured to utilize a traffic shaping utility for controlling the bandwidth that is made available to the particular sub-job. In an embodiment, the system can be configured to control network capacity by creating a “wrapper” around the code that actually accesses the network, wherein the “wrapper” is configured to control network usage by the code.
• Generally, network utilization is a challenging resource to manage. For example, network utilization not only depends upon the amount of network being utilized by a particular sub-job or process, but rather network utilization also depends upon the amount of network utilization that is being used by other sub-jobs and/or processes operating on other parts of the cluster. For example, if a particular first job operating on a first node is utilizing 60% of the network bandwidth that is available for accessing the internet, then a second job being processed by a second node may only have access to the remaining 40% of the network bandwidth for connecting to the internet.
• The second sub-job operating on the second node can only have access to 40% of the network bandwidth notwithstanding the fact that the second job can have 100% access to the local area network from the second node where there are no additional jobs that are being processed on the second node. Accordingly, in order to monitor and allocate network resources, the supervisor controller can be configured to receive resource utilization data from a plurality of agent controller systems in order to determine an aggregate view of network utilization across the cluster. The global knowledge of network utilization can enable the supervisor controller to determine which sub-jobs across the cluster should be reduced in order to ensure that a particular sub-job has sufficient network resources available in order to complete the sub-job.
• In an embodiment, the agent controller system can be configured to control the amount of CPU usage by a particular sub-job on a particular node. In an embodiment, the agent controller system can be configured to utilize the nice functionality provided for in an operating system. The nice functionality is generally provided for in commercially available operating systems. The nice command can enable a process and/or sub-job to have more or less CPU time than other processes or sub-jobs running on the node. The nice command can allow for assigning different processes and/or sub-jobs with a priority level, and based on the priority level that has been assigned to the process and/or sub-job, the CPU can provide more or less processing time to the particular process or sub-job. In an embodiment, the agent controller system can be configured to reduce the CPU usage of a particular sub-job by assigning the sub-job a low priority level using the nice command. Alternatively, the agent controller system can be configured to reduce the CPU usage of a sub-job through the use of cgroups. Generally, cgroups (also known as control groups) provide a mechanism for aggregating and partitioning sets of processes and the future children of the processes into a group having limits on resource utilization. In an embodiment, the agent controller system can be configured to utilize cgroups in order to place limits on the CPU utilization for a particular sub-job that is being processed by a particular node. Alternatively, the agent controller system can be configured to reduce the CPU usage of a sub-job through the use of posix priorities, a scheduler option built into most operating systems, including linux. In an embodiment, the agent controller system can be configured to utilize posix priorities in order to place limits on the CPU utilization for a particular sub-job that is being processed by a particular node. In an embodiment, the system can be configured to control CPU usage by using other kernel mechanisms that are similar to the nice command, cgroups, and posix priorities described above.
• The agent controller system can be configured to control the amount of disk I/O usage by a particular sub-job that is being processed on a particular node. In an embodiment, the agent controller system can be configured to use at least one of the nice command, cgroups, posix priorities, or the sleep command in order to reduce the disk I/O usage of a particular sub-job that is being processed by a particular node. In an embodiment, the system can be configured to control disk I/O by controlling one or more of the following: creating a “wrapper” around the code that actually accesses the disk I/O capacity in order to control access to the disk I/O capacity, and/or using kernel-level controls to adjust disk I/O usage.
• In an embodiment, the system can be configured to control the usage of specific resources, for example, the usage of CPU, RAM, network, and disk I/O, by controlling the resource through the use of a kernel extension added to the computer operating system, for example a loadable kernel module that is dynamically loaded by the operating system kernel.
• In an embodiment, the supervisor controller system can also be configured to control the assignment of sub-jobs to particular nodes on the cluster in order to use resources more efficiently. For example, the supervisor controller system may determine that a given slave node is running primarily sub-jobs that use CPU intensively but do not use RAM or disk I/O intensively, and determine that the given slave node should be assigned additional sub-jobs that require heavy use of RAM or disk I/O but do not require heavy use of CPU.
• The various foregoing embodiments of the resource monitoring and allocation system can be implemented and/or utilized in a variety of computer cluster environments. For example, the resource monitoring and allocation system can be implemented in conjunction with a hadoop cluster system. In an embodiment, the resource monitoring and allocation system can be implemented in conjunction with non-hadoop clusters, such as other types of computer clusters configured to operate a variety of software applications. Software applications include but are not limited to web servers, databases (for example, MySQL or Impala), virtual machines, and the like. In an embodiment, the resource monitoring and allocation system can be implemented with other non-hadoop clusters, such as network appliances.
• In some versions of the hadoop implementation, the resource monitoring and allocation system can be configured to operate in conjunction with the job tracker and the task tracker systems. In an embodiment, the job tracker of the hadoop system divides a new job into a plurality of tasks. The job tracker can be configured to determine the number of available slots or containers in the cluster or in particular nodes to process the various generated tasks. The job tracker can be configured to assign the tasks to various nodes based on the number of slots or containers available at a particular node. In an embodiment, the task tracker of the hadoop system can be configured to transmit to the job tracker the number of available slots or containers for processing various tasks on a particular node. The supervisor controller or the resource monitoring and allocation system can be configured to communicate with one or more agent controllers operating on the various nodes of the cluster. The agent controllers can be configured to communicate with the supervisor controller in order to transmit resource utilization data to the supervisor controller. The resource utilization data can include information about how individual tasks are utilizing various resources (for example, CPU, RAM, disk I/O, network) of the node. In an embodiment, the supervisor controller system and/or the agent controller system can be configured to determine whether a particular task should receive more or less or the same amount of system resources available at the node that is processing the particular task.
• In the context of implementing the resource monitoring and allocation system in conjunction with a non-hadoop cluster, the tasks in a hadoop system are substituted with software applications and other processes. For example, software applications can include but are not limited to web servers, databases, virtual machines, and the like. In such implementations, the agent controller systems can be configured to operate on nodes of a cluster and can be configured to monitor the resource utilization of each software application operating on the node. For example, the agent controller system can be configured to determine the CPU usage, RAM utilization, network usage, and disk I/O usage of a web server operating on the node.
• The agent controller system can be configured to transmit this resource utilization data to a supervisor controller system. The supervisor controller system can be configured to analyze the resource utilization data from a plurality of nodes in the cluster to determine whether resource reallocation is necessary to allow the cluster to operate more efficiently. The supervisor controller system can be configured to transmit resource reallocation instructions to specific agent controller systems operating on particular nodes. The instructions can comprise data necessary for the agent controller system to generate instructions and/or commands to increase and/or reduce the resource utilization of a particular software application or other processes that are operating on the node.
• In implementations where the resource monitoring and allocation system is implemented in a network appliance, such as a network router and/or switch or the like, an agent controller system can be implemented in the network appliance. In an embodiment, the agent controller system can be configured to interrogate the network appliance in order to determine the resource utilization of particular jobs that are being processed by the network appliance. For example, an agent controller system operating on a router and/or switch can be configured to analyze data packets that are coming into the router and/or switch. In an embodiment, the agent controller system can be configured to communicate with a supervisor controller system in order to determine which ports of the router and/or switch through which more data or less data should be processed.
• There are many challenges in implementing the resource monitoring and allocation system. Accordingly, one of ordinary skill in the art will appreciate that the systems, methods, and devices disclosed herein for implementing the resource monitoring and allocation system are novel, unique, and are nonobvious in view of the numerous challenges in implementing such a system. A challenge in implementing the system is the automatic tuning of the allocation of resources to various jobs and sub-jobs being processed by plurality of nodes across a cluster. In an embodiment, the automatic tuning of resource allocations in a cluster is based on desired outcomes inputted into system by the user. For example, a user can define an outcome that is time based. The user can specify that the project needs to be completed by a certain period of time on a particular day.
• Alternatively, the automatic tuning can be based on a desired resource allocation as defined by the user. For example, a user of the cluster may define that a particular job must have 75% of the cluster's network bandwidth capacity as well as 80% of the CPU utilization at a particular node in the cluster. As another example, a user of the cluster may define that a particular job must have access to specifically defined resource minimums, for example at least 100 megabits per second of network bandwidth, 300 megabytes per second of disk I/O, and 1 billion CPU instructions per second.
• The existence of an outcome requirement set by the user can require the resource monitoring and allocation system to have access to global knowledge of the cluster in order to properly monitor and control the various nodes such that the user defined outcomes can be achieved. For example, the resource monitoring and allocation system must globally determine and globally control the network usage of each node in the cluster in order to ensure that 75% of the network bandwidth capacity is dedicated to the particular job or sub-job designated by the user. This can require that the resource monitoring and allocation system reduce the network utilization of certain jobs or sub-jobs operating on other nodes of the cluster in order to provide excess network bandwidth to the particular job or sub-jobs that the user required to have 75% of the network bandwidth of the cluster.
• In an embodiment, the resource monitoring and allocation system can be configured to identify jobs or sub-jobs that have been allocated a certain amount of computer resources but is only utilizing small portion of the resource allocation. By identifying such jobs or sub-jobs, the system can be configured to re-allocate a portion of the resource allocation to another job or sub-job. For example, the system can be configured to identify a first sub-job that is being processed by a first node, wherein the first sub-job has been allocated 75% of the network resource capacity but is only utilizing 25% of the network resource capacity. The system can be configured to reallocate a portion of the network resource capacity from the first sub-job to a second sub-job that is being processed on the first node or another node. Further, the system can be configured to reallocate the portion of the network resource capacity from the second sub-job back to the first sub-job if the system identifies that the performance of the first sub-job declines due to a lack of network resource capacity.
• Determining the available resources across a computer cluster can be challenging because the status of the cluster is continuously changing. Therefore, the resource monitoring and allocation system requires continuous updated information regarding the resource utilization at each node in the cluster. As the information about the status of the various nodes in the cluster changes the resource monitoring and allocation system can be configured to adapt accordingly. Another challenge of the resource monitoring and allocation system is the managing, processing, analyzing, and logging of the large amount of data transmitted to the supervisor controller from the plurality of agent controllers operating in the various nodes of the cluster. In an embodiment, the resource monitoring and allocation system can be configured to receive resource allocation data from each node in the cluster once every 1 second to 5 seconds. The sheer volume of data coming into the monitoring and allocation system makes it impossible for a human being, whether entirely in the person's mind or whether the person is using a pen and paper, to track and/or perform, in real-time or substantially real-time, the activities of the embodiments of the resource management and allocation systems that are disclosed herein.
• FIG. 1 is an embodiment of a schematic diagram illustrating a computer cluster. In an embodiment, thecomputer cluster101 can comprise amaster node104 connected to anetwork108. Thecomputer cluster101 can also comprise a plurality ofnodes110,120,130 that are connected to each other and to themaster node104 throughnetwork108. In an embodiment, thecluster101 can be configured to communicate withclient102. Themaster node104 can be configured to receive from theclient102 jobs for processing on thecluster101. In an embodiment, themaster node104 can be configured to return to theclient102 completed jobs that have been processed by thecluster101.
• Themaster node104 can be configured to analyze jobs received from theclient102. Themaster node104 can be configured to divide the job received from theclient102 into a plurality of smaller jobs or sub-jobs. Themaster node104 can be configured to distribute and/or assign the smaller jobs or sub-jobs tovarious nodes110,120,130 in thecluster101. In assigning the smaller sub-jobs to thevarious nodes110,120,130, themaster node104 may be configured to utilizemanagement software106 for managing and/or tracking the smaller jobs that have been distributed across thecluster101.
• In an embodiment, themanagement software106 is implemented using a hadoop system. In a hadoop system, themanagement software106 can comprise software known as job tracker. Alternatively, themanagement software106 can be implemented using the Yarn software or Yarn resource manager and/or Yarn node manage in a hadoop system. In non-hadoop systems, themanagement software106 can comprise other software applications that are configured to analyze jobs, divide jobs into smaller sub-jobs, and/or distribute the sub-jobs to various nodes in thecluster101 for processing.
• In an embodiment, theslave nodes110,120,130 can comprisesoftware112,122,132 for tracking the sub-jobs that are being processed on the node. In an embodiment, thenodes110,120,130 can comprise astorage device118,128,138 configured to store data and/or software for processing the sub-jobs received from themaster node104. In an embodiment, thesoftware112,122,132 is configured to track sub-jobs114,116,124,126,134,136 that have been received from themaster node104 for further processing on the node. In an embodiment, thesoftware112,122,132 can be configured to communicate with thestorage devices118,128,138 in order to process the sub-jobs.
• FIG. 2 is an embodiment of a schematic diagram illustrating a computer cluster comprising an embodiment of a dynamic monitoring and/or resource allocation system. In an embodiment, acluster201 can be configured to communicate with aclient202. The client can be configured to send a job for processing on thecluster201. Thecluster201 can be configured to return a completed job to theclient202. In an embodiment, thecluster201 can comprise amaster node204 as well as a plurality ofslave nodes210,232. Themaster node204 can be configured to analyze the job received fromclient202. Themaster node204 can comprisesoftware206 for analyzing the job, dividing the job into sub-jobs, and/or distributing the sub-jobs to the various slave nodes in thecluster201. In a hadoop system, thesoftware206 can comprise the job tracker software or the Yarn software. In non-hadoop systems, thesoftware206 can comprise other management software for analyzing jobs, dividing jobs into sub-jobs, and/or distributing sub-jobs across the cluster to various nodes.
• In an embodiment, thesoftware206 can be configured to divide the job into foursub-jobs212,214,228,230. In a hadoop system, the sub-jobs are known as tasks. In non-hadoop systems, the smaller jobs that are generated by themaster node204 are generically known as sub-jobs. As illustrated inFIG. 2, themanagement software206 can be configured to distribute sub-jobs214 to afirst node210 and can be configured to distribute sub-jobs228,230 to asecond node232.
• In an embodiment, theslave nodes210,232 can comprisesoftware216,234 for tracking sub-jobs that have been assigned to a particular node. In a hadoop system, thesoftware216,234 can comprise the task tracker software. In non-hadoop systems, thesoftware216,234 can comprise other node manager software for tracking the sub-jobs that have been assigned to a particular node from amaster node204.
• In an embodiment, the master node can comprise asupervisor controller208. Thesupervisor controller208 can be configured to monitor, track, log, and/or control the allocation of computer resources atparticular nodes210,232. In an embodiment, thenodes210,232 can comprise anagent controller218,236. Theagent controller218,236 can be configured to monitor, track, log and/or control the allocation of computer resources on a particular node. For example, theagent controller218 can be configured to communicate with the kernel of the node or other systems on thenode220 to determine the computer resources being utilized by the sub-jobs222,224 that are operating onnode210.
• In determining the resource utilization of particular sub-jobs operating on a node, theagent controller218,236 can be configured to transmit the resource utilization data to thesupervisor controller208. In an embodiment, thesupervisor controller208 can be configured to analyze the resource utilization data received from theagent controller218,232 in order to determine whether computer resources that are currently being utilized by certain sub-jobs should be reallocated to other sub-jobs. Based on the foregoing determination, thesupervisor controller208 can be configured to generate instructions for transmission to theagent controller218,236. The instructions can be configured to cause theagent controller218,236 to generate further commands to control the allocation of resources on aparticular node210,232 for use byvarious sub-jobs222,224,240,242.
• In an embodiment, theagent controller218,236 can be configured to generate commands for controlling the allocation of resources on a particular node without receiving instructions from asupervisor controller208. For example, anagent controller218,236 can be configured to increase and/or decrease CPU capacity directed to aparticular sub-job222,224,240,242 based on the prioritization of the sub-task. In an embodiment, theagent controller218 can be configured to determine that the sub-job222 has a higher priority than that ofsub-job224. Based on the foregoing determination, theagent controller218 can be configured to increase the CPU capacity directed to sub-job222 while decreasing the CPU capacity forsub-job224. In an embodiment, the foregoing reallocation of computer resources can be performed by theagent controller218 without instructions from thesupervisor controller208.
• FIG. 2A is an embodiment of a schematic diagram illustrating a computer cluster comprising an embodiment of a dynamic monitoring and/or resource allocation system. Similar toFIG. 2, aclient202 can submit jobs for processing oncluster201a.In an embodiment, cluster201acan comprise amaster node204 and aslave node246. In contrast toFIG. 2, thecluster201aas illustrated inFIG. 2A can comprise asupervisor controller208 that operates onnode246 while a job tracker orother management software206 operates onmaster node204.
• The advantage of separating the job tracker orother management software206 from thesupervisor controller208 is to ensure that the job tracker orother management software206 has sufficient computer resources on the master node for processing the job submissions received fromclient202. Similarly, by positioning thesupervisor controller208 on aseparate node246, the operator of thecluster201acan ensure that the supervisor controller has sufficient computer resources dedicated to thesupervisor controller208 such that thesupervisor controller208 can continuously monitor, process, and/or analyze all of the resource data that is being received form the plurality ofagent controllers218,236.
• Additionally, by positioning thesupervisor controller208 on aseparate node246, the operator of thecluster201acan ensure that thesupervisor controller208 has sufficient computer resources for dynamically and automatically generating instructions for controlling in real time or substantially real time the allocation of resources on a particular node for a particular task operating on the node.
• FIG. 2B is an embodiment of a schematic diagram illustrating a computer cluster comprising an embodiment of a dynamic monitoring and/or resource allocation system. Similar toFIGS. 2 and 2A, aclient202 can submit jobs for processing oncluster201b.In contrast toFIGS. 2 and 2A, thecluster201bas illustrated inFIG. 2B can comprise afirst supervisor controller208 that operates onnode246 and asecond supervisor controller209 that operates onnode254. As illustrated inFIG. 2B, the job tracker orother management software206 is positioned onmaster node204.
• The advantage of this configuration is the ability to ensure that the necessary computer resources are being allocated to thesupervisor controller systems208,209 and the job tracker orother management software206. In an embodiment, thefirst supervisor controller208 and thesecond supervisor controller209 can be configured to communicate withdifferent agent controllers218 and236. For example, thefirst supervisor controller208 can be configured to communicate withagent controller218 while thesecond supervisor controller209 can be configured to communicate withagent controller236. In an embodiment, theagent controllers218 and236 communicate only withpredesignated supervisor controllers208,209. For example, theagent controller218 can be configured to only communicate withsupervisor controller208 while theagent controller236 can be configured to only communicate withsupervisor controller209.
• In an embodiment, theagent controllers218,236 can be configured to communicate with thesupervisor controllers208,209 on a first come, first served basis. For example, theagent controller218 can be configured to communicate with either thefirst supervisor controller208 or thesecond supervisor controller209 depending upon which supervisor controller is available at any particular time. Similarly, theagent controller236 can be configured to communicate with either thefirst supervisor controller208 or thesecond supervisor controller209 depending upon which supervisor controller is available at any one particular time.
• The advantage of comprising two or more supervisor controllers in a cluster system is to ensure that the supervisor controllers have sufficient computer resources to continuously monitor, track, analyze, log, and/or control the allocation of computer resources on a particular node for any particular sub-job operating on a node. In an embodiment, the two ormore supervisor controllers208,209 can be configured to communicate with each other in order to share tracking information related to the allocation of computer resources across various nodes in the cluster. The two ormore supervisor controllers208,209 can be configured to communicate with each other in order to coordinate the control of the allocation of computer resources at particular nodes in the cluster.
• FIG. 3 is a flow chart depicting an embodiment of a process for dynamically monitoring and/or allocating resources across a computer cluster. In an embodiment, the process can start atblock302 with a client submitting a job or other submission to the hadoop system. Atblock304, the job tracker of the hadoop system can be configured to receive the submission from the client. Atblock306, the job tracker can be configured to invoke the map reduce function in the hadoop system to use map process in order to divide the submission into various tasks. Atblock306, the job tracker can be configured to invoke the map reduce function of the hadoop system in order to assign the task to various slave nodes in the cluster.
• Atblock308, the slave nodes are configured to receive the assigned task from the job tracker. In an embodiment, the slave node comprises a task tracker that is configured to receive the task from the job tracker. Atblock310, the slave node can be configured to process the task received from the job tracker. Atblock312, the task tracker can be configured to determine if the task has terminated or failed during the processing by the node. If the task has terminated or failed, atblock314, the slave node informs the job tracker. Atblock314, the job tracker reassigns the terminated or failed task to another slave node and returns to block308. If atdecision block312, the task has not terminated, the system moves to block316.
• Atblock316, the agent controller that is operating on the node periodically or continuously accesses or interrogates the slave node to obtain computer resource data from the kernel or other modules. In an embodiment, the agent controller atblock316 can be configured to track the task in the slave node. Atblock320, the agent controller can be configured to transmit the computer resource status data to the supervisor controller. While the node is processing the task that has been assigned to the node atblock310, the agent and supervisor controllers can be configured to track the assigned task atblock318.
• Atblock322, the agent and/or supervisor controllers periodically or in real time determine whether the computer resources that are being allocated to each task at each particular node should be changed. In an embodiment, the system can be configured atblock324 to generate instructions for the slave node to dynamically change the allocation of resources being utilized by particular jobs on a particular node if the agent and/or supervisor controllers determine that the computer resource allocation is above a threshold level for a particular task operating on a particular node. For example, the agent and/or supervisor controllers can be configured to determine that a particular job is utilizing RAM that exceeds a threshold limit or level for a particular node. In response, the agent and/or the supervisor controllers can be configured to instruct the node to terminate the job if the job is utilizing RAM that exceeds a threshold limit or level for a particular node.
• Atblock312, the task tracker can be configured to determine that the task has been terminated and inform the job tracker atblock314. Atblock314, the job tracker can be configured to reassign the terminated task to another slave node. In an embodiment, the supervisor can be configured to use the historical data relating to the previous termination of the job in order to instruct the job tracker to assign the previously terminated task to a node having enough RAM capacity to allocate to the job, thereby preventing the job from being terminated again. Alternatively, the supervisor controller can be configured to directly assign the previously terminated task to a node having enough RAM capacity to allocate to the job, thereby avoiding the need for the job tracker to assign the task to a new node.
• If the agent and/or supervisor controllers determine that the job is operating within an acceptable range or is below a particular threshold level, then the system can be configured to return theblock312 to determine if the task has died or terminated. If the process has not been terminated the system continues to block316 to periodically or continuously access the computer resource status data on a particular node.
• FIG. 3A is a flow chart depicting an embodiment of a process for dynamically monitoring and/or allocating resources across a computer cluster. Similar toFIG. 3, the agent controller can be configured to periodically or continuously access the computer resource status data on a particular slave node. Atblock320, the agent controller can be configured to transmit the computer resource status data to the supervisor controller. While the slave node processes the task atblock310, the agent and/or supervisor controller atblock326 can be configured to track the task on a particular node and determine the priority of the task based on client input when the job was submitted to the job tracker.
• Atblock328, the agent and/or supervisor controllers periodically or in real time determine the resources to be allocated to each task on a slave node based optionally on the prioritization of the task as determined by the client or based optionally on whether the job performance is below a minimum performance guarantee specified by the client. Atblock324, the system can be configured to determine if a resource allocation is above a threshold level for a particular task and/or node or if a job is operating below a designated priority level or if the job performance is below a minimum performance guarantee, then the system can be configured to generate instructions for the slave node to dynamically change the allocation of computer resources to be dedicated to the job in order to bring down the resource allocation below a threshold level, or to ensure that the job is operating at a specific priority level or to ensure that the job performance is above a minimum performance guarantee.
• FIG. 4 is an embodiment of a schematic diagram illustrating a computer cluster comprising an embodiment of a dynamic monitoring and/or resource allocation system. Similar toFIGS. 2, 2A, 2B, aclient402 can communicate with one or more master nodes orother nodes404 in order to submit a job for processing on acomputer cluster401. In an embodiment, themaster node404 can comprise amanagement software406 and asupervisor controller408. In an embodiment, thesupervisor controller408 and themanagement software406 operate on asingle master node404. In an embodiment, thesupervisor controller408 and themanagement software406 operate onseparate master nodes404. In an embodiment, the job that is submitted byclient402 is received by themanagement software406 that is responsible for analyzing the job and dividing the job into smaller sub-jobs. As illustrated inFIG. 4, the system can be implemented in conjunction with a hadoop system; however, one of ordinary skill in the art will appreciate that the systems and methods disclosed herein can be used in conjunction with other cluster systems and not just with hadoop systems.
• The dividedsub-jobs414,410,438,440 can be assigned by themanagement software406 tovarious nodes416,442 in the cluster. In an embodiment, a node manager (or a task tracker in a hadoop system)418,444 can be configured to receive the sub-jobs that have been assigned to a particular node by themanagement software406. Thesupervisor controller408 can be configured to communicate with theagent controllers420,446 that operate on thenodes416,442 of the cluster. While thenodes416,442 are processing the sub-jobs, thenode manager418,444 can be configured to track the sub-jobs being operated on by particular nodes.
• Additionally, theagent controllers420,446 can be configured to also track the sub-jobs being operated by thenodes416,442 in addition to determining the allocation of computer resources to each of the sub-jobs on a particular node. For example,agent controller420 can be configured to communicate with the kernel orother module422 of thenode416 in order to determine the amount ofnetwork capacity424,RAM usage426, disk I/O usage428, andCPU capacitor430 as being utilized by the sub-jobs432,434 that are being operated on by thenode416. Theagent controller420 can be configured to transmit the computer resource allocation data to thesupervisor controller408. Theagent controller420 and/or thesupervisor controller408 either alone or in conjunction with each other, can be configured to determine which sub-jobs432,434 are utilizing acceptable allocations of computer resources of thenode416. For example, theagent controller420 can be configured to determine that afirst sub-job432 is utilizing excess disk I/O capacity428.
• In an embodiment, the foregoing determination can be based on the prioritization assigned to thefirst sub-job432. If the sub-job432 has a low prioritization but is utilizing substantially all of the disk I/O capacity428, theagent controller420 can be configured to independently reduce the amount of disk I/O capacity428 that is allocated to the sub-job432 in order to provide thesecond sub-job434 greater access to the disk I/O capacity428.
• In another example, thesupervisor controller408 and theagent controllers420,446 can be configured to coordinate with each other in order to collectively determine and/or control the resource allocations that are provided to various sub-jobs operating on thenodes416,442. In an embodiment, thesupervisor controller408 can be configured to determine that thethird sub-job458 is utilizing 100% of thenetwork capacity450 by analyzing the resource data transmitted to thesupervisor controller408 from theagent controller446.
• In an embodiment, the 100% utilization of thenetwork capacity450 can result in the 100% network capacity utilization for theentire cluster401. Accordingly, the first sub-job432 operating onnode416 comprises 0% of thenetwork capacity424 fornode416 to process the sub-job432. In an embodiment, thefirst sub-job432 comprises a high priority rating whereas thethird sub-job458 comprises a low priority rating. Thesupervisor controller408 can be configured to generate instructions for instructing theagent controller446 to reduce thenetwork capacity450 that is allocated to thethird sub-job458. Thesupervisor controller408 can also be configured to instruct theagent controller420 to provideadditional network capacity424 to thefirst sub-job432.
• FIG. 5 is a flow chart depicting an embodiment of a process for monitoring and/or allocating cluster resources, such as RAM, network usage, CPU usage, and disk I/O usage. The process can start atblock502 with the agent and/or supervisor controllers accessing the status updates from the slave nodes. Atblock504, the agent and/or supervisor controllers can be configured to determine if the RAM usage is above a threshold level at a particular node for a particular job. If the determination is yes, atblock506, the agent and/or supervisor controllers can be configured to determine a mechanism to reduce the RAM usage for a particular task on a particular node. For example, the agent and/or supervisor controller can be configured to optionally kill a task in order to reduce the RAM usage for a particular task.
• In an embodiment, the agent and/or supervisor controllers can be configured to optionally kill low priority sub-jobs in order to free RAM usage for other high priority jobs operating on the same node. The usage of RAM, unlike other computer resources, is difficult to reduce or limit for a particular task. Generally, a job will require a certain amount of RAM to operate and if the job does not receive the required RAM usage, then the job cannot be performed. Accordingly, there is less discretion in controlling RAM usage as compared to controlling network usage, CPU usage, and disk I/O usage. Alternatively, the agent and/or supervisor controllers can be configured to optionally invoke the garbage collection command of an operating system. For example, the agent and/or supervisor controller can be configured to invoke the JAVA virtual machine garbage collection command for a particular task in order to reduce the RAM usage by that task on a particular node.
• If atblock504, the agent and/or supervisor controllers determine that the actual RAM usage is below a threshold level at a particular node, the agent and/or supervisor controllers atblock508 can be configured to determine whether additional tasks should be assigned to the node. If the determination is yes, then atblock512, the supervisor controller can be configured to instruct the management software206 (for example the job tracker in a hadoop system) to assign new tasks to the slave node. Alternatively, atblock512, the supervisor controller can be configured to assign a new task to the slave node without instructing themanagement software206. If atblock508 the determination is no, the system atblock516 has determined that historically such tasks of this type use excess RAM.
• Atblock518, the agent and/or supervisor controllers can be configured to determine if the network usage is above a threshold level at a particular slave node for a particular job. If the determination is yes, atblock520, the agent and/or supervisor controllers can be configured to determine a mechanism for reducing the network usage. For example, the agent and/or supervisor controllers can be configured to optionally sleep a task atblock524. Alternatively, the agent and/or supervisor controllers can be configured to optionally reduce bandwidth usage atblock526.
• If the determination atblock518 is no, the agent and/or supervisor controller can be configured to determine if additional tasks should be assigned to the node. If the determination is yes, the agent and/or supervisor controllers can be configured to assign atblock528 additional tasks to the node and/or allow a current task more network access. If the determination is no atblock522, the agent and/or supervisor controllers have made a determination that historically such tasks of this type use excess network capacity and therefore no additional tasks should be assigned to this node.
• Atblock532, the agent and/or supervisor controllers can be determined if CPU usage is above a threshold level at a particular node for a particular task. If the determination is yes, the agent and/or supervisor controllers can be configured to determine a mechanism to reduce the CPU usage for a particular task. For example, the agent and/or supervisor controllers can be configured to optionally “nice” a task. Alternatively, the agent and/or supervisor controllers can be configured to optionally invoke a Cgroup command for a task in order to reduce the CPU usage for a particular task.
• If the determination is no atblock532, then the agent and/or supervisor controllers can be configured to determine if additional tasks should be assigned to the node. If the determination is yes, atblock540 the supervisor controller can be configured to instruct themanagement software206 to assign a new sub-job to the slave node. Alternatively, the supervisor controller can be configured to directly assign a new sub-job to the node. If the determination is no atblock536, then atblock544 the agent and/or supervisor controllers have made a determination that historically the job of this type uses excess CPU and therefore no additional sub-jobs should be assigned to this node.
• Atblock546, the agent and/or supervisor determines if disk I/O usage is above a threshold level at a particular slave node. If the determination is yes, then atblock548 the agent and/or supervisor controllers determine a mechanism to reduce the disk I/O usage for a particular task. For example, the agent and/or supervisor controllers can be configured to optionally nice, Cgroup, or sleep a particular sub-job atblock552. If the determination is no atblock546, the agent and/or supervisor controllers can be configured to determine if additional sub-jobs should be assigned to the node. If the determination is yes, then atblock554 the supervisor controller and/or themanagement software206 can be configured to assign a new task to the slave node. If the determination is no atblock550, then atblock556, the agent and/or supervisor controllers have made a determination that historically such sub-jobs of this type use excess disk I/O and therefore no additional sub-jobs should be assigned to this node.
• FIG. 6 is a block diagram depicting a high-level overview of an embodiment of a distributor system. In an embodiment, a supervisor controller, an agent controller, a disk, a network appliance, orother device602 that is in a cluster or connected to a cluster can comprise adistributor604. In an embodiment, adistributor604 can be configured to receive a variety of inputs in order to determine the resource allocations for a particular task operating on a particular node. In an embodiment, thedistributor604 can be configured to receivedata606 regarding the state of a node and/or the computer resource usages at a particular node.
• Thedistributor604 can be configured to analyze thedata606 in order to generate limits and/or allocations of various computer resources for a particular task on a particular node. The limits and/or allocations of various computer resources can be generated asoutputs612 by thedistributor604 wherein theoutput612 can be utilized by the supervisor controller, agent controller, disk, network appliance, orother device602 in order to generate instructions for adding or reducing the allocation of computer resources to a particular job or sub-job.
• In an embodiment, thedistributor604 can be configured to receive as aninput608 operator specified goals and/or properties for a particular job and/or sub-job. For example, an operator, or client, or other user of a cluster system can specify that a job be completed in a less than a specified period of time or that a job must be provided a minimum level of network access in order to complete the job. In an embodiment, thedistributor604 can be configured to analyze the operator specified inputs in order to generate anoutput612 for limiting and/or allocating various computer resources for a particular task operating on a particular node.
• In an embodiment, thedistributor604 can be configured to receive historical data inputs. In an embodiment, historical data inputs can include data relating to how similar jobs of this type require specific CPU usages, RAM usages, network usages, and/or disk I/O usages. In an embodiment, thedistributor604 can be configured to analyze thehistorical data inputs610 in order to generateoutputs612 relating to limitations and/or allocations of various computer resources for particular jobs or sub-jobs on particular nodes.
• FIG. 7 is a flow chart depicting an embodiment of a process for a distributor as illustrated inFIG. 6. In an embodiment, the process can begin atblock702 with the system accessing data atblock704. The data can be related to the state of a cluster(s) and/or parts of a cluster and/or external resources. For example, the system can be configured to access computer resource usage data relating to a particular job operating in a particular node. Atblock706, the system can be configured to access data relating to operator(s) specified goal(s) and/or performance properties for a particular job.
• Atblock708, the system can be configured to access data relating to historical research requirements for similar jobs and/or tasks. In an embodiment, the system can be configured to optionally access historical data relating to historical resource requirements for similar jobs and/or tasks that are performed on particular or similar nodes. Atblock710, the system can be configured to optionally access priority data relating to job submissions in process or in queue to determine global priority of job submissions relative to each other. Atblock712, the system can be configured to analyze the data inputs to determine limits and/or resource allocations for particular jobs and/or tasks operating on particular nodes.
• Atblock714, the system can be configured to generate instructions for limiting and/or allocating resources for particular jobs and/or tasks that are operating on particular nodes. Atblock716, the system can be configured to transmit the instructions to cluster(s) and/or node(s) or sub-jobs or external resources. Atblock716, the process can be configured to end or it can be configured to optionally return to block704 to repeat the process.
• FIG. 8A is a block diagram depicting a high-level overview of an embodiment of virtual clusters. In an embodiment, theclient802 can be configured to submit jobs to a virtual cluster. As illustrated,client802 can be configured to submit a job to amaster node804. The master node can comprise a job tracker orother management software806 and asupervisor controller808. In an embodiment, the job tracker orother management software806 can be configured to analyze the job received from theclient802 in order to divide the job in to a plurality of sub-jobs for distribution and processing by various nodes in the cluster.
• In an embodiment, thesupervisor controller808 can be configured to determine whether the job received from theclient802 should be processed on a firstvirtual cluster805 or whether the job should be processed on a secondvirtual cluster807. As illustrated inFIG. 8A, there is only one physical cluster for processing the job that is received fromclient802. However, thesupervisor controller808 can be configured to dynamically create one or more virtual clusters from one physical cluster. For example, thesupervisor controller808 can be configured to allocatenodes1,2, and3 to form a firstvirtual cluster805 dedicated to processing certain jobs of theclient802 and thesupervisor controller808 can be configured to designatenodes4,5, and6 as a secondvirtual cluster807 that is dedicated to processing another type of job received fromclient802.
• The advantage of creating virtual clusters is an operator need not create separate physical clusters in order to have dedicated clusters for processing certain client jobs. Rather, the operator needs only one cluster that can be divided into one or more virtual clusters that are dedicated to certain client jobs. The advantage of virtual clusters over multiple physical clusters is operational simplicity. The operator need only maintain one physical cluster as opposed to multiple physical clusters. In an embodiment, thesupervisor controller808 can be configured to analyze the sub-jobs and/or the job submitted by theclient802 in order to determine which virtual cluster should process the job and/or sub-jobs.
• In an embodiment, thesupervisor controller808 can be configured to determine that the job submitted by theclient802 is a high priority job. For high priority jobs, thesupervisor controller808 can be configured to submit the related sub-jobs to the secondvirtual cluster807, which can process the sub-jobs faster because the nodes in the secondvirtual cluster807 have been allocated with 75% CPU capacity. In contrast, thesupervisor controller808 can be configured to determine that a client job is a low priority job and therefore should be assigned to the firstvirtual cluster805, which will process the sub-job slower than the secondvirtual cluster807. The reason why the first virtual cluster will process the sub-job slower is because the nodes in the firstvirtual cluster805 have only been allocated 50% of the CPU capacity of each node.
• FIG. 8B is a block diagram depicting a high-level overview of an embodiment of virtual clusters. Similar toFIG. 8A, theclient802 can be configured to submit jobs to themaster node804. In contrast toFIG. 8A, thesupervisor controller808 can be configured to dynamically generate virtual clusters. As illustrated inFIG. 8B, thesupervisor controller808 initially created a firstvirtual cluster824 comprisingnode1,810,node2,812, andnode3,814. Thesupervisor controller808 can be configured to dynamically generate a new firstvirtual cluster826. The dynamic generation of virtual clusters can be advantageous for efficiently utilizing the computer resources of a cluster. For example, thesupervisor controller808 can be configured to analyze the nodes of a cluster in order to determine how to best create a virtual cluster.
• In an embodiment, thesupervisor controller808 created the firstvirtual cluster824 because thesupervisor controller808 determine that at the time there were three nodes having excess CPU capacity of 50%. Thesupervisor controller808 can be configured to determine that the client-submitted job requires 150% of CPU capacity. Accordingly, thesupervisor controller808 can be configured to create the firstvirtual cluster824 in order to satisfy the job requirement of theclient802. However, at another point in time, thesupervisor controller808 can be configured to determine that two additional nodes became free such that 75% of the CPU capacity on each of the nodes was available. In an embodiment, thesupervisor controller808 can be configured to determine that it is more efficient for processing a particular job using two nodes as opposed to processing the job over three nodes. For example, the use of two nodes can be faster for processing jobs. The sharing of data over three nodes requires more time than the sharing of data between two nodes. Accordingly, thesupervisor controller808 can be configured to dynamically create a new firstvirtual cluster826 comprisingnode4,816, andnode5,818, wherein each node can allocate 75% of the CPU capacity of each node to processing the job from theclient802.
• FIG. 8C is a block diagram depicting a high-level overview of an embodiment of virtual clusters. Similar toFIG. 8A and 8B, theclient802 can be configured to submit jobs to themaster node804. In an embodiment, thesupervisor controller808 can be configured to create virtual clusters wherein certain nodes are part of one or more clusters. For example, thesupervisor controller808 can be configured to create the firstvirtual cluster836 comprisingnode1,824,node2,826, andnode3,824. With respect tonode1 and2, thesupervisor controller808 can be configured to designate 100% of the CPU capacity for these nodes to be dedicated for the firstvirtual cluster836.
• With respect tonode3, thesupervisor controller808 can be configured to designate only 50% of the CPU capacity of this node for the firstvirtual cluster836. Thesupervisor controller808 can be configured to generate a secondvirtual cluster838 comprisingnodes3,828,node4,830,node5,932, andnode6,834. In an embodiment, thesupervisor controller808 can be configured to designate that only 50% of the CPU capacity ofnode3 should be dedicated to the secondvirtual cluster838. With respect tonodes4,5, and6, thesupervisor controller808 can be configured to designated 100% of the CPU capacity for these nodes to the secondvirtual cluster838.
• FIG. 8D is a block diagram depicting a high-level overview of an embodiment of virtual clusters. In an embodiment, thesupervisor controller808 can be configured to generate any number of virtual clusters based on the nodes of a single physical cluster. For example, thesupervisor controller808 can be configured to generate three virtual clusters. Thesupervisor controller808 can be configured to generate a first virtualcluster comprising node836,838, and840. Thesupervisor controller808 can be configured to designate thenode836 to dedicate 80% of the CPU capacity to the first virtual cluster while designating only 10% of the CPU capacity ofnode838 to the first virtual cluster and dedicating 50% of the CPU capacity of thenode840 to the first virtual cluster.
• Thesupervisor controller808 can be configured to generate a second virtualcluster comprising nodes836,838,840,842, and844. Thesupervisor controller808 can be configured to designate only 20% of the CPU capacity of thenode836 to the second virtual cluster while dedicating 90% of the CPU capacity of thenode838 to the second virtual cluster and dedicating 50% of the CPU capacity of thenode840 to the second virtual cluster and dedicating 100% of the CPU capacities of thenodes842 and844 to the second virtual cluster. Thesupervisor controller808 can be configured to generate a third virtualcluster comprising node846. Thesupervisor controller808 can be configured to designate that 100% of the CPU capacity of thenode846 be dedicated to the third virtual cluster.
• FIG. 8E is a block diagram depicting a high-level overview of an embodiment of virtual clusters. Similar toFIG. 8A, theclient802 can be configured to submit jobs for processing on a cluster to amaster node804. In an embodiment, thesupervisor controller808 can not only allocate CPU capacity on particular nodes to specific virtual clusters, but also thesupervisor controller808 can be configured to dedicate other computing resources on the node to specific virtual clusters. For example, thesupervisor controller808 can be configured to dedicate 50% of RAM usage onnode848 to the first virtual cluster and to the second virtual cluster.
• In addition to dedicating computer resources at particular nodes to specific virtual clusters, thesupervisor controller808 can also be configured to dedicate computer resources of other devices in the cluster or connected to the cluster to specific virtual clusters. For example, thesupervisor controller808 can be configured to dedicate 30% of the switch utilization of afirst switch860 to the first virtual cluster. Similarly, thesupervisor controller808 can be configured to allocate 0% of a switch usage of asecond switch862.
• FIG. 9 is a flow chart depicting an embodiment of a process for processing jobs using a virtual cluster. Atblock902 the process can begin with a job being received atblock904. The system can be configured to determine whether the submitted job is designated to be processed by a virtual cluster. If the determination is yes, then atblock908, the job tracker or other management software, and/or supervisor controller can be configured to divide the job into various sub-jobs for assignment to nodes in the virtual cluster designated by the system.
• Atblock910, the job tracker or other management software, and/or supervisor controller can be configured to determine which node(s) in the virtual cluster to assign the task or otherwise put the task in a queue for later processing. Atblock912, the supervisor and/or agent controllers can be configured to determine which task in the queue should be assigned to nodes outside the virtual cluster. For example, the supervisor and/or the agent controllers can be configured to determine that the job is a high priority job and therefore should be processed as soon as possible using other nodes outside the virtual cluster.
• In another example, the supervisor and/or the agent controllers can be configured to determine that other nodes outside of the virtual cluster have computer resources available for processing job(s). Accordingly, the supervisor and the agent controllers can be configured to assign sub-jobs in the queue to available nodes outside the virtual cluster atblock914. If atblock912, the supervisor and/or the agent controller determine that a sub-job in the queue should not be assigned to marriage outside of the virtual cluster, the process can return to block910 where the job tracker or other management software, and the supervisor controller can be configured to determine which node in the virtual cluster to assign a sub-job.
• If the determination atblock906 is no, then atblock914 the job tracker or other management software, and supervisor controller can be configured to divide the job submission into sub-jobs for assignment to nodes outside the virtual cluster. Atblock916, the job tracker and/or supervisor controller can be configured to determine which available nodes outside the virtual cluster to assign the sub-jobs, or otherwise put the sub-job in a queue for later processing.
• FIG. 10 is a flowchart depicting an embodiment of a process for processing jobs using a virtual cluster. The process can begin at1002 with receiving a job submission atblock1004. Atblock1006, the system can be configured to determine whether a job submission is designated to be processed by a virtual cluster. Atblock1008, the system can be configured to determine the resources necessary to process the job based on the client specified performance goals. Atblock1010, the system can be configured to generate and/or identify a virtual cluster based on the required resource necessary for processing the job and/or based on the resources available in the nodes of the cluster and/or based on the specified performance goals of the user/client.
• Atblock1012, the system can be configured to assign sub-jobs to the nodes in the created virtual cluster and can be configured to add the assigned sub-jobs to a queue for later processing. Atblock1014, the system can be configured to optionally determine which sub-jobs in the queue should be assigned to nodes outside the virtual cluster. Atblock1014, the system can be configured to optionally return to block1010 where a virtual cluster is identified for processing the jobs in the queue.
• FIG. 11 is a flowchart depicting an embodiment of a process for processing jobs using job groups. The process can begin atblock1102 with receiving a job submission atblock1104. The system can be configured to determine at block1106 a job group type based on the job submission and/or the job submission requirements. Atblock1108, the system can be configured to allocate based on the job group identification CPU capacity, RAM capacity, disk I/O capacity, and/or network capacity. Atblock1110, the job tracker and/or the supervisor controller can be configured to divide the job submission into sub-jobs for assignment to designated nodes with designated resource allocations. Atblock1112, the supervisor and/or the agent controller can be configured to monitor the nodes to determine if resource allocations are sufficient for the jobs to be processed based on the job group designation. If the determination is yes, then system can be configured to optionally return to block1112 to continue monitoring the acceptability of the resource allocation. If the determination atblock1112 is no, then the system can be configured to return toblock1108 in order to allocate nodes with specific CPU capacities, RAM capacities, disk I/O capacity, and/or network capacities for processing the job based on the designated job group.
• FIG. 12 is a flowchart depicting an embodiment of a process for monetizing resources on a computer cluster, for example, selling computer resources on a cluster to customers. In an embodiment, the selling of computer resources is different from selling virtual machines because the latter requires that whole virtual machines be sold to customers whereas the former requires only the computer resources to be sold to the customer. The selling of computer resources can be more efficient and/or more cost effective for the customer and/or the operator of the cluster.
• One of the ordinary skill in the art will appreciate that the monetizing or selling of computer resources need not require the actual sale of computer resources for currency but rather can also be applied to the context where resources are accounted for through intra-company budgeting. For example, the system can be configured to provide computer resources of the cluster to departments (for example, legal department, marketing department, human resources department, and the like) of a company based on a service plan level assigned to the department. In an embodiment, the service plan level assigned to a company department can equate to a budgetary accounting to the department for the company's costs in operating and maintaining the computer cluster.
• The process can begin atblock1202 with the accessing of a job submission atblock1204. The system can be configured to determine at block1206 a customer service plan level for the particular job submission. The system can be configured to determine the customer service plan level by accessing the customer database/serviceplan levels database1220. Customers can select service level requirements and/or plans atblock1218 where such data is stored in the customer database/service planslevels database1220.
• Atblock1208, the system can be configured to determine the resources necessary to complete the job submission. Atblock1212, the job tracker or are other management software and/or the supervisor controller can be configured to divide the job submission into some jobs for assignment to designated nodes with designated resource allocations based on the service plan level of the customer. Atblock1212, the system can be configured to determine if resources are available to process the sub-job based on the service plan level of the customer. If the determination is yes, then atblock1214 the system can be configured to assign the sub-job to an available node based on the service plan level of the customer. If the determination atblock1212 is no, then the system can be configured to add the sub-job to a queue for processing after a node becomes available based on the service plan level of the customer. In an embodiment, the customer selection of service level requirements can be specified differently for a particular job from a customer. For example, a customer may specify a higher service level for an urgent job than for that customer's usual jobs, and if meeting that higher service level requires additional resources, the system can be configured to charge the customer more for running that job or sub-job than if the customer had received the usual service level.
• FIG. 13 is a block diagram depicting a high level overview of an embodiment of a computer cluster comprising heterogeneous nodes. In an embodiment,client1302 can submit jobs for processing on cluster1301 tomaster node1304. In an embodiment, themaster node1304 can comprise a job tracker orother management software1306 that can be configured to receive job submissions from theclient1302. The job tracker orother management software1306 can be configured to analyze the job submission and/or be configured to divide the job into sub-jobs for processing byvarious nodes1310,1324 in the cluster1301.
• In an embodiment, thenodes1310,1324 can comprise a task tracker forother node manager1312,1326 and can comprise anagent controller1314,1328. The task tracker orother node manager1312,1326 can be configured to receive and/or tract the sub-job from the job tracker orother management software1306. In an embodiment, theagent controller1314,1328 can be configured to also track and monitor the processing of the sub-job by the node. In an embodiment, theagent controller1314,1328 can also be configured to determine the total available computer resources that are provided for by aparticular node1310,1324. For example, thenode1310 can provide a total of 100 units ofCPU capacity1316, 100 units ofRAM capacity1318, 100 units ofnetwork capacity1320, and 100 units ofIO capacity1322.
• By comparison, thenode1324 can provide 200 units ofCPU capacity1330, 300 units ofRAM capacity1332, 250 units ofnetwork capacity1334, and 40 units ofIO capacity1336. In determining the total available computer resources provided for by a particular node, theagent controller1314,1328 can be configured to transmit such data to thesupervisor controller1308 in order for the supervisor controller to determine a global awareness of the total amount of computer resources available in the cluster.
• In an embodiment, theagent controller1314,1328 can also be configured to determine the amount of computer resources utilized by the jobs being operated on by a particular noted. For example, theagent controller1314,1328 an be configured to determine that a particular job is utilizing -50 units ofCPU capacity1316 onnode1310. Further, theagent controller1328 can be configured to determine that a second job is utilizing 100 units ofCPU capacity1330 onnode1324. Theagent controller1314,1328 can be configured to transmit the CPU usage data to thesupervisor controller1308.
• In an embodiment theagent controller1314,1328 can be configured to determine the amount of computer resources that are not being utilized at a particular node. For example, theagent controller1314 can be configured to determine that 50 units ofCPU capacity1316 are not being utilized by the job being operated on bynode1310. Similarly, theagent controller1328 can be configured to determine that 100 units ofCPU capacity1330 are not being utilized by the second job that is being operated on bynode1324. Theagent controller1314,1328 can be configured to transmit the available unused computer resource data to thesupervisor controller1308. In an embodiment, thesupervisor controller1308 and/or theagent controller1314,1328 can be configured to allocate additional resources to existing jobs being operated on by nodes in the cluster or can be configured to allocate additional jobs or sub-jobs to the nodes in order to fully utilize the available computer resources that are provided for by the nodes.
• As illustrated inFIG. 13,node1310 andnode1324 provide differing amounts of computer resources. Accordingly thenode1310 and thenode1324 are not homogeneous but rather together make up a heterogeneous cluster because the cluster is said to have different kinds of computer servers that offer varying amounts of computer resources. By tracking the amount of available computer resources not being utilized by current jobs on the nodes, theagent controller1314,1328 can be configured to enable the efficient utilization of heterogeneous clusters.
• In an embodiment, theagent controller1314,1328 in conjunction with thesupervisors controller1308 can be configured to fully utilize the available computer resources being offered by the heterogeneous cluster by allocating as many jobs to each of the different nodes based on each of the nodes available computer resources that can be utilized for processing additional jobs.
• FIG. 14 is a flowchart depicting an embodiment of a process for processing jobs utilizing a heterogeneous computer clusters. The process can begin atblock1402 with the accessing of a job submission atblock1014. Atblock1406, the job tracker or other management software, and/or supervisor controller can be configured to divide the job submission into tasks or sub-jobs for assignment to a first node and a second node. Atblock1408, the agent controls operating on the first node and the second node can be configured to determine if additional computer resources are available for processing additional jobs. If the determination atnode1 is that no computer resources are available atnode1 for processing additional jobs, then atblock1410 the agent controller can be configured to loop back to block1408 to continue to check whether thenode1 has additional resources available for processing other jobs because the utilization of computer resources on any particular node is continuously changing.
• If the determination atnode2 is that there are additional resources available onnode2 for processing additional jobs, then the agent controller operating undernode2 can be configured to transmit the resource availability data ofnode2 to the supervisor controller and/or job tracker or other management software operating on the master node. Atblock414, the job tracker or other management software, and/or the supervisor controller can be configured to assign additional tasks for sub-jobs to the second node. Atblock414 the agent controller can be configured to loop back to block1408 to continuously check whether additional resources become available for processing other jobs. This process can enable the full utilization of heterogeneous clusters because the system continuously checks each node to determine whether additional computer resources are available for processing additional jobs.
• FIG. 15 is a schematic diagram illustrating an embodiment of utilizing job histories for improving resource allocation of a computer cluster. The top half ofFIG. 15 illustrates a standard allocation of sub-jobs and/or tasks. The bottom half ofFIG. 15 illustrates a dynamic allocation of sub-jobs and/or tasks based on job history data. In an embodiment, a first job has a historical resource utilization chart illustrated inchart1508. As can be seen, the first job has at first a high resource utilization at the beginning stages of processing the job and then has a period of low resource utilization in the middle of the period and towards the end of the period the first job has a high resource utilization.
• A second job comprises a resource utilization illustrated inchart1510. At the start, the second job has a low resource utilization and towards the middle period of the job, there is a high resource utilization and towards the end of the job there is very low resource utilization. A typical cluster system would assignjob1 to a first node and would assignjob2 to a second node.Chart1502 illustrates the resource utilization ofjob1 versus the overall resources available for allocation atnode1.Chart1504 illustrates the resource utilization ofjob2 relative to the overall resources available for allocation atnode2. As illustrated incharts1502 and1504, there are significant periods where the computer resources ofnode1 andnode2 are underutilized because of the low resource utilization periods ofjob1 andjob2. Accordingly it would be advantageous to operatejobs1 and2 on a single node in order to have full utilization of a particular node.
• In an embodiment, the resource monitoring and allocation systems disclosed herein can be configured to allow for more efficient utilization of nodes by analyzing the historical resource utilization of jobs and predicting the utilization rates of particular jobs in order to combine certain jobs with other jobs that would allow for more efficient utilization of the resources available for allocation at a particular node. For example, as illustrated inchart1512 and1514,job1 comprises a low resource utilization during the middle of the period for completing the job whilejob2 has a high resource utilization rate during the middle period of completing the job. Accordingly by sending bothjob1 andjob2 to a single node, there can be more efficient overall use of the computer resources available for allocation atnode1 as illustrated inchart1506.
• FIG. 16 is a flowchart depicting an embodiment of a process for generating reports relating to hardware modifications and/or additions to a computer cluster. The process can begin atblock1602 by receiving a job submission atblock1604. Atblock1606, the supervisor controller can be configured to determine the resources necessary to process a job based on client specified performance goals. Atblock1608, the supervisor controller can be configured to determine resources and/or nodes available to process the job. Atblock1610, the job tracker or other management software can be configured to assign the sub-jobs to available nodes. In an embodiment, the supervisor controller can be configured to designate the allocation of computer resources for each sub-job at each node. Atblock1612, the agent controller can be configured to determine periodically or continuously the status and/or resource utilization of each sub-job at each node.
• Atblock1614, the agent controller and/or the supervisor controller can be configured to identify resource limitation bottlenecks in the cluster based on the determining of the status and/or resource utilization at the various nodes in the cluster. Atblock1616, the supervisor controller can be configured to generate a report listing resource limitation bottlenecks and/or hardware modifications and/or additions to mitigate bottlenecks in the cluster. Atblock1616, the system can be configured to optionally loop back to block1612 in order for the agent controller to periodically or continuously determine the status and/or resource utilization of each sub-job at each node.
• FIG. 17 is a flowchart depicting an embodiment of a process for generating reports relating to resource reallocation on a computer cluster. The process can begin atblock1702 by receiving a job submission and at least one of a user identifier, job group, department, user group, or the like atblock1704. Atblock1706, the supervisor controller can be configured to determine the resources necessary to process the job based on the client specified performance goals. Atblock1708, the supervisor controller can be configured to determine the available resources and/or available nodes for processing the job. Atblock1710, the job tracker or other management software or the supervisor controller can be configured to assign sub-jobs to the available nodes.
• Atblock1710, the supervisor controller can be configured to designate the allocation of resources for each sub-job at each node. Atblock1712, the agent controller can be configured to determine periodically or continuously the status and/or resource utilization of each sub-job at each node. Atblock1714, the supervisor controller can be configured to identify resource limitation bottlenecks in the cluster based on the determining of the status and/or resource utilization of the sub-jobs at the various nodes in the cluster. Atblock1716, the supervisor controller can be configured to generate a report listing the resource limitation bottlenecks and/or at least one of the user identifiers, job groups, departments, user groups, or the like that is causing the bottlenecks. Atblock1716, the system can be configured to loop back to block1712 in order for the agent controller to determine periodically or continuously the status and/or resource utilization of each sub-job at each node.
• FIG. 17A is a flowchart depicting an embodiment of a process for determining resource reallocation levels for application to jobs or sub-jobs. In an embodiment, the system can be configured to select a subset of tasks and/or sub-jobs of a particular job and tweak the resource allocation settings or configurations for the selected subset of tasks and/or sub-jobs in order to discover how the tasks and/or sub-jobs react to the resource allocation settings. The system can be configured to apply different resource allocation settings or configurations to different subsets in order to determine the best resource allocation settings for applying to particular sub-jobs. For example, with respect to the java virtual machine heap setting, the system can be configured to set the java virtual machine heap setting to aggressively return unused memory. The system can be configured to monitor the performance characteristics of the sub-jobs based the foregoing setting. The system can be configured to use the resulting information to apply better control for future tasks or sub-jobs of the current job or the future instances of the job.
• Similarly, the system can be configured to determine the actual current capacity of a resource, such as disk I/O capacity or network capacity by dynamically adjusting threshold levels for access to these resources by various tasks or sub-jobs. For example, the system can be configured to increase the network bandwidth requested by all tasks or sub-jobs (added together) over the course of several time intervals until the network stops providing the extra requested bandwidth, then assuming that observed maximum bandwidth provided is the currently available bandwidth. The system can be configured to repeat this process continuously so that each node maintains an estimate of the available maximum capacity for each resource.
• With reference toFIG. 17A, in an embodiment, the process can begin atblock1718 with the system receiving a job submission atblock1720. Atblock1722, the system can be configured to divide the job into a plurality of sub-jobs. Atblock1724, the system can be configured to select one or more subsets of the sub-jobs for applying experiments of resource allocations to determine which resource allocation levels yield the best performance characteristics for the particular type of sub-jobs at issue. Atblock1726, the system can be configured to apply various resource allocation levels to different subsets of sub-jobs. Atblock1728, the system can be configured to monitor performance characteristics of sub-jobs in the various subsets based on the applied resource allocation levels. Atblock1730, the system can be configured to determine which resource allocation levels yield the best performance characteristics for the sub-job type. Atblock1732, the system can be configured to store the resource allocation level that yield the best performance characteristics for future application to similar sub-job types or other sub-jobs that are part of the overall job.
• FIG. 18 is a block diagram depicting a high level overview of an embodiment of a computer cluster comprising a dynamic monitoring and/or resource allocation system. In an embodiment, theclient1802 can submit jobs to themaster node1804 in order to have the job processed by thecluster1801. Themaster node1804 can comprise amanagement software1806 and can comprise asupervisor controller1808. In an embodiment, themanagement software1806 can be configured to analyze the job received from theclient1802 and divide the job into various sub-jobs for processing by the various nodes in the cluster. Thecluster1801 can comprise a plurality ofnodes1822,1842.
• In an embodiment, themanagement software1806 can be configured to send sub-jobs1820,1818,1838,1840 to thevarious nodes1822,1842 for processing. In an embodiment, theother tracking software1824,1844 can be configured to receive the sub-job from the master node in order for the sub-jobs to be processed on the nodes. In an embodiment, theagent controller1826,1846 can be configured to track the progress of the sub-jobs that are being processed by the nodes and can be configured to determine the resource allocation usage of each of the jobs running on each of the nodes.
• In an embodiment theagent controller1826,1846 can be configured to transmit the resource utilization data to thesupervisor controller1808 that operates in themaster node1804. Thesupervisor controller1808 and/or theagent controller1826,1846 can be configured to determine whether the resource allocation of a particular job on a particular node should be reduced or increased or remain the same. In an embodiment, theagent controller1826,1846 can be configured to generate instructions for processing at the kernel, the process, orother module1828,1848 in order to reduce, increase, or keep the resource allocation for the particular sub-job at a particular node.
• In an embodiment, thenodes1822,1842 can be configured to run other software applications including but not limited toweb server1830,1850,database1832,virtual machine1852,impala query engine1834,database query manager1854, andother software applications1836,1856. In an embodiment, theagent controller1826,1846 can be configured to determine the resource utilization of each of the software application running on the various nodes. In an embodiment, theagent controller1826,1846 can be configured to transmit the resource utilization of the software applications operating on each of the nodes to thesupervisor controller1808. Theagent controller1826,1846 and/or thesupervisor controller1808 can be configured to determine that the resources being utilized by a particular software application on a particular node should be reduced, increased, or remain the same.
• In an embodiment, thecluster1801 can comprise anetwork controller1812. Thenetwork controller1812 can comprise a network router, a network switch, or the like. In an embodiment, thenetwork controller1812 can comprise aagent controller1810. Theagent controller1810 can be configured to determine the resource utilization of thenetwork controller1810 by certain nodes, jobs, sub-jobs, or applications. In an embodiment, theagent controller1810 can be configured to transmit the resource utilization data to thesupervisor controller1808. Thesupervisor controller1808 and/or theagent controller1810 can be configured to reallocate the use of resources provided for by thenetwork controller1810 for certain nodes, jobs, sub-jobs, and/or applications.
• In an embodiment, thecluster1801 can be coupled or connected to anexternal resource1816. Theexternal resource1816 can include but is not limited to external databases, data extraction/transformation tools, web services, and the like. In an embodiment, theexternal resource1816 can comprise an agent controller. The agent controller can be configured to determine the usage of resources of theexternal resource1816 by nodes, jobs, sub-jobs, and/or applications. In an embodiment, the agent controller can be configured to transmit the resource utilization data to thesupervisor controller1808. In an embodiment, thesupervisor controller1808 and/or theagent controller1814 can be configured to determine whether the resource utilization of theexternal resource1816 by particular nodes, jobs, sub-jobs, and/or applications on a particular node should be reduced, increased, or remained the same.
Computer System
• In some embodiments, the systems, processes, and methods described above are implemented using a computing system, such as the one illustrated inFIG. 19. Theexample computer system1902 is in communication with one ormore computing systems1920 and/or one ormore data sources1922 via one ormore networks1918. WhileFIG. 19 illustrates an embodiment of acomputing system1902, it is recognized that the functionality provided for in the components and modules ofcomputer system1902 may be combined into fewer components and modules, or further separated into additional components and modules.
Dynamic Resource Monitoring/Allocation Module
• Thecomputer system1902 includes a dynamic resource monitoring/allocation module1914 that carries out the functions, methods, acts, and/or processes described herein. The dynamic resource monitoring/allocation module1914 is executed on thecomputer system1902 by acentral processing unit1910 discussed further below.
• In general the word “module,” as used herein, refers to logic embodied in hardware or firmware or to a collection of software instructions, having entry and exit points. Modules are written in a program language, such as JAVA, C or C++, or the like. Software modules may be compiled or linked into an executable program, installed in a dynamic link library, or may be written in an interpreted language such as BASIC letters, PERL, LUA, or Python. Software modules may be called from other modules or from themselves, and/or may be invoked in response to detected events or interruptions. Modules implemented in hardware include connected logic units such as gates and flip-flops, and/or may include programmable units, such as programmable gate arrays or processors.
• Generally, the modules described herein refer to logical modules that may be combined with other modules or divided into sub-modules despite their physical organization or storage. The modules are executed by one or more computing systems, and may be stored on or within any suitable computer readable medium, or implemented in-whole or in-part within special designed hardware or firmware. Not all calculations, analysis, and/or optimization require the use of computer systems, though any of the above-described methods, calculations, processes, or analyses may be facilitated through the use of computers. Further, in some embodiments, process blocks described herein may be altered, rearranged, combined, and/or omitted.
Computing System Components
• Thecomputer system1902 includes one or more processing units (CPU)1910, which may include a microprocessor. Thecomputer system1902 further includes amemory1912, such as random access memory (RAM) for temporary storage of information, a read only memory (ROM) for permanent storage of information, and amass storage device1904, such as a hard drive, diskette, or optical media storage device. Alternatively, the mass storage device may be implemented in an array of servers. Typically, the components of thecomputer system1902 are connected to the computer using a standards based bus system. The bus system can be implemented using various protocols, such as Peripheral Component Interconnect (PCI), Micro Channel, SCSI, Industrial Standard Architecture (ISA) and Extended ISA (EISA) architectures.
• Thecomputer system1902 includes one or more input/output (I/O) devices andinterfaces1908, such as a keyboard, mouse, touch pad, and printer. The I/O devices andinterfaces1908 can include one or more display devices, such as a monitor, that allows the visual presentation of data to a user. More particularly, a display device provides for the presentation of GUIs as application software data, and multi-media presentations, for example. The I/O devices andinterfaces1908 can also provide a communications interface to various external devices. Thecomputer system1902 may include one or moremulti-media devices1906, such as speakers, video cards, graphics accelerators, and microphones, for example.
Computing System Device/Operating System
• Thecomputer system1902 may run on a variety of computing devices, such as a server, a Windows server, and Structure Query Language server, a Unix Server, a personal computer, a laptop computer, and so forth. In other embodiments, thecomputer system1902 may run on a cluster computer system, a mainframe computer system and/or other computing system suitable for controlling and/or communicating with large databases, performing high volume transaction processing, and generating reports from large databases. Thecomputing system1902 is generally controlled and coordinated by an operating system software, such as z/OS, Windows 95, Windows 98, Windows NT, Windows 2000, Windows XP, Windows Vista, Windows 7, Linux, UNIX, BSD, SunOS, Solaris, or other compatible operating systems, including proprietary operating systems. Operating systems control and schedule computer processes for execution, perform memory management, provide file system, networking, and I/O services, and provide a user interface, such as a graphical user interface (GUI), among other things.
Network
• Thecomputer system1902 illustrated inFIG. 19 is coupled to anetwork1918, such as a LAN, WAN, or the Internet via a communication link1916 (wired, wireless, or a combination thereof).Network1918 communicates with various computing devices and/or other electronic devices.Network1918 is communicating with one ormore computing systems1920 and one ormore data sources1922. The dynamic resource monitoring/allocation module1914 may access or may be accessed by computingsystems1920 and/ordata sources1922 through a web-enabled user access point. Connections may be a direct physical connection, a virtual connection, and other connection type. The web-enabled user access point may include a browser module that uses text, graphics, audio, video, and other media to present data and to allow interaction with data via thenetwork1918.
• The output module may be implemented as a combination of an all-points addressable display such as a cathode ray tube (CRT), a liquid crystal display (LCD), a plasma display, or other types and/or combinations of displays. The output module may be implemented to communicate withinput devices1908 and they also include software with the appropriate interfaces which allow a user to access data through the use of stylized screen elements, such as menus, windows, dialogue boxes, tool bars, and controls (e.g., radio buttons, check boxes, sliding scales, and so forth). Furthermore, the output module may communicate with a set of input and output devices to receive signals from the user.
Other Systems
• Thecomputing system1902 may include one or more internal and/or external data sources (e.g., data sources1922). In some embodiments, one or more of the data repositories and the data sources described above may be implemented using a relational database, such as DB2, Sybase, Oracle, CodeBase, and Microsoft® SQL Server as well as other types of databases such as a flat-file database, an entity relationship database, and object-oriented database, and/or a record-based database.
• Thecomputer system1902 also accesses one ormore databases1922. Thedatabases1922 may be stored in a database or data repository. Thecomputer system1902 may access the one ormore databases1922 through anetwork1918 or may directly access the database or data repository through I/O devices and interfaces1908. The data repository storing the one ormore databases1922 may reside within thecomputer system1902.
• Conditional language, such as, among others, “can,” “could,” “might,” or “may,” unless specifically stated otherwise, or otherwise understood within the context as used, is generally intended to convey that certain embodiments include, while other embodiments do not include, certain features, elements and/or steps. Thus, such conditional language is not generally intended to imply that features, elements and/or steps are in any way required for one or more embodiments or that one or more embodiments necessarily include logic for deciding, with or without user input or prompting, whether these features, elements and/or steps are included or are to be performed in any particular embodiment. The headings used herein are for the convenience of the reader only and are not meant to limit the scope of the inventions or claims.
Additional Embodiments
• Although this invention has been disclosed in the context of certain preferred embodiments and examples, it will be understood by those skilled in the art that the present invention extends beyond the specifically disclosed embodiments to other alternative embodiments and/or uses of the invention and obvious modifications and equivalents thereof. Additionally, the skilled artisan will recognize that any of the above-described methods may be carried out using any appropriate apparatus. Further, the disclosure herein of any particular feature, aspect, method, property, characteristic, quality, attribute, element, or the like in connection with an embodiment may be used in all other embodiments set forth herein. Thus, it is intended that the scope of the present invention herein disclosed should not be limited by the particular disclosed embodiments described above.

Claims (40)

1. (canceled)

2. (canceled)

3. (canceled)

4. (canceled)

5. (canceled)

6. (canceled)

7. (canceled)

8. (canceled)

9. (canceled)

10. (canceled)

11. (canceled)

12. (canceled)

13. (canceled)

14. (canceled)

15. (canceled)

16. (canceled)

17. (canceled)

18. (canceled)

19. (canceled)

20. (canceled)

21. A non-virtual computer cluster system, the system comprising:

a management computing device comprising one or more computer readable storage devices configured to store a plurality of computer executable instructions and one or more hardware computer processors in communication with the one or more computer readable storage devices and configured to execute the plurality of computer executable instructions in order to cause the management computing device to coordinate processing of a plurality of sub-jobs for an overall job; and

a plurality of computer system nodes configured to perform processing of the plurality of sub-jobs for the overall job, each of the plurality of computer system nodes comprising one or more computer readable storage devices configured to store a plurality of computer executable instructions and one or more hardware computer processors in communication with the one or more computer readable storage devices and configured to execute the plurality of computer executable instructions in order to cause each of the plurality of computer system nodes to:

monitor utilization of system resources of the computer system node for processing one or more of the plurality of sub-jobs, the system resources comprising at least one of CPU, electronic storage input/output, network, and memory of the computer system node; and

periodically report the monitored system resource utilization to the management computing device,

wherein the management computing device is a separate computing device from each of the plurality of computer system nodes, and

wherein the management computing device is configured to coordinate processing of the plurality of sub-jobs by the plurality of computer system nodes based at least in part on the monitored system resource utilization of the plurality of computer system nodes.

22. The non-virtual computer cluster system ofclaim 21, wherein the management computing device is further configured to determine progress for completing the overall job by tracking the plurality of sub-jobs being processed by the plurality of computer system nodes.

23. The non-virtual computer cluster system ofclaim 21, wherein the management computing device is further configured to dynamically increase processing of higher priority sub-jobs.

24. The non-virtual computer cluster system ofclaim 21, wherein the management computing device operates a master node of the computer cluster.

25. The non-virtual computer cluster system ofclaim 21, wherein the management computing device is further configured to dynamically adjust system resource allocations for each sub-job on the plurality of computer system nodes.

26. The non-virtual computer cluster system ofclaim 25, wherein the dynamic adjustment of system resource allocations is based at least in part on the periodic reporting from the plurality of computer system nodes.

27. The non-virtual computer cluster system ofclaim 26, wherein the dynamic adjustment of system resource allocations comprises allocating additional central processing unit resources to a sub-job operating on a computer system node based at least in part on the management computing device determining that a total amount of unused central processing unit resources available at the computer system node exceeds a threshold level.

28. The non-virtual computer cluster system ofclaim 27, wherein the dynamic adjustment of system resource allocations comprises allocating additional networking resources to a sub-job operating on a computer system node based at least in part on the management computing device determining that a total amount of unused networking resources available at the computer system node exceeds a threshold level.

29. The non-virtual computer cluster system ofclaim 27, wherein the dynamic adjustment of system resource allocations comprises allocating additional random-access memory resources to a sub-job operating on a computer system node based at least in part on the management computing device determining that a total amount of unused random-access memory resources available at the computer system node exceeds a threshold level.

30. The non-virtual computer cluster system ofclaim 27, wherein the dynamic adjustment of system resource allocations comprises allocating additional electronic storage input-output resources to a sub-job operating on a computer system node based at least in part on the management computing device determining that a total amount of unused electronic storage input-output resources available at the computer system node exceeds a threshold level.

31. The non-virtual computer cluster system ofclaim 21, wherein the management computing device is further configured to receive goals specified by an operator of the computer cluster system, wherein the goals comprise a priority level for one or more of the plurality of sub-jobs.

32. The non-virtual computer cluster system ofclaim 31, wherein the management computing device is configured to assign a first level of system resource allocations to a first sub-job and a second level of system resource allocations to a second sub-job, wherein the first level is greater than the second level, wherein the first sub-job comprises a first priority level and the second sub-job comprises a second priority level, wherein the first priority level is greater than the second priority level.

33. The non-virtual computer cluster system ofclaim 31, wherein the goals specified by the operator of the computer cluster comprises a user group preference for one or more of the plurality of sub-jobs.

34. The non-virtual computer cluster system ofclaim 33, wherein the management computing device is configured to assign a first level of system resource allocations to a first sub-job and a second level of system resource allocations to a second sub-job, wherein the first level is greater than the second level, wherein the first sub-job comprises a first user group preference and the second sub-job comprises a second user group preference, wherein sub-jobs associated with the first user group preference are processed with greater priority than sub-jobs associated with the second user group preference.

35. A computer-implemented method for coordinating processing of a plurality of sub-jobs for an overall job by a plurality of computer system nodes in a non-virtual computer cluster system, the method comprising:

monitoring, by each of the plurality of computer system nodes of the computer cluster system, utilization of system resources of each of the plurality of computer system nodes for processing one or more of the plurality of sub-jobs for the overall job, the system resources comprising at least one of CPU, electronic storage input/output, network, and memory of the particular computer system node;

periodically reporting, by each of the plurality of computer system nodes of the computer cluster system, the monitored system resource utilization to the management computing device of the computer cluster system;

receiving, by the management computing device, the monitored system resource utilization of each of the plurality of computer system nodes of the computer cluster system; and

coordinating, by the management computing device, processing of the plurality of sub-jobs by the plurality of computer system nodes based at least in part on the monitored system resource utilization of the plurality of computer system nodes,

wherein each of the plurality of computer system nodes and the management computing device comprises a computer processor and an electronic storage medium.

36. The computer-implemented method ofclaim 35 further comprising determining, by the management computing device, progress for completing the overall job by tracking the plurality of sub-jobs being processed by the plurality of computer system nodes.

37. The computer-implemented method ofclaim 35 further comprising receiving, by the management computing device, a priority level for one or more of the plurality of sub-jobs from an operator of the computer cluster system.

38. The computer-implemented method ofclaim 37 further comprising assigning, by the management computing device, a first level of system resource allocations to a first sub-job and a second level of system resource allocations to a second sub-job, wherein the first level is greater than the second level, wherein the first sub-job comprises a first priority level and the second sub-job comprises a second priority level, wherein the first priority level is greater than the second priority level.

39. The computer-implemented method ofclaim 35, further comprising dynamically adjusting, by the management computing device, system resource allocations for one or more of the plurality of sub-jobs being processed by the plurality of computer system nodes.

40. The computer-implemented method ofclaim 39, wherein the dynamic adjustment of system resource allocations is based at least in part on the periodic reporting from the plurality of computer system nodes.

Images