

技术领域technical field
本发明涉及一种服务网格调度器及调度作业的方法,尤其是一种设置委派执行作业执行模块的网格作业调度器,以及利用该网格作业调度器对作业进行调度的方法。The present invention relates to a service grid scheduler and a method for scheduling jobs, in particular to a grid job scheduler for setting a delegated execution job execution module, and a method for scheduling jobs using the grid job scheduler.
背景技术Background technique
随着社会网络化、信息化的程度的不断提高,在从商业分析到科学研究等各个领域内对计算能力的要求大大提高,目前,传统的基于单机/单节点的计算系统已经远远不能满足如处理、存储数据等各类应用对计算能力的需求。为了满足对计算能力的需求,网格计算应运而生,网格计算是利用网络中一些闲置的处理能力来解决复杂问题的计算模式,即利用互联网技术,把分散在不同地理位置的计算机组成一台虚拟超级计算机。每一台参与的计算机就是其中的一个“节点”,所有的计算机就组成了一张节点网—网格,网格的计算能力强且费用低。在实质上来说“网格计算”是一种分布式应用,网格中的每一台计算机只是完成工作的一个小部分,虽然单台计算机的运算能力有限,但成千上万台计算机组合起来的计算能力就可以达到超级计算机的计算能力,因此,利用网格技术是目前提高计算能力的一主要途径。在网格技术中,有效进行作业管理、作业调度和资源管理是网格技术必须解决的问题。With the continuous improvement of social networking and informatization, the requirements for computing power in various fields from business analysis to scientific research have been greatly increased. At present, the traditional single-machine/single-node computing system is far from being able to meet Various applications such as processing and storing data require computing power. In order to meet the demand for computing power, grid computing came into being. Grid computing is a computing model that uses some idle processing power in the network to solve complex problems. A virtual supercomputer. Each participating computer is one of the "nodes", and all the computers form a node network—grid, which has strong computing power and low cost. In essence, "grid computing" is a distributed application. Each computer in the grid is only a small part of the work. Although the computing power of a single computer is limited, thousands of computers combined The computing power of a supercomputer can reach the computing power of a supercomputer. Therefore, the use of grid technology is currently a major way to improve computing power. In grid technology, effective job management, job scheduling and resource management are problems that grid technology must solve.
现有技术中,OMII(Open Middleware Infrastructure Institute,开放的中间件基础结构研究所)开发出的网格作业调度器中采用了一个标准的基于Web服务的作业提交和监控GridSAM模块,有效实现了对作业提交和监控,但利用该网格作业调度器调度作业时存在以下缺陷:利用该网格作业调度器只能简单的对提交的作业进行调度,而在调度作业时没有考虑作业的类型,致使有些类型的作业背调度后不能被执行,给用户带来极大不便;同时,利用该网格作业调度器不能对提交的作业进行再调度,即不能在整个网格范围内根据作业执行节点的负载均衡分配作业,而只能分配给与该调度器相连接的作业执行模块进行执行,这样,当某一执行作业的节点地负载过重时,便影响了对作业处理的速度和质量,同时也没有充分利用闲余的计算资源,没有从根本上实现对作业的调度。In the prior art, the grid job scheduler developed by OMII (Open Middleware Infrastructure Institute) adopts a standard Web service-based job submission and monitoring GridSAM module, which effectively realizes the Job submission and monitoring, but there are the following defects when using the grid job scheduler to schedule jobs: the grid job scheduler can only simply schedule the submitted jobs, and does not consider the type of jobs when scheduling jobs, resulting in Some types of jobs cannot be executed after back-scheduling, which brings great inconvenience to users; at the same time, the submitted jobs cannot be rescheduled by using the grid job scheduler, that is, the jobs cannot be executed according to the number of nodes in the entire grid. Load balancing distributes jobs, and can only be assigned to the job execution module connected to the scheduler for execution. In this way, when the load of a node executing a job is too heavy, the speed and quality of job processing are affected, and at the same time It also does not make full use of idle computing resources, and does not fundamentally realize job scheduling.
发明内容Contents of the invention
本发明的第一目的在于针对现有技术中的网格作业调度器不能实现对作业的调度的现状,提供一种网格作业调度器,在网格作业调度器中设置一委派执行作业执行模块,通过该委派执行作业执行模块实现对作业的有效调度。The first purpose of the present invention is to provide a grid job scheduler in view of the current situation that the grid job scheduler in the prior art cannot realize job scheduling, and a delegated execution job execution module is set in the grid job scheduler , the effective scheduling of jobs is realized through the delegated execution job execution module.
本发明的第二目的在于针对现有技术不能实现对作业的有效调度的现状,提供一种调度作业的方法,通过网格作业调度器中设置的委派执行作业执行模块对作业及资源的分析、匹配实现对作业的有效调度。The second purpose of the present invention is to provide a method for scheduling jobs in view of the current situation that the prior art cannot realize effective scheduling of jobs, which analyzes the jobs and resources through the delegated execution job execution module set in the grid job scheduler, Matching enables efficient scheduling of jobs.
为实现上述第一目的,本发明提供一种网格作业调度器,包括依次连接的作业提交描述语言解析模块、调度模块及信息服务获取模块,相互连接的资源匹配模块及POSIX作业调度模块,所述调度模块还与所述资源匹配模块连接,所述作业提交描述语言解析模块与所述调度模块中的作业管理模块连接,所述信息服务获取模块与所述调度模块中的资源获取模块连接,所述资源匹配模块与所述调度模块中的作业调度模块连接,所述资源获取模块及所述作业调度模块同时与所述作业管理模块连接;另外,该网格作业调度器还包括:In order to achieve the above-mentioned first purpose, the present invention provides a grid job scheduler, including sequentially connected job submission description language parsing module, scheduling module and information service acquisition module, resource matching module and POSIX job scheduling module connected to each other, so The scheduling module is also connected to the resource matching module, the job submission description language parsing module is connected to the job management module in the scheduling module, and the information service obtaining module is connected to the resource obtaining module in the scheduling module, The resource matching module is connected to the job scheduling module in the scheduling module, and the resource acquisition module and the job scheduling module are connected to the job management module at the same time; in addition, the grid job scheduler also includes:
一委派执行作业执行模块,与所述资源匹配模块连接,提供与作业执行模块及其他的网格作业调度器的作业提交描述语言解析模块连接的接口,用于根据作业类型及与所述委派执行作业执行模块连接的作业执行模块的工作性能信息,将作业委派给与所述委派执行作业执行模块连接的其他的网格作业调度器进行调度。A delegated execution job execution module, connected with the resource matching module, providing an interface connected with the job execution module and other job submission description language parsing modules of the grid job scheduler, for executing according to the type of the job and the assignment Based on the work performance information of the job execution module connected to the job execution module, the job is delegated to other grid job schedulers connected to the delegated job execution module for scheduling.
为实现上述第二目的,本发明提供一种调度作业的方法,其执行以下步骤:In order to achieve the above-mentioned second purpose, the present invention provides a method for scheduling jobs, which performs the following steps:
步骤1、用户输入待处理信息;Step 1. The user inputs the information to be processed;
步骤2、作业提交描述语言解析模块对所述待处理信息进行分析并构造作业;
步骤3、所述作业提交描述语言解析模块将所述作业保存在调度模块中的作业管理模块中;
步骤4、所述调度模块中的资源获取模块通过信息服务获取模块获取资源信息并为所述作业匹配资源;
步骤5、所述调度模块中的作业调度模块将已被匹配了资源的待调度作业发送给资源匹配模块,该资源匹配模块判断该资源匹配模块所在节点的作业执行模块是否支持所述待调度作业,若支持,则执行步骤6;否则,跳转执行步骤8;
步骤6、所述资源匹配模块判断所述待调度作业是否为POSIX类型的作业,是则执行步骤7;否则,跳转执行步骤8;
步骤7、所述资源匹配模块将所述待调度作业转发给POSIX作业调度模块,由该POSIX作业调度模块将该待调度作业发送给一作业执行模块以对所述待调度作业进行处理,然后跳转执行步骤9;
步骤8、所述资源匹配模块将所述待调度作业转发给委派执行作业执行模块,该委派执行作业执行模块根据所述待调度作业的类型及与该委派执行作业执行模块连接的节点的工作性能信息将所述待调度作业委派给一网格作业调度器以对该待调度作业进行调度及执行;Step 8. The resource matching module forwards the job to be scheduled to the delegated job execution module, and the delegated job execution module is based on the type of the job to be scheduled and the performance of the nodes connected to the delegated job execution module information to delegate the unscheduled job to a grid job scheduler for scheduling and execution of the unscheduled job;
步骤9、结束。Step 9, end.
由上述网格作业调度器及调度方法的技术方案可知,本发明在网格作业调度器中引入了委派执行作业执行模块,该模块可根据待调度作业的类型对该作业分配作业,有效保证了被分配作业的可执行性;同时,还根据与该网格作业调度器相连接的其它调度器节点的工作性能信息决定是否对该作业进行再调度,在整个网格环境内实现了资源的均衡,提高了处理作业的速度和质量,有效利用了资源,方便了用户。It can be known from the above-mentioned technical solutions of the grid job scheduler and the scheduling method that the present invention introduces a delegated execution job execution module into the grid job scheduler, which can assign jobs to the job according to the type of job to be scheduled, effectively ensuring Executability of the assigned job; at the same time, it also decides whether to reschedule the job according to the work performance information of other scheduler nodes connected to the grid job scheduler, and realizes the balance of resources in the entire grid environment , improve the speed and quality of processing jobs, effectively use resources, and facilitate users.
下面通过附图和实施例,对本发明的技术方案做进一步的详细描述。The technical solutions of the present invention will be described in further detail below with reference to the accompanying drawings and embodiments.
附图说明Description of drawings
图1为本发明网格作业调度器的一结构示意图;Fig. 1 is a schematic structural diagram of the grid job scheduler of the present invention;
图2为本发明网格作业调度器的另一结构示意图;Fig. 2 is another schematic structural diagram of the grid job scheduler of the present invention;
图3为本发明网格作业调度方法的流程图。FIG. 3 is a flow chart of the grid job scheduling method of the present invention.
具体实施方式Detailed ways
参见图1,图1所示为本发明网格作业调度器的一具体结构示意图,由图1可知,该网格作业调度器包括依次连接的作业提交描述语言(JobSubmission Description Language,JSDL)解析模块1、调度模块2及信息服务获取模块3,相互连接的资源匹配模块4及POSIX(Portable Operating SystemInterface Standard)作业调度模块5,以及与资源匹配模块4连接的委派执行作业执行模块6。其中,调度模块2包括依次连接的资源获取模块21、作业管理模块22和作业调度模块23,JSDL解析模块1与调度模块2连接具体是与作业管理模块22连接,信息服务获取模块3与调度模块2连接具体是与资源获取模块21连接,资源匹配模块4与作业调度模块23连接。委派执行作业执行模块6提供与作业调度器的JSDL解析模块及作业执行模块连接的接口,可根据待调度作业的类型以及与该委派执行作业执行模块6连接的其它调度器节点的工作性能如CPU负载、剩余内存、硬盘可用空间等信息将作业分配给该委派执行作业执行模块6所在节点的合适的作业执行模块进行执行,或将该作业委派给其它的网格作业调度器以进行再调度,最终将该作业分配给负载较轻的节点执行,在整个网格环境内实现了资源的均衡,提高了处理作业的速度和质量,有效利用了资源,方便了用户。Referring to Fig. 1, Fig. 1 shows a specific structural schematic diagram of the grid job scheduler of the present invention, as can be seen from Fig. 1, the grid job scheduler includes sequentially connected job submission description language (JobSubmission Description Language, JSDL) parsing modules 1.
参见图2,图2所示为本发明网格作业调度器的另一具体结构示意图,由图2可知,该网格作业调度器在图1所示的网格作业调度器内,还增设了一Web作业调度模块7,该Web作业调度模块7与资源匹配模块4连接,提供Web服务接口,用于调用Web服务执行模块执行Web服务类型的作业,使得对Web服务类型的作业的处理快捷方便,进一步提高了调度作业的速度。Referring to Fig. 2, Fig. 2 shows another specific structural schematic diagram of the grid job scheduler of the present invention, as can be seen from Fig. 2, the grid job scheduler is added in the grid job scheduler shown in Fig. 1 A Web
参见图3,图3所示为本发明网格作业调度方法的一具体流程,其具体执行以下步骤:Referring to Fig. 3, Fig. 3 shows a specific process of the grid job scheduling method of the present invention, which specifically performs the following steps:
步骤31、用户通过客户端输入一个待处理信息;
步骤32、客户端将用户输入的待处理信息传送给JSDL。解析模块,该JSDL,解析模块对待处理信息进行分析并构造作业,具体操作步骤如下:
步骤321、客户端将用户输入的待处理信息传送给JSDL解析模块;Step 321, the client sends the pending information input by the user to the JSDL parsing module;
步骤322、JSDL解析模块分析待处理信息是JSDL文件或Job对象或JSDL文档对象,若为Job对象,执行步骤324;若为JSDL文档对象323,执行步骤;若为JSDL文件,则将该JSDL文件文件解析为一JSDL文档对象;Step 322, the JSDL parsing module analyzes that the information to be processed is a JSDL file or a Job object or a JSDL document object, if it is a Job object, execute step 324; if it is a JSDL document object 323, execute step; if it is a JSDL file, then the JSDL file The file is parsed into a JSDL document object;
步骤323、JSDL解析模块将JSDL文档对象解析为一Job对象;Step 323, the JSDL parsing module parses the JSDL document object into a Job object;
步骤324、JSDL解析模块根据Job对象构造作业;Step 324, the JSDL parsing module constructs a job according to the Job object;
步骤33、JSDL解析模块将作业发送给调度模块,调度模块将该作业保存在作业管理模块中;
步骤34、资源获取模块对作业管理模块进行扫描,当发现该作业管理模块中存在未被匹配资源的新作业时,通过信息服务获取模块获取资源信息,并根据资源信息为新作业匹配资源;
步骤35、调度模块中的作业调度模块对作业管理模块进行扫描,当发现该作业管理模块中存在已被匹配了资源但尚未被调度的待调度作业时,将该待调度作业发送给资源匹配模块,该资源匹配模块判断其所在节点的各作业执行模块是否支持该待调度作业,若支持,则执行步骤36;否则,跳转执行步骤38;
步骤36、资源匹配模块判断该待调度作业是否为POSIX类型的作业,是则执行步骤37;否则,跳转执行步骤38;
步骤37、资源匹配模块将待调度作业转发给POSIX作业调度模块,该POSIX作业调度模块将待调度作业根据连接的作业执行模块的性能如负载,CPU等信息选择一作业执行模块,将该待调度作业发送给选定的作业执行模块以进行处理,然后跳转执行步骤39;
步骤38、资源匹配模块将待调度作业转发给委派执行作业执行模块,该委派执行作业执行模块首先判断待调度作业的类型,根据该作业类型在与该委派执行作业执行模块连接的作业执行模块中选择一负载最小且性能较高的作业执行模块执行该作业;当与该委派执行作业执行模块连接的作业执行模块与作业类型都不匹配或其性能较低或负载较高时,该委派执行作业执行模块便将该作业委托给其它的网格作业调度器进行调度;
通过委派执行作业执行模块实现了对作业的按照类型及作业执行模块的性能的有效匹配,避免了因作业类型不匹配或作业执行模块的性能低下对处理作业的速度和质量的影响,保证了被分配作业的可执行性;同时,通过对作业的再调度,在整个网格范围内实现了资源的均衡,提高了处理作业的速度和质量,有效利用了资源,方便了用户;By delegating the execution of the job execution module, the effective matching of the type of the job and the performance of the job execution module is realized, which avoids the impact on the speed and quality of the processing job due to the mismatch of the job type or the low performance of the job execution module, and ensures the quality of the job being processed. Executability of assigned jobs; at the same time, by rescheduling jobs, resource balance is realized in the entire grid range, the speed and quality of processing jobs are improved, resources are effectively utilized, and users are convenient;
步骤39、结束。
在上述网格作业调度方法中,步骤35中,资源匹配模块判断该待调度作业的类型时,若待调度作业为POSIX类型的作业,则执行步骤37;若为Web服务类型的作业,执行步骤351;若既不是POSIX类型的作业也不是Web服务类型的作业,则跳转执行步骤38;In the above-mentioned grid job scheduling method, in
步骤351、资源匹配模块将待调度作业转发给Web作业调度模块,该Web作业调度模块选择一具体的Web服务执行模块执行该待调度作业;然后执行步骤39。Step 351 , the resource matching module forwards the job to be scheduled to the Web job scheduling module, and the Web job scheduling module selects a specific Web service execution module to execute the job to be scheduled; and then executes
最后所应说明的是,以上实施例仅用以说明本发明的技术方案而非限制,尽管参照较佳实施例对本发明进行了详细说明,本领域的普通技术人员应当理解,可以对本发明的技术方案进行修改或者等同替换,而不脱离本发明技术方案的精神和范围。Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention without limitation. Although the present invention has been described in detail with reference to the preferred embodiments, those of ordinary skill in the art should understand that the technical solutions of the present invention can be The scheme shall be modified or equivalently replaced without departing from the spirit and scope of the technical scheme of the present invention.
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB2005101325393ACN100373343C (en) | 2005-12-26 | 2005-12-26 | Service grid scheduler and method for scheduling jobs |
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CNB2005101325393ACN100373343C (en) | 2005-12-26 | 2005-12-26 | Service grid scheduler and method for scheduling jobs |
| Publication Number | Publication Date |
|---|---|
| CN1790275A CN1790275A (en) | 2006-06-21 |
| CN100373343Ctrue CN100373343C (en) | 2008-03-05 |
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CNB2005101325393AExpired - Fee RelatedCN100373343C (en) | 2005-12-26 | 2005-12-26 | Service grid scheduler and method for scheduling jobs |
| Country | Link |
|---|---|
| CN (1) | CN100373343C (en) |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102103518A (en)* | 2011-02-23 | 2011-06-22 | 运软网络科技(上海)有限公司 | System for managing resources in virtual environment and implementation method thereof |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN101196832B (en)* | 2007-12-29 | 2012-12-19 | 北京航空航天大学 | Resource proxy machine facing illumination variant equipment |
| CN103049326B (en)* | 2013-01-16 | 2015-04-15 | 浪潮(北京)电子信息产业有限公司 | Method and system for managing job program of job management and scheduling system |
| CN103701894A (en)* | 2013-12-25 | 2014-04-02 | 浙江省公众信息产业有限公司 | Method and system for dispatching dynamic resource |
| CN107656741B (en)* | 2016-07-25 | 2020-07-28 | 中国科学院计算机网络信息中心 | A method and system for using application encapsulation in high performance computing environment |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1581806A (en)* | 2003-08-05 | 2005-02-16 | 国际商业机器公司 | Method and system for balancing working load in network computing environment |
| CN1671135A (en)* | 2004-03-19 | 2005-09-21 | 联想(北京)有限公司 | A network-oriented machine group working management system and realizing method thereof |
| US20050283782A1 (en)* | 2004-06-17 | 2005-12-22 | Platform Computing Corporation | Job-centric scheduling in a grid environment |
| US20050283534A1 (en)* | 2004-06-17 | 2005-12-22 | Platform Computing Corporation | Goal-oriented predictive scheduling in a grid environment |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1581806A (en)* | 2003-08-05 | 2005-02-16 | 国际商业机器公司 | Method and system for balancing working load in network computing environment |
| CN1671135A (en)* | 2004-03-19 | 2005-09-21 | 联想(北京)有限公司 | A network-oriented machine group working management system and realizing method thereof |
| US20050283782A1 (en)* | 2004-06-17 | 2005-12-22 | Platform Computing Corporation | Job-centric scheduling in a grid environment |
| US20050283534A1 (en)* | 2004-06-17 | 2005-12-22 | Platform Computing Corporation | Goal-oriented predictive scheduling in a grid environment |
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102103518A (en)* | 2011-02-23 | 2011-06-22 | 运软网络科技(上海)有限公司 | System for managing resources in virtual environment and implementation method thereof |
| Publication number | Publication date |
|---|---|
| CN1790275A (en) | 2006-06-21 |
| Publication | Publication Date | Title |
|---|---|---|
| Adhikari et al. | DPTO: A deadline and priority-aware task offloading in fog computing framework leveraging multilevel feedback queueing | |
| JP7197620B2 (en) | Streaming computing method and apparatus based on DAG interactions | |
| Li et al. | SSLB: self-similarity-based load balancing for large-scale fog computing | |
| Elmroth et al. | A grid resource broker supporting advance reservations and benchmark-based resource selection | |
| Zeng et al. | Joint optimization of task scheduling and image placement in fog computing supported software-defined embedded system | |
| CN107087019A (en) | A Device-Cloud Collaborative Computing Architecture and Task Scheduling Device and Method | |
| US20100162245A1 (en) | Runtime task with inherited dependencies for batch processing | |
| CN104915407A (en) | Resource scheduling method under Hadoop-based multi-job environment | |
| WO2013023473A1 (en) | Management method and system of virtual machine resources | |
| Bhullar et al. | Intelligent stress calculation and scheduling in segmented processor systems using buddy approach | |
| CN110806928A (en) | Job submitting method and system | |
| US8743387B2 (en) | Grid computing system with virtual printer | |
| CN101820365B (en) | Service bus supporting dynamic optimization of QoS of composite service | |
| CN118265973A (en) | Message processing method and device | |
| CN110781180A (en) | Data screening method and data screening device | |
| CN100373343C (en) | Service grid scheduler and method for scheduling jobs | |
| CN117633329A (en) | Data acquisition method and system for multiple data sources | |
| CN118069349A (en) | A variable depth resource management method and system for multiple scenarios | |
| Kasula | Performance analysis of layered architecture to integrate mobile devices and grid computing with a resource scheduling algorithm | |
| Östberg et al. | Mediation of service overhead in service-oriented grid architectures | |
| In et al. | Sphinx: A scheduling middleware for data intensive applications on a grid | |
| CN115714693A (en) | Low-power consumption control method and medium of network message pre-classification processor and low-power consumption processor | |
| CN109145015B (en) | Data query method, device and system based on structured query language | |
| Liao et al. | Efficient and fair scheduler of multiple resources for MapReduce system | |
| Jain et al. | PriDynSim a simulator for dynamic priority based I/O scheduling for cloud applications |
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| C14 | Grant of patent or utility model | ||
| GR01 | Patent grant | ||
| C17 | Cessation of patent right | ||
| CF01 | Termination of patent right due to non-payment of annual fee | Granted publication date:20080305 Termination date:20121226 |