CN103761146A - Method for dynamically setting quantities of slots for MapReduce - Google Patents
Method for dynamically setting quantities of slots for MapReduceDownload PDFInfo
- Publication number
- CN103761146A CN103761146ACN201410004521.4ACN201410004521ACN103761146ACN 103761146 ACN103761146 ACN 103761146ACN 201410004521 ACN201410004521 ACN 201410004521ACN 103761146 ACN103761146 ACN 103761146A
- Authority
- CN
- China
- Prior art keywords
- slots
- node
- tasktracker
- memory
- reduce
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Landscapes
- Devices For Executing Special Programs (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
Description
Translated fromChinese技术领域technical field
本发明涉及计算机技术领域，具体的说是一种MapReduce动态设定slots数量的方法。The invention relates to the field of computer technology, in particular to a method for dynamically setting the number of slots in MapReduce.
背景技术Background technique
如今的互联网技术不断发展，数据成爆炸式增长，网络上数据规模急剧增加，混乱的数据中蕴藏着巨大的商机，可以从海量的数据中提取价值。但随之而来的问题是单机的数据处理能力无法满足当今海量数据应用的处理要求，基于大规模计算集群的分布式计算成为未来数据性能提升的主要途径。针对Hadoop的核心技术MapReduce计算模型进行研究，针对MapReduce中每个节点中默认设定相同个数的map、reduce数量问题提出了一种MapReduce动态设定slots数量的策略。根据集群中不同节点的硬件配置不同设置不同的map数量和reduce数量。Today's Internet technology continues to develop, data has exploded, and the scale of data on the network has increased dramatically. There are huge business opportunities in chaotic data, and value can be extracted from massive data. But the ensuing problem is that the data processing capability of a single machine cannot meet the processing requirements of today's massive data applications, and distributed computing based on large-scale computing clusters has become the main way to improve data performance in the future. Research on the MapReduce computing model, the core technology of Hadoop, proposes a strategy to dynamically set the number of slots in MapReduce to solve the problem of setting the same number of maps and reduce numbers in each node in MapReduce by default. Set different map numbers and reduce numbers according to different hardware configurations of different nodes in the cluster.
目前对于mapreduce中map个数和reduce个数设定如下：map task的数量即mapred.tasktracker.map.tasks.maximu的参数值，但一个TaskTracker能配置多少个slot，还是与它的物理环境有关。每个task是由新启动的JVM独立执行，有多个task的时候就会有多个JVM，每个JVM消耗一部分内存，再加上DataNode和TaskTracker的内存消耗，机器内存可能就会不够用。这样除了考虑调配每个新启动JVM的内存限制外，还得关注下到底需要多少个新启动JVM，也就是map slot 和 reduce slot的数目。它们的设置还与机器的处理器数目有关。具体的配置还得从集群的实际运行效果来观察和分析。Input Split的大小，决定了一个Job拥有多少个map。然而，如果输入的数据量巨大，那么默认的block会有几万甚至几十万的Map Task，集群的网络传输会很大，最严重的是给Job Tracker的调度、队列、内存都会带来很大压力。因此要设定适当地符合机器计算能力的slots数量。Currently, the number of maps and reduces in mapreduce is set as follows: the number of map tasks is the parameter value of mapred.tasktracker.map.tasks.maximu, but how many slots a TaskTracker can configure is still related to its physical environment. Each task is executed independently by the newly started JVM. When there are multiple tasks, there will be multiple JVMs. Each JVM consumes part of the memory. In addition to the memory consumption of DataNode and TaskTracker, the machine memory may not be enough. In this way, in addition to considering the memory limit of each newly started JVM, you also need to pay attention to how many newly started JVMs are needed, that is, the number of map slots and reduce slots. Their settings are also related to the number of processors on the machine. The specific configuration has to be observed and analyzed from the actual running effect of the cluster. The size of the Input Split determines how many maps a Job has. However, if the amount of input data is huge, the default block will have tens of thousands or even hundreds of thousands of Map Tasks. huge pressure. Therefore, it is necessary to set the number of slots appropriately corresponding to the computing power of the machine.
在Hadoop中，使用slot表示各个TaskTraker上的资源，一个slot代表固定的资源组合，在执行mapreduce程序时，每个TaskTracker上的Map slot个数及Reduce slot个数是通过mapred.tasktracker.map.tasks.maximum和mapred.tasktracker.reduce.tasks.maximum进行配置。这两个参数一旦配置后，不能动态修改。由于不同作用的任务需要的资源量不同，集群中的节点硬件配置也不尽相同，因此，针对节点资源的不同，提出一种 MapReduce动态设定slots数量的策略。该策略可以根据节点计算能力动态设定slot数量，提高MapReduce程序执行的性能。In Hadoop, slots are used to represent the resources on each TaskTracker, and a slot represents a fixed combination of resources. When executing the mapreduce program, the number of Map slots and Reduce slots on each TaskTracker is passed through mapred.tasktracker.map.tasks .maximum and mapred.tasktracker.reduce.tasks.maximum for configuration. Once these two parameters are configured, they cannot be modified dynamically. Because tasks with different roles require different amounts of resources, the hardware configurations of nodes in the cluster are also different. Therefore, according to the differences in node resources, a strategy for dynamically setting the number of slots in MapReduce is proposed. This strategy can dynamically set the number of slots according to the computing power of nodes, and improve the performance of MapReduce program execution.
发明内容Contents of the invention
本发明的技术任务是解决现有技术的不足，提供一种MapReduce动态设定slots数量的方法。The technical task of the present invention is to solve the deficiencies of the prior art and provide a method for dynamically setting the number of slots in MapReduce.
本发明的技术方案是按以下方式实现的，该一种MapReduce动态设定slots数量的方法，其具体设定过程为：The technical scheme of the present invention is realized in the following manner, the method for dynamically setting the number of slots in this MapReduce, and its concrete setting process is:
首先确定集群节点中CPU的数量，然后根据每个节点中CPU的核心的数量由主从模式框架MapReduce动态设定决定slots数量：根据job队列和TaskTracker节点的资源情况作为输入，其中TaskTracker的资源情况包括CPU的核心数量及节点的内存大小，然后再根据节点的计算能力设定slots数量；First determine the number of CPUs in the cluster nodes, and then according to the number of CPU cores in each node, the number of slots is determined dynamically by the master-slave mode framework MapReduce: according to the job queue and the resource status of TaskTracker nodes as input, the resource status of TaskTracker Including the number of CPU cores and the memory size of the node, and then set the number of slots according to the computing power of the node;
在主-从模式框架MapReduce的主节点上运行JobTracker,其负责监控机群，任务调度；从节点上运行TaskTracker，其负责监控任务执行，报告进度；Run JobTracker on the master node of the master-slave mode framework MapReduce, which is responsible for monitoring the cluster and task scheduling; run TaskTracker on the slave node, which is responsible for monitoring task execution and reporting progress;
TaskTracker定期向JobTracker 发送心跳信息，该信息中携带本节点的资源使用情况；TaskTracker periodically sends heartbeat information to JobTracker, which carries the resource usage of the node;
在心跳到达时，主节点中的调度发生，若TaskTracker 报告自己有空闲资源， 则JobTracker 使用调度算法选择一个任务发射到该节点运行。When the heartbeat arrives, the scheduling in the master node occurs. If the TaskTracker reports that it has idle resources, the JobTracker uses the scheduling algorithm to select a task to send to the node for execution.
在设定slots数量时需要设计两个变量，一个是map slot，一个是reduce slot：首先修改TaskTracker中的代码，将map slot数量初步设定为节点中CPU的核心数量，reduce slot数量初步设定为节点中CPU的核心数量的一半；然后在类方法中，根据 slots 数量来决定申请内存的大小，task的总内存分配大小等于map slot数量与TaskTracker中单个map slot内存大小之积再加上resuce slot数量与TaskTracker中单个reduce slot内存大小之积；如果task的总内存分配与集群中相应节点的剩余内存相比小，则将slots设定为该值；如果task的总内存分配与集群中相应节点的剩余内存小，则减少map slot数量或reduce slot数量，交替的较少slots数量，直到满足节点中内存条件为止。When setting the number of slots, two variables need to be designed, one is map slot and the other is reduce slot: first modify the code in TaskTracker, and initially set the number of map slots to the number of CPU cores in the node, and initially set the number of reduce slots It is half of the number of CPU cores in the node; then in the class method, the size of the requested memory is determined according to the number of slots, and the total memory allocation size of the task is equal to the product of the number of map slots and the memory size of a single map slot in TaskTracker plus resource The product of the number of slots and the memory size of a single reduce slot in TaskTracker; if the total memory allocation of the task is smaller than the remaining memory of the corresponding node in the cluster, set slots to this value; if the total memory allocation of the task is the same as the corresponding node in the cluster If the remaining memory of the node is small, reduce the number of map slots or reduce slots, and alternately reduce the number of slots until the memory conditions in the node are met.
本发明与现有技术相比所产生的有益效果是：The beneficial effect that the present invention produces compared with prior art is:
本发明的一种MapReduce动态设定slots数量的方法通过分析Hadoop集群中节点的计算能力，利用每个节点的CPU及内存情况确定出slots数量，然后根据该数量得到合理的map数量及reduce数量，该策略使得整个集群处理MapReduce任务的性能大大提升，并优化资源的合理利用，实用性强，易于推广。A method for dynamically setting the number of slots for MapReduce of the present invention determines the number of slots by analyzing the computing power of the nodes in the Hadoop cluster, using the CPU and memory conditions of each node, and then obtains a reasonable map number and reduce number according to the number, This strategy greatly improves the performance of the entire cluster in processing MapReduce tasks, and optimizes the rational use of resources. It is practical and easy to promote.
附图说明Description of drawings
附图1是本发明的作业job执行流程图。Accompanying drawing 1 is the job execution flowchart of the present invention.
附图2是本发明的设定slots数量的流程图。Accompanying drawing 2 is the flowchart of setting the number of slots of the present invention.
具体实施方式Detailed ways
下面结合附图对本发明的一种MapReduce动态设定slots数量的方法作以下详细说明。A method for dynamically setting the number of slots for MapReduce of the present invention will be described in detail below in conjunction with the accompanying drawings.
本发明涉及当前大数据Hadoop集群中MapReduce急需解决的一个重要问题，即根据集群中节点硬件配置不同、计算能力不同动态设定map、reduce数量的问题。通过本方法提出的MapReduce动态设定slots数量的策略，该策略可以有效解决动态设定slots数量的问题，并使整个集群处理MapReduce任务的性能大大提升。The present invention relates to an important problem urgently needed to be solved by MapReduce in the current big data Hadoop cluster, that is, the problem of dynamically setting the number of map and reduce according to different hardware configurations and computing capabilities of nodes in the cluster. Through the strategy of dynamically setting the number of slots in MapReduce proposed by this method, this strategy can effectively solve the problem of dynamically setting the number of slots, and greatly improve the performance of the entire cluster in processing MapReduce tasks.
本发明依赖于主-从模式框架MapReduce，该框架采用Master/Slave的架构，它主要有以下4个部分组成：The present invention relies on master-slave pattern framework MapReduce, and this framework adopts the framework of Master/Slave, and it mainly has following 4 parts to form:
1）Client。1) Client.
2）JobTracker：JobTracke负责资源监控和作业调度。JobTracker 监控所有TaskTracker 与job的健康状况，一旦发现失败，就将相应的任务转移到其他节点；同时，JobTracker 会跟踪任务的执行进度、资源使用量等信息，并将这些信息告诉任务调度器，而调度器会在资源出现空闲时，选择合适的任务使用这些资源。在Hadoop 中，任务调度器是一个可插拔的模块，用户可以根据自己的需要设计相应的调度器。2) JobTracker: JobTracke is responsible for resource monitoring and job scheduling. JobTracker monitors the health status of all TaskTrackers and jobs, and transfers the corresponding tasks to other nodes once a failure is found; at the same time, JobTracker will track the task execution progress, resource usage and other information, and inform the task scheduler of this information, and The scheduler will select appropriate tasks to use these resources when the resources are free. In Hadoop, the task scheduler is a pluggable module, and users can design the corresponding scheduler according to their needs.
3）TaskTracker：TaskTracker 会周期性地通过Heartbeat 将本节点上资源的使用情况和任务的运行进度汇报给JobTracker，同时接收JobTracker 发送过来的命令并执行相应的操作（如启动新任务、杀死任务等）。TaskTracker 使用“slot”等量划分本节点上的资源量。“slot”代表计算资源（CPU、内存等）。一个Task 获取到一个slot 后才有机会运行，而Hadoop 调度器的作用就是将各个TaskTracker 上的空闲slot 分配给Task 使用。slot 分为Map slot 和Reduce slot 两种，分别供MapTask 和Reduce Task 使用。TaskTracker 通过slot 数目（可配置参数）限定Task 的并发度。3) TaskTracker: TaskTracker will periodically report the usage of resources on this node and the running progress of tasks to JobTracker through Heartbeat, and at the same time receive commands sent by JobTracker and perform corresponding operations (such as starting new tasks, killing tasks, etc. ). TaskTracker uses "slot" to divide the amount of resources on this node. A "slot" represents a computing resource (CPU, memory, etc.). A Task has a chance to run after obtaining a slot, and the role of the Hadoop scheduler is to allocate the idle slots on each TaskTracker to the Task. There are two types of slots: Map slot and Reduce slot, which are used by MapTask and Reduce Task respectively. TaskTracker limits the concurrency of Task through the number of slots (configurable parameters).
4）Task：Task 分为Map Task 和Reduce Task 两种，均由TaskTracker 启动。HDFS 以固定大小的block 为基本单位存储数据，而对于MapReduce 而言，其处理单位是split。split 是一个逻辑概念，它只包含一些元数据信息，比如数据起始位置、数据长度、数据所在节点等。它的划分方法完全由用户自己决定。但需要注意的是，split 的多少决定了Map Task 的数目，因为每个split 只会交给一个Map Task 处理。4) Task: Task is divided into Map Task and Reduce Task, both of which are started by TaskTracker. HDFS stores data with a fixed-size block as the basic unit, while for MapReduce, its processing unit is split. Split is a logical concept that only contains some metadata information, such as the starting position of the data, the length of the data, and the node where the data is located. Its division method is entirely up to the user to decide. But it should be noted that the number of splits determines the number of Map Tasks, because each split will only be handed over to one Map Task for processing.
如附图1、图2所示，本发明提供的一种MapReduce动态设定slots数量的方法，该策略主要是根据集群节点中计算能力来设定slots数量，节点计算能力是根据CPU个数及内存两个因素所确定的。 首先确定集群节点中CPU的数量，然后根据每个节点中CPU的核心的数量决定slots数量，这样可以根据不同节点计算能力来处理任务，使得mapreduce任务更高效的执行，提高性能。MapReduce动态设定slots数量的策略中内存因素，是根据slots数量来决定申请内存的大小，再根据节点的内存情况相应的调整slots数量，若在申请过程中内存不足时则可以减少slots数量知道达到内存条件，反之将slots数量设定为依据CPU数量设定的slots数量，最终根据slots数量确定map、reduce数量。其具体设定过程为：As shown in accompanying drawing 1, Fig. 2, a kind of method of MapReduce dynamically setting slots quantity provided by the present invention, this strategy is mainly to set the slots quantity according to computing power in the cluster node, node computing power is according to CPU number and Memory is determined by two factors. First determine the number of CPUs in the cluster nodes, and then determine the number of slots based on the number of CPU cores in each node, so that tasks can be processed according to the computing power of different nodes, so that mapreduce tasks can be executed more efficiently and performance can be improved. The memory factor in MapReduce's strategy of dynamically setting the number of slots is to determine the size of the application memory according to the number of slots, and then adjust the number of slots accordingly according to the memory status of the node. If the memory is insufficient during the application process, the number of slots can be reduced until it reaches Memory conditions, otherwise, set the number of slots to the number of slots set according to the number of CPUs, and finally determine the number of map and reduce according to the number of slots. The specific setting process is:
首先确定集群节点中CPU的数量，然后根据每个节点中CPU的核心的数量由主从模式框架MapReduce动态设定决定slots数量：根据job队列和TaskTracker节点的资源情况作为输入，其中TaskTracker的资源情况包括CPU的核心数量及节点的内存大小，然后再根据节点的计算能力设定slots数量；First determine the number of CPUs in the cluster nodes, and then according to the number of CPU cores in each node, the number of slots is determined dynamically by the master-slave mode framework MapReduce: according to the job queue and the resource status of TaskTracker nodes as input, the resource status of TaskTracker Including the number of CPU cores and the memory size of the node, and then set the number of slots according to the computing power of the node;
在主-从模式框架MapReduce的主节点上运行JobTracker,其负责监控机群，任务调度；从节点上运行TaskTracker，其负责监控任务执行，报告进度；Run JobTracker on the master node of the master-slave mode framework MapReduce, which is responsible for monitoring the cluster and task scheduling; run TaskTracker on the slave node, which is responsible for monitoring task execution and reporting progress;
TaskTracker定期向JobTracker 发送心跳信息，该信息中携带本节点的资源使用情况；TaskTracker periodically sends heartbeat information to JobTracker, which carries the resource usage of the node;
在心跳到达时，主节点中的调度发生，若TaskTracker 报告自己有空闲资源， 则JobTracker 使用调度算法选择一个任务发射到该节点运行。When the heartbeat arrives, the scheduling in the master node occurs. If the TaskTracker reports that it has idle resources, the JobTracker uses the scheduling algorithm to select a task to send to the node for execution.
在设定slots数量时需要设计两个变量，一个是map slot，一个是reduce slot：首先修改TaskTracker中的代码，将map slot数量初步设定为节点中CPU的核心数量，reduce slot数量初步设定为节点中CPU的核心数量的一半；然后在类方法中，根据 slots 数量来决定申请内存的大小，task的总内存分配大小等于map slot数量与TaskTracker中单个map slot内存大小之积再加上resuce slot数量与TaskTracker中单个reduce slot内存大小之积；如果task的总内存分配与集群中相应节点的剩余内存相比小，则将slots设定为该值；如果task的总内存分配与集群中相应节点的剩余内存小，则减少map slot数量或reduce slot数量，交替的较少slots数量，直到满足节点中内存条件为止。When setting the number of slots, two variables need to be designed, one is map slot and the other is reduce slot: first modify the code in TaskTracker, and initially set the number of map slots to the number of CPU cores in the node, and initially set the number of reduce slots It is half of the number of CPU cores in the node; then in the class method, the size of the requested memory is determined according to the number of slots, and the total memory allocation size of the task is equal to the product of the number of map slots and the memory size of a single map slot in TaskTracker plus resource The product of the number of slots and the memory size of a single reduce slot in TaskTracker; if the total memory allocation of the task is smaller than the remaining memory of the corresponding node in the cluster, set slots to this value; if the total memory allocation of the task is the same as the corresponding node in the cluster If the remaining memory of the node is small, reduce the number of map slots or reduce slots, and alternately reduce the number of slots until the memory conditions in the node are met.
本发明的目的是针对分布式计算框架来动态设定slots数量。该策略的思想是根据Hadoop集群中每个节点的计算能力不同来动态设定slots数量。从节点拥有的CPU与内存情况来设定map数量与reduce数量，该技术问题是将节点中CPU数量与slots数量合理的联系起来；通过内存的限定约束slots的数量，使得能符合集群中节点的处理能力，使得任务更高效。The purpose of the present invention is to dynamically set the number of slots for the distributed computing framework. The idea of this strategy is to dynamically set the number of slots according to the computing power of each node in the Hadoop cluster. Set the number of maps and reduce from the CPU and memory of the node. The technical problem is to reasonably link the number of CPUs in the node with the number of slots; the number of slots is restricted by the limitation of memory so that it can meet the requirements of the nodes in the cluster. processing power to make tasks more efficient.
节点中CPU数量与slots数量合理的联系中，统计每个节点的CPU数量，将slots数量设置成节点中CPU的core数量，因为每个core可以单独处理一个Task，而不用等待，在map Task或reduce Task执行时很会快速的执行。In the reasonable relationship between the number of CPUs in a node and the number of slots, count the number of CPUs in each node, and set the number of slots to the number of CPU cores in the node, because each core can process a Task independently without waiting. The reduce Task executes very quickly.
在内存限定约束中，可以根据slots数量来决定申请内存的大小，再根据节点的内存情况相应的调整slots数量，若在申请过程中内存不足时则可以减少slots数量，直到达到内存限制的要求，反之将slots数量设定为依据CPU数量设定的slots数量。 In the memory limit constraint, the size of the application memory can be determined according to the number of slots, and then the number of slots can be adjusted accordingly according to the memory condition of the node. If the memory is insufficient during the application process, the number of slots can be reduced until the memory limit requirement is met. Otherwise, set the number of slots to the number of slots set according to the number of CPUs. the
下面参照附图1和附图2，对本发明的内容以一个具体实例来详细描述。Referring to accompanying drawing 1 and accompanying drawing 2 below, the content of the present invention is described in detail with a specific example.
首先部署分布式集群环境，使用具有11 个节点的Hadoop 机群, 其中一个节点作为master, 其余十个作为slave. 其中10个节点均采用Xeon E5-2620 2.00GHz CPU, core的数量是24, 96GB 内存,12*2T硬盘, 操作系统是centos6.3 ,另外一个节点配置是Xeon E7- 8837 2.67GHz CPU, core的数量是128， 500GB 内存,5*2T硬盘, 操作系统是centos6.3。在操作系统是centos6.3上按照官方文档安装hadoop组件。然后将hdfs、mapreduce服务开启。First deploy a distributed cluster environment, using a Hadoop cluster with 11 nodes, one of which is the master, and the remaining ten are slaves. The 10 nodes all use Xeon E5-2620 2.00GHz CPU, the number of cores is 24, and the memory is 96GB , 12*2T hard disk, the operating system is centos6.3, the other node configuration is Xeon E7- 8837 2.67GHz CPU, the number of cores is 128, 500GB memory, 5*2T hard disk, the operating system is centos6.3. Install hadoop components according to the official documentation on the operating system centos6.3. Then enable the hdfs and mapreduce services.
作业job执行流程图如附图1所示，首先确定MapReduce的输入文件或目录应该在File system上存在，如果MapReduce依赖于HDFS，就得先将本地的文件上传到HDFS上。Client会向JobTracker申请一个Jobid来作为job的标识符。然后MapReduce就需要将job执行必要的资源文件复制到HDFS上。接下来才是作业job提交过程，对输入文件做数据分片(input split)。数据分片是为了在mapper执行之前就确定它处理数据的范围，并且分片的数量决定map task的数量，它们之间一一对应。这种数据分片(split)只是逻辑分片，记录它应当访问哪个block，及在这个block上的起始index及数据长度的信息。然后初始化作业，JobTracker将负责分发任务给TaskTracker，TaskTracker在运行时会周期性地向JobTracker发送心跳请求，汇报TaskTracker的状态数据、TaskTracker上task执行状态及希望从JobTracker得到可以执行的task。而在真正TaskTracker节点中运行的map数量和reduce数量是由map slots和reduce slots数量决定的。因此，根据集群中相应节点的计算能力确定每个节点中map slots 和reduce slots数量很重要，直接影响着任务的运行效率。The job execution flow chart is shown in Figure 1. First, make sure that the input files or directories of MapReduce should exist on the File system. If MapReduce depends on HDFS, you must first upload the local files to HDFS. Client will apply for a Jobid from JobTracker as the job identifier. Then MapReduce needs to copy the resource files necessary for job execution to HDFS. The next step is the job submission process, where the input file is split. Data sharding is to determine the range of data it processes before the mapper is executed, and the number of shards determines the number of map tasks, and there is a one-to-one correspondence between them. This kind of data fragmentation (split) is just a logical fragmentation, which records which block it should access, and information about the starting index and data length on this block. Then initialize the job, the JobTracker will be responsible for distributing tasks to the TaskTracker, and the TaskTracker will periodically send a heartbeat request to the JobTracker when it is running, reporting the status data of the TaskTracker, the execution status of the task on the TaskTracker, and the tasks that can be executed from the JobTracker. The number of maps and reduces running in the real TaskTracker node is determined by the number of map slots and reduce slots. Therefore, it is very important to determine the number of map slots and reduce slots in each node according to the computing power of the corresponding nodes in the cluster, which directly affects the operating efficiency of the task.
设定slots数量的流程图如附图2所示，首先获取集群中的每个节点的CPU的core数量， map slot数量初步设定为节点中CPU的core数量，reduce slot数量初步设定为节点中CPU的core数量的一半；然后获取每个节点中的剩余内存大小，在类方法 initializeMemoryManagement() 中 ，根据 slots 数量来决定申请内存的大小，task的总内存分配大小等于map slot数量与TaskTracker中单个map slot内存大小之积再加上reduce slot数量与TaskTracker中单个reduce slot内存大小之积。如果task的总内存分配与集群中相应节点的剩余内存相比小，则将map slots设定为节点中CPU的core数量，reduce slot数量为map slot数量的一半；否则如果task的总内存分配与集群中相应节点的剩余内存小，则减少map slot数量或reduce slot数量，交替的较少slots数量，直到满足节点中内存条件为止，这时map slots设定为满足条件的map slots数量，reduce slot数量为满足条件的reduce slots数量。然后根据根据类方法 TaskTracker.initialize() 中两个 TaskLauncher 线程，分别负责启动 Mapper 和 Reduce 任务，在TaskLauncher中需要传入相应slots数量，然后执行相应的Task，如map task或reduce task。在执行结束后，释放所占有的资源。该方法将节点的计算能力用CPU的core数量及内存大小来决定的，对于节点中CPU的core数量多且内存大的节点设置较大的map和reduce数量，对于一些节点中CPU的core数量少且内存相对较少的节点设置较小的map和reduce数量。在该集群中，采用Xeon E5-2620 2.00GHz CPU, core的数量是24, 96GB 内存的10个节点均将map设置为24，reduce设置为12。另一个节点配置是Xeon E7- 8837 2.67GHz CPU, core的数量是128， 500GB 内存，将map设置为128，reduce为64。这样的设置比每个机器节点设置的map数量和reduce数量的任务执行效率高，同时达到优化资源的合理利用。The flowchart for setting the number of slots is shown in Figure 2. First, obtain the number of CPU cores of each node in the cluster. The number of map slots is initially set to the number of CPU cores in the node, and the number of reduce slots is initially set to the number of nodes. half of the number of cores in the CPU; then obtain the remaining memory size in each node, in the class method initializeMemoryManagement() , determine the size of the application memory according to the number of slots, the total memory allocation size of the task is equal to the number of map slots and the number of TaskTracker The product of the memory size of a single map slot plus the product of the number of reduce slots and the memory size of a single reduce slot in TaskTracker. If the total memory allocation of the task is smaller than the remaining memory of the corresponding node in the cluster, set the map slots to the number of CPU cores in the node, and the number of reduce slots to half of the number of map slots; otherwise, if the total memory allocation of the task is equal to If the remaining memory of the corresponding node in the cluster is small, reduce the number of map slots or reduce slots, and alternately reduce the number of slots until the memory conditions in the node are met. At this time, the map slots are set to the number of map slots that meet the conditions, and the reduce slot The number is the number of reduce slots that meet the conditions. Then according to the two TaskLauncher threads in the class method TaskTracker.initialize(), they are responsible for starting the Mapper and Reduce tasks respectively. In the TaskLauncher, the corresponding number of slots needs to be passed in, and then the corresponding Task, such as map task or reduce task, should be executed. After the execution ends, release the occupied resources. This method determines the computing power of a node by the number of CPU cores and memory size. For nodes with a large number of CPU cores and large memory, set a larger number of map and reduce. For some nodes, the number of CPU cores is small. And nodes with relatively less memory set a smaller number of map and reduce. In this cluster, Xeon E5-2620 2.00GHz CPU is used, the number of cores is 24, and the 10 nodes with 96GB memory all set map to 24 and reduce to 12. Another node configuration is Xeon E7- 8837 2.67GHz CPU, the number of cores is 128, 500GB memory, map is set to 128, and reduce is 64. This setting is more efficient than the number of maps and reduce tasks set by each machine node, and at the same time achieves a reasonable utilization of optimized resources.
以上所述仅为本发明的实施例而已，凡在本发明的精神和原则之内，所作的任何修改、等同替换、改进等，均应包含在本发明的保护范围之内。The above description is only an embodiment of the present invention, and any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.
Claims (2)
Translated fromChinesePriority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410004521.4ACN103761146B (en) | 2014-01-06 | 2014-01-06 | A kind of method that MapReduce dynamically sets slots quantity |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201410004521.4ACN103761146B (en) | 2014-01-06 | 2014-01-06 | A kind of method that MapReduce dynamically sets slots quantity |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN103761146Atrue CN103761146A (en) | 2014-04-30 |
| CN103761146B CN103761146B (en) | 2017-10-31 |
Family
ID=50528389
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201410004521.4AActiveCN103761146B (en) | 2014-01-06 | 2014-01-06 | A kind of method that MapReduce dynamically sets slots quantity |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN103761146B (en) |
Cited By (11)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104270412A (en)* | 2014-06-24 | 2015-01-07 | 南京邮电大学 | A three-level cache method based on Hadoop distributed file system |
| CN105607955A (en)* | 2015-12-23 | 2016-05-25 | 浪潮集团有限公司 | Calculation task distribution method and apparatus |
| CN105868025A (en)* | 2016-03-30 | 2016-08-17 | 华中科技大学 | System for settling fierce competition of memory resources in big data processing system |
| WO2017107456A1 (en)* | 2015-12-25 | 2017-06-29 | 乐视控股（北京）有限公司 | Method and apparatus for determining resources consumed by task |
| CN107203422A (en)* | 2016-08-28 | 2017-09-26 | 深圳晶泰科技有限公司 | A kind of job scheduling method towards high-performance calculation cloud platform |
| CN107766138A (en)* | 2017-10-20 | 2018-03-06 | 北京集奥聚合科技有限公司 | A kind of hadoop Mission Monitors method and system |
| CN108170530A (en)* | 2017-12-26 | 2018-06-15 | 北京工业大学 | A kind of Hadoop Load Balancing Task Scheduling methods based on mixing meta-heuristic algorithm |
| CN108268316A (en)* | 2016-12-30 | 2018-07-10 | 北京国双科技有限公司 | The method and device of job scheduling |
| CN110618865A (en)* | 2019-09-20 | 2019-12-27 | 中国银行股份有限公司 | Hadoop task scheduling method and device |
| CN111914007A (en)* | 2020-06-15 | 2020-11-10 | 武汉达梦数据库有限公司 | Method and device for Hadoop cluster to run ETL process |
| CN115378949A (en)* | 2022-08-25 | 2022-11-22 | 公安部第一研究所 | A data processing method based on dynamic cluster and data dynamic fragmentation technology |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102073546A (en)* | 2010-12-13 | 2011-05-25 | 北京航空航天大学 | Task-dynamic dispatching method under distributed computation mode in cloud computing environment |
| CN102541645A (en)* | 2012-01-04 | 2012-07-04 | 北京航空航天大学 | Dynamic adjustment method for node task slot based on node state feedbacks |
| CN102609303A (en)* | 2012-01-18 | 2012-07-25 | 华为技术有限公司 | Slow-task dispatching method and slow-task dispatching device of Map Reduce system |
| US20120304186A1 (en)* | 2011-05-26 | 2012-11-29 | International Business Machines Corporation | Scheduling Mapreduce Jobs in the Presence of Priority Classes |
| US20130191843A1 (en)* | 2011-08-23 | 2013-07-25 | Infosys Limited | System and method for job scheduling optimization |
- 2014
- 2014-01-06CNCN201410004521.4Apatent/CN103761146B/enactiveActive
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN102073546A (en)* | 2010-12-13 | 2011-05-25 | 北京航空航天大学 | Task-dynamic dispatching method under distributed computation mode in cloud computing environment |
| US20120304186A1 (en)* | 2011-05-26 | 2012-11-29 | International Business Machines Corporation | Scheduling Mapreduce Jobs in the Presence of Priority Classes |
| US20130191843A1 (en)* | 2011-08-23 | 2013-07-25 | Infosys Limited | System and method for job scheduling optimization |
| CN102541645A (en)* | 2012-01-04 | 2012-07-04 | 北京航空航天大学 | Dynamic adjustment method for node task slot based on node state feedbacks |
| CN102609303A (en)* | 2012-01-18 | 2012-07-25 | 华为技术有限公司 | Slow-task dispatching method and slow-task dispatching device of Map Reduce system |
Cited By (16)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN104270412A (en)* | 2014-06-24 | 2015-01-07 | 南京邮电大学 | A three-level cache method based on Hadoop distributed file system |
| CN105607955A (en)* | 2015-12-23 | 2016-05-25 | 浪潮集团有限公司 | Calculation task distribution method and apparatus |
| WO2017107456A1 (en)* | 2015-12-25 | 2017-06-29 | 乐视控股（北京）有限公司 | Method and apparatus for determining resources consumed by task |
| CN105868025B (en)* | 2016-03-30 | 2019-05-10 | 华中科技大学 | A system to solve the fierce competition of memory resources in big data processing system |
| CN105868025A (en)* | 2016-03-30 | 2016-08-17 | 华中科技大学 | System for settling fierce competition of memory resources in big data processing system |
| CN107203422A (en)* | 2016-08-28 | 2017-09-26 | 深圳晶泰科技有限公司 | A kind of job scheduling method towards high-performance calculation cloud platform |
| CN107203422B (en)* | 2016-08-28 | 2020-09-01 | 深圳晶泰科技有限公司 | Job scheduling method for high-performance computing cloud platform |
| CN108268316A (en)* | 2016-12-30 | 2018-07-10 | 北京国双科技有限公司 | The method and device of job scheduling |
| CN107766138A (en)* | 2017-10-20 | 2018-03-06 | 北京集奥聚合科技有限公司 | A kind of hadoop Mission Monitors method and system |
| CN108170530A (en)* | 2017-12-26 | 2018-06-15 | 北京工业大学 | A kind of Hadoop Load Balancing Task Scheduling methods based on mixing meta-heuristic algorithm |
| CN108170530B (en)* | 2017-12-26 | 2021-08-17 | 北京工业大学 | A Hadoop Load Balancing Task Scheduling Method Based on Hybrid Metaheuristic Algorithm |
| CN110618865A (en)* | 2019-09-20 | 2019-12-27 | 中国银行股份有限公司 | Hadoop task scheduling method and device |
| CN110618865B (en)* | 2019-09-20 | 2022-07-05 | 中国银行股份有限公司 | Hadoop task scheduling method and device |
| CN111914007A (en)* | 2020-06-15 | 2020-11-10 | 武汉达梦数据库有限公司 | Method and device for Hadoop cluster to run ETL process |
| CN111914007B (en)* | 2020-06-15 | 2024-02-02 | 武汉达梦数据库股份有限公司 | Method and device for hadoop cluster to run ETL flow |
| CN115378949A (en)* | 2022-08-25 | 2022-11-22 | 公安部第一研究所 | A data processing method based on dynamic cluster and data dynamic fragmentation technology |
Also Published As
| Publication number | Publication date |
|---|---|
| CN103761146B (en) | 2017-10-31 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN103761146B (en) | A kind of method that MapReduce dynamically sets slots quantity | |
| CN104915407B (en) | A kind of resource regulating method based under Hadoop multi-job environment | |
| CN103530189B (en) | It is a kind of towards the automatic telescopic of stream data and the method and device of migration | |
| Rao et al. | Performance issues of heterogeneous hadoop clusters in cloud computing | |
| CN110908788B (en) | Data processing method, device, computer equipment and storage medium based on Spark Streaming | |
| CN103365726B (en) | A kind of method for managing resource towards GPU cluster and system | |
| CN103810023B (en) | The intelligent deployment method of Distributed Application and system in a kind of cloud platform | |
| CN103036946B (en) | A kind of method and system processing file backup task for cloud platform | |
| CN107222531B (en) | Container cloud resource scheduling method | |
| CN102521014B (en) | Deploying method and deploying device for virtual machine | |
| CN108845878A (en) | Big data processing method and device based on serverless computing | |
| CN104243617B (en) | Towards the method for scheduling task and system of mixed load in a kind of isomeric group | |
| CN109614227B (en) | Task resource allocation method, apparatus, electronic device, and computer-readable medium | |
| CN109726004B (en) | Data processing method and device | |
| CN107291536B (en) | Application task flow scheduling method in cloud computing environment | |
| CN108572873A (en) | A load balancing method and device for solving Spark data skew problem | |
| CN106656525B (en) | A data broadcasting system, data broadcasting method and device | |
| CN112231098A (en) | Task processing method, device, equipment and storage medium | |
| CN105138679A (en) | Data processing system and method based on distributed caching | |
| CN109992373A (en) | Resource scheduling method, information management method and device, and task deployment system | |
| CN102999317B (en) | Towards the elasticity multi-process service processing method of many tenants | |
| WO2023274278A1 (en) | Resource scheduling method and device and computing node | |
| WO2018131556A1 (en) | Resource setting control device, resource setting control system, resource setting control method, and computer-readable recording medium | |
| CN104408047A (en) | Method for uploading text file to HDFS (hadoop distributed file system) in multi-machine parallel mode based on NFS (network file system) file server | |
| CN107528871B (en) | Data analysis in storage systems |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| C06 | Publication | ||
| PB01 | Publication | ||
| C10 | Entry into substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |