Movatterモバイル変換

1 © Hortonworks Inc. 2011 – 2016. All Rights ReservedApache Hadoop YARN:Past, Present andFutureMelbourne, Aug.31 2016Junping Du

2 © Hortonworks Inc. 2011 – 2016. All Rights ReservedWho.JSON{"name" : "Junping Du" ,"job_title" : "Lead Software Engineer @ Hortonworks YARN core team","experiences" : [ {"software_industry_years" : 10,"hadoop_experience" : "Hadoop contributor before YARN comes out, ApacheHadoop committer & PMC, Release Manager for Apache Hadoop 2.6",”non_hadoop_experience" : “Architect in cloud computing and enterprise software"}],"email" : "junping_du@apache.org"}

3 © Hortonworks Inc. 2011 – 2016. All Rights ReservedWhat is Apache Hadoop YARN ?⬢ YARN is short for “Yet Another Resource Negotiator”⬢ Big Data Operating System–Resource Management and Scheduling–Support for “colorful” applications, like: Batch, Interactive,Real-Time, etc.⬢ Enterprise adoption accelerating–Secure mode becoming more widespread–Multi-tenant support–Diverse workloads⬢ SLAs–Tolerance for slow running jobs decreasing–Consistent performance desired

4 © Hortonworks Inc. 2011 – 2016. All Rights Reserved4 © Hortonworks Inc. 2011 – 2016. All Rights ReservedPast

5 © Hortonworks Inc. 2011 – 2016. All Rights ReservedA brief Timeline1st line of Code Open sourced First 2.0 alpha First 2.0 betaJune-July 2010 August 2011 May 2012 August 2013⬢ Sub-project of Apache Hadoop⬢ Releases tied to Hadoop releases⬢ Alphas and betas

6 © Hortonworks Inc. 2011 – 2016. All Rights ReservedGA Releases2.2 2.3 2.4 2.5Oct. 2013 Feb. 2014 Apr. 2014 Aug. 2014• 1st GA• MR binarycompatibility• YARN APIcleanup• Testing!• 1st Post GA• Bug fixes• Alpha features- Load simulator- LCEenhancements• RM Fail-over• CS Preemption• TimelineService V1• Writable RESTAPIs• TimelineService V1security

7 © Hortonworks Inc. 2011 – 2016. All Rights ReservedGA Releases (Recent + Planning)2.6 2.7 2.8/2.9 3.0Nov. 2014 Apr. 2015 2nd H 2016 (estimated) TBD• KMS• Long runningservice support• RollingUpgrade• Node LabelSupport• DockerContainer• PluggableAuthorization• SharedResourceCache• TimelineService V1.5• GracefulDecommission• Log CLIEnhancement• TimelineService V2

8 © Hortonworks Inc. 2011 – 2016. All Rights ReservedOutstanding YARN Features released in 2.6/2.7Default PartitionPartition BGPUsPartition CWindowsJDK 8 JDK 7 JDK 7⬢ Rolling UpgradeNode LabelPluggable ACLs

9 © Hortonworks Inc. 2011 – 2016. All Rights ReservedRecent Maintenance Releases Updates⬢ 2.6 and 2.7 maintenance releases are carried out–Only blockers and critical fixes are added⬢ Apache Hadoop 2.6–2.6.4 released in Feb. 2016–2.6.3 released in Dec. 2015–2.6.2 released in Oct. 2015⬢ Apache Hadoop 2.7–2.7.3 released in Aug. 2016–2.7.2 released in Jan. 2016–2.7.1 released in Jul. 2015

10© Hortonworks Inc. 2011 – 2016. All Rights Reserved10© Hortonworks Inc. 2011 – 2016. All Rights ReservedPresent

11© Hortonworks Inc. 2011 – 2016. All Rights ReservedYARN in Modern Data Architecture⬢ Modern Data Architecture–Enable applications to have access to allyour enterprise data through anefficient centralized platform–Supported with a centralized approachgovernance, security and operations–Versatile to handle any applications anddatasets no matter the size or type⬢ YARN’s Evolution–The “CORE” of Modern DataArchitecture–Centralized resource management, highefficient scheduling, flexible resourcemodel, isolation in security andperformance, “colorful” applicationssupport, etc.

12© Hortonworks Inc. 2011 – 2016. All Rights ReservedApache Hadoop YARNResourceManager(active)ResourceManager(standby)NodeManager1NodeManager2NodeManager3NodeManager4Resources: 128G, 16 vcoresAuto-calculate node resourcesLabel: SASDynamically update noderesources

13© Hortonworks Inc. 2011 – 2016. All Rights ReservedNodeManager Resource Management⬢ Options to report NM resources based on node hardware–YARN-160–Restart of the NM required to enable feature⬢ Alternatively, admins can use the rmadmin command to update the node’s resources–YARN-291–Looks at the dynamic-resource.xml–No restart of the NM or the RM required

14© Hortonworks Inc. 2011 – 2016. All Rights ReservedApache Hadoop YARN SchedulerInter queue pre-emptionImprovements to pre-emptionApplicationQueue B – 25%Queue C – 25%Label: SAS (non-exclusive)Queue A – 50%Priority/FIFO, FairResourceManager(active)Application, Queue A, 4G, 1 vcoreSupport for application priorityReservation for applicationSupport for cost based placementagentUser

15© Hortonworks Inc. 2011 – 2016. All Rights ReservedCapacity scheduler⬢ Support for application priority within a queue–YARN-1963–Users can specify application priority–Specified as an integer, higher number is higher priority–Application priority can be updated while it’s running⬢ Improvements to reservations–YARN-2572–Support for cost based placement agent added in addition to greedy⬢ Queue allocation policy can be switched to fair sharing–YARN-3319–Containers allocated on a fair share basis instead of FIFO

16© Hortonworks Inc. 2011 – 2016. All Rights ReservedCapacity scheduler⬢ Support for non-exclusive node labels–YARN-3214–Improvement over partition that existed earlier–Better for cluster utilization⬢ Improvements to pre-emption

17© Hortonworks Inc. 2011 – 2016. All Rights ReservedNode 1NodeManagerSupport added for gracefuldecomissioning128G, 16 vcoresLaunch Applicaton 1 AMAM process/Docker container(alpha)Launch AM process viaContainerExecutor – DCE, LCE, WSCE.Monitor/isolate memory and cpu.Support added for disk and networkisolation via CGroups(alpha)Apache Hadoop YARN Application LifecycleResourceManager(active)Request containersAllocate containersSupport added to resize containers. Container 1 process/Dockercontainer(alpha)Container 2 process/Dockercontainer(alpha)Launch containers on node using DCE,LCE, WSCE. Monitor/isolate memory andcpu. Support added for disk and networkisolation using Cgroups(alpha).History Server(ATS 1.5– leveldb+ HDFS, JHS - HDFS)HDFSLog aggregation

18© Hortonworks Inc. 2011 – 2016. All Rights ReservedApache Hadoop YARN⬢ Graceful decommissioning of NodeManagers–YARN-914–Drains a node that’s being decommissioned to allow running containers to finish⬢ Resource isolation support for disk and network–YARN-2619, YARN-2140–Containers get a fair share of disk and network resources using CGroups–Alpha feature⬢ Docker support in LinuxContainerExecutor–YARN-3853–Support to launch Docker containers alongside process containers–Alpha feature

19© Hortonworks Inc. 2011 – 2016. All Rights ReservedApache Hadoop YARN⬢ Support for container resizing–YARN-1197–Allows applications to change the size of an existing container⬢ ATS 1.5–YARN-4233–Store timeline events on HDFS–Better scalability and reliability

20© Hortonworks Inc. 2011 – 2016. All Rights ReservedOperational support⬢ Improvements to existing tools (like yarn logs)⬢ New tools added (yarn top)⬢ Improvements to the RM UI to expose more details about running applications

21© Hortonworks Inc. 2011 – 2016. All Rights Reserved21© Hortonworks Inc. 2011 – 2016. All Rights ReservedFuture

22© Hortonworks Inc. 2011 – 2016. All Rights ReservedPackaging Containers– Lightweight mechanism for packaging and resource isolation– Popularized and made accessible by Docker– Can replace VMs in some cases– Or more accurately, VMs got used in places where they didn’tneed to be Native integration ++ in YARN– Support for “Container Runtimes” in LCE: YARN-3611– Process runtime– Docker runtime

23© Hortonworks Inc. 2011 – 2016. All Rights ReservedAPIs Applications need simple APIs Need to be deployable “easily” Simple REST API layer fronting YARN– https://issues.apache.org/jira/browse/YARN-4793– [Umbrella] Simplified API layer for services and beyond Spawn services & Manage them

24© Hortonworks Inc. 2011 – 2016. All Rights ReservedYARN as a Platform YARN itself is evolving to support services and complex apps– https://issues.apache.org/jira/browse/YARN-4692– [Umbrella] Simplified and first-class support for services in YARN Scheduling– Application priorities: YARN-1963– Affinity / anti-affinity: YARN-1042– Services as first-class citizens: Preemption, reservations etc

25© Hortonworks Inc. 2011 – 2016. All Rights ReservedYARN as a Platform (Contd) Application & Services upgrades– ”Do an upgrade of my Spark / HBase apps with minimal impact to end-users”– YARN-4726 Simplified discovery of services via DNS mechanisms: YARN-4757 YARN Federation – to infinity and beyond: YARN-2915

26© Hortonworks Inc. 2011 – 2016. All Rights ReservedYARN Service Framework Platform is only as good as the tools A native YARN framework– https://issues.apache.org/jira/browse/YARN-4692– [Umbrella] Native YARN framework layer for services andbeyond Slider supporting a DAG of apps:– https://issues.apache.org/jira/browse/SLIDER-875

27© Hortonworks Inc. 2011 – 2016. All Rights ReservedOperational and User Experience Modern YARN web UI - YARN-3368 Enhanced shell interfaces Metrics: Timeline Service V2 – YARN-2928 Application & Services monitoring, integration with other systems First class support for YARN hosted services in Ambari– https://issues.apache.org/jira/browse/AMBARI-17353

28© Hortonworks Inc. 2011 – 2016. All Rights ReservedUse-cases.. Assemble!YARN and Other Platform ServicesStorageResourceManagement SecurityServiceDiscovery ManagementMonitoringAlertsHoliday AssemblyHBaseWebServerIOT AssemblyKafka Storm HBase SolrGovernanceMR Tez Spark …

29© Hortonworks Inc. 2011 – 2016. All Rights ReservedFuture Work List (I)⬢ Arbitrary resource types–YARN-3926–Admins can decide what resource types tosupport–Resource types read via a config file⬢ New scheduler features–YARN-4902–Support richer placement strategies such asaffinity, anti-affinity⬢ Distributed scheduling–YARN-2877, YARN-4742–NMs run a local scheduler–Allows faster scheduling turnaround⬢ YARN federation–YARN-2915–Allows YARN to scale out to tens of thousands ofnodes–Cluster of clusters which appear as a single clusterto an end user⬢ Better support for disk and network isolation–Tied to supporting arbitrary resource types

30© Hortonworks Inc. 2011 – 2016. All Rights ReservedFuture Work List (II)⬢ Simplified and first-class support forservices in YARN–YARN-4692–Container restart (YARN-3988)•Allow container restart withoutlosing allocation–Service discovery via DNS (YARN-4757)•Running services can bediscovered via DNS–Allocation re-use (YARN-4726)•Allow AMs to stop a container butnot lose resources on the node⬢ Enhance Docker support–YARN-3611–Support to mount volumes–Isolate containers using CGroups⬢ ATS v2 Phase 2–YARN-2928 (Phase 1), YARN-5355 (Phase 2)–Run timeline service on Hbase–Support for more data, better performance⬢ Also in the pipeline–Switch to Java 8 with Hadoop 3.0–Add support for GPU isolation–Better tools to detect limping nodes–New RM UI – YARN-3368

Movatterモバイル変換

Change Language

Apache Hadoop YARN: Past, Present and Future

Embed presentation

Recommended

More Related Content

What's hot

Viewers also liked

Similar to Apache Hadoop YARN: Past, Present and Future

More from DataWorks Summit/Hadoop Summit

Recently uploaded

Apache Hadoop YARN: Past, Present and Future