NotificationsYou must be signed in to change notification settings
Fork9k
Star15k

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Jump to bottom

YARN-11781. Implement Dynamic Requests Handling in CapacityScheduler#7448

Open

TaoYang526 wants to merge2 commits intoapache:trunk

base:trunk

Choose a base branch

fromTaoYang526:YARN-11781

Open

YARN-11781. Implement Dynamic Requests Handling in CapacityScheduler#7448

TaoYang526 wants to merge2 commits intoapache:trunkfromTaoYang526:YARN-11781

+1,010 −0

Conversation

Copy link

Contributor

TaoYang526 commentedMar 1, 2025

Description of PR

Details:

Dynamic Request Handling: The RequestsHandler class has been added to handle and modify resource and scheduling requests at the beginning of the CapacityScheduler#allocate method.
Configuration-Driven Updates: The behavior of the request handler can be configured dynamically via capacity-scheduler.xml configuration, allowing for updates to be applied without restarting the ResourceManager.
Scriptable Matching: Requests can be matched and modified based on JavaScript expressions, providing flexibility in defining which requests should be updated and how.
Support for ResourceRequest to SchedulingRequest Conversion: The handler supports converting ResourceRequest objects to SchedulingRequest objects, enabling more advanced scheduling capabilities.

Usage: To enable and configure the request handler, administrators can set the following properties in capacity-scheduler.xml:
yarn.scheduler.capacity.request-handler.enabled: Enables or disables the request handler.
yarn.scheduler.capacity.request-handler.updates: Specifies the JSON configuration for the request updates.

Configuration example:

<property>  <name>yarn.scheduler.capacity.request-handler.enabled</name>  <value>true</value></property><property>  <name>yarn.scheduler.capacity.request-handler.updates</name>  <value>    {      "items": [        {          "appMatchExpr": "queue == 'default'",          "requestMatchExpr": "priority > 10",          "isRRToSR": true,          "partition": "partitionA",          "executionType": "GUARANTEED",          "allocationTags": ["tag1", "tag2"],          "placementConstraint": "and(in,rack,hbase:notin,node,zk)"        }      ]    }  </value></property>

How was this patch tested?

For code changes:

Does the title or this PR starts with the corresponding JIRA issue id (e.g. 'HADOOP-17799. Your PR title ...')?
Object storage: have the integration tests been executed and the endpoint declared according to the connector-specific documentation?
If adding new dependencies to the code, are these dependencies licensed in a way that is compatible for inclusion underASF 2.0?
If applicable, have you updated theLICENSE,LICENSE-binary,NOTICE-binary files?

github-actionsbot added YARN trunk labels

Mar 1, 2025

This comment was marked as outdated.

YARN-11781. Implement dynamic requests handling in CapacityScheduler.

4a0a769

TaoYang526 force-pushed theYARN-11781 branch fromb204aa9 to4a0a769Compare

March 4, 2025 06:37

Copy link

hadoop-yetus commentedMar 4, 2025

💔-1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 48s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 1s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any@author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	40m 35s		trunk passed
+1 💚	compile	1m 0s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 52s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	0m 57s		trunk passed
+1 💚	mvnsite	0m 56s		trunk passed
+1 💚	javadoc	0m 56s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 48s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 55s		trunk passed
+1 💚	shadedclient	39m 44s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 44s		the patch passed
+1 💚	compile	0m 53s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 53s		the patch passed
+1 💚	compile	0m 45s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	0m 45s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
-0⚠️	checkstyle	0m 45s	/results-checkstyle-hadoop-yarn-project_hadoop-yarn_hadoop-yarn-server_hadoop-yarn-server-resourcemanager.txt	hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager: The patch generated 1 new + 117 unchanged - 0 fixed = 118 total (was 117)
+1 💚	mvnsite	0m 47s		the patch passed
+1 💚	javadoc	0m 44s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 39s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 53s		the patch passed
+1 💚	shadedclient	40m 20s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	116m 9s		hadoop-yarn-server-resourcemanager in the patch passed.
-1 ❌	asflicense	0m 33s	/results-asflicense.txt	The patch generated 2 ASF License warnings.
		252m 24s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base:https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7448/2/artifact/out/Dockerfile
GITHUB PR	#7448
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux e9f585671c81 5.15.0-131-generic#141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk /`4a0a769`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7448/2/testReport/
Max. process+thread count	918 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7448/2/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0https://yetus.apache.org

This message was automatically generated.

fix checkstyle/license warnings.

abdedf1

Copy link

hadoop-yetus commentedMar 5, 2025

🎊+1 overall

Vote	Subsystem	Runtime	Logfile	Comment
+0 🆗	reexec	0m 49s		Docker mode activated.
			_ Prechecks _
+1 💚	dupname	0m 0s		No case conflicting files found.
+0 🆗	codespell	0m 0s		codespell was not available.
+0 🆗	detsecrets	0m 0s		detect-secrets was not available.
+1 💚	@author	0m 0s		The patch does not contain any@author tags.
+1 💚	test4tests	0m 0s		The patch appears to include 3 new or modified test files.
			_ trunk Compile Tests _
+1 💚	mvninstall	41m 44s		trunk passed
+1 💚	compile	1m 2s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	compile	0m 52s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	checkstyle	0m 57s		trunk passed
+1 💚	mvnsite	0m 56s		trunk passed
+1 💚	javadoc	0m 59s		trunk passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 49s		trunk passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 57s		trunk passed
+1 💚	shadedclient	40m 4s		branch has no errors when building and testing our client artifacts.
			_ Patch Compile Tests _
+1 💚	mvninstall	0m 45s		the patch passed
+1 💚	compile	0m 53s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javac	0m 53s		the patch passed
+1 💚	compile	0m 43s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	javac	0m 43s		the patch passed
+1 💚	blanks	0m 0s		The patch has no blanks issues.
+1 💚	checkstyle	0m 45s		the patch passed
+1 💚	mvnsite	0m 47s		the patch passed
+1 💚	javadoc	0m 44s		the patch passed with JDK Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04
+1 💚	javadoc	0m 39s		the patch passed with JDK Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
+1 💚	spotbugs	1m 52s		the patch passed
+1 💚	shadedclient	40m 35s		patch has no errors when building and testing our client artifacts.
			_ Other Tests _
+1 💚	unit	114m 53s		hadoop-yarn-server-resourcemanager in the patch passed.
+1 💚	asflicense	0m 35s		The patch does not generate ASF License warnings.
		253m 3s

Subsystem	Report/Notes
Docker	ClientAPI=1.48 ServerAPI=1.48 base:https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7448/3/artifact/out/Dockerfile
GITHUB PR	#7448
Optional Tests	dupname asflicense compile javac javadoc mvninstall mvnsite unit shadedclient spotbugs checkstyle codespell detsecrets
uname	Linux eefadf7d94ac 5.15.0-131-generic#141-Ubuntu SMP Fri Jan 10 21:18:28 UTC 2025 x86_64 x86_64 x86_64 GNU/Linux
Build tool	maven
Personality	dev-support/bin/hadoop.sh
git revision	trunk /`abdedf1`
Default Java	Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Multi-JDK versions	/usr/lib/jvm/java-11-openjdk-amd64:Ubuntu-11.0.26+4-post-Ubuntu-1ubuntu120.04 /usr/lib/jvm/java-8-openjdk-amd64:Private Build-1.8.0_442-8u442-b06~~us1-0ubuntu1~~20.04-b06
Test Results	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7448/3/testReport/
Max. process+thread count	873 (vs. ulimit of 5500)
modules	C: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager U: hadoop-yarn-project/hadoop-yarn/hadoop-yarn-server/hadoop-yarn-server-resourcemanager
Console output	https://ci-hadoop.apache.org/job/hadoop-multibranch/job/PR-7448/3/console
versions	git=2.25.1 maven=3.6.3 spotbugs=4.2.2
Powered by	Apache Yetus 0.14.0https://yetus.apache.org

This message was automatically generated.

zeekling suggested changes

Mar 9, 2025

View reviewed changes

...java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/CapacityScheduler.java

		try {
		reinitRequestsHandler(this.conf);
		} catch (Throwable innerT) {
		LOG.error("Failed to re-init requests handler : {}",

Copy link

zeeklingMar 9, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Will the basic functions be affected if the initialization fails?

Copy link

ContributorAuthor

TaoYang526Mar 10, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Thanks@zeekling for the review.
oldConf is a configuration that has already been validated and is in effect, the invocation of CapacityScheduler#reinitRequestsHandler here will not throw an exception actually. The purpose of this try/catch block is only included to make sure that the original rollback logic remains unaffected.

Copy link

zeeklingMar 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

got it

zeekling reviewed

Mar 15, 2025

View reviewed changes

...n/java/org/apache/hadoop/yarn/server/resourcemanager/scheduler/capacity/RequestsHandler.java

		public RequestsHandleResponse handle(FiCaSchedulerApp app,
		List<ResourceRequest> resourceRequests,
		List<SchedulingRequest> schedulingRequests) {
		readLock.lock();

Copy link

zeeklingMar 15, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This is an update operator, why use readLock?

Copy link

ContributorAuthor

TaoYang526Mar 17, 2025

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

The readLock and writeLock are used to protect fields in RequestsHandler such as enabled and updateItems, which can be used concurrently, for example, they could be updated while refreshQueues is called in a RPC thread, meanwhile could be read to handle requests in other RPC threads. So that these fields should be modified exclusively in the initialize method (requiring writeLock for mutual exclusion) but only read concurrently in the handle method (where readLock enables multiple threads to access the shared state simultaneously without blocking)‌.

Labels

trunk YARN

3 participants

Movatterモバイル変換

Navigation Menu

Search code, repositories, users, issues, pull requests...

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

YARN-11781. Implement Dynamic Requests Handling in CapacityScheduler#7448

Are you sure you want to change the base?

YARN-11781. Implement Dynamic Requests Handling in CapacityScheduler#7448

Conversation

TaoYang526 commentedMar 1, 2025

Description of PR

How was this patch tested?

For code changes:

This comment was marked as outdated.

hadoop-yetus commentedMar 4, 2025

hadoop-yetus commentedMar 5, 2025

zeeklingMar 9, 2025

Choose a reason for hiding this comment

TaoYang526Mar 10, 2025

Choose a reason for hiding this comment

zeeklingMar 15, 2025

Choose a reason for hiding this comment

zeeklingMar 15, 2025

Choose a reason for hiding this comment

TaoYang526Mar 17, 2025

Choose a reason for hiding this comment