- Notifications
You must be signed in to change notification settings - Fork586
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
-
AIM: Increase the consumption of a message at the Consumer End using Thread Pool in Spring Application (or even any other recommended way if possible) with Rabbitmq as message broker Scenario: I have a scenario in which our application takes around ~15 seconds for consumption of 1 message whereas we producing 1 message per second, in simple way. So, by the time 1st message is consumed after 15 seconds. We have 14 more messages and so on. Is there any way to reduce this gap on the consumer and producer side by "increasing Consumption at consumer end"? Existing Understanding: I tried to increase Thread Pool so that each consumer has 15 threads. This will ensure Existing Implementation: With the above understanding, I implemented above implementation, but don't see any significant improvement on the consumption rate at Consumer end. I found consumption rate at consumer end independent to Thread Pool Is above implementation correct or missing something? Are there any other ways to solve this issue? Added it at Stackoverflow (https://stackoverflow.com/q/72798968/5662835) coz I'm not aware which Channel is actively looked upon by Rabbitmq Team |
BetaWas this translation helpful?Give feedback.
All reactions
Replies: 1 comment 10 replies
-
Consider using discussions for questions. Increasing the number of consumer work pool threads will help in some cases and won't make any difference or be a net negative in others. We cannot suggest much without knowing how many cores the consumer application has available to it and what it does. In fact, onlybenchmarking of your consumer will tell the whole story. |
BetaWas this translation helpful?Give feedback.
All reactions
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
-
Everything is okay, just that the logic contains a REST call that takes around 15-30 seconds in worst case. I'm thinking to make system resilient when majority of messages are having REST call that takes around 15-30 seconds in worst case by introducing Consumer Thread Pool or by any other way possible.
I was thinking of a case where internally library might using Divide and Conquer Approach (or something similar) to optimize a particular flow. |
BetaWas this translation helpful?Give feedback.
All reactions
-
It's hard to say, but you can use a few dozens in a given connection without problems. You can start experimenting with a few channels (10, 20, etc) and one consumer per channel. Considering the message processing is long compared to the actual delivery, set Qos (prefetch count) to 2, this way you should have a message ready when you're done processing. Set the size of the thread pool to the number of channels. |
BetaWas this translation helpful?Give feedback.
All reactions
👍 1
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
-
There seem to be relation between |
BetaWas this translation helpful?Give feedback.
All reactions
-
Thanks for the follow-up and the answer on SO, users will appreciate it for sure. |
BetaWas this translation helpful?Give feedback.
All reactions
Uh oh!
There was an error while loading.Please reload this page.
Uh oh!
There was an error while loading.Please reload this page.
-
Yes, Prefetch count can be configured also, and should be low in scenario where processing of Single Message is time consuming coz in that case 250 (by default), 249 will be blocked and will get a change late and new message can be proceeded before them in some other consumer. |
BetaWas this translation helpful?Give feedback.
All reactions
This discussion was converted from issue #804 on June 29, 2022 10:21.