There's a lot of policy in here, and a lot of knobs to go with it. Do you have a sense for how all of this is going to behave in the real-world, and if/how someone would utilize these knobs effectively? How did you arrive at this specific set and also the defaults employed?

Some of the criteria used:

• Have a good replacement for setting theMinThreads as a workaround
  • This would now be to setThreadsToAddWithoutDelay to the equivalent and setMaxThreadsToAddBeforeFallback to something higher to give some buffer for spikes that may need more threads
  • MaxThreadsToAddBeforeFallback could also be set to a large value to effectively unlimit the heuristic
• Use progressive delays to avoid creating too many threads too quickly
  • Without that, it would be conceivable thatMaxThreadsToAddBeforeFallback threads would be created in short order to respond to even a short sync-over-async IO before the IO even completes (if there are that many work items that would block on the async work)
  • The delay also helps to create a high watermark of how many threads were necessary last time to unblock, so that when there's a limit to how many work items would block, it would quickly release existing waiting threads to let other work be done meanwhile
  • The larger the number of threads that get unblocked all at once, the higher the latency of their processing would be after unblocking. There's probably not a good solution to this.
  • Ideally it would not require as many threads for an async IO completion to unblock waiting threads, sort of like on Windows where a separate pool of threads handles IO completions, needs some experimentation
  • It's not always clear that adding more threads would help, more so in starvation-type cases
• No one set of defaults will work well for all cases, use conservative defaults to start with
  • The current defaults are much more agressive than before
  • TheMaxThreadsToAddBeforeFallback_ProcCountFactor of10 came from a prior discussion where we felt that adding 10x the proc count relatively quickly may not be too bad
  • The defaults can be made more aggressive easily, but it would be difficult to make the defaults more conservative since apps that work well with the defaults may not after that without configuration
  • The really bad cases where many 100s or even 1000s of threads need to be created will likely need to configure for the app's situation based on expected workload and how bad it can get, in order to work around the issue
• Make things sufficiently configurable
  • It would have been nice to make configurable the delay threshold for detecting starvation and the delay used to add threads during continuous starvation. Now, for sync-over-async the delay and rate of progression in delays can be adjusted.
  • Similarly to hill climbing config values, the config values don't have to be used but it can be helpful to enable the freedom to configure them
  • I expect I would suggest most users running into bad blocking due to sync-over-async to configureThreadsToAddWithoutDelay andMaxThreadsToAddBeforeFallback, and perhapsMaxDelayUntilFallbackMs to control the thread injection latency for spikes
• I intend to use the same config values (maybe with a couple of others) for improving the starvation heuristic similarly in the future
  • Starvation is a bit different and may need a few quirks, but hopefully we can use something similar

Decided to removeMaxThreadsToAddBeforeFallback and renamedMaxDelayBeforeFallbackMs toMaxDelayMs in the latest commit. The max threads limit before falling back to starvation seems unnecessary, it would be unlimited for starvation anyway. Now the only time it would fall back to starvation is in low-memory situations.