version: 6.5.0

Hi team,

We're experiencing intermittent issue with requests processing where the requests are stuck in queue but not processed. I'll try to explain in detail about the situation and my understanding about Scribe Java lib:

• Whenever requests are made using scribe java, there won't be any response back to the client and the requests stuck there for hours until restart of jvm is performed.
• Scribejava has ApacheHttpClient that does actual requests to target server. This underlying httpclient has queue mechanism for processing the requests where the requests are added into a queue and processed in async manner.

   java.lang.Thread.State: WAITING (parking)at sun.misc.Unsafe.park(Native Method)- parking to wait for  <0x00000000b150e370> (a java.util.concurrent.CountDownLatch$Sync)at java.util.concurrent.locks.LockSupport.park(LockSupport.java:175)at java.util.concurrent.locks.AbstractQueuedSynchronizer.parkAndCheckInterrupt(AbstractQueuedSynchronizer.java:836)at java.util.concurrent.locks.AbstractQueuedSynchronizer.doAcquireSharedInterruptibly(AbstractQueuedSynchronizer.java:997)at java.util.concurrent.locks.AbstractQueuedSynchronizer.acquireSharedInterruptibly(AbstractQueuedSynchronizer.java:1304)at java.util.concurrent.CountDownLatch.await(CountDownLatch.java:231)at com.github.scribejava.httpclient.apache.OAuthAsyncCompletionHandler.getResult(OAuthAsyncCompletionHandler.java:91)at com.github.scribejava.httpclient.apache.ApacheHttpFuture.get(ApacheHttpFuture.java:37)at com.github.scribejava.core.httpclient.AbstractAsyncOnlyHttpClient.execute(AbstractAsyncOnlyHttpClient.java:35)at com.github.scribejava.core.oauth.OAuthService.execute(OAuthService.java:105)

• Looking at stacktrace from our thread dump, thread is waiting for the response from this client and we were under assumption that is could be connect timeout issues, this is what we're focusing on from last few weeks.
• Therefore, we added connection timeout setting for all calls.
• I can see that issue is reproducing even after applying connect timeout setting for client object. There is something else that is causing the issue for which I started analyzing heap dump for.

Heap dump analysis:

• From the heap dump, I can see that there are requests stuck inrequestQueue but not picked up by ioReactor thread.
  
• InternalHttpAsyncClient 'status' is STOPPED.
  
• IOReactor 'status' is ACTIVE
  
• IOReactor execute() looks like stuck or terminated when creating dispatcher threads.
  

I tried to understand the flow of execution, and if I understood it correctly somehow the execute() method fromAbstractMultiworkerIOReactor got terminated at marked section below:

I would like to request you to look at this issue and help me figuring out what's causing the behavior, is it a bug in code or due to performance reasons? I know it is bit older version but upgrading to latest version can fix the issue? It is not easily reproducible and happens randomly without any pattern.

Note that heap dump is collected after 2 hrs of the issue is reproduced.

Appreciate your help!

Thanks & Regards,
Karthik