With a large PHPUnit tests suite, I receive this error "Failed to open stream: Too many open files".
I found the original code from@nicolas-grekas here and with small adjustments, it resolves my issue.

Warning: tests for HttpClient are OK but would someone try with a real HTTP/2 client push? I was not able to test it.

carsonbot added Status: Needs Review Bug HttpClient labels

Jan 19, 2022

carsonbot added this to the6.1 milestone

Jan 19, 2022

nicolas-grekas changed the base branch from6.1 to4.4

January 19, 2022 09:39

nicolas-grekas modified the milestones:6.1,4.4

Jan 19, 2022

adrienfr mentioned this pull request

Jan 19, 2022

[HttpClient] Failed to open stream: Too many open files#44900

Closed

nicolas-grekas reviewed

Jan 19, 2022

View reviewed changes

src/Symfony/Component/HttpClient/Internal/CurlClientState.php OutdatedShow resolvedHide resolved

Copy link

ContributorAuthor

adrienfr commentedJan 19, 2022•
edited
Loading

@nicolas-grekas I think the issue is related to$this->handle.
If I keep thestatic way and create this :

$multi = (object) [//'handle' => $this->handle,//'handle' => &$this->handle,'logger' => &$this->logger,'pushedResponses' => &$this->pushedResponses,'dnsCache' => &$this->dnsCache,'handlesActivity' => &$this->handlesActivity,'openHandles' => &$this->openHandles,'execCounter' => &$this->execCounter,'pauseExpiries' => &$this->pauseExpiries,'lastTimeout' => &$this->lastTimeout,  ];// Keep a dummy "onPush" reference to work around a refcount bug in PHPcurl_multi_setopt($this->handle, \CURLMOPT_PUSHFUNCTION,$this->onPush =staticfunction ($parent,$pushed,array$requestHeaders)use ($multi,$maxPendingPushes) {returnself::handlePush($parent,$pushed,$requestHeaders,$multi,$maxPendingPushes);  });

It works. But I soon as I uncomment one of thehandle key in my$multi object, I have the "failed to open stream: Too many open files" error, as if this$this->handle is not properly "reset".

One other thing I noticed, a lot moreCurlMultiHandle usage in newer version:

SF 5.3.7 :
CurlHttpClient just beforecurl_multi_setopt($this->multi->handle, \CURLMOPT_PUSHFUNCTION, ...

var_dump($this->multi->handle);

.  854 / 4775 ( 17%)resource(15232) oftype (curl_multi).resource(15342) oftype (curl_multi).resource(15383) oftype (curl_multi)

SF 5.3-dev :
CurlClientState juste beforecurl_multi_setopt($this->handle, \CURLMOPT_PUSHFUNCTION, $this->onPush, ...

var_dump($this->handle);

.  854 / 4775 ( 17%)resource(26271) oftype (curl_multi).resource(26500) oftype (curl_multi).resource(26569) oftype (curl_multi)

Copy link

ContributorAuthor

adrienfr commentedJan 19, 2022•
edited
Loading

@nicolas-grekas might have found a "leak" inCurlClientState::reset() this block has been removed in recent PRs:

if (\is_resource($this->handle) ||$this->handleinstanceof \CurlMultiHandle) {if (\defined('CURLMOPT_PUSHFUNCTION')) {curl_multi_setopt($this->handle, \CURLMOPT_PUSHFUNCTION,null);    }$active =0;while (\CURLM_CALL_MULTI_PERFORM ===curl_multi_exec($this->handle,$active));}foreach ($this->openHandlesas [$ch]) {if (\is_resource($ch) ||$chinstanceof \CurlHandle) {curl_setopt($ch, \CURLOPT_VERBOSE,false);    }}

If I add these 2 blocks (especially the one with thecurl_multi_setopt($this->handle, \CURLMOPT_PUSHFUNCTION, null); call), no more issue. Do you want me to update the PR by re-adding these 2 blocks?

Copy link

ContributorAuthor

adrienfr commentedJan 19, 2022

Regarding this part added in your last PR, I don't really understand the goal but shouldn't we have a call tocurl_share_close() somewhere? This block also seems responsible for having a lot more open streams than before (5.3.7)

$this->share =curl_share_init();curl_share_setopt($this->share, \CURLSHOPT_SHARE, \CURL_LOCK_DATA_DNS);curl_share_setopt($this->share, \CURLSHOPT_SHARE, \CURL_LOCK_DATA_SSL_SESSION);if (\defined('CURL_LOCK_DATA_CONNECT')) {curl_share_setopt($this->share, \CURLSHOPT_SHARE, \CURL_LOCK_DATA_CONNECT);}

Copy link

Member

nicolas-grekas commentedJan 19, 2022

I wish I could reproduce locally. If you can share the code privately with me, you can reach me on Symfony's Slack.

Copy link

Member

nicolas-grekas commentedJan 19, 2022

Can you please try this patch instead?

--- a/src/Symfony/Component/HttpClient/Internal/CurlClientState.php+++ b/src/Symfony/Component/HttpClient/Internal/CurlClientState.php@@ -23,9 +23,9 @@ use Symfony\Component\HttpClient\Response\CurlResponse;  */ final class CurlClientState extends ClientState {-    /** @var \CurlMultiHandle|resource */+    /** @var \CurlMultiHandle|resource|null */     public $handle;-    /** @var \CurlShareHandle|resource */+    /** @var \CurlShareHandle|resource|null */     public $share;     /** @var PushedResponse[] */     public $pushedResponses = [];@@ -65,8 +65,17 @@ final class CurlClientState extends ClientState             return;         }-        curl_multi_setopt($this->handle, \CURLMOPT_PUSHFUNCTION, function ($parent, $pushed, array $requestHeaders) use ($maxPendingPushes) {-            return $this->handlePush($parent, $pushed, $requestHeaders, $maxPendingPushes);+        // Clone to prevent a circular reference+        $multi = clone $this;+        $multi->handle = null;+        $multi->share = null;+        $multi->pushedResponses = &$this->pushedResponses;+        $multi->logger = &$this->logger;+        $multi->handlesActivity = &$this->handlesActivity;+        $multi->openHandles = &$this->openHandles;++        curl_multi_setopt($this->handle, \CURLMOPT_PUSHFUNCTION, static function ($parent, $pushed, array $requestHeaders) use ($multi, $maxPendingPushes) {+            return $multi->handlePush($parent, $pushed, $requestHeaders, $maxPendingPushes);         });     }