We already haveHTTPERROR_ALLOWED_CODES that overridehandle_httpstatus_list but the setting name is not genericHTTPERROR_, we could use onRedirectMiddleware instead of creating a new setting.

Right,handle_httpstatus_list (the spider attr and the meta key) are mostly forHttpErrorMiddleware,RedirectMiddleware is their additional user. Similarly forHTTPERROR_ALLOW_ALL (the setting) andhandle_httpstatus_all (the spider attr and the meta key). But out of theseRedirectMiddleware only uses both meta keys and thehandle_httpstatus_list spider attr, not using the spider attr and the setting forall. This is inconsistent (though I haven't considered what will the spider behavior be if this made consistent).

I don’t like the HTTP errors middleware reading theREDIRECT_ALLOWED_HTTP_CODES setting…

I wonder if we should take a step back and try to think through what the ideal “settings and meta API” should be, and then try to move towards it in a backward-compatible way with deprecations where needed.

I feel this should be similar to dont_filter, where you have a generic setting/meta and a specific setting/meta (e.g. allow_offsite). I think the specific should override the generic, i.e. no merging, e.g. if generic is [401] and want to extend with 402 for a specific middleware, you set the specific setting/meta to [401,402], not just [402]. But I am not completely sure.

Is this test useless/duplicate withoutmin_spider?

It seems, if you see the only assert is:assert spider.download_delay == expected lmk if you want to keep and drop themin_spider no sure what we would check in this case.

OK, this is stupid :)

The extension setsspider.download_delay in_spider_opened() (based onAUTOTHROTTLE_START_DELAY,DOWNLOAD_DELAY and the original value ofspider.download_delay, just like the params of this test), so that the downloader could use it as the start delay value for newly created slots inscrapy.core.downloader._get_concurrency_delay(). So if we want to stop usingspider.download_delay we need this to work in some other way (and I don't think we can change theDOWNLOAD_DELAY setting value here).

@Gallaecio thoughts?

Puff, headache incoming… 😓

Maybe we should keep download_delay until#6660 is addressed?

I'm fine with keeping it. Note that we need to revert a part of#6994 before the next release for this:#7117

I wonder if we should try to find a way to trigger this only once per spider, but no strong opinion. Maybe louder is better 🙂

I wonder if we should handle this somewhat differently.

i.e., when both the setting and the spider attribute are defined, the spider attribute, even if deprecated, should be considered an override of the setting value, and the setting value should be ignored.

In fact, I wonder if we should also consider the meta value an override of both the setting and the spider attribute, and hence ignore those if the meta value is set. But I am fine with keeping that out of scope here.

Either way, we should make sure we clarify this in the docs and the release notes.