hey@nejch, thanks for looking at my PR. I guess I should've looked at open issues before opening it :) To your questions: - I ran into the rate limit on the
/users/:id: endpoint. - Yes, attached to the end of this message. I first ran a short loop to make sure I've reached the rate limit, and then ran my code that fetched the data that I need. I've set
max_retries=1 which is why you will see only 1 retry instead of the default 10.
And yes it seems that these headers no longer exist for gitlab-com; the reporter of#1889 mentioned that as well. It is possible that Cloudflare is not passing these headers along. 2022-06-20T11:14:03+0300 [DEBUG] urllib3.connectionpool - Starting new HTTPS connection (1): gitlab.com:443send: b'GET /api/v4/groups?per_page=100&min_access_level=40 HTTP/1.1\r\nHost: gitlab.com\r\nUser-Agent: python-gitlab/3.5.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nPRIVATE-TOKEN: ***\r\nContent-type: application/json\r\n\r\n'reply: 'HTTP/1.1 200 OK\r\n'header: Date: Mon, 20 Jun 2022 08:14:04 GMTheader: Content-Type: application/jsonheader: Transfer-Encoding: chunkedheader: Connection: keep-aliveheader: Cache-Control: max-age=0, private, must-revalidateheader: Etag: W/"c5ab47387a7f6b720554bb2425b3b2fe"header: Link: <https://gitlab.com/api/v4/groups?min_access_level=40&order_by=name&owned=false&page=1&per_page=100&sort=asc&statistics=false&with_custom_attributes=false>; rel="first", <https://gitlab.com/api/v4/groups?min_access_level=40&order_by=name&owned=false&page=1&per_page=100&sort=asc&statistics=false&with_custom_attributes=false>; rel="last"header: Vary: Originheader: X-Content-Type-Options: nosniffheader: X-Frame-Options: SAMEORIGINheader: X-Next-Page: header: X-Page: 1header: X-Per-Page: 100header: X-Prev-Page: header: X-Request-Id: 01G602T2JSZ1GM92BS4FV1QQX8header: X-Runtime: 0.202389header: X-Total: 20header: X-Total-Pages: 1header: Strict-Transport-Security: max-age=31536000header: Referrer-Policy: strict-origin-when-cross-originheader: GitLab-LB: fe-13-lb-gprdheader: GitLab-SV: api-gke-us-east1-cheader: CF-Cache-Status: DYNAMICheader: Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"header: Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=Kax%2F2w4Ke3oK4eC%2BxRSkST%2B%2BlgKoD6aicEEb%2FtyX1TuxPR08UuWw0NwK%2BoWAVRnw9SKyUwFX%2BN9Q1PPn6AGTs%2Bbo4rqhIKnSg15srLa3YteFCMD264OgEhlRJq%2BXOicr8l2G6klgO%2BE%3D"}],"group":"cf-nel","max_age":604800}header: NEL: {"success_fraction":0.01,"report_to":"cf-nel","max_age":604800}header: Server: cloudflareheader: CF-RAY: 71e30c7c6f697d9e-TLVheader: Content-Encoding: gzip2022-06-20T11:14:04+0300 [DEBUG] urllib3.connectionpool - https://gitlab.com:443 "GET /api/v4/groups?per_page=100&min_access_level=40 HTTP/1.1" 200 Nonesend: b'GET /api/v4/groups/16076036/members?per_page=100&min_access_level=40 HTTP/1.1\r\nHost: gitlab.com\r\nUser-Agent: python-gitlab/3.5.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nPRIVATE-TOKEN: ***\r\nContent-type: application/json\r\n\r\n'reply: 'HTTP/1.1 200 OK\r\n'header: Date: Mon, 20 Jun 2022 08:14:05 GMTheader: Content-Type: application/jsonheader: Transfer-Encoding: chunkedheader: Connection: keep-aliveheader: Cache-Control: max-age=0, private, must-revalidateheader: Etag: W/"c3cd59286c4e3c98c5bd093619940c0c"header: Link: <https://gitlab.com/api/v4/groups/16076036/members?id=16076036&min_access_level=40&page=1&per_page=100>; rel="first", <https://gitlab.com/api/v4/groups/16076036/members?id=16076036&min_access_level=40&page=1&per_page=100>; rel="last"header: Vary: Originheader: X-Content-Type-Options: nosniffheader: X-Frame-Options: SAMEORIGINheader: X-Next-Page: header: X-Page: 1header: X-Per-Page: 100header: X-Prev-Page: header: X-Request-Id: 01G602T309ZWENB37JYNER4GH7header: X-Runtime: 0.129332header: X-Total: 12header: X-Total-Pages: 1header: Strict-Transport-Security: max-age=31536000header: Referrer-Policy: strict-origin-when-cross-originheader: GitLab-LB: fe-14-lb-gprdheader: GitLab-SV: api-gke-us-east1-dheader: CF-Cache-Status: DYNAMICheader: Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"header: Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=tXC%2F%2FLFtAda5ObhwRZUsRARnvdhywJpEFy%2Fm5gVyAMe86Uo1U9bLadeX%2FIq0iwF3AvOVKuJ2L%2BXWpBWL6gLgiTPKMgGlBQ7ysk6lNkN8WQ92XGDQOUmzzmBByB5WOWhjwHEMpO%2FAXKY%3D"}],"group":"cf-nel","max_age":604800}header: NEL: {"success_fraction":0.01,"report_to":"cf-nel","max_age":604800}header: Server: cloudflareheader: CF-RAY: 71e30c7f4c297d9e-TLVheader: Content-Encoding: gzip2022-06-20T11:14:04+0300 [DEBUG] urllib3.connectionpool - https://gitlab.com:443 "GET /api/v4/groups/16076036/members?per_page=100&min_access_level=40 HTTP/1.1" 200 Nonesend: b'GET /api/v4/users/10821733/ HTTP/1.1\r\nHost: gitlab.com\r\nUser-Agent: python-gitlab/3.5.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nPRIVATE-TOKEN: ***\r\nContent-type: application/json\r\n\r\n'reply: 'HTTP/1.1 429 Too Many Requests\r\n'header: Date: Mon, 20 Jun 2022 08:14:05 GMTheader: Content-Type: application/jsonheader: Content-Length: 89header: Connection: keep-aliveheader: Cache-Control: no-cacheheader: Vary: Originheader: X-Content-Type-Options: nosniffheader: X-Frame-Options: SAMEORIGINheader: X-Request-Id: 01G602T3CRRZ0AYAQA74CK41VHheader: X-Runtime: 0.044223header: Strict-Transport-Security: max-age=31536000header: Referrer-Policy: strict-origin-when-cross-originheader: GitLab-LB: fe-25-lb-gprdheader: GitLab-SV: api-gke-us-east1-cheader: CF-Cache-Status: DYNAMICheader: Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"header: Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=SCMaLNWcLLooykPtjJMaWYXilZsmlyr9FWguBvRtg5%2FQqF8h3Z9hfCO7ecFxErFy8M2UOG6wjEFso1zbpLR6ztI334%2FZTaDiHOZmPGczQbovD0dhCiQ0Q4rqFr0ApOLLML2rO83sanA%3D"}],"group":"cf-nel","max_age":604800}header: NEL: {"success_fraction":0.01,"report_to":"cf-nel","max_age":604800}header: Server: cloudflareheader: CF-RAY: 71e30c8188497d9e-TLV2022-06-20T11:14:05+0300 [DEBUG] urllib3.connectionpool - https://gitlab.com:443 "GET /api/v4/users/10821733/ HTTP/1.1" 429 89send: b'GET /api/v4/users/10821733/ HTTP/1.1\r\nHost: gitlab.com\r\nUser-Agent: python-gitlab/3.5.0\r\nAccept-Encoding: gzip, deflate\r\nAccept: */*\r\nConnection: keep-alive\r\nPRIVATE-TOKEN: ***\r\nContent-type: application/json\r\n\r\n'reply: 'HTTP/1.1 429 Too Many Requests\r\n'header: Date: Mon, 20 Jun 2022 08:14:05 GMTheader: Content-Type: application/jsonheader: Content-Length: 89header: Connection: keep-aliveheader: Cache-Control: no-cacheheader: Vary: Originheader: X-Content-Type-Options: nosniffheader: X-Frame-Options: SAMEORIGINheader: X-Request-Id: 01G602T3RSA1HGKBB4NW2GD366header: X-Runtime: 0.027737header: Strict-Transport-Security: max-age=31536000header: Referrer-Policy: strict-origin-when-cross-originheader: GitLab-LB: fe-22-lb-gprdheader: GitLab-SV: api-gke-us-east1-cheader: CF-Cache-Status: DYNAMICheader: Expect-CT: max-age=604800, report-uri="https://report-uri.cloudflare.com/cdn-cgi/beacon/expect-ct"header: Report-To: {"endpoints":[{"url":"https:\/\/a.nel.cloudflare.com\/report\/v3?s=jfa01qENyS5pkJ6u2UEIEYxEbSPMDM74JeI1wKLru9GbKCdcUe8iToj6SAIu4MO3V4KCe1L7Bgt4FcNdEAKQtK5KLZzg2srWszdc3aR1eB1nDBZd7zc4C2aF21d4h6hMAlD0MtJDGCo%3D"}],"group":"cf-nel","max_age":604800}header: NEL: {"success_fraction":0.01,"report_to":"cf-nel","max_age":604800}header: Server: cloudflareheader: CF-RAY: 71e30c843d047d9e-TLV2022-06-20T11:14:05+0300 [DEBUG] urllib3.connectionpool - https://gitlab.com:443 "GET /api/v4/users/10821733/ HTTP/1.1" 429 892022-06-20T11:14:05+0300 [WARNING] authomize.gitlab.client.gitlab_client - Hit rate limit not covered by SDK. Waiting 1 minute(s).
|
While debugging my code against gitlab.com (SaaS, not self-hosted) I realized that the SDK bombards the API with the same request.
I found out that for some reason, headers such as
Retry-Afterare missing from the response so only the defaultwait_timeis taken into account. This means that the SDK basically doesn't wait at all.Since for Gitlab.com most rate limits are calculated per 1 minute, I think it's better to set the default timeout to that value in case rate limit headers are missing.