Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

🔨 Update FastAPI People Experts script, refactor and optimize data fetching to handle rate limits#13267

Merged
tiangolo merged 8 commits intomasterfrom
people
Jan 28, 2025
Merged

🔨 Update FastAPI People Experts script, refactor and optimize data fetching to handle rate limits#13267
tiangolo merged 8 commits intomasterfrom
people

Conversation

@tiangolo
Copy link
Member

@tiangolotiangolo commentedJan 28, 2025
edited
Loading

🔨 Update FastAPI People Experts script, refactor and optimize data fetching to handle rate limits

FastAPI People Experts have not been updated in a few months because the script was hitting GitHub API rate limits.

The first step was to separate the logic into different scripts, I started with that before, so now contributors, reviewers, translators, and sponsors are computed in different scripts and GitHub Actions.

Then I was still hitting rate limits, I was thinking I would need to store the data in an external DB and update it gradually and continuously to avoid the rate limits, that would have been a big side project just to get that working. 😅

Then with this, I played a bit with different ways to get the data, how to limit the results, and also a bit to analyze the distribution of results to optimize the queries.

I discovered that (it seems) GitHub rate limits the GraphQL API based on an estimate of how much it would take to compute the queries in a worst-case scenario, not what they actually take to compute. So, using a query that would get 100 comments per discussion, I was consuming the rate limit per hour quickly. Nevertheless, there are only a handful of discussions that have more than 50 comments. When I updated the query to fetch only the 50 comments per discussion, that solved the rate limit. That's how I concluded that GitHub was assigning the rate limit "points" based on what would be a worst-case cost of computing each querybefore executing the query, so I was charged rate-limit-points before they were used.

And all this is just to say that the main thing that solved the problem was to set the query to fetch the first 50 comments instead of the first 100, even though most discussions have less than 50.

I also had a temporary snippet to do some quick stats of the fetched data and calculate how many had how many comments:

max_comments=0max_replies=0max_comments_discussion=Nonemax_replies_discussion=Nonecomments_counter=Counter[int]()comments_by_counts=defaultdict(list)replies_counter=Counter[int]()replies_by_counts=defaultdict(list)fordiscussionindiscussion_nodes:comments_counter[discussion.comments.totalCount]+=1comments_by_counts[discussion.comments.totalCount].append(discussion)ifdiscussion.comments.totalCount>max_comments:max_comments=discussion.comments.totalCountmax_comments_discussion=discussionforcommentindiscussion.comments.nodes:replies_counter[comment.replies.totalCount]+=1replies_by_counts[comment.replies.totalCount].append(comment)ifcomment.replies.totalCount>max_replies:max_replies=comment.replies.totalCountmax_replies_discussion=discussion

This was run in an interactive window, so it's not even a full script.

The results as of the moment of making this PR:

comments_counter==Counter({1:1272,2:851,3:640,4:463,0:461,5:346,6:251,7:173,8:129,9:120,10:86,11:53,12:50,13:45,14:40,18:26,15:25,16:21,17:19,21:19,19:15,20:13,24:8,23:7,22:7,25:6,29:4,28:4,31:3,34:3,27:3,33:3,43:2,40:2,42:2,30:2,105:1,46:1,36:1,26:1,64:1,53:1,39:1,35:1,55:1,98:1,66:1,37:1,77:1,52:1})

and:

replies_counter==Counter({0:19342,1:991,2:332,3:201,4:101,5:65,6:45,7:19,8:16,9:16,10:4,13:4,12:3,15:2,22:1,18:1,14:1})

@github-actionsgithub-actionsbot added the docsDocumentation about how to use FastAPI labelJan 28, 2025
@github-actions
Copy link
Contributor

@github-actions
Copy link
Contributor

@github-actions
Copy link
Contributor

@github-actions
Copy link
Contributor

@tiangolotiangolo added internal and removed docsDocumentation about how to use FastAPI labelsJan 28, 2025
@tiangolotiangolo marked this pull request as ready for reviewJanuary 28, 2025 20:33
@tiangolotiangolo merged commitff68d08 intomasterJan 28, 2025
56 checks passed
@tiangolotiangolo deleted the people branchJanuary 28, 2025 20:34
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

No reviews

Assignees

No one assigned

Labels

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

1 participant

@tiangolo

Comments


[8]ページ先頭

©2009-2026 Movatter.jp