Movatterモバイル変換


[0]ホーム

URL:


Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

Sign up
Appearance settings

Improve__setattr__ performance of Pydantic models by caching setter functions#10868

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
sydney-runkle merged 5 commits intopydantic:mainfromMarkusSintonen:fast-setattr
Nov 19, 2024

Conversation

MarkusSintonen
Copy link
Contributor

@MarkusSintonenMarkusSintonen commentedNov 18, 2024
edited by pydantic-hookybot
Loading

Change Summary

Attribute setting has been pretty slow forBaseModel due to the extensive checks it has been doing for every__setattr__ call. PR improves performance of__setattr__ by memoizing the attribute specific handlers to the model class. This makes the attribute assigning some 7x faster. Also add missing benchmarks for attribute usage.

fromtimeitimporttimeitfrompydanticimportBaseModelclassModel(BaseModel):field:intmodel=Model(field=1)defrun():model.field=2# Before 1.048# After 0.147print(timeit(run,number=1000000))

Related issue number

fix#10853

Checklist

  • The pull request title is a good summary of the changes - it will be used in the changelog
  • Unit tests for the changes exist
  • Tests pass on CI
  • Documentation reflects the changes where applicable
  • My PR is ready to review,please add a comment including the phrase "please review" to assign reviewers

Selected Reviewer:@sydney-runkle

@github-actionsgithub-actionsbot added the relnotes-fixUsed for bugfixes. labelNov 18, 2024
@codspeed-hqCodSpeed HQ
Copy link

codspeed-hqbot commentedNov 18, 2024
edited
Loading

CodSpeed Performance Report

Merging#10868 willnot alter performance

ComparingMarkusSintonen:fast-setattr (404b8b7) withmain (30ee4f4)

Summary

✅ 44 untouched benchmarks

🆕 2 new benchmarks

Benchmarks breakdown

BenchmarkmainMarkusSintonen:fast-setattrChange
🆕test_getattrN/A54 µsN/A
🆕test_setattrN/A87.7 µsN/A

@github-actionsGitHub Actions
Copy link
Contributor

github-actionsbot commentedNov 18, 2024
edited
Loading

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  pydantic
  main.py
  pydantic/_internal
  _model_construction.py
Project Total 

This report was generated bypython-coverage-comment-action

@MarkusSintonen
Copy link
ContributorAuthor

please review

pydantic-hooky[bot] reacted with thumbs up emoji

Copy link
Contributor

@sydney-runklesydney-runkle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Hi@MarkusSintonen,

Cool idea, thanks! I think memoization could be helpful here. Let me circle back with some colleagues to verify.

Specifically,@dmontagu, wdyt about this? I recall you've done a lot of work on these setattr branches.

@sydney-runklesydney-runkle added relnotes-performanceUsed for performance improvements. and removed relnotes-fixUsed for bugfixes. labelsNov 18, 2024
Copy link
Contributor

@dmontagudmontagu left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

This is great!

Smart idea to memoize most of the checks that are really only dependent on the class and attribute name. I made a couple notes and generally defer to Sydney on specific style nits, but the overall approach seems sensible to me and a good improvement.

Consider the PR approved by me, at least conceptually; I'm just not explicitly approving due to the nit comments maybe meriting some minor changes before merging.

sydney-runkle and MarkusSintonen reacted with thumbs up emoji
@sydney-runkle
Copy link
Contributor

sydney-runkle commentedNov 18, 2024
edited
Loading

Consider the PR approved by me, at least conceptually

Fantastic, thanks for the prompt review.

I've give this a nit-picky review this evening, then we can move forward!

This makes me think more about what else we could potentially memoize in the schema gen department...

One other thing I want to make sure of - this doesn't leave us with any pickling issues? I don't think so, given passing tests, but we should check.

Copy link
Member

@ViicosViicos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Thanks, I think this is a smart idea. We might have to worry about the size of the cache for large models (with a lot of fields). If we encounter such issues, we could use proper functions defined once instead of creating lambdas everytime

@MarkusSintonen
Copy link
ContributorAuthor

Thanks, I think this is a smart idea. We might have to worry about the size of the cache for large models (with a lot of fields). If we encounter such issues, we could use proper functions defined once instead of creating lambdas everytime

I wouldnt worry about size of it as anyways all the fields are listed in various ClassVars. However if we want to remove the tiny overhead of field name strs we could push the handler fn into eg FieldInfo/ModelPrivateAttr.

functions defined once

I purposely didnt want to touch the model generation side to not make it anymore heavier than it already is. Because of the mentioned large models it could just do work for no good reason in case fields are not even used like this.

@Viicos
Copy link
Member

Viicos commentedNov 19, 2024
edited
Loading

Not sure exactly what you mean byfield name strs /didnt want to touch the model generation side, but what I wanted to say is we could do something like this:

HANDLERS= {'descriptor':lambdam,val:attribute.__set__(m,val),'cached_property':lambdam,val:m.__dict__.__setitem__(name,val),    ...}def_setattr_handler(name:str,value:Any):    ...ifhasattr(attribute,'__set__'):returnHANDLERS['descriptor']    ...elifisinstance(attr,cached_property):returnHANDLERS['cached_property']

So that we don't create a new function every time.

sydney-runkle reacted with thumbs up emoji

@ViicosViicos changed the titleFix slowBaseModel.__setattr__Improve__setattr__ performance of Pydantic models by caching setter functionsNov 19, 2024
@MarkusSintonen
Copy link
ContributorAuthor

but what I wanted to say is we could do something like this

Ah yes I see! That would make sense yes

@MarkusSintonen
Copy link
ContributorAuthor

@Viicos the suggestion now done here7dfbe50

sydney-runkle reacted with heart emoji

Copy link
Member

@ViicosViicos left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Thanks for adding the simple dict. This looks good to me.

Overall this approach can look a bit weird as calling__setattr__ (i.e. at theinstance level) mutates a class variable that will be valid for every instance. But functionally it makes sense.

Copy link
Contributor

@sydney-runklesydney-runkle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Looking good! A few more questions / comments.

Copy link
Contributor

@sydney-runklesydney-runkle left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Great work, thank you!

@sydney-runklesydney-runkle merged commitaddf1f9 intopydantic:mainNov 19, 2024
53 checks passed
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment
Reviewers

@dmontagudmontagudmontagu left review comments

@sydney-runklesydney-runklesydney-runkle approved these changes

@ViicosViicosViicos approved these changes

Assignees

@sydney-runklesydney-runkle

Labels
ready for reviewrelnotes-performanceUsed for performance improvements.
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Slow setting an value of a basic Model
4 participants
@MarkusSintonen@sydney-runkle@Viicos@dmontagu

[8]ページ先頭

©2009-2025 Movatter.jp