Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

chore: add files cache for reading template tar archives from db#17141

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
aslilac merged 7 commits intomainfromlilac/files-cache
Apr 2, 2025

Conversation

aslilac
Copy link
Member

@aslilacaslilac commentedMar 27, 2025•
edited
Loading

Closes#17052

Requirements

  • Multiple template versions can share a single "file"

    • This is solved by using a map fromfile id tofs.FS interface pointers.

  • Files should be "ref counted" so that they're released when no one is using them.

    • This is solved by using a ref count, obviously :^)

  • You should be able to fetch multiple different files in parallel, but you should not fetch the same file multiple times in parallel.

    • This is solved by using a globalsync.Mutex and alazy.ValueWithError per file ID.

Implementation

  • Uses afero for turning the tar data from the database into a readonlyio/fs.FS

  • Adds alazy.ValueWithError as a counterpart tolazy.Value, but with the ability to fail

Emyrk reacted with heart emoji
@aslilacaslilac requested a review fromEmyrkMarch 28, 2025 17:48
@aslilacaslilac marked this pull request as ready for reviewMarch 28, 2025 17:50
Copy link
Member

@EmyrkEmyrk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

Overall LG!
Maybe we should add some metric on theentry for when it was created. At some point some metrics like how long an entry exists would be useful.

We might even enforce some TTL or something 🤷

@aslilac
Copy link
MemberAuthor

aslilac commentedApr 1, 2025•
edited
Loading

We might even enforce some TTL or something

what would it look like if the entry got evicted while someone was holding it? the gc wouldn't actually free it, but then if someone requests it again we'd end up with a separate copy of everything because the cache would no longer see it. I kinda feel like an expiration time would only mask bugs.

@aslilacaslilac requested a review fromEmyrkApril 1, 2025 22:07
@Emyrk
Copy link
Member

We might even enforce some TTL or something

what would it look like if the entry got evicted while someone was holding it? the gc wouldn't actually free it, but then if someone requests it again we'd end up with a separate copy of everything because the cache would no longer see it. I kinda feel like an expiration time would only mask bugs.

You are right. Can we add an issue to throw some prometheus metrics around this then? So if something does happen, we have somewhere to look.

Copy link
Member

@EmyrkEmyrk left a comment

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others.Learn more.

👍 LGTM, left a few nits

Some things to follow up:

  • Add an issue for adding prometheus metrics to this cache
  • Potentially revisit the API when we get to usage. If we can guaranteeRelease is called, that would be ideal 🙏

@aslilacaslilac merged commitac7ea08 intomainApr 2, 2025
29 checks passed
@aslilacaslilac deleted the lilac/files-cache branchApril 2, 2025 22:42
@github-actionsgithub-actionsbot locked and limited conversation to collaboratorsApr 2, 2025
Sign up for freeto subscribe to this conversation on GitHub. Already have an account?Sign in.
Reviewers

@EmyrkEmyrkEmyrk approved these changes

Assignees

@aslilacaslilac

Labels
None yet
Projects
None yet
Milestone
No milestone
Development

Successfully merging this pull request may close these issues.

Add filesystem cache for dynamic parameters loading tf files
2 participants
@aslilac@Emyrk
[8]ページ先頭
©2009-2025 Movatter.jp