Movatterモバイル変換

[0]ホーム
URL:

Skip to content

Navigation Menu

Sign in
Appearance settings

Search code, repositories, users, issues, pull requests...

Provide feedback

We read every piece of feedback, and take your input very seriously.

Saved searches

Use saved searches to filter your results more quickly

 Sign up
Appearance settings

image embedding data cache#16556

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to ourterms of service andprivacy statement. We’ll occasionally send you account related emails.

Already on GitHub?Sign in to your account

Merged
catboxanon merged 1 commit intodevfromimage-embedding-data-cache
Oct 29, 2024
Merged

Conversation

@w-e-w
Copy link
Collaborator

@w-e-ww-e-w commentedOct 17, 2024•
edited
Loading

Description

tldr

  • reduce future unnecessary disk read for determining if an image is an image embedding or not
  • if addition option is enabled potentially reduce embedding load time by cacheing the embedding data

in embedding dir webui will scan and loadimage embedding

images that has textual inversion data embedded inside the image file

allpngwebpjxlavif in the dir will be considered as potentialimage embedding unless the image file name has a second suffix ofpreview
webui will attempt to read and extract the textual inversion data from the image file
this is a rather slow operation as it requires reading of the entire image from disk

if a user does not know about or forgot about image embedding (I don't think this is commonly used)
they may decide to sotre extra image files that alongside the embedding file, like previews but don't have the.preview 2nd suffix
if there are lots of images then they may found that loading embedded is taking exceedingly long

this happened to me, when I decided to dump lots of PNGs files that doesn't have.preview 2nd suffix into the embedded dir, took me some time to realize what happened

to solve this I use the diskcache from cache models to caching the results of loading an image as embedding
this was able to drastically reduces the time that it takes to load the textual inversion

I implemented so that if an image is not an image embedding then the result is always cache, so next time it won't need to read from disk

if actual image embeddings won't be cached by default unless optiontextual_inversion_image_embedding_data_cache : Cache the data of image embeddings is enabled
the reason why is an option is because even though I believe this reduce load time for the "actual image embedding" it will also takes up extra disk space

Notes:

I don't think I've done any changes that will cause things to break
but I don't have any actual image embedding so this PR tested not teste with real data
if someone that regularly use image embedding please verify this PR

Checklist:

@w-e-w
Copy link
CollaboratorAuthor

this PR can be summed up as

  1. take a block of existing code, move it into a function
  2. add persistent cache mechanism using existing modules

@catboxanoncatboxanon merged commitdeb3803 intodevOct 29, 2024
@catboxanoncatboxanon deleted the image-embedding-data-cache branchOctober 29, 2024 14:28
bluelovers added a commit to bluelovers/stable-diffusion-webui that referenced this pull requestNov 26, 2024
* dev-a1111:  fix passing of literal backslash (AUTOMATIC1111#16671)  fix prompt-bracket-checker miscounting of literal tokens (AUTOMATIC1111#16669)  Bump safetensors to v0.4.5  Honor lossless webp compression option in API  use shared.hf_endpoint (AUTOMATIC1111#16611)  Warn if WebUI is installed under a dot directory (AUTOMATIC1111#16584)  sd_xl_v.yaml: use_checkpoint = False  XYZ option to disable grid (AUTOMATIC1111#16416)  Fix Default system None filter logic (AUTOMATIC1111#16309)  Fix weighting config for SDXL v-pred  InputAccordion duplicate elem_id handling (AUTOMATIC1111#16381)  addEventListener {passive: false} (AUTOMATIC1111#16575)  pyenv-win compatibility - another approach (AUTOMATIC1111#16287)  Disable Hires checkpoint if same as First pass checkpoint (AUTOMATIC1111#16269)  Fix postprocessing_enable_in_main_ui ScriptPostprocessing elem_id (AUTOMATIC1111#16373)  extra_only / main_ui_only ScriptPostprocessing (AUTOMATIC1111#16374)  Allow newline in Extra Network activation text (AUTOMATIC1111#16428)  image embedding data cache (AUTOMATIC1111#16556)  Fix typo
THClements added a commit to THClements/stable-diffusion-webui that referenced this pull requestFeb 1, 2025
commitdc34c00Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Fri Dec 27 22:10:13 2024 +0900    fix shadows name 'k' from outer scope    while syntactically correct this triggers a false Unresolved reference 'k' error in PyCharmscommitfc54833Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Fri Dec 27 08:58:44 2024 +0900    Authenticated Sysinfocommit078d04eAuthor: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Fri Dec 27 10:40:15 2024 +0900    ruff <path> is deprecated. Use ruff check <path> (AUTOMATIC1111#16753)commit1a773bfMerge:f25c3fcf113474Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Thu Dec 26 06:33:04 2024 +0900    Merge pull requestAUTOMATIC1111#16751 from Neokmi/master    Fix  Codeformer and gfpgan extension , Inconsistent overlay layer types when visibility value is less than 1commitf113474Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Thu Dec 26 06:26:47 2024 +0900    lintcommit6577e06Author: klx <128426349+Neokmi@users.noreply.github.com>Date:   Thu Dec 26 02:16:05 2024 +0800    Update postprocessing_gfpgan.py    Fix  gfpgan extension , Inconsistent overlay layer types when visibility value is less than 1commit7953c57Author: klx <128426349+Neokmi@users.noreply.github.com>Date:   Thu Dec 26 02:14:49 2024 +0800    Update postprocessing_codeformer.py    Fix  Codeformer extension , Inconsistent overlay layer types when visibility value is less than 1commitf25c3fcAuthor: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Wed Dec 25 05:43:55 2024 +0900    fix sd_vae_explanation (AUTOMATIC1111#16748)commitfc0952aMerge:04903afb414c62Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Tue Dec 24 22:58:43 2024 +0900    Merge pull requestAUTOMATIC1111#16745 from Sanchows/removed-unused-import-modules-errors    removed unnecessary import 'modules.errors'commitb414c62Author: Alexander Sachenko <35726682+Sanchows@users.noreply.github.com>Date:   Tue Dec 24 15:45:10 2024 +0300    removed unnecessary import modules.errorscommit04903afMerge:e8c3b1f9568622Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Wed Dec 18 03:21:48 2024 +0900    Merge pull requestAUTOMATIC1111#16604 from Haoming02/ext-updt-parallel    Check for Extension Updates in Parallelcommite8c3b1fMerge:01207688bf30e3Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Wed Dec 18 02:37:30 2024 +0900    Merge pull requestAUTOMATIC1111#16718 from Haoming02/bracket-checker-order    [Bracket Checker] Also check for the order of bracketscommit8bf30e3Author: Haoming <hmstudy02@gmail.com>Date:   Wed Dec 18 01:02:40 2024 +0800    revert IIFEcommitfbc51faAuthor: Haoming <hmstudy02@gmail.com>Date:   Mon Dec 16 09:47:38 2024 +0800    skip escapedcommit7025a2cAuthor: Haoming <hmstudy02@gmail.com>Date:   Thu Dec 12 16:08:15 2024 +0800    check-for-ordercommit0120768Merge:023454bb425b97Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Thu Nov 28 17:39:12 2024 +0900    Merge pull requestAUTOMATIC1111#16687 from Haoming02/dropdown4format    Use gr.Dropdown for Image Formatscommitb425b97Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Thu Nov 28 16:56:10 2024 +0900    improve img fromat descriptioncommit539ea39Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Thu Nov 28 14:02:33 2024 +0900    use DropdownEditable    use DropdownEditable so user can input other formats if they require it    make the default png the first on the listcommit65bd61eAuthor: Haoming <hmstudy02@gmail.com>Date:   Wed Nov 27 10:42:50 2024 +0800    format-dropdowncommit023454bAuthor: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Thu Nov 21 11:33:59 2024 +0900    fix passing of literal backslash (AUTOMATIC1111#16671)commitcd869bbAuthor: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Thu Nov 21 02:29:41 2024 +0900    fix prompt-bracket-checker miscounting of literal tokens (AUTOMATIC1111#16669)commit957888aMerge:7799859d2c9efbAuthor: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Tue Nov 19 14:41:37 2024 +0900    Merge pull requestAUTOMATIC1111#16667 from AUTOMATIC1111/fix/safetensors-bump    Bump safetensors to v0.4.5commitd2c9efbAuthor: catboxanon <122327233+catboxanon@users.noreply.github.com>Date:   Mon Nov 18 20:48:36 2024 -0500    Bump safetensors to v0.4.5ResolvesAUTOMATIC1111#16650commit7799859Merge:1b16c62ca3bedbAuthor: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Sat Nov 2 01:28:12 2024 +0900    Merge pull requestAUTOMATIC1111#16620 from AUTOMATIC1111/fix/api-webp-lossless    Honor lossless WebP compression option in APIcommitca3bedbAuthor: catboxanon <122327233+catboxanon@users.noreply.github.com>Date:   Fri Nov 1 11:32:52 2024 -0400    Honor lossless webp compression option in APIcommit1b16c62Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Thu Oct 31 02:01:32 2024 +0900    use shared.hf_endpoint (AUTOMATIC1111#16611)commit91de919Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Wed Oct 30 22:34:37 2024 +0900    Warn if WebUI is installed under a dot directory (AUTOMATIC1111#16584)commitaa52408Merge:28323cfe6f36d9Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Wed Oct 30 09:51:41 2024 +0900    Merge pull requestAUTOMATIC1111#16606 from AUTOMATIC1111/fix/vweighting    Fix config for SDXL v-predcommite6f36d9Author: catboxanon <122327233+catboxanon@users.noreply.github.com>Date:   Tue Oct 29 13:27:32 2024 -0400    sd_xl_v.yaml: use_checkpoint = False    In accordance withAUTOMATIC1111#15803commit28323cfAuthor: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Wed Oct 30 02:18:38 2024 +0900    XYZ option to disable grid (AUTOMATIC1111#16416)commit533c7b7Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Wed Oct 30 02:13:16 2024 +0900    Fix Default system None filter logic (AUTOMATIC1111#16309)commitac28cadAuthor: catboxanon <122327233+catboxanon@users.noreply.github.com>Date:   Tue Oct 29 11:49:09 2024 -0400    Fix weighting config for SDXL v-pred    Fixes a small oversight I made.commit5206b93Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Wed Oct 30 00:03:21 2024 +0900    InputAccordion duplicate elem_id handling (AUTOMATIC1111#16381)commit5948143Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Tue Oct 29 23:59:04 2024 +0900    addEventListener {passive: false} (AUTOMATIC1111#16575)commitf31faf6Author: viking1304 <viking1304@gmail.com>Date:   Tue Oct 29 15:54:58 2024 +0100    pyenv-win compatibility - another approach (AUTOMATIC1111#16287)commit14c6d6cAuthor: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Tue Oct 29 23:45:45 2024 +0900    Disable Hires checkpoint if same as First pass checkpoint (AUTOMATIC1111#16269)commit4ec10bcAuthor: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Tue Oct 29 23:38:55 2024 +0900    Fix postprocessing_enable_in_main_ui ScriptPostprocessing elem_id (AUTOMATIC1111#16373)commit0bf36cfAuthor: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Tue Oct 29 23:35:46 2024 +0900    extra_only / main_ui_only ScriptPostprocessing (AUTOMATIC1111#16374)commit820fe8dAuthor: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Tue Oct 29 23:30:08 2024 +0900    Allow newline in Extra Network activation text (AUTOMATIC1111#16428)commitdeb3803Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Tue Oct 29 23:28:21 2024 +0900    image embedding data cache (AUTOMATIC1111#16556)commit9568622Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Tue Oct 29 20:16:15 2024 +0900    limit number of simultaneous updates    shared.opts.concurrent_git_fetch_limitcommitdf74c3cAuthor: Haoming <hmstudy02@gmail.com>Date:   Tue Oct 29 14:12:42 2024 +0800    threadingcommitd88a3c1Merge:38c8043ee0ad5cAuthor: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Sun Oct 27 10:28:23 2024 +0900    Merge pull requestAUTOMATIC1111#16588 from bluelovers/patch-3    chore(js): avoid lots of `Wake Lock is not supported.`commit38c8043Merge:984b952d8ad364Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Sun Oct 27 01:09:41 2024 +0900    Merge pull requestAUTOMATIC1111#16523 from changeworld/fix/typo    Fix typo: Github -> GitHubcommitee0ad5cAuthor: bluelovers <codelovers@users.sourceforge.net>Date:   Fri Oct 25 09:59:45 2024 +0800    chore(js): avoid lots of `Wake Lock is not supported.`commit984b952Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Thu Oct 24 22:05:51 2024 +0900    Fix DAT models download (AUTOMATIC1111#16302)commit5865da2Merge:bb1f391c2ce1d3Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Sun Oct 20 11:40:02 2024 +0900    Merge pull requestAUTOMATIC1111#16569 from AUTOMATIC1111/feat/ztsnr-auto    Automatically enable ztSNR based on existence of key in `state_dict`commitbb1f391Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Sun Oct 20 09:58:53 2024 +0900    clarify readme: weget ... chmod +x webui.sh (AUTOMATIC1111#16251)commit6a59766Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Sun Oct 20 09:56:12 2024 +0900    Add Skip Early CFG to XYZ (AUTOMATIC1111#16282)    Co-authored-by: Yevhenii Hurin <evgeny.gurin@gmail.com>commit65423d2Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Sun Oct 20 09:52:47 2024 +0900    MIME type text/css (AUTOMATIC1111#16406)commitc2bc187Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Sun Oct 20 09:51:59 2024 +0900    fix modalImageViewer preview/result flicker (AUTOMATIC1111#16426)commitd0b27dcMerge:bb4cbafcbaaf0aAuthor: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Sun Oct 20 09:42:21 2024 +0900    Merge pull requestAUTOMATIC1111#16300 from hello2564/fix_NGMS_pr_typo    fix NGMS pr typocommitc2ce1d3Author: catboxanon <122327233+catboxanon@users.noreply.github.com>Date:   Sat Oct 19 19:58:13 2024 -0400    Automatically enable ztSNR based on existence of key in state_dictcommitbb4cbafMerge:c462e5a9677b09Author: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Sun Oct 20 08:27:12 2024 +0900    Merge pull requestAUTOMATIC1111#16341 from gutris1/devv    add break-word for geninfo in pnginfocommitc462e5aMerge:8b19b75c9a06d1Author: catboxanon <122327233+catboxanon@users.noreply.github.com>Date:   Sat Oct 19 10:51:16 2024 -0400    Merge pull requestAUTOMATIC1111#16460 from AUTOMATIC1111/sd-1.5-urlcommit8b19b75Merge:907bfb51ae073cAuthor: AUTOMATIC1111 <16777216c@gmail.com>Date:   Sat Oct 19 17:40:56 2024 +0300    Merge pull requestAUTOMATIC1111#16567 from AUTOMATIC1111/feat/sdxl-vpred    Support and automatically detect SDXL V-prediction modelscommit907bfb5Author: AUTOMATIC1111 <16777216c@gmail.com>Date:   Sat Oct 19 17:33:58 2024 +0300    add w-e-w and catboxanon to codeowners filecommit1ae073cAuthor: catboxanon <122327233+catboxanon@users.noreply.github.com>Date:   Sat Oct 19 06:53:19 2024 -0400    Support SDXL v-pred modelscommitc9a06d1Author: missionfloyd <missionfloyd@users.noreply.github.com>Date:   Tue Oct 8 16:50:39 2024 -0600    Use stable-diffusion-v1-5 repo insteadcommitd8ad364Author: Takashi Takebayashi <changesworlds@gmail.com>Date:   Thu Oct 3 14:33:37 2024 +0900    Fix typo    Github -> GitHubcommitf57ec2bAuthor: missionfloyd <missionfloyd@users.noreply.github.com>Date:   Tue Sep 3 19:58:29 2024 -0600    Update stable diffusion 1.5 URLcommit9677b09Author: gutris1 <132797949+gutris1@users.noreply.github.com>Date:   Wed Aug 7 17:37:23 2024 +0700    add break-word for geninfo in pnginfocommitcbaaf0aAuthor: hello2564 <ksncdos@163.com>Date:   Wed Jul 31 14:55:30 2024 +0800    fix NGMS pr typocommit4823909Merge:850e14982a973cAuthor: AUTOMATIC1111 <16777216c@gmail.com>Date:   Sat Jul 27 15:50:26 2024 +0300    Merge branch 'master' into devcommit850e149Merge:834297b8e0881dAuthor: AUTOMATIC1111 <16777216c@gmail.com>Date:   Sat Jul 27 15:47:49 2024 +0300    Merge pull requestAUTOMATIC1111#16275 from AUTOMATIC1111/fix-image-upscale-on-cpu    fix image upscale on cpucommit8e0881dAuthor: w-e-w <40751091+w-e-w@users.noreply.github.com>Date:   Sat Jul 27 21:10:20 2024 +0900    fix image upscale on cpu    for some reason upscale using cpu will fail with    RuntimeError: Inplace update to inference tensor outside InferenceMode    switch from no_grad to inference_mode seems to have fixed itcommit834297bMerge:9f5a98dc19d044Author: AUTOMATIC1111 <16777216c@gmail.com>Date:   Sat Jul 27 07:09:08 2024 +0300    Merge branch 'master' into dev
Sign up for freeto join this conversation on GitHub. Already have an account?Sign in to comment

Reviewers

@catboxanoncatboxanoncatboxanon approved these changes

@AUTOMATIC1111AUTOMATIC1111Awaiting requested review from AUTOMATIC1111AUTOMATIC1111 is a code owner

Assignees

No one assigned

Labels

None yet

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

2 participants

@w-e-w@catboxanon
[8]ページ先頭
©2009-2025 Movatter.jp