What is this?

Install

$ pip install deep-daze

Windows Install

  pip install deep-daze

Examples

$ imagine"a house in the forest"

  imagine"a house in the forest"

$ imagine"shattered plates on the ground" --deeper

Advanced

$ imagine"stranger in strange lands" --num-layers 32

Usage

CLI

NAME    imagineSYNOPSIS    imagine TEXT<flags>POSITIONAL ARGUMENTS    TEXT        (required) A phrase less than 77 tokens which you would like to visualize.FLAGS    --img=IMAGE_PATH        Default: None        Path to png/jpg image or PIL image to optimize on    --encoding=ENCODING        Default: None        User-created custom CLIP encoding. If used, replaces any text or image that was used.    --create_story=CREATE_STORY        Default: False        Creates a story by optimizing each epoch on a new sliding-window of the input words. If this is enabled, much longer texts than 77 tokens can be used. Requires save_progress to visualize the transitions of the story.    --story_start_words=STORY_START_WORDS        Default: 5        Only usedif create_story is True. How many words to optimize onfor the first epoch.    --story_words_per_epoch=STORY_WORDS_PER_EPOCH        Default: 5        Only usedif create_story is True. How many words to add to the optimization goal per epoch after the first one.    --story_separator:        Default: None        Only usedif create_story is True. Defines a separator like'.' that splits the text into groupsforeach epoch. Separator needs to bein the text otherwise it will be ignored    --lower_bound_cutout=LOWER_BOUND_CUTOUT        Default: 0.1        Lower bound of the sampling of the size of the random cut-out of the SIREN image per batch. Should be smaller than 0.8.    --upper_bound_cutout=UPPER_BOUND_CUTOUT        Default: 1.0        Upper bound of the sampling of the size of the random cut-out of the SIREN image per batch. Should probably stay at 1.0.    --saturate_bound=SATURATE_BOUND        Default: False        If True, the LOWER_BOUND_CUTOUT is linearly increased to 0.75 during training.    --learning_rate=LEARNING_RATE        Default: 1e-05        The learning rate of the neural net.    --num_layers=NUM_LAYERS        Default: 16        The number of hidden layers to usein the Siren neural net.    --batch_size=BATCH_SIZE        Default: 4        The number of generated images to pass into Siren before calculating loss. Decreasing this can lower memory and accuracy.    --gradient_accumulate_every=GRADIENT_ACCUMULATE_EVERY        Default: 4        Calculate a weighted loss of n samplesfor each iteration. Increasing this canhelp increase accuracy with lower batch sizes.    --epochs=EPOCHS        Default: 20        The number of epochs to run.    --iterations=ITERATIONS        Default: 1050        The number oftimes to calculate and backpropagate lossin a given epoch.    --save_every=SAVE_EVERY        Default: 100        Generate an image everytime iterations is a multiple of this number.    --image_width=IMAGE_WIDTH        Default: 512        The desired resolution of the image.    --deeper=DEEPER        Default: False        Uses a Siren neural net with 32 hidden layers.    --overwrite=OVERWRITE        Default: False        Whether or not to overwrite existing generated images of the same name.    --save_progress=SAVE_PROGRESS        Default: False        Whether or not to save images generated before training Siren is complete.    --seed=SEED        Type: Optional[]        Default: None        A seed to be usedfor deterministic runs.    --open_folder=OPEN_FOLDER        Default: True        Whether or not to open a folder showing your generated images.    --save_date_time=SAVE_DATE_TIME        Default: False        Save files with a timestamp prepended e.g.`%y%m%d-%H%M%S-my_phrase_here`    --start_image_path=START_IMAGE_PATH        Default: None        The generator is trained first on a starting image before steered towards the textual input    --start_image_train_iters=START_IMAGE_TRAIN_ITERS        Default: 50        The number of stepsfor the initial training on the starting image    --theta_initial=THETA_INITIAL        Default: 30.0        Hyperparameter describing the frequency of the color space. Only applies to the first layer of the network.    --theta_hidden=THETA_INITIAL        Default: 30.0        Hyperparameter describing the frequency of the color space. Only applies to the hidden layers of the network.    --save_gif=SAVE_GIF        Default: False        Whether or not to save a GIF animation of the generation procedure. Only worksif save_progress isset to True.

Priming

$ imagine'a clear night sky filled with stars' --start_image_path ./cloudy-night-sky.jpg

Optimize for the interpretation of an image

$ imagine --img samples/Autumn_1875_Frederic_Edwin_Church.jpg

Optimize for text and image combined

$ imagine"A psychedelic experience." --img samples/hot-dog.jpg

New: Create a story

Python

Invokedeep_daze.Imagine in Python

fromdeep_dazeimportImagineimagine=Imagine(text='cosmic love and attention',num_layers=24,)imagine()

Save progress every fourth iteration

imagine=Imagine(text=text,save_every=4,save_progress=True)

Prepend current timestamp on each image.

imagine=Imagine(text=text,save_every=4,save_progress=True,save_date_time=True,)

High GPU memory usage

imagine=Imagine(text=text,num_layers=42,batch_size=64,gradient_accumulate_every=1,)

Average GPU memory usage

imagine=Imagine(text=text,num_layers=24,batch_size=16,gradient_accumulate_every=2)

Very low GPU memory usage (less than 4 GiB)

imagine=Imagine(text=text,image_width=256,num_layers=16,batch_size=1,gradient_accumulate_every=16# Increase gradient_accumulate_every to correct for loss in low batch sizes)

VRAM and speed benchmarks:

Where is this going?

Alternatives

Citations

@misc{unpublished2021clip,title  ={CLIP: Connecting Text and Images},author ={Alec Radford, Ilya Sutskever, Jong Wook Kim, Gretchen Krueger, Sandhini Agarwal},year   ={2021}}

@misc{sitzmann2020implicit,title   ={Implicit Neural Representations with Periodic Activation Functions},author  ={Vincent Sitzmann and Julien N. P. Martel and Alexander W. Bergman and David B. Lindell and Gordon Wetzstein},year    ={2020},eprint  ={2006.09661},archivePrefix ={arXiv},primaryClass ={cs.CV}}