fixed filenames

README.md

@@ -86,6 +86,21 @@ completely). The default is 0.75, and ranges from 0.25-0.75 give interesting res
 
 ## Changes
 
+* v1.05 (22 August 2022 - after the drop)
+   * Filenames now use the following formats:
+       000010.95183149.png      -- Two files produced by the same command (e.g. -n2),
+       000010.26742632.png      -- distinguished by a different seed.
+
+       000011.455191342.01.png  -- Two files produced by the same command using
+       000011.455191342.02.png  -- a batch size>1 (e.g. -b2). They have the same seed.
+
+       000011.4160627868.grid#1-4.png  -- a grid of four images (-g); the whole grid can
+                                          be regenerated with the indicated key
+
+    * It should no longer be possible for one image to overwrite another
+    * You can use the "cd" and "pwd" commands at the dream> prompt to set and retrieve
+      the path of the output directory.
+     
 * v1.04 (22 August 2022 - after the drop)
    * Updated README to reflect installation of the released weights.
    * Suppressed very noisy and inconsequential warning when loading the frozen CLIP
@@ -110,6 +125,8 @@ completely). The default is 0.75, and ranges from 0.25-0.75 give interesting res
 
 ## Installation
 
+There are separate installation walkthroughs for [Linux/Mac](#linuxmac) and [Windows](#windows).
+
 ### Linux/Mac
 
 1. You will need to install the following prerequisites if they are not already available. Use your
@@ -149,24 +166,27 @@ After these steps, your command prompt will be prefixed by "(ldm)" as shown abov
 (ldm) ~/stable-diffusion$ python3 scripts/preload_models.py
 ```
 
+Note that this step is necessary because I modified the original
+just-in-time model loading scheme to allow the script to work on GPU
+machines that are not internet connected. See [Workaround for machines with limited internet connectivity](#workaround-for-machines-with-limited-internet-connectivity)
+
 7. Now you need to install the weights for the stable diffusion model.
 
 For running with the released weights, you will first need to set up an acount with Hugging Face (https://huggingface.co).
-Use your credentials to log in, and then point browser at https://huggingface.co/CompVis/stable-diffusion-v-1-4-original.
+Use your credentials to log in, and then point your browser at https://huggingface.co/CompVis/stable-diffusion-v-1-4-original.
 You may be asked to sign a license agreement at this point.
 
 Click on "Files and versions" near the top of the page, and then click on the file named "sd-v1-4.ckpt". You'll be taken
-to a page that prompts you to click the "download" link. Now save the file somewhere safe on your local machine.
+to a page that prompts you to click the "download" link. Save the file somewhere safe on your local machine.
 
-Now run the following commands from within the stable-diffusion directory to point it to the weights file.
+Now run the following commands from within the stable-diffusion directory. This will create a symbolic
+link from the stable-diffusion model.ckpt file, to the true location of the sd-v1-4.ckpt file.
    
 ```
 (ldm) ~/stable-diffusion$ mkdir -p models/ldm/stable-diffusion-v1
 (ldm) ~/stable-diffusion$ ln -sf /path/to/sd-v1-4.ckpt models/ldm/stable-diffusion-v1/model.ckpt
 ```
 
-The weight file is >4 GB in size, so downloading may take a while.
-
 8. Start generating images!
 ```
 # for the pre-release weights use the -l or --liaon400m switch
@@ -181,7 +201,7 @@ The weight file is >4 GB in size, so downloading may take a while.
 9. Subsequently, to relaunch the script, be sure to run "conda activate ldm" (step 5, second command), enter the "stable-diffusion" 
 directory, and then launch the dream script (step 8). If you forget to activate the ldm environment, the script will fail with multiple ModuleNotFound errors.
 
-### Updating to newer versions of the script
+#### Updating to newer versions of the script
 
 This distribution is changing rapidly. If you used the "git clone" method (step 5) to download the stable-diffusion directory, then to update to the latest and greatest version, launch the Anaconda window, enter "stable-diffusion", and type:
 ```
@@ -232,7 +252,7 @@ downloaded just-in-time)
 
 For running with the released weights, you will first need to set up
 an acount with Hugging Face (https://huggingface.co).  Use your
-credentials to log in, and then point browser at
+credentials to log in, and then point your browser at
 https://huggingface.co/CompVis/stable-diffusion-v-1-4-original.  You
 may be asked to sign a license agreement at this point.
 
@@ -243,15 +263,16 @@ safe on your local machine.  The weight file is >4 GB in size, so
 downloading may take a while.
 
 Now run the following commands from **within the stable-diffusion
-directory** to point it to the weights file:
+directory** to copy the weights file to the right place:
    
 ```
 mkdir -p models/ldm/stable-diffusion-v1
 copy C:\path\to\sd-v1-4.ckpt models\ldm\stable-diffusion-v1\model.ckpt
 ```
-
-Instead of copying the file, you may instead create a shortcut within the
-models\ldm\stable-diffusion-v1\ directory that points to it.
+Please replace "C:\path\to\sd-v1.4.ckpt" with the correct path to wherever
+you stashed this file. If you prefer not to copy or move the .ckpt file, 
+you may instead create a shortcut to it from within
+"models\ldm\stable-diffusion-v1\".
 
 10. Start generating images!
 ```
@@ -263,7 +284,7 @@ python scripts\dream.py
 ```
 11. Subsequently, to relaunch the script, first activate the Anaconda command window (step 4), enter the stable-diffusion directory (step 6, "cd \path\to\stable-diffusion"), run "conda activate ldm" (step 7b), and then launch the dream script (step 10).
 
-### Updating to newer versions of the script
+#### Updating to newer versions of the script
 
 This distribution is changing rapidly. If you used the "git clone" method (step 5) to download the stable-diffusion directory, then to update to the latest and greatest version, launch the Anaconda window, enter "stable-diffusion", and type:
 ```

ldm/simplet2i.py

@@ -60,6 +60,7 @@ from torch import autocast
 from contextlib import contextmanager, nullcontext
 import time
 import math
+import re
 
 from ldm.util import instantiate_from_config
 from ldm.models.diffusion.ddim     import DDIMSampler
@@ -171,7 +172,6 @@ The vast majority of these arguments default to reasonable values.
 
         # make directories and establish names for the output files
         os.makedirs(outdir, exist_ok=True)
-        base_count = len(os.listdir(outdir))-1
 
         start_code = None
         if self.fixed_code:
@@ -185,7 +185,7 @@ The vast majority of these arguments default to reasonable values.
         sampler         = self.sampler
         images = list()
         seeds  = list()
-
+        filename = None
         tic    = time.time()
         
         with torch.no_grad():
@@ -218,10 +218,11 @@ The vast majority of these arguments default to reasonable values.
                             if not grid:
                                 for x_sample in x_samples_ddim:
                                     x_sample = 255. * rearrange(x_sample.cpu().numpy(), 'c h w -> h w c')
-                                    filename = os.path.join(outdir, f"{base_count:05}.png")
+                                    filename = self._unique_filename(outdir,previousname=filename,
+                                                                     seed=seed,isbatch=(batch_size>1))
+                                    assert not os.path.exists(filename)
                                     Image.fromarray(x_sample.astype(np.uint8)).save(filename)
                                     images.append([filename,seed])
-                                    base_count += 1
                             else:
                                 all_samples.append(x_samples_ddim)
                                 seeds.append(seed)
@@ -283,7 +284,6 @@ The vast majority of these arguments default to reasonable values.
 
         # make directories and establish names for the output files
         os.makedirs(outdir, exist_ok=True)
-        base_count = len(os.listdir(outdir))-1
 
         assert os.path.isfile(init_img)
         init_image = self._load_img(init_img).to(self.device)
@@ -304,7 +304,8 @@ The vast majority of these arguments default to reasonable values.
 
         images = list()
         seeds  = list()
-
+        filename = None
+        
         tic    = time.time()
         
         with torch.no_grad():
@@ -333,10 +334,10 @@ The vast majority of these arguments default to reasonable values.
                             if not grid:
                                 for x_sample in x_samples:
                                     x_sample = 255. * rearrange(x_sample.cpu().numpy(), 'c h w -> h w c')
-                                    filename = os.path.join(outdir, f"{base_count:05}.png")
+                                    filename = self._unique_filename(outdir,filename,seed=seed,isbatch=(batch_size>1))
+                                    assert not os.path.exists(filename)
                                     Image.fromarray(x_sample.astype(np.uint8)).save(filename)
                                     images.append([filename,seed])
-                                    base_count += 1
                             else:
                                 all_samples.append(x_samples)
                                 seeds.append(seed)
@@ -357,7 +358,6 @@ The vast majority of these arguments default to reasonable values.
 
     def _make_grid(self,samples,seeds,batch_size,iterations,outdir):
         images = list()
-        base_count = len(os.listdir(outdir))-1
         n_rows = batch_size if batch_size>1 else int(math.sqrt(batch_size * iterations))
         # save as grid
         grid = torch.stack(samples, 0)
@@ -366,7 +366,7 @@ The vast majority of these arguments default to reasonable values.
 
         # to image
         grid = 255. * rearrange(grid, 'c h w -> h w c').cpu().numpy()
-        filename = os.path.join(outdir, f"{base_count:05}.png")
+        filename = self._unique_filename(outdir,seed=seeds[0],grid_count=batch_size*iterations)
         Image.fromarray(grid.astype(np.uint8)).save(filename)
         for s in seeds:
             images.append([filename,s])
@@ -430,3 +430,40 @@ The vast majority of these arguments default to reasonable values.
         image = image[None].transpose(0, 3, 1, 2)
         image = torch.from_numpy(image)
         return 2.*image - 1.
+
+    def _unique_filename(self,outdir,previousname=None,seed=0,isbatch=False,grid_count=None):
+        revision = 1
+
+        if previousname is None:
+            # count up until we find an unfilled slot
+            dir_list  = [a.split('.',1)[0] for a in os.listdir(outdir)]
+            uniques   = dict.fromkeys(dir_list,True)
+            basecount = 1
+            while f'{basecount:06}' in uniques:
+                basecount += 1
+            if grid_count is not None:
+                grid_label = f'grid#1-{grid_count}'
+                filename = f'{basecount:06}.{seed}.{grid_label}.png'
+            elif isbatch:
+                filename = f'{basecount:06}.{seed}.01.png'
+            else:
+                filename = f'{basecount:06}.{seed}.png'
+            
+            return os.path.join(outdir,filename)
+
+        else:
+            previousname = os.path.basename(previousname)
+            x = re.match('^(\d+)\..*\.png',previousname)
+            if not x:
+                return self._unique_filename(outdir,previousname,seed)
+
+            basecount = int(x.groups()[0])
+            series = 0 
+            finished = False
+            while not finished:
+                series += 1
+                filename = f'{basecount:06}.{seed}.png'
+                if isbatch or os.path.exists(os.path.join(outdir,filename)):
+                    filename = f'{basecount:06}.{seed}.{series:02}.png'
+                finished = not os.path.exists(os.path.join(outdir,filename))
+            return os.path.join(outdir,filename)

scripts/dream.py

@@ -70,9 +70,9 @@ def main():
     # preload the model
     if not debugging:
         t2i.load_model()
-    print("\n* Initialization done! Awaiting your command (-h for help, q to quit)...")
+    print("\n* Initialization done! Awaiting your command (-h for help, 'q' to quit, 'cd' to change output dir, 'pwd' to print output dir)...")
 
-    log_path   = os.path.join(opt.outdir,"dream_log.txt")
+    log_path   = os.path.join(opt.outdir,'..','dream_log.txt')
     with open(log_path,'a') as log:
         cmd_parser = create_cmd_parser()
         main_loop(t2i,cmd_parser,log)
@@ -90,13 +90,31 @@ def main_loop(t2i,parser,log):
             done = True
             break
 
-        elements = shlex.split(command)
+        try:
+            elements = shlex.split(command)
+        except ValueError as e:
+            print(str(e))
+            continue
+        
         if len(elements)==0:
             continue
         
-        if elements[0]=='q':  # 
+        if elements[0]=='q':
             done = True
             break
+
+        if elements[0]=='cd' and len(elements)>1:
+            if os.path.exists(elements[1]):
+                print(f"setting image output directory to {elements[1]}")
+                t2i.outdir=elements[1]
+            else:
+                print(f"directory {elements[1]} does not exist")
+            continue
+
+        if elements[0]=='pwd':
+            print(f"current output directory is {t2i.outdir}")
+            continue
+        
         if elements[0].startswith('!dream'): # in case a stored prompt still contains the !dream command
             elements.pop(0)
             
@@ -133,6 +151,10 @@ def main_loop(t2i,parser,log):
         except KeyboardInterrupt:
             print('*interrupted*')
             continue
+        except RuntimeError as e:
+            print(str(e))
+            continue
+            
 
     print("goodbye!")
 
@@ -147,7 +169,13 @@ def write_log_message(t2i,opt,results,logfile):
     img_num    = 1
     batch_size = opt.batch_size or t2i.batch_size
     seenit     = {}
-    
+
+    seeds = [a[1] for a in results]
+    if batch_size > 1:
+        seeds = f"(seeds for each batch row: {seeds})"
+    else:
+        seeds = f"(seeds for individual images: {seeds})"
+
     for r in results:
         seed = r[1]
         log_message = (f'{r[0]}: {prompt_str} -S{seed}')
@@ -166,7 +194,10 @@ def write_log_message(t2i,opt,results,logfile):
         if r[0] not in seenit:
             seenit[r[0]] = True
             try:
-                _write_prompt_to_png(r[0],f'{prompt_str} -S{seed}')
+                if opt.grid:
+                    _write_prompt_to_png(r[0],f'{prompt_str} -g -S{seed} {seeds}')
+                else:
+                    _write_prompt_to_png(r[0],f'{prompt_str} -S{seed}')
             except FileNotFoundError:
                 print(f"Could not open file '{r[0]}' for reading")
 
@@ -244,7 +275,8 @@ def create_cmd_parser():
 
 if readline_available:
     def setup_readline():
-        readline.set_completer(Completer(['--steps','-s','--seed','-S','--iterations','-n','--batch_size','-b',
+        readline.set_completer(Completer(['cd','pwd',
+                                          '--steps','-s','--seed','-S','--iterations','-n','--batch_size','-b',
                                           '--width','-W','--height','-H','--cfg_scale','-C','--grid','-g',
                                           '--individual','-i','--init_img','-I','--strength','-f']).complete)
         readline.set_completer_delims(" ")
@@ -266,8 +298,13 @@ if readline_available:
             return
 
         def complete(self,text,state):
-            if text.startswith('-I') or text.startswith('--init_img'):
-                return self._image_completions(text,state)
+            buffer = readline.get_line_buffer()
+            
+            if text.startswith(('-I','--init_img')):
+                return self._path_completions(text,state,('.png'))
+
+            if buffer.strip().endswith('cd') or text.startswith(('.','/')):
+                return self._path_completions(text,state,())
 
             response = None
             if state == 0:
@@ -287,12 +324,14 @@ if readline_available:
                 response = None
             return response
 
-        def _image_completions(self,text,state):
+        def _path_completions(self,text,state,extensions):
             # get the path so far
             if text.startswith('-I'):
                 path = text.replace('-I','',1).lstrip()
             elif text.startswith('--init_img='):
                 path = text.replace('--init_img=','',1).lstrip()
+            else:
+                path = text
 
             matches  = list()
 
@@ -309,7 +348,7 @@ if readline_available:
                     if full_path.startswith(path):
                         if os.path.isdir(full_path):
                             matches.append(os.path.join(os.path.dirname(text),n)+'/')
-                        elif n.endswith('.png'):
+                        elif n.endswith(extensions):
                             matches.append(os.path.join(os.path.dirname(text),n))
 
             try:
@@ -317,7 +356,6 @@ if readline_available:
             except IndexError:
                 response = None
             return response
-        
 
 if __name__ == "__main__":
     main()