Change tiling strategy to make TiledStableDiffusionRefineInvocation work with more tile shapes and overlaps.

- Fix type errors
- Enable SilenceWarnings to be used as both a context manager and a decorator
- Remove duplicate implementation
- Check the initial verbosity on __enter__() rather than __init__()

docs/contributing/DOWNLOAD_QUEUE.md

@@ -128,8 +128,7 @@ The queue operates on a series of download job objects. These objects
 specify the source and destination of the download, and keep track of
 the progress of the download.
 
-Two job types are defined. `DownloadJob` and
-`MultiFileDownloadJob`. The former is a pydantic object with the
+The only job type currently implemented is `DownloadJob`, a pydantic object with the
 following fields:
 
 | **Field**      | **Type**        |  **Default**  | **Description** |
@@ -139,7 +138,7 @@ following fields:
 | `dest`           | Path            |               | Where to download to              |
 | `access_token`   | str             |               | [optional] string containing authentication token for access |
 | `on_start`       | Callable        |               | [optional] callback when the download starts |
-| `on_progress` | Callable | | [optional] callback called at intervals during download progress |
+| `on_progress`    | Callable        |               | [optional] callback called at intervals during download progress |
 | `on_complete`    | Callable        |               | [optional] callback called after successful download completion |
 | `on_error`       | Callable        |               | [optional] callback called after an error occurs  |
 | `id`             | int             | auto assigned | Job ID, an integer >= 0           |
@@ -191,33 +190,6 @@ A cancelled job will have status `DownloadJobStatus.ERROR` and an
 `error_type` field of "DownloadJobCancelledException". In addition,
 the job's `cancelled` property will be set to True.
 
-The `MultiFileDownloadJob` is used for diffusers model downloads,
-which contain multiple files and directories under a common root:
-
-| **Field**      | **Type**        |  **Default**  | **Description** |
-|----------------|-----------------|---------------|-----------------|
-| _Fields passed in at job creation time_                               |
-| `download_parts` | Set[DownloadJob]|               | Component download jobs |
-| `dest`           | Path            |               | Where to download to              |
-| `on_start`       | Callable        |               | [optional] callback when the download starts |
-| `on_progress` | Callable | | [optional] callback called at intervals during download progress |
-| `on_complete`    | Callable        |               | [optional] callback called after successful download completion |
-| `on_error`       | Callable        |               | [optional] callback called after an error occurs  |
-| `id`             | int             | auto assigned | Job ID, an integer >= 0           |
-| _Fields updated over the course of the download task_
-| `status`         | DownloadJobStatus|              | Status code                                |
-| `download_path`  | Path |              | Path to the root of the downloaded files |
-| `bytes`          | int              | 0            | Bytes downloaded so far   |
-| `total_bytes`    | int              | 0            | Total size of the file at the remote site  |
-| `error_type`     | str              |              | String version of the exception that caused an error during download |
-| `error`          | str              |              | String version of the traceback associated with an error |
-| `cancelled`      | bool             | False        | Set to true if the job was cancelled by the caller|
-
-Note that the MultiFileDownloadJob does not support the `priority`,
-`job_started`, `job_ended` or `content_type` attributes. You can get
-these from the individual download jobs in `download_parts`.
-
-
 ### Callbacks
 
 Download jobs can be associated with a series of callbacks, each with
@@ -279,40 +251,11 @@ jobs using `list_jobs()`, fetch a single job by its with
 running jobs with `cancel_all_jobs()`, and wait for all jobs to finish
 with `join()`.
 
-#### job = queue.download(source, dest, priority, access_token, on_start, on_progress, on_complete, on_cancelled, on_error)
+#### job = queue.download(source, dest, priority, access_token)
 
 Create a new download job and put it on the queue, returning the
 DownloadJob object.
 
-#### multifile_job = queue.multifile_download(parts, dest, access_token, on_start, on_progress, on_complete, on_cancelled, on_error)
-
-This is similar to download(), but instead of taking a single source,
-it accepts a `parts` argument consisting of a list of
-`RemoteModelFile` objects. Each part corresponds to a URL/Path pair,
-where the URL is the location of the remote file, and the Path is the
-destination.
-
-`RemoteModelFile` can be imported from `invokeai.backend.model_manager.metadata`, and
-consists of a url/path pair. Note that the path *must* be relative.
-
-The method returns a `MultiFileDownloadJob`.
-
-
-```
-from invokeai.backend.model_manager.metadata import RemoteModelFile
-remote_file_1 = RemoteModelFile(url='http://www.foo.bar/my/pytorch_model.safetensors'',
-                                path='my_model/textencoder/pytorch_model.safetensors'
-			 			  )
-remote_file_2 = RemoteModelFile(url='http://www.bar.baz/vae.ckpt',
-                                path='my_model/vae/diffusers_model.safetensors'
-			 			  )
-job = queue.multifile_download(parts=[remote_file_1, remote_file_2],
-                               dest='/tmp/downloads',
-                               on_progress=TqdmProgress().update)
-queue.wait_for_job(job)
-print(f"The files were downloaded to {job.download_path}")
-```
-
 #### jobs = queue.list_jobs()
 
 Return a list of all active and inactive `DownloadJob`s.

docs/contributing/MODEL_MANAGER.md

@@ -397,25 +397,26 @@ In the event you wish to create a new installer, you may use the
 following initialization pattern:
 
 ```
-from invokeai.app.services.config import get_config
+from invokeai.app.services.config import InvokeAIAppConfig
 from invokeai.app.services.model_records import ModelRecordServiceSQL
 from invokeai.app.services.model_install import ModelInstallService
 from invokeai.app.services.download import DownloadQueueService
-from invokeai.app.services.shared.sqlite.sqlite_database import SqliteDatabase
+from invokeai.app.services.shared.sqlite import SqliteDatabase
 from invokeai.backend.util.logging import InvokeAILogger
 
-config = get_config()
+config = InvokeAIAppConfig.get_config()
+config.parse_args()
 
 logger = InvokeAILogger.get_logger(config=config)
-db = SqliteDatabase(config.db_path, logger)
+db = SqliteDatabase(config, logger)
 record_store = ModelRecordServiceSQL(db)
 queue = DownloadQueueService()
 queue.start()
 
-installer = ModelInstallService(app_config=config,
+installer = ModelInstallService(app_config=config, 
                                 record_store=record_store,
-                                download_queue=queue
-                                )
+              download_queue=queue
+           )
 installer.start()
 ```
 
@@ -1366,20 +1367,12 @@ the in-memory loaded model:
 | `model`        | AnyModel               | The instantiated model (details below) |
 | `locker`       | ModelLockerBase        | A context manager that mediates the movement of the model into VRAM |
 
-### get_model_by_key(key, [submodel]) -> LoadedModel
-
-The `get_model_by_key()` method will retrieve the model using its
-unique database key. For example:
-
-loaded_model = loader.get_model_by_key('f13dd932c0c35c22dcb8d6cda4203764', SubModelType('vae'))
-
-`get_model_by_key()` may raise any of the following exceptions:
-
-* `UnknownModelException`   -- key not in database
-* `ModelNotFoundException`  -- key in database but model not found at path
-* `NotImplementedException` -- the loader doesn't know how to load this type of model
-
-### Using the Loaded Model in Inference
+Because the loader can return multiple model types, it is typed to
+return `AnyModel`, a Union `ModelMixin`, `torch.nn.Module`,
+`IAIOnnxRuntimeModel`, `IPAdapter`, `IPAdapterPlus`, and
+`EmbeddingModelRaw`. `ModelMixin` is the base class of all diffusers
+models, `EmbeddingModelRaw` is used for LoRA and TextualInversion
+models. The others are obvious.
 
 `LoadedModel` acts as a context manager. The context loads the model
 into the execution device (e.g. VRAM on CUDA systems), locks the model
@@ -1387,33 +1380,17 @@ in the execution device for the duration of the context, and returns
 the model. Use it like this:
 
 ```
-loaded_model_= loader.get_model_by_key('f13dd932c0c35c22dcb8d6cda4203764', SubModelType('vae'))
-with loaded_model as vae:
+model_info = loader.get_model_by_key('f13dd932c0c35c22dcb8d6cda4203764', SubModelType('vae'))
+with model_info as vae:
  image = vae.decode(latents)[0]
 ```
 
-The object returned by the LoadedModel context manager is an
-`AnyModel`, which is a Union of `ModelMixin`, `torch.nn.Module`,
-`IAIOnnxRuntimeModel`, `IPAdapter`, `IPAdapterPlus`, and
-`EmbeddingModelRaw`. `ModelMixin` is the base class of all diffusers
-models, `EmbeddingModelRaw` is used for LoRA and TextualInversion
-models. The others are obvious.
-
-In addition, you may call `LoadedModel.model_on_device()`, a context
-manager that returns a tuple of the model's state dict in CPU and the
-model itself in VRAM. It is used to optimize the LoRA patching and
-unpatching process:
-
-```
-loaded_model_= loader.get_model_by_key('f13dd932c0c35c22dcb8d6cda4203764', SubModelType('vae'))
-with loaded_model.model_on_device() as (state_dict, vae):
- image = vae.decode(latents)[0]
-```
-
-Since not all models have state dicts, the `state_dict` return value
-can be None.
-
+`get_model_by_key()` may raise any of the following exceptions:
 
+* `UnknownModelException`   -- key not in database
+* `ModelNotFoundException`  -- key in database but model not found at path
+* `NotImplementedException` -- the loader doesn't know how to load this type of model
+  
 ### Emitting model loading events
 
 When the `context` argument is passed to `load_model_*()`, it will
@@ -1601,59 +1578,3 @@ This method takes a model key, looks it up using the
 `ModelRecordServiceBase` object in `mm.store`, and passes the returned
 model configuration to `load_model_by_config()`.  It may raise a
 `NotImplementedException`.
-
-## Invocation Context Model Manager API
-
-Within invocations, the following methods are available from the
-`InvocationContext` object:
-
-### context.download_and_cache_model(source) -> Path
-
-This method accepts a `source` of a remote model, downloads and caches
-it locally, and then returns a Path to the local model. The source can
-be a direct download URL or a HuggingFace repo_id.
-
-In the case of HuggingFace repo_id, the following variants are
-recognized:
-
-* stabilityai/stable-diffusion-v4           -- default model
-* stabilityai/stable-diffusion-v4:fp16      -- fp16 variant
-* stabilityai/stable-diffusion-v4:fp16:vae  -- the fp16 vae subfolder
-* stabilityai/stable-diffusion-v4:onnx:vae  -- the onnx variant vae subfolder
-
-You can also point at an arbitrary individual file within a repo_id
-directory using this syntax:
-
-* stabilityai/stable-diffusion-v4::/checkpoints/sd4.safetensors
-
-### context.load_local_model(model_path, [loader]) -> LoadedModel
-
-This method loads a local model from the indicated path, returning a
-`LoadedModel`. The optional loader is a Callable that accepts a Path
-to the object, and returns a `AnyModel` object. If no loader is
-provided, then the method will use `torch.load()` for a .ckpt or .bin
-checkpoint file, `safetensors.torch.load_file()` for a safetensors
-checkpoint file, or `cls.from_pretrained()` for a directory that looks
-like a diffusers directory.
-
-### context.load_remote_model(source, [loader]) -> LoadedModel
-
-This method accepts a `source` of a remote model, downloads and caches
-it locally, loads it, and returns a `LoadedModel`. The source can be a
-direct download URL or a HuggingFace repo_id.
-
-In the case of HuggingFace repo_id, the following variants are
-recognized:
-
-* stabilityai/stable-diffusion-v4           -- default model
-* stabilityai/stable-diffusion-v4:fp16      -- fp16 variant
-* stabilityai/stable-diffusion-v4:fp16:vae  -- the fp16 vae subfolder
-* stabilityai/stable-diffusion-v4:onnx:vae  -- the onnx variant vae subfolder
-
-You can also point at an arbitrary individual file within a repo_id
-directory using this syntax:
-
-* stabilityai/stable-diffusion-v4::/checkpoints/sd4.safetensors
-
-
-

invokeai/app/api/dependencies.py

@@ -93,7 +93,7 @@ class ApiDependencies:
         conditioning = ObjectSerializerForwardCache(
             ObjectSerializerDisk[ConditioningFieldData](output_folder / "conditioning", ephemeral=True)
         )
-        download_queue_service = DownloadQueueService(app_config=configuration, event_bus=events)
+        download_queue_service = DownloadQueueService(event_bus=events)
         model_images_service = ModelImageFileStorageDisk(model_images_folder / "model_images")
         model_manager = ModelManagerService.build_model_manager(
             app_config=configuration,

invokeai/app/invocations/blend_latents.py

@@ -0,0 +1,98 @@
+from typing import Any, Union
+
+import numpy as np
+import numpy.typing as npt
+import torch
+
+from invokeai.app.invocations.baseinvocation import BaseInvocation, invocation
+from invokeai.app.invocations.fields import FieldDescriptions, Input, InputField, LatentsField
+from invokeai.app.invocations.primitives import LatentsOutput
+from invokeai.app.services.shared.invocation_context import InvocationContext
+from invokeai.backend.util.devices import TorchDevice
+
+
+@invocation(
+    "lblend",
+    title="Blend Latents",
+    tags=["latents", "blend"],
+    category="latents",
+    version="1.0.3",
+)
+class BlendLatentsInvocation(BaseInvocation):
+    """Blend two latents using a given alpha. Latents must have same size."""
+
+    latents_a: LatentsField = InputField(
+        description=FieldDescriptions.latents,
+        input=Input.Connection,
+    )
+    latents_b: LatentsField = InputField(
+        description=FieldDescriptions.latents,
+        input=Input.Connection,
+    )
+    alpha: float = InputField(default=0.5, description=FieldDescriptions.blend_alpha)
+
+    def invoke(self, context: InvocationContext) -> LatentsOutput:
+        latents_a = context.tensors.load(self.latents_a.latents_name)
+        latents_b = context.tensors.load(self.latents_b.latents_name)
+
+        if latents_a.shape != latents_b.shape:
+            raise Exception("Latents to blend must be the same size.")
+
+        device = TorchDevice.choose_torch_device()
+
+        def slerp(
+            t: Union[float, npt.NDArray[Any]],  # FIXME: maybe use np.float32 here?
+            v0: Union[torch.Tensor, npt.NDArray[Any]],
+            v1: Union[torch.Tensor, npt.NDArray[Any]],
+            DOT_THRESHOLD: float = 0.9995,
+        ) -> Union[torch.Tensor, npt.NDArray[Any]]:
+            """
+            Spherical linear interpolation
+            Args:
+                t (float/np.ndarray): Float value between 0.0 and 1.0
+                v0 (np.ndarray): Starting vector
+                v1 (np.ndarray): Final vector
+                DOT_THRESHOLD (float): Threshold for considering the two vectors as
+                                    colineal. Not recommended to alter this.
+            Returns:
+                v2 (np.ndarray): Interpolation vector between v0 and v1
+            """
+            inputs_are_torch = False
+            if not isinstance(v0, np.ndarray):
+                inputs_are_torch = True
+                v0 = v0.detach().cpu().numpy()
+            if not isinstance(v1, np.ndarray):
+                inputs_are_torch = True
+                v1 = v1.detach().cpu().numpy()
+
+            dot = np.sum(v0 * v1 / (np.linalg.norm(v0) * np.linalg.norm(v1)))
+            if np.abs(dot) > DOT_THRESHOLD:
+                v2 = (1 - t) * v0 + t * v1
+            else:
+                theta_0 = np.arccos(dot)
+                sin_theta_0 = np.sin(theta_0)
+                theta_t = theta_0 * t
+                sin_theta_t = np.sin(theta_t)
+                s0 = np.sin(theta_0 - theta_t) / sin_theta_0
+                s1 = sin_theta_t / sin_theta_0
+                v2 = s0 * v0 + s1 * v1
+
+            if inputs_are_torch:
+                v2_torch: torch.Tensor = torch.from_numpy(v2).to(device)
+                return v2_torch
+            else:
+                assert isinstance(v2, np.ndarray)
+                return v2
+
+        # blend
+        bl = slerp(self.alpha, latents_a, latents_b)
+        assert isinstance(bl, torch.Tensor)
+        blended_latents: torch.Tensor = bl  # for type checking convenience
+
+        # https://discuss.huggingface.co/t/memory-usage-by-later-pipeline-stages/23699
+        blended_latents = blended_latents.to("cpu")
+
+        TorchDevice.empty_cache()
+
+        name = context.tensors.save(tensor=blended_latents)
+        return LatentsOutput.build(latents_name=name, latents=blended_latents, seed=self.latents_a.seed)

invokeai/app/invocations/compel.py

@@ -81,13 +81,9 @@ class CompelInvocation(BaseInvocation):
 
         with (
             # apply all patches while the model is on the target device
-            text_encoder_info.model_on_device() as (model_state_dict, text_encoder),
+            text_encoder_info as text_encoder,
             tokenizer_info as tokenizer,
-            ModelPatcher.apply_lora_text_encoder(
-                text_encoder,
-                loras=_lora_loader(),
-                model_state_dict=model_state_dict,
-            ),
+            ModelPatcher.apply_lora_text_encoder(text_encoder, _lora_loader()),
             # Apply CLIP Skip after LoRA to prevent LoRA application from failing on skipped layers.
             ModelPatcher.apply_clip_skip(text_encoder, self.clip.skipped_layers),
             ModelPatcher.apply_ti(tokenizer, text_encoder, ti_list) as (
@@ -176,14 +172,9 @@ class SDXLPromptInvocationBase:
 
         with (
             # apply all patches while the model is on the target device
-            text_encoder_info.model_on_device() as (state_dict, text_encoder),
+            text_encoder_info as text_encoder,
             tokenizer_info as tokenizer,
-            ModelPatcher.apply_lora(
-                text_encoder,
-                loras=_lora_loader(),
-                prefix=lora_prefix,
-                model_state_dict=state_dict,
-            ),
+            ModelPatcher.apply_lora(text_encoder, _lora_loader(), lora_prefix),
             # Apply CLIP Skip after LoRA to prevent LoRA application from failing on skipped layers.
             ModelPatcher.apply_clip_skip(text_encoder, clip_field.skipped_layers),
             ModelPatcher.apply_ti(tokenizer, text_encoder, ti_list) as (

invokeai/app/invocations/constants.py

@@ -1,6 +1,7 @@
 from typing import Literal
 
 from invokeai.backend.stable_diffusion.schedulers import SCHEDULER_MAP
+from invokeai.backend.util.devices import TorchDevice
 
 LATENT_SCALE_FACTOR = 8
 """
@@ -15,3 +16,5 @@ SCHEDULER_NAME_VALUES = Literal[tuple(SCHEDULER_MAP.keys())]
 
 IMAGE_MODES = Literal["L", "RGB", "RGBA", "CMYK", "YCbCr", "LAB", "HSV", "I", "F"]
 """A literal type for PIL image modes supported by Invoke"""
+
+DEFAULT_PRECISION = TorchDevice.choose_torch_dtype()

invokeai/app/invocations/controlnet_image_processors.py

@@ -2,7 +2,6 @@
 # initial implementation by Gregg Helt, 2023
 # heavily leverages controlnet_aux package: https://github.com/patrickvonplaten/controlnet_aux
 from builtins import bool, float
-from pathlib import Path
 from typing import Dict, List, Literal, Union
 
 import cv2
@@ -37,13 +36,12 @@ from invokeai.app.invocations.util import validate_begin_end_step, validate_weig
 from invokeai.app.services.shared.invocation_context import InvocationContext
 from invokeai.app.util.controlnet_utils import CONTROLNET_MODE_VALUES, CONTROLNET_RESIZE_VALUES, heuristic_resize
 from invokeai.backend.image_util.canny import get_canny_edges
-from invokeai.backend.image_util.depth_anything import DEPTH_ANYTHING_MODELS, DepthAnythingDetector
-from invokeai.backend.image_util.dw_openpose import DWPOSE_MODELS, DWOpenposeDetector
+from invokeai.backend.image_util.depth_anything import DepthAnythingDetector
+from invokeai.backend.image_util.dw_openpose import DWOpenposeDetector
 from invokeai.backend.image_util.hed import HEDProcessor
 from invokeai.backend.image_util.lineart import LineartProcessor
 from invokeai.backend.image_util.lineart_anime import LineartAnimeProcessor
 from invokeai.backend.image_util.util import np_to_pil, pil_to_np
-from invokeai.backend.util.devices import TorchDevice
 
 from .baseinvocation import BaseInvocation, BaseInvocationOutput, Classification, invocation, invocation_output
 
@@ -141,7 +139,6 @@ class ImageProcessorInvocation(BaseInvocation, WithMetadata, WithBoard):
         return context.images.get_pil(self.image.image_name, "RGB")
 
     def invoke(self, context: InvocationContext) -> ImageOutput:
-        self._context = context
         raw_image = self.load_image(context)
         # image type should be PIL.PngImagePlugin.PngImageFile ?
         processed_image = self.run_processor(raw_image)
@@ -287,8 +284,7 @@ class MidasDepthImageProcessorInvocation(ImageProcessorInvocation):
     # depth_and_normal not supported in controlnet_aux v0.0.3
     # depth_and_normal: bool = InputField(default=False, description="whether to use depth and normal mode")
 
-    def run_processor(self, image: Image.Image) -> Image.Image:
-        # TODO: replace from_pretrained() calls with context.models.download_and_cache() (or similar)
+    def run_processor(self, image):
         midas_processor = MidasDetector.from_pretrained("lllyasviel/Annotators")
         processed_image = midas_processor(
             image,
@@ -315,7 +311,7 @@ class NormalbaeImageProcessorInvocation(ImageProcessorInvocation):
     detect_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.detect_res)
     image_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.image_res)
 
-    def run_processor(self, image: Image.Image) -> Image.Image:
+    def run_processor(self, image):
         normalbae_processor = NormalBaeDetector.from_pretrained("lllyasviel/Annotators")
         processed_image = normalbae_processor(
             image, detect_resolution=self.detect_resolution, image_resolution=self.image_resolution
@@ -334,7 +330,7 @@ class MlsdImageProcessorInvocation(ImageProcessorInvocation):
     thr_v: float = InputField(default=0.1, ge=0, description="MLSD parameter `thr_v`")
     thr_d: float = InputField(default=0.1, ge=0, description="MLSD parameter `thr_d`")
 
-    def run_processor(self, image: Image.Image) -> Image.Image:
+    def run_processor(self, image):
         mlsd_processor = MLSDdetector.from_pretrained("lllyasviel/Annotators")
         processed_image = mlsd_processor(
             image,
@@ -357,7 +353,7 @@ class PidiImageProcessorInvocation(ImageProcessorInvocation):
     safe: bool = InputField(default=False, description=FieldDescriptions.safe_mode)
     scribble: bool = InputField(default=False, description=FieldDescriptions.scribble_mode)
 
-    def run_processor(self, image: Image.Image) -> Image.Image:
+    def run_processor(self, image):
         pidi_processor = PidiNetDetector.from_pretrained("lllyasviel/Annotators")
         processed_image = pidi_processor(
             image,
@@ -385,7 +381,7 @@ class ContentShuffleImageProcessorInvocation(ImageProcessorInvocation):
     w: int = InputField(default=512, ge=0, description="Content shuffle `w` parameter")
     f: int = InputField(default=256, ge=0, description="Content shuffle `f` parameter")
 
-    def run_processor(self, image: Image.Image) -> Image.Image:
+    def run_processor(self, image):
         content_shuffle_processor = ContentShuffleDetector()
         processed_image = content_shuffle_processor(
             image,
@@ -409,7 +405,7 @@ class ContentShuffleImageProcessorInvocation(ImageProcessorInvocation):
 class ZoeDepthImageProcessorInvocation(ImageProcessorInvocation):
     """Applies Zoe depth processing to image"""
 
-    def run_processor(self, image: Image.Image) -> Image.Image:
+    def run_processor(self, image):
         zoe_depth_processor = ZoeDetector.from_pretrained("lllyasviel/Annotators")
         processed_image = zoe_depth_processor(image)
         return processed_image
@@ -430,7 +426,7 @@ class MediapipeFaceProcessorInvocation(ImageProcessorInvocation):
     detect_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.detect_res)
     image_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.image_res)
 
-    def run_processor(self, image: Image.Image) -> Image.Image:
+    def run_processor(self, image):
         mediapipe_face_processor = MediapipeFaceDetector()
         processed_image = mediapipe_face_processor(
             image,
@@ -458,7 +454,7 @@ class LeresImageProcessorInvocation(ImageProcessorInvocation):
     detect_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.detect_res)
     image_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.image_res)
 
-    def run_processor(self, image: Image.Image) -> Image.Image:
+    def run_processor(self, image):
         leres_processor = LeresDetector.from_pretrained("lllyasviel/Annotators")
         processed_image = leres_processor(
             image,
@@ -500,8 +496,8 @@ class TileResamplerProcessorInvocation(ImageProcessorInvocation):
         np_img = cv2.resize(np_img, (W, H), interpolation=cv2.INTER_AREA)
         return np_img
 
-    def run_processor(self, image: Image.Image) -> Image.Image:
-        np_img = np.array(image, dtype=np.uint8)
+    def run_processor(self, img):
+        np_img = np.array(img, dtype=np.uint8)
         processed_np_image = self.tile_resample(
             np_img,
             # res=self.tile_size,
@@ -524,7 +520,7 @@ class SegmentAnythingProcessorInvocation(ImageProcessorInvocation):
     detect_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.detect_res)
     image_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.image_res)
 
-    def run_processor(self, image: Image.Image) -> Image.Image:
+    def run_processor(self, image):
         # segment_anything_processor = SamDetector.from_pretrained("ybelkada/segment-anything", subfolder="checkpoints")
         segment_anything_processor = SamDetectorReproducibleColors.from_pretrained(
             "ybelkada/segment-anything", subfolder="checkpoints"
@@ -570,7 +566,7 @@ class ColorMapImageProcessorInvocation(ImageProcessorInvocation):
 
     color_map_tile_size: int = InputField(default=64, ge=1, description=FieldDescriptions.tile_size)
 
-    def run_processor(self, image: Image.Image) -> Image.Image:
+    def run_processor(self, image: Image.Image):
         np_image = np.array(image, dtype=np.uint8)
         height, width = np_image.shape[:2]
 
@@ -605,18 +601,12 @@ class DepthAnythingImageProcessorInvocation(ImageProcessorInvocation):
     )
     resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.image_res)
 
-    def run_processor(self, image: Image.Image) -> Image.Image:
-        def loader(model_path: Path):
-            return DepthAnythingDetector.load_model(
-                model_path, model_size=self.model_size, device=TorchDevice.choose_torch_device()
-            )
+    def run_processor(self, image: Image.Image):
+        depth_anything_detector = DepthAnythingDetector()
+        depth_anything_detector.load_model(model_size=self.model_size)
 
-        with self._context.models.load_remote_model(
-            source=DEPTH_ANYTHING_MODELS[self.model_size], loader=loader
-        ) as model:
-            depth_anything_detector = DepthAnythingDetector(model, TorchDevice.choose_torch_device())
-            processed_image = depth_anything_detector(image=image, resolution=self.resolution)
-            return processed_image
+        processed_image = depth_anything_detector(image=image, resolution=self.resolution)
+        return processed_image
 
 
 @invocation(
@@ -634,11 +624,8 @@ class DWOpenposeImageProcessorInvocation(ImageProcessorInvocation):
     draw_hands: bool = InputField(default=False)
     image_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.image_res)
 
-    def run_processor(self, image: Image.Image) -> Image.Image:
-        onnx_det = self._context.models.download_and_cache_model(DWPOSE_MODELS["yolox_l.onnx"])
-        onnx_pose = self._context.models.download_and_cache_model(DWPOSE_MODELS["dw-ll_ucoco_384.onnx"])
-
-        dw_openpose = DWOpenposeDetector(onnx_det=onnx_det, onnx_pose=onnx_pose)
+    def run_processor(self, image: Image.Image):
+        dw_openpose = DWOpenposeDetector()
         processed_image = dw_openpose(
             image,
             draw_face=self.draw_face,

invokeai/app/invocations/create_denoise_mask.py

@@ -0,0 +1,80 @@
+from typing import Optional
+
+import torch
+import torchvision.transforms as T
+from PIL import Image
+from torchvision.transforms.functional import resize as tv_resize
+
+from invokeai.app.invocations.baseinvocation import BaseInvocation, invocation
+from invokeai.app.invocations.constants import DEFAULT_PRECISION
+from invokeai.app.invocations.fields import FieldDescriptions, ImageField, Input, InputField
+from invokeai.app.invocations.image_to_latents import ImageToLatentsInvocation
+from invokeai.app.invocations.model import VAEField
+from invokeai.app.invocations.primitives import DenoiseMaskOutput
+from invokeai.app.services.shared.invocation_context import InvocationContext
+from invokeai.backend.stable_diffusion.diffusers_pipeline import image_resized_to_grid_as_tensor
+
+
+@invocation(
+    "create_denoise_mask",
+    title="Create Denoise Mask",
+    tags=["mask", "denoise"],
+    category="latents",
+    version="1.0.2",
+)
+class CreateDenoiseMaskInvocation(BaseInvocation):
+    """Creates mask for denoising model run."""
+
+    vae: VAEField = InputField(description=FieldDescriptions.vae, input=Input.Connection, ui_order=0)
+    image: Optional[ImageField] = InputField(default=None, description="Image which will be masked", ui_order=1)
+    mask: ImageField = InputField(description="The mask to use when pasting", ui_order=2)
+    tiled: bool = InputField(default=False, description=FieldDescriptions.tiled, ui_order=3)
+    fp32: bool = InputField(
+        default=DEFAULT_PRECISION == torch.float32,
+        description=FieldDescriptions.fp32,
+        ui_order=4,
+    )
+
+    def prep_mask_tensor(self, mask_image: Image.Image) -> torch.Tensor:
+        if mask_image.mode != "L":
+            mask_image = mask_image.convert("L")
+        mask_tensor: torch.Tensor = image_resized_to_grid_as_tensor(mask_image, normalize=False)
+        if mask_tensor.dim() == 3:
+            mask_tensor = mask_tensor.unsqueeze(0)
+        # if shape is not None:
+        #    mask_tensor = tv_resize(mask_tensor, shape, T.InterpolationMode.BILINEAR)
+        return mask_tensor
+
+    @torch.no_grad()
+    def invoke(self, context: InvocationContext) -> DenoiseMaskOutput:
+        if self.image is not None:
+            image = context.images.get_pil(self.image.image_name)
+            image_tensor = image_resized_to_grid_as_tensor(image.convert("RGB"))
+            if image_tensor.dim() == 3:
+                image_tensor = image_tensor.unsqueeze(0)
+        else:
+            image_tensor = None
+
+        mask = self.prep_mask_tensor(
+            context.images.get_pil(self.mask.image_name),
+        )
+
+        if image_tensor is not None:
+            vae_info = context.models.load(self.vae.vae)
+
+            img_mask = tv_resize(mask, image_tensor.shape[-2:], T.InterpolationMode.BILINEAR, antialias=False)
+            masked_image = image_tensor * torch.where(img_mask < 0.5, 0.0, 1.0)
+            # TODO:
+            masked_latents = ImageToLatentsInvocation.vae_encode(vae_info, self.fp32, self.tiled, masked_image.clone())
+
+            masked_latents_name = context.tensors.save(tensor=masked_latents)
+        else:
+            masked_latents_name = None
+
+        mask_name = context.tensors.save(tensor=mask)
+
+        return DenoiseMaskOutput.build(
+            mask_name=mask_name,
+            masked_latents_name=masked_latents_name,
+            gradient=False,
+        )

invokeai/app/invocations/create_gradient_mask.py

@@ -0,0 +1,138 @@
+from typing import Literal, Optional
+
+import numpy as np
+import torch
+import torchvision.transforms as T
+from PIL import Image, ImageFilter
+from torchvision.transforms.functional import resize as tv_resize
+
+from invokeai.app.invocations.baseinvocation import BaseInvocation, BaseInvocationOutput, invocation, invocation_output
+from invokeai.app.invocations.constants import DEFAULT_PRECISION
+from invokeai.app.invocations.fields import (
+    DenoiseMaskField,
+    FieldDescriptions,
+    ImageField,
+    Input,
+    InputField,
+    OutputField,
+)
+from invokeai.app.invocations.image_to_latents import ImageToLatentsInvocation
+from invokeai.app.invocations.model import UNetField, VAEField
+from invokeai.app.services.shared.invocation_context import InvocationContext
+from invokeai.backend.model_manager import LoadedModel
+from invokeai.backend.model_manager.config import MainConfigBase, ModelVariantType
+from invokeai.backend.stable_diffusion.diffusers_pipeline import image_resized_to_grid_as_tensor
+
+
+@invocation_output("gradient_mask_output")
+class GradientMaskOutput(BaseInvocationOutput):
+    """Outputs a denoise mask and an image representing the total gradient of the mask."""
+
+    denoise_mask: DenoiseMaskField = OutputField(description="Mask for denoise model run")
+    expanded_mask_area: ImageField = OutputField(
+        description="Image representing the total gradient area of the mask. For paste-back purposes."
+    )
+
+
+@invocation(
+    "create_gradient_mask",
+    title="Create Gradient Mask",
+    tags=["mask", "denoise"],
+    category="latents",
+    version="1.1.0",
+)
+class CreateGradientMaskInvocation(BaseInvocation):
+    """Creates mask for denoising model run."""
+
+    mask: ImageField = InputField(default=None, description="Image which will be masked", ui_order=1)
+    edge_radius: int = InputField(
+        default=16, ge=0, description="How far to blur/expand the edges of the mask", ui_order=2
+    )
+    coherence_mode: Literal["Gaussian Blur", "Box Blur", "Staged"] = InputField(default="Gaussian Blur", ui_order=3)
+    minimum_denoise: float = InputField(
+        default=0.0, ge=0, le=1, description="Minimum denoise level for the coherence region", ui_order=4
+    )
+    image: Optional[ImageField] = InputField(
+        default=None,
+        description="OPTIONAL: Only connect for specialized Inpainting models, masked_latents will be generated from the image with the VAE",
+        title="[OPTIONAL] Image",
+        ui_order=6,
+    )
+    unet: Optional[UNetField] = InputField(
+        description="OPTIONAL: If the Unet is a specialized Inpainting model, masked_latents will be generated from the image with the VAE",
+        default=None,
+        input=Input.Connection,
+        title="[OPTIONAL] UNet",
+        ui_order=5,
+    )
+    vae: Optional[VAEField] = InputField(
+        default=None,
+        description="OPTIONAL: Only connect for specialized Inpainting models, masked_latents will be generated from the image with the VAE",
+        title="[OPTIONAL] VAE",
+        input=Input.Connection,
+        ui_order=7,
+    )
+    tiled: bool = InputField(default=False, description=FieldDescriptions.tiled, ui_order=8)
+    fp32: bool = InputField(
+        default=DEFAULT_PRECISION == torch.float32,
+        description=FieldDescriptions.fp32,
+        ui_order=9,
+    )
+
+    @torch.no_grad()
+    def invoke(self, context: InvocationContext) -> GradientMaskOutput:
+        mask_image = context.images.get_pil(self.mask.image_name, mode="L")
+        if self.edge_radius > 0:
+            if self.coherence_mode == "Box Blur":
+                blur_mask = mask_image.filter(ImageFilter.BoxBlur(self.edge_radius))
+            else:  # Gaussian Blur OR Staged
+                # Gaussian Blur uses standard deviation. 1/2 radius is a good approximation
+                blur_mask = mask_image.filter(ImageFilter.GaussianBlur(self.edge_radius / 2))
+
+            blur_tensor: torch.Tensor = image_resized_to_grid_as_tensor(blur_mask, normalize=False)
+
+            # redistribute blur so that the original edges are 0 and blur outwards to 1
+            blur_tensor = (blur_tensor - 0.5) * 2
+
+            threshold = 1 - self.minimum_denoise
+
+            if self.coherence_mode == "Staged":
+                # wherever the blur_tensor is less than fully masked, convert it to threshold
+                blur_tensor = torch.where((blur_tensor < 1) & (blur_tensor > 0), threshold, blur_tensor)
+            else:
+                # wherever the blur_tensor is above threshold but less than 1, drop it to threshold
+                blur_tensor = torch.where((blur_tensor > threshold) & (blur_tensor < 1), threshold, blur_tensor)
+
+        else:
+            blur_tensor: torch.Tensor = image_resized_to_grid_as_tensor(mask_image, normalize=False)
+
+        mask_name = context.tensors.save(tensor=blur_tensor.unsqueeze(1))
+
+        # compute a [0, 1] mask from the blur_tensor
+        expanded_mask = torch.where((blur_tensor < 1), 0, 1)
+        expanded_mask_image = Image.fromarray((expanded_mask.squeeze(0).numpy() * 255).astype(np.uint8), mode="L")
+        expanded_image_dto = context.images.save(expanded_mask_image)
+
+        masked_latents_name = None
+        if self.unet is not None and self.vae is not None and self.image is not None:
+            # all three fields must be present at the same time
+            main_model_config = context.models.get_config(self.unet.unet.key)
+            assert isinstance(main_model_config, MainConfigBase)
+            if main_model_config.variant is ModelVariantType.Inpaint:
+                mask = blur_tensor
+                vae_info: LoadedModel = context.models.load(self.vae.vae)
+                image = context.images.get_pil(self.image.image_name)
+                image_tensor = image_resized_to_grid_as_tensor(image.convert("RGB"))
+                if image_tensor.dim() == 3:
+                    image_tensor = image_tensor.unsqueeze(0)
+                img_mask = tv_resize(mask, image_tensor.shape[-2:], T.InterpolationMode.BILINEAR, antialias=False)
+                masked_image = image_tensor * torch.where(img_mask < 0.5, 0.0, 1.0)
+                masked_latents = ImageToLatentsInvocation.vae_encode(
+                    vae_info, self.fp32, self.tiled, masked_image.clone()
+                )
+                masked_latents_name = context.tensors.save(tensor=masked_latents)
+
+        return GradientMaskOutput(
+            denoise_mask=DenoiseMaskField(mask_name=mask_name, masked_latents_name=masked_latents_name, gradient=True),
+            expanded_mask_area=ImageField(image_name=expanded_image_dto.image_name),
+        )

invokeai/app/invocations/crop_latents.py

@@ -0,0 +1,61 @@
+from invokeai.app.invocations.baseinvocation import BaseInvocation, invocation
+from invokeai.app.invocations.constants import LATENT_SCALE_FACTOR
+from invokeai.app.invocations.fields import FieldDescriptions, Input, InputField, LatentsField
+from invokeai.app.invocations.primitives import LatentsOutput
+from invokeai.app.services.shared.invocation_context import InvocationContext
+
+
+# The Crop Latents node was copied from @skunkworxdark's implementation here:
+# https://github.com/skunkworxdark/XYGrid_nodes/blob/74647fa9c1fa57d317a94bd43ca689af7f0aae5e/images_to_grids.py#L1117C1-L1167C80
+@invocation(
+    "crop_latents",
+    title="Crop Latents",
+    tags=["latents", "crop"],
+    category="latents",
+    version="1.0.2",
+)
+# TODO(ryand): Named `CropLatentsCoreInvocation` to prevent a conflict with custom node `CropLatentsInvocation`.
+# Currently, if the class names conflict then 'GET /openapi.json' fails.
+class CropLatentsCoreInvocation(BaseInvocation):
+    """Crops a latent-space tensor to a box specified in image-space. The box dimensions and coordinates must be
+    divisible by the latent scale factor of 8.
+    """
+
+    latents: LatentsField = InputField(
+        description=FieldDescriptions.latents,
+        input=Input.Connection,
+    )
+    x: int = InputField(
+        ge=0,
+        multiple_of=LATENT_SCALE_FACTOR,
+        description="The left x coordinate (in px) of the crop rectangle in image space. This value will be converted to a dimension in latent space.",
+    )
+    y: int = InputField(
+        ge=0,
+        multiple_of=LATENT_SCALE_FACTOR,
+        description="The top y coordinate (in px) of the crop rectangle in image space. This value will be converted to a dimension in latent space.",
+    )
+    width: int = InputField(
+        ge=1,
+        multiple_of=LATENT_SCALE_FACTOR,
+        description="The width (in px) of the crop rectangle in image space. This value will be converted to a dimension in latent space.",
+    )
+    height: int = InputField(
+        ge=1,
+        multiple_of=LATENT_SCALE_FACTOR,
+        description="The height (in px) of the crop rectangle in image space. This value will be converted to a dimension in latent space.",
+    )
+
+    def invoke(self, context: InvocationContext) -> LatentsOutput:
+        latents = context.tensors.load(self.latents.latents_name)
+
+        x1 = self.x // LATENT_SCALE_FACTOR
+        y1 = self.y // LATENT_SCALE_FACTOR
+        x2 = x1 + (self.width // LATENT_SCALE_FACTOR)
+        y2 = y1 + (self.height // LATENT_SCALE_FACTOR)
+
+        cropped_latents = latents[..., y1:y2, x1:x2]
+
+        name = context.tensors.save(tensor=cropped_latents)
+
+        return LatentsOutput.build(latents_name=name, latents=cropped_latents)

invokeai/app/invocations/ideal_size.py

@@ -0,0 +1,65 @@
+import math
+from typing import Tuple
+
+from invokeai.app.invocations.baseinvocation import BaseInvocation, BaseInvocationOutput, invocation, invocation_output
+from invokeai.app.invocations.constants import LATENT_SCALE_FACTOR
+from invokeai.app.invocations.fields import FieldDescriptions, InputField, OutputField
+from invokeai.app.invocations.model import UNetField
+from invokeai.app.services.shared.invocation_context import InvocationContext
+from invokeai.backend.model_manager.config import BaseModelType
+
+
+@invocation_output("ideal_size_output")
+class IdealSizeOutput(BaseInvocationOutput):
+    """Base class for invocations that output an image"""
+
+    width: int = OutputField(description="The ideal width of the image (in pixels)")
+    height: int = OutputField(description="The ideal height of the image (in pixels)")
+
+
+@invocation(
+    "ideal_size",
+    title="Ideal Size",
+    tags=["latents", "math", "ideal_size"],
+    version="1.0.3",
+)
+class IdealSizeInvocation(BaseInvocation):
+    """Calculates the ideal size for generation to avoid duplication"""
+
+    width: int = InputField(default=1024, description="Final image width")
+    height: int = InputField(default=576, description="Final image height")
+    unet: UNetField = InputField(default=None, description=FieldDescriptions.unet)
+    multiplier: float = InputField(
+        default=1.0,
+        description="Amount to multiply the model's dimensions by when calculating the ideal size (may result in "
+        "initial generation artifacts if too large)",
+    )
+
+    def trim_to_multiple_of(self, *args: int, multiple_of: int = LATENT_SCALE_FACTOR) -> Tuple[int, ...]:
+        return tuple((x - x % multiple_of) for x in args)
+
+    def invoke(self, context: InvocationContext) -> IdealSizeOutput:
+        unet_config = context.models.get_config(self.unet.unet.key)
+        aspect = self.width / self.height
+        dimension: float = 512
+        if unet_config.base == BaseModelType.StableDiffusion2:
+            dimension = 768
+        elif unet_config.base == BaseModelType.StableDiffusionXL:
+            dimension = 1024
+        dimension = dimension * self.multiplier
+        min_dimension = math.floor(dimension * 0.5)
+        model_area = dimension * dimension  # hardcoded for now since all models are trained on square images
+
+        if aspect > 1.0:
+            init_height = max(min_dimension, math.sqrt(model_area / aspect))
+            init_width = init_height * aspect
+        else:
+            init_width = max(min_dimension, math.sqrt(model_area * aspect))
+            init_height = init_width / aspect
+
+        scaled_width, scaled_height = self.trim_to_multiple_of(
+            math.floor(init_width),
+            math.floor(init_height),
+        )
+
+        return IdealSizeOutput(width=scaled_width, height=scaled_height)

invokeai/app/invocations/image_to_latents.py

@@ -0,0 +1,125 @@
+from functools import singledispatchmethod
+
+import einops
+import torch
+from diffusers.models.attention_processor import (
+    AttnProcessor2_0,
+    LoRAAttnProcessor2_0,
+    LoRAXFormersAttnProcessor,
+    XFormersAttnProcessor,
+)
+from diffusers.models.autoencoders.autoencoder_kl import AutoencoderKL
+from diffusers.models.autoencoders.autoencoder_tiny import AutoencoderTiny
+
+from invokeai.app.invocations.baseinvocation import BaseInvocation, invocation
+from invokeai.app.invocations.constants import DEFAULT_PRECISION
+from invokeai.app.invocations.fields import (
+    FieldDescriptions,
+    ImageField,
+    Input,
+    InputField,
+)
+from invokeai.app.invocations.model import VAEField
+from invokeai.app.invocations.primitives import LatentsOutput
+from invokeai.app.services.shared.invocation_context import InvocationContext
+from invokeai.backend.model_manager import LoadedModel
+from invokeai.backend.stable_diffusion.diffusers_pipeline import image_resized_to_grid_as_tensor
+
+
+@invocation(
+    "i2l",
+    title="Image to Latents",
+    tags=["latents", "image", "vae", "i2l"],
+    category="latents",
+    version="1.0.2",
+)
+class ImageToLatentsInvocation(BaseInvocation):
+    """Encodes an image into latents."""
+
+    image: ImageField = InputField(
+        description="The image to encode",
+    )
+    vae: VAEField = InputField(
+        description=FieldDescriptions.vae,
+        input=Input.Connection,
+    )
+    tiled: bool = InputField(default=False, description=FieldDescriptions.tiled)
+    fp32: bool = InputField(default=DEFAULT_PRECISION == torch.float32, description=FieldDescriptions.fp32)
+
+    @staticmethod
+    def vae_encode(vae_info: LoadedModel, upcast: bool, tiled: bool, image_tensor: torch.Tensor) -> torch.Tensor:
+        with vae_info as vae:
+            assert isinstance(vae, torch.nn.Module)
+            orig_dtype = vae.dtype
+            if upcast:
+                vae.to(dtype=torch.float32)
+
+                use_torch_2_0_or_xformers = hasattr(vae.decoder, "mid_block") and isinstance(
+                    vae.decoder.mid_block.attentions[0].processor,
+                    (
+                        AttnProcessor2_0,
+                        XFormersAttnProcessor,
+                        LoRAXFormersAttnProcessor,
+                        LoRAAttnProcessor2_0,
+                    ),
+                )
+                # if xformers or torch_2_0 is used attention block does not need
+                # to be in float32 which can save lots of memory
+                if use_torch_2_0_or_xformers:
+                    vae.post_quant_conv.to(orig_dtype)
+                    vae.decoder.conv_in.to(orig_dtype)
+                    vae.decoder.mid_block.to(orig_dtype)
+                # else:
+                #    latents = latents.float()
+
+            else:
+                vae.to(dtype=torch.float16)
+                # latents = latents.half()
+
+            if tiled:
+                vae.enable_tiling()
+            else:
+                vae.disable_tiling()
+
+            # non_noised_latents_from_image
+            image_tensor = image_tensor.to(device=vae.device, dtype=vae.dtype)
+            with torch.inference_mode():
+                latents = ImageToLatentsInvocation._encode_to_tensor(vae, image_tensor)
+
+            latents = vae.config.scaling_factor * latents
+            latents = latents.to(dtype=orig_dtype)
+
+        return latents
+
+    @torch.no_grad()
+    def invoke(self, context: InvocationContext) -> LatentsOutput:
+        image = context.images.get_pil(self.image.image_name)
+
+        vae_info = context.models.load(self.vae.vae)
+
+        image_tensor = image_resized_to_grid_as_tensor(image.convert("RGB"))
+        if image_tensor.dim() == 3:
+            image_tensor = einops.rearrange(image_tensor, "c h w -> 1 c h w")
+
+        latents = self.vae_encode(vae_info, self.fp32, self.tiled, image_tensor)
+
+        latents = latents.to("cpu")
+        name = context.tensors.save(tensor=latents)
+        return LatentsOutput.build(latents_name=name, latents=latents, seed=None)
+
+    @singledispatchmethod
+    @staticmethod
+    def _encode_to_tensor(vae: AutoencoderKL, image_tensor: torch.FloatTensor) -> torch.FloatTensor:
+        assert isinstance(vae, torch.nn.Module)
+        image_tensor_dist = vae.encode(image_tensor).latent_dist
+        latents: torch.Tensor = image_tensor_dist.sample().to(
+            dtype=vae.dtype
+        )  # FIXME: uses torch.randn. make reproducible!
+        return latents
+
+    @_encode_to_tensor.register
+    @staticmethod
+    def _(vae: AutoencoderTiny, image_tensor: torch.FloatTensor) -> torch.FloatTensor:
+        assert isinstance(vae, torch.nn.Module)
+        latents: torch.FloatTensor = vae.encode(image_tensor).latents
+        return latents

invokeai/app/invocations/infill.py

@@ -42,16 +42,15 @@ class InfillImageProcessorInvocation(BaseInvocation, WithMetadata, WithBoard):
         """Infill the image with the specified method"""
         pass
 
-    def load_image(self) -> tuple[Image.Image, bool]:
+    def load_image(self, context: InvocationContext) -> tuple[Image.Image, bool]:
         """Process the image to have an alpha channel before being infilled"""
-        image = self._context.images.get_pil(self.image.image_name)
+        image = context.images.get_pil(self.image.image_name)
         has_alpha = True if image.mode == "RGBA" else False
         return image, has_alpha
 
     def invoke(self, context: InvocationContext) -> ImageOutput:
-        self._context = context
         # Retrieve and process image to be infilled
-        input_image, has_alpha = self.load_image()
+        input_image, has_alpha = self.load_image(context)
 
         # If the input image has no alpha channel, return it
         if has_alpha is False:
@@ -134,12 +133,8 @@ class LaMaInfillInvocation(InfillImageProcessorInvocation):
     """Infills transparent areas of an image using the LaMa model"""
 
     def infill(self, image: Image.Image):
-        with self._context.models.load_remote_model(
-            source="https://github.com/Sanster/models/releases/download/add_big_lama/big-lama.pt",
-            loader=LaMA.load_jit_model,
-        ) as model:
-            lama = LaMA(model)
-            return lama(image)
+        lama = LaMA()
+        return lama(image)
 
 
 @invocation("infill_cv2", title="CV2 Infill", tags=["image", "inpaint"], category="inpaint", version="1.2.2")

invokeai/app/invocations/latents_to_image.py

@@ -0,0 +1,127 @@
+import torch
+from diffusers.image_processor import VaeImageProcessor
+from diffusers.models.attention_processor import (
+    AttnProcessor2_0,
+    LoRAAttnProcessor2_0,
+    LoRAXFormersAttnProcessor,
+    XFormersAttnProcessor,
+)
+from diffusers.models.autoencoders.autoencoder_kl import AutoencoderKL
+from diffusers.models.autoencoders.autoencoder_tiny import AutoencoderTiny
+from PIL import Image
+
+from invokeai.app.invocations.baseinvocation import BaseInvocation, invocation
+from invokeai.app.invocations.constants import DEFAULT_PRECISION
+from invokeai.app.invocations.fields import (
+    FieldDescriptions,
+    Input,
+    InputField,
+    LatentsField,
+    WithBoard,
+    WithMetadata,
+)
+from invokeai.app.invocations.model import VAEField
+from invokeai.app.invocations.primitives import ImageOutput
+from invokeai.app.services.shared.invocation_context import InvocationContext
+from invokeai.backend.model_manager.load.load_base import LoadedModel
+from invokeai.backend.stable_diffusion import set_seamless
+from invokeai.backend.util.devices import TorchDevice
+
+
+@invocation(
+    "l2i",
+    title="Latents to Image",
+    tags=["latents", "image", "vae", "l2i"],
+    category="latents",
+    version="1.2.2",
+)
+class LatentsToImageInvocation(BaseInvocation, WithMetadata, WithBoard):
+    """Generates an image from latents."""
+
+    latents: LatentsField = InputField(
+        description=FieldDescriptions.latents,
+        input=Input.Connection,
+    )
+    vae: VAEField = InputField(
+        description=FieldDescriptions.vae,
+        input=Input.Connection,
+    )
+    tiled: bool = InputField(default=False, description=FieldDescriptions.tiled)
+    fp32: bool = InputField(default=DEFAULT_PRECISION == torch.float32, description=FieldDescriptions.fp32)
+
+    @staticmethod
+    def vae_decode(
+        context: InvocationContext,
+        vae_info: LoadedModel,
+        seamless_axes: list[str],
+        latents: torch.Tensor,
+        use_fp32: bool,
+        use_tiling: bool,
+    ) -> Image.Image:
+        assert isinstance(vae_info.model, (AutoencoderKL, AutoencoderTiny))
+        with set_seamless(vae_info.model, seamless_axes), vae_info as vae:
+            assert isinstance(vae, (AutoencoderKL, AutoencoderTiny))
+            latents = latents.to(vae.device)
+            if use_fp32:
+                vae.to(dtype=torch.float32)
+
+                use_torch_2_0_or_xformers = hasattr(vae.decoder, "mid_block") and isinstance(
+                    vae.decoder.mid_block.attentions[0].processor,
+                    (
+                        AttnProcessor2_0,
+                        XFormersAttnProcessor,
+                        LoRAXFormersAttnProcessor,
+                        LoRAAttnProcessor2_0,
+                    ),
+                )
+                # if xformers or torch_2_0 is used attention block does not need
+                # to be in float32 which can save lots of memory
+                if use_torch_2_0_or_xformers:
+                    vae.post_quant_conv.to(latents.dtype)
+                    vae.decoder.conv_in.to(latents.dtype)
+                    vae.decoder.mid_block.to(latents.dtype)
+                else:
+                    latents = latents.float()
+
+            else:
+                vae.to(dtype=torch.float16)
+                latents = latents.half()
+
+            if use_tiling or context.config.get().force_tiled_decode:
+                vae.enable_tiling()
+            else:
+                vae.disable_tiling()
+
+            # clear memory as vae decode can request a lot
+            TorchDevice.empty_cache()
+
+            with torch.inference_mode():
+                # copied from diffusers pipeline
+                latents = latents / vae.config.scaling_factor
+                image = vae.decode(latents, return_dict=False)[0]
+                image = (image / 2 + 0.5).clamp(0, 1)  # denormalize
+                # we always cast to float32 as this does not cause significant overhead and is compatible with bfloat16
+                np_image = image.cpu().permute(0, 2, 3, 1).float().numpy()
+
+                image = VaeImageProcessor.numpy_to_pil(np_image)[0]
+
+        TorchDevice.empty_cache()
+
+        return image
+
+    @torch.no_grad()
+    def invoke(self, context: InvocationContext) -> ImageOutput:
+        latents = context.tensors.load(self.latents.latents_name)
+        vae_info = context.models.load(self.vae.vae)
+
+        image = self.vae_decode(
+            context=context,
+            vae_info=vae_info,
+            seamless_axes=self.vae.seamless_axes,
+            latents=latents,
+            use_fp32=self.fp32,
+            use_tiling=self.tiled,
+        )
+        image_dto = context.images.save(image=image)
+
+        return ImageOutput.build(image_dto)

invokeai/app/invocations/resize_latents.py

@@ -0,0 +1,103 @@
+from typing import Literal
+
+import torch
+
+from invokeai.app.invocations.baseinvocation import BaseInvocation, invocation
+from invokeai.app.invocations.constants import LATENT_SCALE_FACTOR
+from invokeai.app.invocations.fields import (
+    FieldDescriptions,
+    Input,
+    InputField,
+    LatentsField,
+)
+from invokeai.app.invocations.primitives import LatentsOutput
+from invokeai.app.services.shared.invocation_context import InvocationContext
+from invokeai.backend.util.devices import TorchDevice
+
+LATENTS_INTERPOLATION_MODE = Literal["nearest", "linear", "bilinear", "bicubic", "trilinear", "area", "nearest-exact"]
+
+
+@invocation(
+    "lresize",
+    title="Resize Latents",
+    tags=["latents", "resize"],
+    category="latents",
+    version="1.0.2",
+)
+class ResizeLatentsInvocation(BaseInvocation):
+    """Resizes latents to explicit width/height (in pixels). Provided dimensions are floor-divided by 8."""
+
+    latents: LatentsField = InputField(
+        description=FieldDescriptions.latents,
+        input=Input.Connection,
+    )
+    width: int = InputField(
+        ge=64,
+        multiple_of=LATENT_SCALE_FACTOR,
+        description=FieldDescriptions.width,
+    )
+    height: int = InputField(
+        ge=64,
+        multiple_of=LATENT_SCALE_FACTOR,
+        description=FieldDescriptions.width,
+    )
+    mode: LATENTS_INTERPOLATION_MODE = InputField(default="bilinear", description=FieldDescriptions.interp_mode)
+    antialias: bool = InputField(default=False, description=FieldDescriptions.torch_antialias)
+
+    def invoke(self, context: InvocationContext) -> LatentsOutput:
+        latents = context.tensors.load(self.latents.latents_name)
+        device = TorchDevice.choose_torch_device()
+
+        resized_latents = torch.nn.functional.interpolate(
+            latents.to(device),
+            size=(self.height // LATENT_SCALE_FACTOR, self.width // LATENT_SCALE_FACTOR),
+            mode=self.mode,
+            antialias=self.antialias if self.mode in ["bilinear", "bicubic"] else False,
+        )
+
+        # https://discuss.huggingface.co/t/memory-usage-by-later-pipeline-stages/23699
+        resized_latents = resized_latents.to("cpu")
+
+        TorchDevice.empty_cache()
+
+        name = context.tensors.save(tensor=resized_latents)
+        return LatentsOutput.build(latents_name=name, latents=resized_latents, seed=self.latents.seed)
+
+
+@invocation(
+    "lscale",
+    title="Scale Latents",
+    tags=["latents", "resize"],
+    category="latents",
+    version="1.0.2",
+)
+class ScaleLatentsInvocation(BaseInvocation):
+    """Scales latents by a given factor."""
+
+    latents: LatentsField = InputField(
+        description=FieldDescriptions.latents,
+        input=Input.Connection,
+    )
+    scale_factor: float = InputField(gt=0, description=FieldDescriptions.scale_factor)
+    mode: LATENTS_INTERPOLATION_MODE = InputField(default="bilinear", description=FieldDescriptions.interp_mode)
+    antialias: bool = InputField(default=False, description=FieldDescriptions.torch_antialias)
+
+    def invoke(self, context: InvocationContext) -> LatentsOutput:
+        latents = context.tensors.load(self.latents.latents_name)
+
+        device = TorchDevice.choose_torch_device()
+
+        # resizing
+        resized_latents = torch.nn.functional.interpolate(
+            latents.to(device),
+            scale_factor=self.scale_factor,
+            mode=self.mode,
+            antialias=self.antialias if self.mode in ["bilinear", "bicubic"] else False,
+        )
+
+        # https://discuss.huggingface.co/t/memory-usage-by-later-pipeline-stages/23699
+        resized_latents = resized_latents.to("cpu")
+        TorchDevice.empty_cache()
+
+        name = context.tensors.save(tensor=resized_latents)
+        return LatentsOutput.build(latents_name=name, latents=resized_latents, seed=self.latents.seed)

invokeai/app/invocations/scheduler.py

@@ -0,0 +1,34 @@
+from invokeai.app.invocations.baseinvocation import BaseInvocation, BaseInvocationOutput, invocation, invocation_output
+from invokeai.app.invocations.constants import SCHEDULER_NAME_VALUES
+from invokeai.app.invocations.fields import (
+    FieldDescriptions,
+    InputField,
+    OutputField,
+    UIType,
+)
+from invokeai.app.services.shared.invocation_context import InvocationContext
+
+
+@invocation_output("scheduler_output")
+class SchedulerOutput(BaseInvocationOutput):
+    scheduler: SCHEDULER_NAME_VALUES = OutputField(description=FieldDescriptions.scheduler, ui_type=UIType.Scheduler)
+
+
+@invocation(
+    "scheduler",
+    title="Scheduler",
+    tags=["scheduler"],
+    category="latents",
+    version="1.0.0",
+)
+class SchedulerInvocation(BaseInvocation):
+    """Selects a scheduler."""
+
+    scheduler: SCHEDULER_NAME_VALUES = InputField(
+        default="euler",
+        description=FieldDescriptions.scheduler,
+        ui_type=UIType.Scheduler,
+    )
+
+    def invoke(self, context: InvocationContext) -> SchedulerOutput:
+        return SchedulerOutput(scheduler=self.scheduler)

invokeai/app/invocations/tiled_stable_diffusion_refine.py

@@ -0,0 +1,384 @@
+from contextlib import ExitStack
+from typing import Iterator, Tuple
+
+import numpy as np
+import numpy.typing as npt
+import torch
+from diffusers.models.unets.unet_2d_condition import UNet2DConditionModel
+from PIL import Image
+from pydantic import field_validator
+
+from invokeai.app.invocations.baseinvocation import BaseInvocation, invocation
+from invokeai.app.invocations.constants import DEFAULT_PRECISION, LATENT_SCALE_FACTOR, SCHEDULER_NAME_VALUES
+from invokeai.app.invocations.fields import (
+    ConditioningField,
+    FieldDescriptions,
+    ImageField,
+    Input,
+    InputField,
+    UIType,
+)
+from invokeai.app.invocations.image_to_latents import ImageToLatentsInvocation
+from invokeai.app.invocations.latent import DenoiseLatentsInvocation, get_scheduler
+from invokeai.app.invocations.latents_to_image import LatentsToImageInvocation
+from invokeai.app.invocations.model import ModelIdentifierField, UNetField, VAEField
+from invokeai.app.invocations.noise import get_noise
+from invokeai.app.invocations.primitives import ImageOutput
+from invokeai.app.services.shared.invocation_context import InvocationContext
+from invokeai.app.util.controlnet_utils import CONTROLNET_MODE_VALUES, CONTROLNET_RESIZE_VALUES, prepare_control_image
+from invokeai.backend.lora import LoRAModelRaw
+from invokeai.backend.model_patcher import ModelPatcher
+from invokeai.backend.stable_diffusion.diffusers_pipeline import ControlNetData, image_resized_to_grid_as_tensor
+from invokeai.backend.tiles.tiles import calc_tiles_with_overlap, merge_tiles_with_linear_blending
+from invokeai.backend.tiles.utils import Tile
+from invokeai.backend.util.devices import TorchDevice
+from invokeai.backend.util.hotfixes import ControlNetModel
+
+
+@invocation(
+    "tiled_stable_diffusion_refine",
+    title="Tiled Stable Diffusion Refine",
+    tags=["upscale", "denoise"],
+    category="latents",
+    version="1.0.0",
+)
+class TiledStableDiffusionRefineInvocation(BaseInvocation):
+    """A tiled Stable Diffusion pipeline for refining high resolution images. This invocation is intended to be used to
+    refine an image after upscaling i.e. it is the second step in a typical "tiled upscaling" workflow.
+    """
+
+    image: ImageField = InputField(description="Image to be refined.")
+
+    positive_conditioning: ConditioningField = InputField(
+        description=FieldDescriptions.positive_cond, input=Input.Connection
+    )
+    negative_conditioning: ConditioningField = InputField(
+        description=FieldDescriptions.negative_cond, input=Input.Connection
+    )
+    # TODO(ryand): Add multiple-of validation.
+    tile_height: int = InputField(default=512, gt=0, description="Height of the tiles.")
+    tile_width: int = InputField(default=512, gt=0, description="Width of the tiles.")
+    tile_overlap: int = InputField(
+        default=16,
+        gt=0,
+        description="Target overlap between adjacent tiles (the last row/column may overlap more than this).",
+    )
+    steps: int = InputField(default=18, gt=0, description=FieldDescriptions.steps)
+    cfg_scale: float | list[float] = InputField(default=6.0, description=FieldDescriptions.cfg_scale, title="CFG Scale")
+    denoising_start: float = InputField(
+        default=0.65,
+        ge=0,
+        le=1,
+        description=FieldDescriptions.denoising_start,
+    )
+    denoising_end: float = InputField(default=1.0, ge=0, le=1, description=FieldDescriptions.denoising_end)
+    scheduler: SCHEDULER_NAME_VALUES = InputField(
+        default="euler",
+        description=FieldDescriptions.scheduler,
+        ui_type=UIType.Scheduler,
+    )
+    unet: UNetField = InputField(
+        description=FieldDescriptions.unet,
+        input=Input.Connection,
+        title="UNet",
+    )
+    cfg_rescale_multiplier: float = InputField(
+        title="CFG Rescale Multiplier", default=0, ge=0, lt=1, description=FieldDescriptions.cfg_rescale_multiplier
+    )
+    vae: VAEField = InputField(
+        description=FieldDescriptions.vae,
+        input=Input.Connection,
+    )
+    vae_fp32: bool = InputField(
+        default=DEFAULT_PRECISION == torch.float32, description="Whether to use float32 precision when running the VAE."
+    )
+    # HACK(ryand): We probably want to allow the user to control all of the parameters in ControlField. But, we akwardly
+    # don't want to use the image field. Figure out how best to handle this.
+    # TODO(ryand): Currently, there is no ControlNet preprocessor applied to the tile images. In other words, we pretty
+    # much assume that it is a tile ControlNet. We need to decide how we want to handle this. E.g. find a way to support
+    # CN preprocessors, raise a clear warning when a non-tile CN model is selected, hardcode the supported CN models,
+    # etc.
+    control_model: ModelIdentifierField = InputField(
+        description=FieldDescriptions.controlnet_model, ui_type=UIType.ControlNetModel
+    )
+    control_weight: float = InputField(default=0.6)
+
+    @field_validator("cfg_scale")
+    def ge_one(cls, v: list[float] | float) -> list[float] | float:
+        """Validate that all cfg_scale values are >= 1"""
+        if isinstance(v, list):
+            for i in v:
+                if i < 1:
+                    raise ValueError("cfg_scale must be greater than 1")
+        else:
+            if v < 1:
+                raise ValueError("cfg_scale must be greater than 1")
+        return v
+
+    @staticmethod
+    def crop_latents_to_tile(latents: torch.Tensor, image_tile: Tile) -> torch.Tensor:
+        """Crop the latent-space tensor to the area corresponding to the image-space tile.
+        The tile coordinates must be divisible by the LATENT_SCALE_FACTOR.
+        """
+        for coord in [image_tile.coords.top, image_tile.coords.left, image_tile.coords.right, image_tile.coords.bottom]:
+            if coord % LATENT_SCALE_FACTOR != 0:
+                raise ValueError(
+                    f"The tile coordinates must all be divisible by the latent scale factor"
+                    f" ({LATENT_SCALE_FACTOR}). {image_tile.coords=}."
+                )
+        assert latents.dim() == 4  # We expect: (batch_size, channels, height, width).
+
+        top = image_tile.coords.top // LATENT_SCALE_FACTOR
+        left = image_tile.coords.left // LATENT_SCALE_FACTOR
+        bottom = image_tile.coords.bottom // LATENT_SCALE_FACTOR
+        right = image_tile.coords.right // LATENT_SCALE_FACTOR
+        return latents[..., top:bottom, left:right]
+
+    def run_controlnet(
+        self,
+        image: Image.Image,
+        controlnet_model: ControlNetModel,
+        weight: float,
+        do_classifier_free_guidance: bool,
+        width: int,
+        height: int,
+        device: torch.device,
+        dtype: torch.dtype,
+        control_mode: CONTROLNET_MODE_VALUES = "balanced",
+        resize_mode: CONTROLNET_RESIZE_VALUES = "just_resize_simple",
+    ) -> ControlNetData:
+        control_image = prepare_control_image(
+            image=image,
+            do_classifier_free_guidance=do_classifier_free_guidance,
+            width=width,
+            height=height,
+            device=device,
+            dtype=dtype,
+            control_mode=control_mode,
+            resize_mode=resize_mode,
+        )
+        return ControlNetData(
+            model=controlnet_model,
+            image_tensor=control_image,
+            weight=weight,
+            begin_step_percent=0.0,
+            end_step_percent=1.0,
+            control_mode=control_mode,
+            # Any resizing needed should currently be happening in prepare_control_image(), but adding resize_mode to
+            # ControlNetData in case needed in the future.
+            resize_mode=resize_mode,
+        )
+
+    @torch.no_grad()
+    def invoke(self, context: InvocationContext) -> ImageOutput:
+        # TODO(ryand): Expose the seed parameter.
+        seed = 0
+
+        # Load the input image.
+        input_image = context.images.get_pil(self.image.image_name)
+
+        # Calculate the tile locations to cover the image.
+        # We have selected this tiling strategy to make it easy to achieve tile coords that are multiples of 8. This
+        # facilitates conversions between image space and latent space.
+        # TODO(ryand): Expose these tiling parameters. (Keep in mind the multiple-of constraints on these params.)
+        tiles = calc_tiles_with_overlap(
+            image_height=input_image.height,
+            image_width=input_image.width,
+            tile_height=self.tile_height,
+            tile_width=self.tile_width,
+            overlap=self.tile_overlap,
+        )
+
+        # Convert the input image to a torch.Tensor.
+        input_image_torch = image_resized_to_grid_as_tensor(input_image.convert("RGB"), multiple_of=LATENT_SCALE_FACTOR)
+        input_image_torch = input_image_torch.unsqueeze(0)  # Add a batch dimension.
+        # Validate our assumptions about the shape of input_image_torch.
+        assert input_image_torch.dim() == 4  # We expect: (batch_size, channels, height, width).
+        assert input_image_torch.shape[:2] == (1, 3)
+
+        # Split the input image into tiles in torch.Tensor format.
+        image_tiles_torch: list[torch.Tensor] = []
+        for tile in tiles:
+            image_tile = input_image_torch[
+                :,
+                :,
+                tile.coords.top : tile.coords.bottom,
+                tile.coords.left : tile.coords.right,
+            ]
+            image_tiles_torch.append(image_tile)
+
+        # Split the input image into tiles in numpy format.
+        # TODO(ryand): We currently maintain both np.ndarray and torch.Tensor tiles. Ideally, all operations should work
+        # with torch.Tensor tiles.
+        input_image_np = np.array(input_image)
+        image_tiles_np: list[npt.NDArray[np.uint8]] = []
+        for tile in tiles:
+            image_tile_np = input_image_np[
+                tile.coords.top : tile.coords.bottom,
+                tile.coords.left : tile.coords.right,
+                :,
+            ]
+            image_tiles_np.append(image_tile_np)
+
+        # VAE-encode each image tile independently.
+        # TODO(ryand): Is there any advantage to VAE-encoding the entire image before splitting it into tiles? What
+        # about for decoding?
+        vae_info = context.models.load(self.vae.vae)
+        latent_tiles: list[torch.Tensor] = []
+        for image_tile_torch in image_tiles_torch:
+            latent_tiles.append(
+                ImageToLatentsInvocation.vae_encode(
+                    vae_info=vae_info, upcast=self.vae_fp32, tiled=False, image_tensor=image_tile_torch
+                )
+            )
+
+        # Generate noise with dimensions corresponding to the full image in latent space.
+        # It is important that the noise tensor is generated at the full image dimension and then tiled, rather than
+        # generating for each tile independently. This ensures that overlapping regions between tiles use the same
+        # noise.
+        assert input_image_torch.shape[2] % LATENT_SCALE_FACTOR == 0
+        assert input_image_torch.shape[3] % LATENT_SCALE_FACTOR == 0
+        global_noise = get_noise(
+            width=input_image_torch.shape[3],
+            height=input_image_torch.shape[2],
+            device=TorchDevice.choose_torch_device(),
+            seed=seed,
+            downsampling_factor=LATENT_SCALE_FACTOR,
+            use_cpu=True,
+        )
+
+        # Crop the global noise into tiles.
+        noise_tiles = [self.crop_latents_to_tile(latents=global_noise, image_tile=t) for t in tiles]
+
+        # Prepare an iterator that yields the UNet's LoRA models and their weights.
+        def _lora_loader() -> Iterator[Tuple[LoRAModelRaw, float]]:
+            for lora in self.unet.loras:
+                lora_info = context.models.load(lora.lora)
+                assert isinstance(lora_info.model, LoRAModelRaw)
+                yield (lora_info.model, lora.weight)
+                del lora_info
+
+        # Load the UNet model.
+        unet_info = context.models.load(self.unet.unet)
+
+        refined_latent_tiles: list[torch.Tensor] = []
+        with ExitStack() as exit_stack, unet_info as unet, ModelPatcher.apply_lora_unet(unet, _lora_loader()):
+            assert isinstance(unet, UNet2DConditionModel)
+            scheduler = get_scheduler(
+                context=context,
+                scheduler_info=self.unet.scheduler,
+                scheduler_name=self.scheduler,
+                seed=seed,
+            )
+            pipeline = DenoiseLatentsInvocation.create_pipeline(unet=unet, scheduler=scheduler)
+
+            # Prepare the prompt conditioning data. The same prompt conditioning is applied to all tiles.
+            # Assume that all tiles have the same shape.
+            _, _, latent_height, latent_width = latent_tiles[0].shape
+            conditioning_data = DenoiseLatentsInvocation.get_conditioning_data(
+                context=context,
+                positive_conditioning_field=self.positive_conditioning,
+                negative_conditioning_field=self.negative_conditioning,
+                unet=unet,
+                latent_height=latent_height,
+                latent_width=latent_width,
+                cfg_scale=self.cfg_scale,
+                steps=self.steps,
+                cfg_rescale_multiplier=self.cfg_rescale_multiplier,
+            )
+
+            # Load the ControlNet model.
+            # TODO(ryand): Support multiple ControlNet models.
+            controlnet_model = exit_stack.enter_context(context.models.load(self.control_model))
+            assert isinstance(controlnet_model, ControlNetModel)
+
+            # Denoise (i.e. "refine") each tile independently.
+            for image_tile_np, latent_tile, noise_tile in zip(image_tiles_np, latent_tiles, noise_tiles, strict=True):
+                assert latent_tile.shape == noise_tile.shape
+
+                # Prepare a PIL Image for ControlNet processing.
+                # TODO(ryand): This is a bit awkward that we have to prepare both torch.Tensor and PIL.Image versions of
+                # the tiles. Ideally, the ControlNet code should be able to work with Tensors.
+                image_tile_pil = Image.fromarray(image_tile_np)
+
+                # Run the ControlNet on the image tile.
+                height, width, _ = image_tile_np.shape
+                # The height and width must be evenly divisible by LATENT_SCALE_FACTOR. This is enforced earlier, but we
+                # validate this assumption here.
+                assert height % LATENT_SCALE_FACTOR == 0
+                assert width % LATENT_SCALE_FACTOR == 0
+                controlnet_data = self.run_controlnet(
+                    image=image_tile_pil,
+                    controlnet_model=controlnet_model,
+                    weight=self.control_weight,
+                    do_classifier_free_guidance=True,
+                    width=width,
+                    height=height,
+                    device=controlnet_model.device,
+                    dtype=controlnet_model.dtype,
+                    control_mode="balanced",
+                    resize_mode="just_resize_simple",
+                )
+
+                num_inference_steps, timesteps, init_timestep, scheduler_step_kwargs = (
+                    DenoiseLatentsInvocation.init_scheduler(
+                        scheduler,
+                        device=unet.device,
+                        steps=self.steps,
+                        denoising_start=self.denoising_start,
+                        denoising_end=self.denoising_end,
+                        seed=seed,
+                    )
+                )
+
+                # TODO(ryand): Think about when/if latents/noise should be moved off of the device to save VRAM.
+                latent_tile = latent_tile.to(device=unet.device, dtype=unet.dtype)
+                noise_tile = noise_tile.to(device=unet.device, dtype=unet.dtype)
+                refined_latent_tile = pipeline.latents_from_embeddings(
+                    latents=latent_tile,
+                    timesteps=timesteps,
+                    init_timestep=init_timestep,
+                    noise=noise_tile,
+                    seed=seed,
+                    mask=None,
+                    masked_latents=None,
+                    gradient_mask=None,
+                    num_inference_steps=num_inference_steps,
+                    scheduler_step_kwargs=scheduler_step_kwargs,
+                    conditioning_data=conditioning_data,
+                    control_data=[controlnet_data],
+                    ip_adapter_data=None,
+                    t2i_adapter_data=None,
+                    callback=lambda x: None,
+                )
+                refined_latent_tiles.append(refined_latent_tile)
+
+        # VAE-decode each refined latent tile independently.
+        refined_image_tiles: list[Image.Image] = []
+        for refined_latent_tile in refined_latent_tiles:
+            refined_image_tile = LatentsToImageInvocation.vae_decode(
+                context=context,
+                vae_info=vae_info,
+                seamless_axes=self.vae.seamless_axes,
+                latents=refined_latent_tile,
+                use_fp32=self.vae_fp32,
+                use_tiling=False,
+            )
+            refined_image_tiles.append(refined_image_tile)
+
+        # TODO(ryand): I copied this from DenoiseLatentsInvocation. I'm not sure if it's actually important.
+        TorchDevice.empty_cache()
+
+        # Merge the refined image tiles back into a single image.
+        refined_image_tiles_np = [np.array(t) for t in refined_image_tiles]
+        merged_image_np = np.zeros(shape=(input_image.height, input_image.width, 3), dtype=np.uint8)
+        # TODO(ryand): Tune the blend_amount. Should this be exposed as a parameter?
+        merge_tiles_with_linear_blending(
+            dst_image=merged_image_np, tiles=tiles, tile_images=refined_image_tiles_np, blend_amount=self.tile_overlap
+        )
+
+        # Save the refined image and return its reference.
+        merged_image_pil = Image.fromarray(merged_image_np)
+        image_dto = context.images.save(image=merged_image_pil)
+
+        return ImageOutput.build(image_dto)

invokeai/app/invocations/upscale.py

@@ -1,4 +1,5 @@
 # Copyright (c) 2022 Kyle Schouviller (https://github.com/kyle0654) & the InvokeAI Team
+from pathlib import Path
 from typing import Literal
 
 import cv2
@@ -9,8 +10,10 @@ from pydantic import ConfigDict
 from invokeai.app.invocations.fields import ImageField
 from invokeai.app.invocations.primitives import ImageOutput
 from invokeai.app.services.shared.invocation_context import InvocationContext
+from invokeai.app.util.download_with_progress import download_with_progress_bar
 from invokeai.backend.image_util.basicsr.rrdbnet_arch import RRDBNet
 from invokeai.backend.image_util.realesrgan.realesrgan import RealESRGAN
+from invokeai.backend.util.devices import TorchDevice
 
 from .baseinvocation import BaseInvocation, invocation
 from .fields import InputField, WithBoard, WithMetadata
@@ -49,6 +52,7 @@ class ESRGANInvocation(BaseInvocation, WithMetadata, WithBoard):
 
         rrdbnet_model = None
         netscale = None
+        esrgan_model_path = None
 
         if self.model_name in [
             "RealESRGAN_x4plus.pth",
@@ -91,25 +95,28 @@ class ESRGANInvocation(BaseInvocation, WithMetadata, WithBoard):
             context.logger.error(msg)
             raise ValueError(msg)
 
-        loadnet = context.models.load_remote_model(
-            source=ESRGAN_MODEL_URLS[self.model_name],
+        esrgan_model_path = Path(context.config.get().models_path, f"core/upscaling/realesrgan/{self.model_name}")
+
+        # Downloads the ESRGAN model if it doesn't already exist
+        download_with_progress_bar(
+            name=self.model_name, url=ESRGAN_MODEL_URLS[self.model_name], dest_path=esrgan_model_path
         )
 
-        with loadnet as loadnet_model:
-            upscaler = RealESRGAN(
-                scale=netscale,
-                loadnet=loadnet_model,
-                model=rrdbnet_model,
-                half=False,
-                tile=self.tile_size,
-            )
+        upscaler = RealESRGAN(
+            scale=netscale,
+            model_path=esrgan_model_path,
+            model=rrdbnet_model,
+            half=False,
+            tile=self.tile_size,
+        )
 
-            # prepare image - Real-ESRGAN uses cv2 internally, and cv2 uses BGR vs RGB for PIL
-            # TODO: This strips the alpha... is that okay?
-            cv2_image = cv2.cvtColor(np.array(image.convert("RGB")), cv2.COLOR_RGB2BGR)
-            upscaled_image = upscaler.upscale(cv2_image)
+        # prepare image - Real-ESRGAN uses cv2 internally, and cv2 uses BGR vs RGB for PIL
+        # TODO: This strips the alpha... is that okay?
+        cv2_image = cv2.cvtColor(np.array(image.convert("RGB")), cv2.COLOR_RGB2BGR)
+        upscaled_image = upscaler.upscale(cv2_image)
+        pil_image = Image.fromarray(cv2.cvtColor(upscaled_image, cv2.COLOR_BGR2RGB)).convert("RGBA")
 
-            pil_image = Image.fromarray(cv2.cvtColor(upscaled_image, cv2.COLOR_BGR2RGB)).convert("RGBA")
+        TorchDevice.empty_cache()
 
         image_dto = context.images.save(image=pil_image)
 

invokeai/app/services/config/config_default.py

@@ -86,7 +86,6 @@ class InvokeAIAppConfig(BaseSettings):
         patchmatch: Enable patchmatch inpaint code.
         models_dir: Path to the models directory.
         convert_cache_dir: Path to the converted models cache directory. When loading a non-diffusers model, it will be converted and store on disk at this location.
-        download_cache_dir: Path to the directory that contains dynamically downloaded models.
         legacy_conf_dir: Path to directory of legacy checkpoint config files.
         db_dir: Path to InvokeAI databases directory.
         outputs_dir: Path to directory for outputs.
@@ -147,8 +146,7 @@ class InvokeAIAppConfig(BaseSettings):
 
     # PATHS
     models_dir:                    Path = Field(default=Path("models"),     description="Path to the models directory.")
-    convert_cache_dir:             Path = Field(default=Path("models/.convert_cache"), description="Path to the converted models cache directory. When loading a non-diffusers model, it will be converted and store on disk at this location.")
-    download_cache_dir:            Path = Field(default=Path("models/.download_cache"), description="Path to the directory that contains dynamically downloaded models.")
+    convert_cache_dir:             Path = Field(default=Path("models/.cache"), description="Path to the converted models cache directory. When loading a non-diffusers model, it will be converted and store on disk at this location.")
     legacy_conf_dir:               Path = Field(default=Path("configs"), description="Path to directory of legacy checkpoint config files.")
     db_dir:                        Path = Field(default=Path("databases"),  description="Path to InvokeAI databases directory.")
     outputs_dir:                   Path = Field(default=Path("outputs"),    description="Path to directory for outputs.")
@@ -305,11 +303,6 @@ class InvokeAIAppConfig(BaseSettings):
         """Path to the converted cache models directory, resolved to an absolute path.."""
         return self._resolve(self.convert_cache_dir)
 
-    @property
-    def download_cache_path(self) -> Path:
-        """Path to the downloaded models directory, resolved to an absolute path.."""
-        return self._resolve(self.download_cache_dir)
-
     @property
     def custom_nodes_path(self) -> Path:
         """Path to the custom nodes directory, resolved to an absolute path.."""

invokeai/app/services/download/__init__.py

@@ -1,17 +1,10 @@
 """Init file for download queue."""
 
-from .download_base import (
-    DownloadJob,
-    DownloadJobStatus,
-    DownloadQueueServiceBase,
-    MultiFileDownloadJob,
-    UnknownJobIDException,
-)
+from .download_base import DownloadJob, DownloadJobStatus, DownloadQueueServiceBase, UnknownJobIDException
 from .download_default import DownloadQueueService, TqdmProgress
 
 __all__ = [
     "DownloadJob",
-    "MultiFileDownloadJob",
     "DownloadQueueServiceBase",
     "DownloadQueueService",
     "TqdmProgress",

invokeai/app/services/download/download_base.py

@@ -5,13 +5,11 @@ from abc import ABC, abstractmethod
 from enum import Enum
 from functools import total_ordering
 from pathlib import Path
-from typing import Any, Callable, List, Optional, Set, Union
+from typing import Any, Callable, List, Optional
 
 from pydantic import BaseModel, Field, PrivateAttr
 from pydantic.networks import AnyHttpUrl
 
-from invokeai.backend.model_manager.metadata import RemoteModelFile
-
 
 class DownloadJobStatus(str, Enum):
     """State of a download job."""
@@ -35,23 +33,30 @@ class ServiceInactiveException(Exception):
     """This exception is raised when user attempts to initiate a download before the service is started."""
 
 
-SingleFileDownloadEventHandler = Callable[["DownloadJob"], None]
-SingleFileDownloadExceptionHandler = Callable[["DownloadJob", Optional[Exception]], None]
-MultiFileDownloadEventHandler = Callable[["MultiFileDownloadJob"], None]
-MultiFileDownloadExceptionHandler = Callable[["MultiFileDownloadJob", Optional[Exception]], None]
-DownloadEventHandler = Union[SingleFileDownloadEventHandler, MultiFileDownloadEventHandler]
-DownloadExceptionHandler = Union[SingleFileDownloadExceptionHandler, MultiFileDownloadExceptionHandler]
+DownloadEventHandler = Callable[["DownloadJob"], None]
+DownloadExceptionHandler = Callable[["DownloadJob", Optional[Exception]], None]
 
 
-class DownloadJobBase(BaseModel):
-    """Base of classes to monitor and control downloads."""
+@total_ordering
+class DownloadJob(BaseModel):
+    """Class to monitor and control a model download request."""
 
+    # required variables to be passed in on creation
+    source: AnyHttpUrl = Field(description="Where to download from. Specific types specified in child classes.")
+    dest: Path = Field(description="Destination of downloaded model on local disk; a directory or file path")
+    access_token: Optional[str] = Field(default=None, description="authorization token for protected resources")
     # automatically assigned on creation
     id: int = Field(description="Numeric ID of this job", default=-1)  # default id is a sentinel
+    priority: int = Field(default=10, description="Queue priority; lower values are higher priority")
 
-    dest: Path = Field(description="Initial destination of downloaded model on local disk; a directory or file path")
-    download_path: Optional[Path] = Field(default=None, description="Final location of downloaded file or directory")
+    # set internally during download process
     status: DownloadJobStatus = Field(default=DownloadJobStatus.WAITING, description="Status of the download")
+    download_path: Optional[Path] = Field(default=None, description="Final location of downloaded file")
+    job_started: Optional[str] = Field(default=None, description="Timestamp for when the download job started")
+    job_ended: Optional[str] = Field(
+        default=None, description="Timestamp for when the download job ende1d (completed or errored)"
+    )
+    content_type: Optional[str] = Field(default=None, description="Content type of downloaded file")
     bytes: int = Field(default=0, description="Bytes downloaded so far")
     total_bytes: int = Field(default=0, description="Total file size (bytes)")
 
@@ -69,6 +74,14 @@ class DownloadJobBase(BaseModel):
     _on_cancelled: Optional[DownloadEventHandler] = PrivateAttr(default=None)
     _on_error: Optional[DownloadExceptionHandler] = PrivateAttr(default=None)
 
+    def __hash__(self) -> int:
+        """Return hash of the string representation of this object, for indexing."""
+        return hash(str(self))
+
+    def __le__(self, other: "DownloadJob") -> bool:
+        """Return True if this job's priority is less than another's."""
+        return self.priority <= other.priority
+
     def cancel(self) -> None:
         """Call to cancel the job."""
         self._cancelled = True
@@ -85,11 +98,6 @@ class DownloadJobBase(BaseModel):
         """Return true if job completed without errors."""
         return self.status == DownloadJobStatus.COMPLETED
 
-    @property
-    def waiting(self) -> bool:
-        """Return true if the job is waiting to run."""
-        return self.status == DownloadJobStatus.WAITING
-
     @property
     def running(self) -> bool:
         """Return true if the job is running."""
@@ -146,37 +154,6 @@ class DownloadJobBase(BaseModel):
         self._on_cancelled = on_cancelled
 
 
-@total_ordering
-class DownloadJob(DownloadJobBase):
-    """Class to monitor and control a model download request."""
-
-    # required variables to be passed in on creation
-    source: AnyHttpUrl = Field(description="Where to download from. Specific types specified in child classes.")
-    access_token: Optional[str] = Field(default=None, description="authorization token for protected resources")
-    priority: int = Field(default=10, description="Queue priority; lower values are higher priority")
-
-    # set internally during download process
-    job_started: Optional[str] = Field(default=None, description="Timestamp for when the download job started")
-    job_ended: Optional[str] = Field(
-        default=None, description="Timestamp for when the download job ende1d (completed or errored)"
-    )
-    content_type: Optional[str] = Field(default=None, description="Content type of downloaded file")
-
-    def __hash__(self) -> int:
-        """Return hash of the string representation of this object, for indexing."""
-        return hash(str(self))
-
-    def __le__(self, other: "DownloadJob") -> bool:
-        """Return True if this job's priority is less than another's."""
-        return self.priority <= other.priority
-
-
-class MultiFileDownloadJob(DownloadJobBase):
-    """Class to monitor and control multifile downloads."""
-
-    download_parts: Set[DownloadJob] = Field(default_factory=set, description="List of download parts.")
-
-
 class DownloadQueueServiceBase(ABC):
     """Multithreaded queue for downloading models via URL."""
 
@@ -224,48 +201,6 @@ class DownloadQueueServiceBase(ABC):
         """
         pass
 
-    @abstractmethod
-    def multifile_download(
-        self,
-        parts: List[RemoteModelFile],
-        dest: Path,
-        access_token: Optional[str] = None,
-        submit_job: bool = True,
-        on_start: Optional[DownloadEventHandler] = None,
-        on_progress: Optional[DownloadEventHandler] = None,
-        on_complete: Optional[DownloadEventHandler] = None,
-        on_cancelled: Optional[DownloadEventHandler] = None,
-        on_error: Optional[DownloadExceptionHandler] = None,
-    ) -> MultiFileDownloadJob:
-        """
-        Create and enqueue a multifile download job.
-
-        :param parts: Set of URL / filename pairs
-        :param dest: Path to download to. See below.
-        :param access_token: Access token to download the indicated files. If not provided,
-         each file's URL may be matched to an access token using the config file matching
-         system.
-        :param submit_job: If true [default] then submit the job for execution. Otherwise,
-         you will need to pass the job to submit_multifile_download().
-        :param on_start, on_progress, on_complete, on_error: Callbacks for the indicated
-         events.
-        :returns: A MultiFileDownloadJob object for monitoring the state of the download.
-
-        The `dest` argument is a Path object pointing to a directory. All downloads
-        with be placed inside this directory. The callbacks will receive the
-        MultiFileDownloadJob.
-        """
-        pass
-
-    @abstractmethod
-    def submit_multifile_download(self, job: MultiFileDownloadJob) -> None:
-        """
-        Enqueue a previously-created multi-file download job.
-
-        :param job: A MultiFileDownloadJob created with multifile_download()
-        """
-        pass
-
     @abstractmethod
     def submit_download_job(
         self,
@@ -317,7 +252,7 @@ class DownloadQueueServiceBase(ABC):
         pass
 
     @abstractmethod
-    def cancel_job(self, job: DownloadJobBase) -> None:
+    def cancel_job(self, job: DownloadJob) -> None:
         """Cancel the job, clearing partial downloads and putting it into ERROR state."""
         pass
 
@@ -327,7 +262,7 @@ class DownloadQueueServiceBase(ABC):
         pass
 
     @abstractmethod
-    def wait_for_job(self, job: DownloadJobBase, timeout: int = 0) -> DownloadJobBase:
+    def wait_for_job(self, job: DownloadJob, timeout: int = 0) -> DownloadJob:
         """Wait until the indicated download job has reached a terminal state.
 
         This will block until the indicated install job has completed,

invokeai/app/services/download/download_default.py

@@ -8,28 +8,23 @@ import time
 import traceback
 from pathlib import Path
 from queue import Empty, PriorityQueue
-from typing import TYPE_CHECKING, Any, Dict, List, Literal, Optional, Set
+from typing import TYPE_CHECKING, Any, Dict, List, Optional, Set
 
 import requests
 from pydantic.networks import AnyHttpUrl
 from requests import HTTPError
 from tqdm import tqdm
 
-from invokeai.app.services.config import InvokeAIAppConfig, get_config
-from invokeai.app.services.events.events_base import EventServiceBase
 from invokeai.app.util.misc import get_iso_timestamp
-from invokeai.backend.model_manager.metadata import RemoteModelFile
 from invokeai.backend.util.logging import InvokeAILogger
 
 from .download_base import (
     DownloadEventHandler,
     DownloadExceptionHandler,
     DownloadJob,
-    DownloadJobBase,
     DownloadJobCancelledException,
     DownloadJobStatus,
     DownloadQueueServiceBase,
-    MultiFileDownloadJob,
     ServiceInactiveException,
     UnknownJobIDException,
 )
@@ -47,24 +42,20 @@ class DownloadQueueService(DownloadQueueServiceBase):
     def __init__(
         self,
         max_parallel_dl: int = 5,
-        app_config: Optional[InvokeAIAppConfig] = None,
         event_bus: Optional["EventServiceBase"] = None,
         requests_session: Optional[requests.sessions.Session] = None,
     ):
         """
         Initialize DownloadQueue.
 
-        :param app_config: InvokeAIAppConfig object
         :param max_parallel_dl: Number of simultaneous downloads allowed [5].
         :param requests_session: Optional requests.sessions.Session object, for unit tests.
         """
-        self._app_config = app_config or get_config()
         self._jobs: Dict[int, DownloadJob] = {}
-        self._download_part2parent: Dict[AnyHttpUrl, MultiFileDownloadJob] = {}
         self._next_job_id = 0
         self._queue: PriorityQueue[DownloadJob] = PriorityQueue()
         self._stop_event = threading.Event()
-        self._job_terminated_event = threading.Event()
+        self._job_completed_event = threading.Event()
         self._worker_pool: Set[threading.Thread] = set()
         self._lock = threading.Lock()
         self._logger = InvokeAILogger.get_logger("DownloadQueueService")
@@ -116,16 +107,18 @@ class DownloadQueueService(DownloadQueueServiceBase):
             raise ServiceInactiveException(
                 "The download service is not currently accepting requests. Please call start() to initialize the service."
             )
-        job.id = self._next_id()
-        job.set_callbacks(
-            on_start=on_start,
-            on_progress=on_progress,
-            on_complete=on_complete,
-            on_cancelled=on_cancelled,
-            on_error=on_error,
-        )
-        self._jobs[job.id] = job
-        self._queue.put(job)
+        with self._lock:
+            job.id = self._next_job_id
+            self._next_job_id += 1
+            job.set_callbacks(
+                on_start=on_start,
+                on_progress=on_progress,
+                on_complete=on_complete,
+                on_cancelled=on_cancelled,
+                on_error=on_error,
+            )
+            self._jobs[job.id] = job
+            self._queue.put(job)
 
     def download(
         self,
@@ -148,7 +141,7 @@ class DownloadQueueService(DownloadQueueServiceBase):
             source=source,
             dest=dest,
             priority=priority,
-            access_token=access_token or self._lookup_access_token(source),
+            access_token=access_token,
         )
         self.submit_download_job(
             job,
@@ -160,63 +153,10 @@ class DownloadQueueService(DownloadQueueServiceBase):
         )
         return job
 
-    def multifile_download(
-        self,
-        parts: List[RemoteModelFile],
-        dest: Path,
-        access_token: Optional[str] = None,
-        submit_job: bool = True,
-        on_start: Optional[DownloadEventHandler] = None,
-        on_progress: Optional[DownloadEventHandler] = None,
-        on_complete: Optional[DownloadEventHandler] = None,
-        on_cancelled: Optional[DownloadEventHandler] = None,
-        on_error: Optional[DownloadExceptionHandler] = None,
-    ) -> MultiFileDownloadJob:
-        mfdj = MultiFileDownloadJob(dest=dest, id=self._next_id())
-        mfdj.set_callbacks(
-            on_start=on_start,
-            on_progress=on_progress,
-            on_complete=on_complete,
-            on_cancelled=on_cancelled,
-            on_error=on_error,
-        )
-
-        for part in parts:
-            url = part.url
-            path = dest / part.path
-            assert path.is_relative_to(dest), "only relative download paths accepted"
-            job = DownloadJob(
-                source=url,
-                dest=path,
-                access_token=access_token,
-            )
-            mfdj.download_parts.add(job)
-            self._download_part2parent[job.source] = mfdj
-        if submit_job:
-            self.submit_multifile_download(mfdj)
-        return mfdj
-
-    def submit_multifile_download(self, job: MultiFileDownloadJob) -> None:
-        for download_job in job.download_parts:
-            self.submit_download_job(
-                download_job,
-                on_start=self._mfd_started,
-                on_progress=self._mfd_progress,
-                on_complete=self._mfd_complete,
-                on_cancelled=self._mfd_cancelled,
-                on_error=self._mfd_error,
-            )
-
     def join(self) -> None:
         """Wait for all jobs to complete."""
         self._queue.join()
 
-    def _next_id(self) -> int:
-        with self._lock:
-            id = self._next_job_id
-            self._next_job_id += 1
-        return id
-
     def list_jobs(self) -> List[DownloadJob]:
         """List all the jobs."""
         return list(self._jobs.values())
@@ -238,14 +178,14 @@ class DownloadQueueService(DownloadQueueServiceBase):
         except KeyError as excp:
             raise UnknownJobIDException("Unrecognized job") from excp
 
-    def cancel_job(self, job: DownloadJobBase) -> None:
+    def cancel_job(self, job: DownloadJob) -> None:
         """
         Cancel the indicated job.
 
         If it is running it will be stopped.
         job.status will be set to DownloadJobStatus.CANCELLED
         """
-        if job.status in [DownloadJobStatus.WAITING, DownloadJobStatus.RUNNING]:
+        with self._lock:
             job.cancel()
 
     def cancel_all_jobs(self) -> None:
@@ -254,12 +194,12 @@ class DownloadQueueService(DownloadQueueServiceBase):
             if not job.in_terminal_state:
                 self.cancel_job(job)
 
-    def wait_for_job(self, job: DownloadJobBase, timeout: int = 0) -> DownloadJobBase:
+    def wait_for_job(self, job: DownloadJob, timeout: int = 0) -> DownloadJob:
         """Block until the indicated job has reached terminal state, or when timeout limit reached."""
         start = time.time()
         while not job.in_terminal_state:
-            if self._job_terminated_event.wait(timeout=0.25):  # in case we miss an event
-                self._job_terminated_event.clear()
+            if self._job_completed_event.wait(timeout=0.25):  # in case we miss an event
+                self._job_completed_event.clear()
             if timeout > 0 and time.time() - start > timeout:
                 raise TimeoutError("Timeout exceeded")
         return job
@@ -288,25 +228,22 @@ class DownloadQueueService(DownloadQueueServiceBase):
                 job.job_started = get_iso_timestamp()
                 self._do_download(job)
                 self._signal_job_complete(job)
-            except DownloadJobCancelledException:
-                self._signal_job_cancelled(job)
-                self._cleanup_cancelled_job(job)
-            except Exception as excp:
+            except (OSError, HTTPError) as excp:
                 job.error_type = excp.__class__.__name__ + f"({str(excp)})"
                 job.error = traceback.format_exc()
                 self._signal_job_error(job, excp)
+            except DownloadJobCancelledException:
+                self._signal_job_cancelled(job)
+                self._cleanup_cancelled_job(job)
+
             finally:
                 job.job_ended = get_iso_timestamp()
-                self._job_terminated_event.set()  # signal a change to terminal state
-                self._download_part2parent.pop(job.source, None)  # if this is a subpart of a multipart job, remove it
-                self._job_terminated_event.set()
+                self._job_completed_event.set()  # signal a change to terminal state
                 self._queue.task_done()
-
         self._logger.debug(f"Download queue worker thread {threading.current_thread().name} exiting.")
 
     def _do_download(self, job: DownloadJob) -> None:
         """Do the actual download."""
-
         url = job.source
         header = {"Authorization": f"Bearer {job.access_token}"} if job.access_token else {}
         open_mode = "wb"
@@ -398,29 +335,38 @@ class DownloadQueueService(DownloadQueueServiceBase):
     def _in_progress_path(self, path: Path) -> Path:
         return path.with_name(path.name + ".downloading")
 
-    def _lookup_access_token(self, source: AnyHttpUrl) -> Optional[str]:
-        # Pull the token from config if it exists and matches the URL
-        token = None
-        for pair in self._app_config.remote_api_tokens or []:
-            if re.search(pair.url_regex, str(source)):
-                token = pair.token
-                break
-        return token
-
     def _signal_job_started(self, job: DownloadJob) -> None:
         job.status = DownloadJobStatus.RUNNING
-        self._execute_cb(job, "on_start")
+        if job.on_start:
+            try:
+                job.on_start(job)
+            except Exception as e:
+                self._logger.error(
+                    f"An error occurred while processing the on_start callback: {traceback.format_exception(e)}"
+                )
         if self._event_bus:
             self._event_bus.emit_download_started(job)
 
     def _signal_job_progress(self, job: DownloadJob) -> None:
-        self._execute_cb(job, "on_progress")
+        if job.on_progress:
+            try:
+                job.on_progress(job)
+            except Exception as e:
+                self._logger.error(
+                    f"An error occurred while processing the on_progress callback: {traceback.format_exception(e)}"
+                )
         if self._event_bus:
             self._event_bus.emit_download_progress(job)
 
     def _signal_job_complete(self, job: DownloadJob) -> None:
         job.status = DownloadJobStatus.COMPLETED
-        self._execute_cb(job, "on_complete")
+        if job.on_complete:
+            try:
+                job.on_complete(job)
+            except Exception as e:
+                self._logger.error(
+                    f"An error occurred while processing the on_complete callback: {traceback.format_exception(e)}"
+                )
         if self._event_bus:
             self._event_bus.emit_download_complete(job)
 
@@ -428,21 +374,26 @@ class DownloadQueueService(DownloadQueueServiceBase):
         if job.status not in [DownloadJobStatus.RUNNING, DownloadJobStatus.WAITING]:
             return
         job.status = DownloadJobStatus.CANCELLED
-        self._execute_cb(job, "on_cancelled")
+        if job.on_cancelled:
+            try:
+                job.on_cancelled(job)
+            except Exception as e:
+                self._logger.error(
+                    f"An error occurred while processing the on_cancelled callback: {traceback.format_exception(e)}"
+                )
         if self._event_bus:
             self._event_bus.emit_download_cancelled(job)
 
-        # if multifile download, then signal the parent
-        if parent_job := self._download_part2parent.get(job.source, None):
-            if not parent_job.in_terminal_state:
-                parent_job.status = DownloadJobStatus.CANCELLED
-                self._execute_cb(parent_job, "on_cancelled")
-
     def _signal_job_error(self, job: DownloadJob, excp: Optional[Exception] = None) -> None:
         job.status = DownloadJobStatus.ERROR
         self._logger.error(f"{str(job.source)}: {traceback.format_exception(excp)}")
-        self._execute_cb(job, "on_error", excp)
-
+        if job.on_error:
+            try:
+                job.on_error(job, excp)
+            except Exception as e:
+                self._logger.error(
+                    f"An error occurred while processing the on_error callback: {traceback.format_exception(e)}"
+                )
         if self._event_bus:
             self._event_bus.emit_download_error(job)
 
@@ -455,97 +406,6 @@ class DownloadQueueService(DownloadQueueServiceBase):
         except OSError as excp:
             self._logger.warning(excp)
 
-    ########################################
-    # callbacks used for multifile downloads
-    ########################################
-    def _mfd_started(self, download_job: DownloadJob) -> None:
-        self._logger.info(f"File download started: {download_job.source}")
-        with self._lock:
-            mf_job = self._download_part2parent[download_job.source]
-            if mf_job.waiting:
-                mf_job.total_bytes = sum(x.total_bytes for x in mf_job.download_parts)
-                mf_job.status = DownloadJobStatus.RUNNING
-                assert download_job.download_path is not None
-                path_relative_to_destdir = download_job.download_path.relative_to(mf_job.dest)
-                mf_job.download_path = (
-                    mf_job.dest / path_relative_to_destdir.parts[0]
-                )  # keep just the first component of the path
-                self._execute_cb(mf_job, "on_start")
-
-    def _mfd_progress(self, download_job: DownloadJob) -> None:
-        with self._lock:
-            mf_job = self._download_part2parent[download_job.source]
-            if mf_job.cancelled:
-                for part in mf_job.download_parts:
-                    self.cancel_job(part)
-            elif mf_job.running:
-                mf_job.total_bytes = sum(x.total_bytes for x in mf_job.download_parts)
-                mf_job.bytes = sum(x.total_bytes for x in mf_job.download_parts)
-                self._execute_cb(mf_job, "on_progress")
-
-    def _mfd_complete(self, download_job: DownloadJob) -> None:
-        self._logger.info(f"Download complete: {download_job.source}")
-        with self._lock:
-            mf_job = self._download_part2parent[download_job.source]
-
-            # are there any more active jobs left in this task?
-            if mf_job.running and all(x.complete for x in mf_job.download_parts):
-                mf_job.status = DownloadJobStatus.COMPLETED
-                self._execute_cb(mf_job, "on_complete")
-
-            # we're done with this sub-job
-            self._job_terminated_event.set()
-
-    def _mfd_cancelled(self, download_job: DownloadJob) -> None:
-        with self._lock:
-            mf_job = self._download_part2parent[download_job.source]
-            assert mf_job is not None
-
-            if not mf_job.in_terminal_state:
-                self._logger.warning(f"Download cancelled: {download_job.source}")
-                mf_job.cancel()
-
-            for s in mf_job.download_parts:
-                self.cancel_job(s)
-
-    def _mfd_error(self, download_job: DownloadJob, excp: Optional[Exception] = None) -> None:
-        with self._lock:
-            mf_job = self._download_part2parent[download_job.source]
-            assert mf_job is not None
-            if not mf_job.in_terminal_state:
-                mf_job.status = download_job.status
-                mf_job.error = download_job.error
-                mf_job.error_type = download_job.error_type
-                self._execute_cb(mf_job, "on_error", excp)
-                self._logger.error(
-                    f"Cancelling {mf_job.dest} due to an error while downloading {download_job.source}: {str(excp)}"
-                )
-                for s in [x for x in mf_job.download_parts if x.running]:
-                    self.cancel_job(s)
-                self._download_part2parent.pop(download_job.source)
-                self._job_terminated_event.set()
-
-    def _execute_cb(
-        self,
-        job: DownloadJob | MultiFileDownloadJob,
-        callback_name: Literal[
-            "on_start",
-            "on_progress",
-            "on_complete",
-            "on_cancelled",
-            "on_error",
-        ],
-        excp: Optional[Exception] = None,
-    ) -> None:
-        if callback := getattr(job, callback_name, None):
-            args = [job, excp] if excp else [job]
-            try:
-                callback(*args)
-            except Exception as e:
-                self._logger.error(
-                    f"An error occurred while processing the {callback_name} callback: {traceback.format_exception(e)}"
-                )
-
 
 def get_pc_name_max(directory: str) -> int:
     if hasattr(os, "pathconf"):

invokeai/app/services/model_install/model_install_base.py

@@ -13,7 +13,7 @@ from invokeai.app.services.events.events_base import EventServiceBase
 from invokeai.app.services.invoker import Invoker
 from invokeai.app.services.model_install.model_install_common import ModelInstallJob, ModelSource
 from invokeai.app.services.model_records import ModelRecordServiceBase
-from invokeai.backend.model_manager import AnyModelConfig
+from invokeai.backend.model_manager.config import AnyModelConfig
 
 
 class ModelInstallServiceBase(ABC):
@@ -243,11 +243,12 @@ class ModelInstallServiceBase(ABC):
         """
 
     @abstractmethod
-    def download_and_cache_model(self, source: str | AnyHttpUrl) -> Path:
+    def download_and_cache(self, source: Union[str, AnyHttpUrl], access_token: Optional[str] = None) -> Path:
         """
         Download the model file located at source to the models cache and return its Path.
 
-        :param source: A string representing a URL or repo_id.
+        :param source: A Url or a string that can be converted into one.
+        :param access_token: Optional access token to access restricted resources.
 
         The model file will be downloaded into the system-wide model cache
         (`models/.cache`) if it isn't already there. Note that the model cache

invokeai/app/services/model_install/model_install_common.py

@@ -8,7 +8,7 @@ from pydantic import BaseModel, Field, PrivateAttr, field_validator
 from pydantic.networks import AnyHttpUrl
 from typing_extensions import Annotated
 
-from invokeai.app.services.download import DownloadJob, MultiFileDownloadJob
+from invokeai.app.services.download import DownloadJob
 from invokeai.backend.model_manager import AnyModelConfig, ModelRepoVariant
 from invokeai.backend.model_manager.config import ModelSourceType
 from invokeai.backend.model_manager.metadata import AnyModelRepoMetadata
@@ -26,6 +26,13 @@ class InstallStatus(str, Enum):
     CANCELLED = "cancelled"  # terminated with an error message
 
 
+class ModelInstallPart(BaseModel):
+    url: AnyHttpUrl
+    path: Path
+    bytes: int = 0
+    total_bytes: int = 0
+
+
 class UnknownInstallJobException(Exception):
     """Raised when the status of an unknown job is requested."""
 
@@ -162,7 +169,6 @@ class ModelInstallJob(BaseModel):
     )
     # internal flags and transitory settings
     _install_tmpdir: Optional[Path] = PrivateAttr(default=None)
-    _multifile_job: Optional[MultiFileDownloadJob] = PrivateAttr(default=None)
     _exception: Optional[Exception] = PrivateAttr(default=None)
 
     def set_error(self, e: Exception) -> None:

invokeai/app/services/model_install/model_install_default.py

@@ -5,22 +5,21 @@ import os
 import re
 import threading
 import time
+from hashlib import sha256
 from pathlib import Path
 from queue import Empty, Queue
 from shutil import copyfile, copytree, move, rmtree
 from tempfile import mkdtemp
-from typing import TYPE_CHECKING, Any, Dict, List, Optional, Tuple, Type, Union
+from typing import TYPE_CHECKING, Any, Dict, List, Optional, Union
 
 import torch
 import yaml
 from huggingface_hub import HfFolder
 from pydantic.networks import AnyHttpUrl
-from pydantic_core import Url
 from requests import Session
 
 from invokeai.app.services.config import InvokeAIAppConfig
-from invokeai.app.services.download import DownloadQueueServiceBase, MultiFileDownloadJob
-from invokeai.app.services.events.events_base import EventServiceBase
+from invokeai.app.services.download import DownloadJob, DownloadQueueServiceBase, TqdmProgress
 from invokeai.app.services.invoker import Invoker
 from invokeai.app.services.model_install.model_install_base import ModelInstallServiceBase
 from invokeai.app.services.model_records import DuplicateModelException, ModelRecordServiceBase
@@ -45,7 +44,6 @@ from invokeai.backend.model_manager.search import ModelSearch
 from invokeai.backend.util import InvokeAILogger
 from invokeai.backend.util.catch_sigint import catch_sigint
 from invokeai.backend.util.devices import TorchDevice
-from invokeai.backend.util.util import slugify
 
 from .model_install_common import (
     MODEL_SOURCE_TO_TYPE_MAP,
@@ -93,7 +91,7 @@ class ModelInstallService(ModelInstallServiceBase):
         self._downloads_changed_event = threading.Event()
         self._install_completed_event = threading.Event()
         self._download_queue = download_queue
-        self._download_cache: Dict[int, ModelInstallJob] = {}
+        self._download_cache: Dict[AnyHttpUrl, ModelInstallJob] = {}
         self._running = False
         self._session = session
         self._install_thread: Optional[threading.Thread] = None
@@ -212,12 +210,33 @@ class ModelInstallService(ModelInstallServiceBase):
         access_token: Optional[str] = None,
         inplace: Optional[bool] = False,
     ) -> ModelInstallJob:
-        """Install a model using pattern matching to infer the type of source."""
-        source_obj = self._guess_source(source)
-        if isinstance(source_obj, LocalModelSource):
-            source_obj.inplace = inplace
-        elif isinstance(source_obj, HFModelSource) or isinstance(source_obj, URLModelSource):
-            source_obj.access_token = access_token
+        variants = "|".join(ModelRepoVariant.__members__.values())
+        hf_repoid_re = f"^([^/:]+/[^/:]+)(?::({variants})?(?::/?([^:]+))?)?$"
+        source_obj: Optional[StringLikeSource] = None
+
+        if Path(source).exists():  # A local file or directory
+            source_obj = LocalModelSource(path=Path(source), inplace=inplace)
+        elif match := re.match(hf_repoid_re, source):
+            source_obj = HFModelSource(
+                repo_id=match.group(1),
+                variant=match.group(2) if match.group(2) else None,  # pass None rather than ''
+                subfolder=Path(match.group(3)) if match.group(3) else None,
+                access_token=access_token,
+            )
+        elif re.match(r"^https?://[^/]+", source):
+            # Pull the token from config if it exists and matches the URL
+            _token = access_token
+            if _token is None:
+                for pair in self.app_config.remote_api_tokens or []:
+                    if re.search(pair.url_regex, source):
+                        _token = pair.token
+                        break
+            source_obj = URLModelSource(
+                url=AnyHttpUrl(source),
+                access_token=_token,
+            )
+        else:
+            raise ValueError(f"Unsupported model source: '{source}'")
         return self.import_model(source_obj, config)
 
     def import_model(self, source: ModelSource, config: Optional[Dict[str, Any]] = None) -> ModelInstallJob:  # noqa D102
@@ -278,9 +297,8 @@ class ModelInstallService(ModelInstallServiceBase):
     def cancel_job(self, job: ModelInstallJob) -> None:
         """Cancel the indicated job."""
         job.cancel()
-        self._logger.warning(f"Cancelling {job.source}")
-        if dj := job._multifile_job:
-            self._download_queue.cancel_job(dj)
+        with self._lock:
+            self._cancel_download_parts(job)
 
     def prune_jobs(self) -> None:
         """Prune all completed and errored jobs."""
@@ -328,7 +346,7 @@ class ModelInstallService(ModelInstallServiceBase):
                     legacy_config_path = stanza.get("config")
                     if legacy_config_path:
                         # In v3, these paths were relative to the root. Migrate them to be relative to the legacy_conf_dir.
-                        legacy_config_path = self._app_config.root_path / legacy_config_path
+                        legacy_config_path: Path = self._app_config.root_path / legacy_config_path
                         if legacy_config_path.is_relative_to(self._app_config.legacy_conf_path):
                             legacy_config_path = legacy_config_path.relative_to(self._app_config.legacy_conf_path)
                         config["config_path"] = str(legacy_config_path)
@@ -368,92 +386,38 @@ class ModelInstallService(ModelInstallServiceBase):
             rmtree(model_path)
         self.unregister(key)
 
-    @classmethod
-    def _download_cache_path(cls, source: Union[str, AnyHttpUrl], app_config: InvokeAIAppConfig) -> Path:
-        escaped_source = slugify(str(source))
-        return app_config.download_cache_path / escaped_source
-
-    def download_and_cache_model(
+    def download_and_cache(
         self,
-        source: str | AnyHttpUrl,
+        source: Union[str, AnyHttpUrl],
+        access_token: Optional[str] = None,
+        timeout: int = 0,
     ) -> Path:
         """Download the model file located at source to the models cache and return its Path."""
-        model_path = self._download_cache_path(str(source), self._app_config)
+        model_hash = sha256(str(source).encode("utf-8")).hexdigest()[0:32]
+        model_path = self._app_config.convert_cache_path / model_hash
 
-        # We expect the cache directory to contain one and only one downloaded file or directory.
+        # We expect the cache directory to contain one and only one downloaded file.
         # We don't know the file's name in advance, as it is set by the download
         # content-disposition header.
         if model_path.exists():
-            contents: List[Path] = list(model_path.iterdir())
+            contents = [x for x in model_path.iterdir() if x.is_file()]
             if len(contents) > 0:
                 return contents[0]
 
         model_path.mkdir(parents=True, exist_ok=True)
-        model_source = self._guess_source(str(source))
-        remote_files, _ = self._remote_files_from_source(model_source)
-        job = self._multifile_download(
+        job = self._download_queue.download(
+            source=AnyHttpUrl(str(source)),
             dest=model_path,
-            remote_files=remote_files,
-            subfolder=model_source.subfolder if isinstance(model_source, HFModelSource) else None,
+            access_token=access_token,
+            on_progress=TqdmProgress().update,
         )
-        files_string = "file" if len(remote_files) == 1 else "files"
-        self._logger.info(f"Queuing model download: {source} ({len(remote_files)} {files_string})")
-        self._download_queue.wait_for_job(job)
+        self._download_queue.wait_for_job(job, timeout)
         if job.complete:
             assert job.download_path is not None
             return job.download_path
         else:
             raise Exception(job.error)
 
-    def _remote_files_from_source(
-        self, source: ModelSource
-    ) -> Tuple[List[RemoteModelFile], Optional[AnyModelRepoMetadata]]:
-        metadata = None
-        if isinstance(source, HFModelSource):
-            metadata = HuggingFaceMetadataFetch(self._session).from_id(source.repo_id, source.variant)
-            assert isinstance(metadata, ModelMetadataWithFiles)
-            return metadata.download_urls(
-                variant=source.variant or self._guess_variant(),
-                subfolder=source.subfolder,
-                session=self._session,
-            ), metadata
-
-        if isinstance(source, URLModelSource):
-            try:
-                fetcher = self.get_fetcher_from_url(str(source.url))
-                kwargs: dict[str, Any] = {"session": self._session}
-                metadata = fetcher(**kwargs).from_url(source.url)
-                assert isinstance(metadata, ModelMetadataWithFiles)
-                return metadata.download_urls(session=self._session), metadata
-            except ValueError:
-                pass
-
-            return [RemoteModelFile(url=source.url, path=Path("."), size=0)], None
-
-        raise Exception(f"No files associated with {source}")
-
-    def _guess_source(self, source: str) -> ModelSource:
-        """Turn a source string into a ModelSource object."""
-        variants = "|".join(ModelRepoVariant.__members__.values())
-        hf_repoid_re = f"^([^/:]+/[^/:]+)(?::({variants})?(?::/?([^:]+))?)?$"
-        source_obj: Optional[StringLikeSource] = None
-
-        if Path(source).exists():  # A local file or directory
-            source_obj = LocalModelSource(path=Path(source))
-        elif match := re.match(hf_repoid_re, source):
-            source_obj = HFModelSource(
-                repo_id=match.group(1),
-                variant=ModelRepoVariant(match.group(2)) if match.group(2) else None,  # pass None rather than ''
-                subfolder=Path(match.group(3)) if match.group(3) else None,
-            )
-        elif re.match(r"^https?://[^/]+", source):
-            source_obj = URLModelSource(
-                url=Url(source),
-            )
-        else:
-            raise ValueError(f"Unsupported model source: '{source}'")
-        return source_obj
-
     # --------------------------------------------------------------------------------------------
     # Internal functions that manage the installer threads
     # --------------------------------------------------------------------------------------------
@@ -514,19 +478,16 @@ class ModelInstallService(ModelInstallServiceBase):
         job.config_out = self.record_store.get_model(key)
         self._signal_job_completed(job)
 
-    def _set_error(self, install_job: ModelInstallJob, excp: Exception) -> None:
-        multifile_download_job = install_job._multifile_job
-        if multifile_download_job and any(
-            x.content_type is not None and "text/html" in x.content_type for x in multifile_download_job.download_parts
-        ):
-            install_job.set_error(
+    def _set_error(self, job: ModelInstallJob, excp: Exception) -> None:
+        if any(x.content_type is not None and "text/html" in x.content_type for x in job.download_parts):
+            job.set_error(
                 InvalidModelConfigException(
-                    f"At least one file in {install_job.local_path} is an HTML page, not a model. This can happen when an access token is required to download."
+                    f"At least one file in {job.local_path} is an HTML page, not a model. This can happen when an access token is required to download."
                 )
             )
         else:
-            install_job.set_error(excp)
-        self._signal_job_errored(install_job)
+            job.set_error(excp)
+        self._signal_job_errored(job)
 
     # --------------------------------------------------------------------------------------------
     # Internal functions that manage the models directory
@@ -552,6 +513,7 @@ class ModelInstallService(ModelInstallServiceBase):
         This is typically only used during testing with a new DB or when using the memory DB, because those are the
         only situations in which we may have orphaned models in the models directory.
         """
+
         installed_model_paths = {
             (self._app_config.models_path / x.path).resolve() for x in self.record_store.all_models()
         }
@@ -563,13 +525,8 @@ class ModelInstallService(ModelInstallServiceBase):
             if resolved_path in installed_model_paths:
                 return True
             # Skip core models entirely - these aren't registered with the model manager.
-            for special_directory in [
-                self.app_config.models_path / "core",
-                self.app_config.convert_cache_dir,
-                self.app_config.download_cache_dir,
-            ]:
-                if resolved_path.is_relative_to(special_directory):
-                    return False
+            if str(resolved_path).startswith(str(self.app_config.models_path / "core")):
+                return False
             try:
                 model_id = self.register_path(model_path)
                 self._logger.info(f"Registered {model_path.name} with id {model_id}")
@@ -684,15 +641,20 @@ class ModelInstallService(ModelInstallServiceBase):
             inplace=source.inplace or False,
         )
 
-    def _import_from_hf(
-        self,
-        source: HFModelSource,
-        config: Optional[Dict[str, Any]] = None,
-    ) -> ModelInstallJob:
+    def _import_from_hf(self, source: HFModelSource, config: Optional[Dict[str, Any]]) -> ModelInstallJob:
         # Add user's cached access token to HuggingFace requests
-        if source.access_token is None:
-            source.access_token = HfFolder.get_token()
-        remote_files, metadata = self._remote_files_from_source(source)
+        source.access_token = source.access_token or HfFolder.get_token()
+        if not source.access_token:
+            self._logger.info("No HuggingFace access token present; some models may not be downloadable.")
+
+        metadata = HuggingFaceMetadataFetch(self._session).from_id(source.repo_id, source.variant)
+        assert isinstance(metadata, ModelMetadataWithFiles)
+        remote_files = metadata.download_urls(
+            variant=source.variant or self._guess_variant(),
+            subfolder=source.subfolder,
+            session=self._session,
+        )
+
         return self._import_remote_model(
             source=source,
             config=config,
@@ -700,12 +662,22 @@ class ModelInstallService(ModelInstallServiceBase):
             metadata=metadata,
         )
 
-    def _import_from_url(
-        self,
-        source: URLModelSource,
-        config: Optional[Dict[str, Any]],
-    ) -> ModelInstallJob:
-        remote_files, metadata = self._remote_files_from_source(source)
+    def _import_from_url(self, source: URLModelSource, config: Optional[Dict[str, Any]]) -> ModelInstallJob:
+        # URLs from HuggingFace will be handled specially
+        metadata = None
+        fetcher = None
+        try:
+            fetcher = self.get_fetcher_from_url(str(source.url))
+        except ValueError:
+            pass
+        kwargs: dict[str, Any] = {"session": self._session}
+        if fetcher is not None:
+            metadata = fetcher(**kwargs).from_url(source.url)
+        self._logger.debug(f"metadata={metadata}")
+        if metadata and isinstance(metadata, ModelMetadataWithFiles):
+            remote_files = metadata.download_urls(session=self._session)
+        else:
+            remote_files = [RemoteModelFile(url=source.url, path=Path("."), size=0)]
         return self._import_remote_model(
             source=source,
             config=config,
@@ -720,9 +692,12 @@ class ModelInstallService(ModelInstallServiceBase):
         metadata: Optional[AnyModelRepoMetadata],
         config: Optional[Dict[str, Any]],
     ) -> ModelInstallJob:
+        # TODO: Replace with tempfile.tmpdir() when multithreading is cleaned up.
+        # Currently the tmpdir isn't automatically removed at exit because it is
+        # being held in a daemon thread.
         if len(remote_files) == 0:
             raise ValueError(f"{source}: No downloadable files found")
-        destdir = Path(
+        tmpdir = Path(
             mkdtemp(
                 dir=self._app_config.models_path,
                 prefix=TMPDIR_PREFIX,
@@ -733,28 +708,55 @@ class ModelInstallService(ModelInstallServiceBase):
             source=source,
             config_in=config or {},
             source_metadata=metadata,
-            local_path=destdir,  # local path may change once the download has started due to content-disposition handling
+            local_path=tmpdir,  # local path may change once the download has started due to content-disposition handling
             bytes=0,
             total_bytes=0,
         )
-        # remember the temporary directory for later removal
-        install_job._install_tmpdir = destdir
-        install_job.total_bytes = sum((x.size or 0) for x in remote_files)
+        # In the event that there is a subfolder specified in the source,
+        # we need to remove it from the destination path in order to avoid
+        # creating unwanted subfolders
+        if isinstance(source, HFModelSource) and source.subfolder:
+            root = Path(remote_files[0].path.parts[0])
+            subfolder = root / source.subfolder
+        else:
+            root = Path(".")
+            subfolder = Path(".")
 
-        multifile_job = self._multifile_download(
-            remote_files=remote_files,
-            dest=destdir,
-            subfolder=source.subfolder if isinstance(source, HFModelSource) else None,
-            access_token=source.access_token,
-            submit_job=False,  # Important! Don't submit the job until we have set our _download_cache dict
-        )
-        self._download_cache[multifile_job.id] = install_job
-        install_job._multifile_job = multifile_job
+        # we remember the path up to the top of the tmpdir so that it may be
+        # removed safely at the end of the install process.
+        install_job._install_tmpdir = tmpdir
+        assert install_job.total_bytes is not None  # to avoid type checking complaints in the loop below
 
-        files_string = "file" if len(remote_files) == 1 else "files"
-        self._logger.info(f"Queueing model install: {source} ({len(remote_files)} {files_string})")
+        files_string = "file" if len(remote_files) == 1 else "file"
+        self._logger.info(f"Queuing model install: {source} ({len(remote_files)} {files_string})")
         self._logger.debug(f"remote_files={remote_files}")
-        self._download_queue.submit_multifile_download(multifile_job)
+        for model_file in remote_files:
+            url = model_file.url
+            path = root / model_file.path.relative_to(subfolder)
+            self._logger.debug(f"Downloading {url} => {path}")
+            install_job.total_bytes += model_file.size
+            assert hasattr(source, "access_token")
+            dest = tmpdir / path.parent
+            dest.mkdir(parents=True, exist_ok=True)
+            download_job = DownloadJob(
+                source=url,
+                dest=dest,
+                access_token=source.access_token,
+            )
+            self._download_cache[download_job.source] = install_job  # matches a download job to an install job
+            install_job.download_parts.add(download_job)
+
+        # only start the jobs once install_job.download_parts is fully populated
+        for download_job in install_job.download_parts:
+            self._download_queue.submit_download_job(
+                download_job,
+                on_start=self._download_started_callback,
+                on_progress=self._download_progress_callback,
+                on_complete=self._download_complete_callback,
+                on_error=self._download_error_callback,
+                on_cancelled=self._download_cancelled_callback,
+            )
+
         return install_job
 
     def _stat_size(self, path: Path) -> int:
@@ -766,104 +768,87 @@ class ModelInstallService(ModelInstallServiceBase):
                 size += sum(self._stat_size(Path(root, x)) for x in files)
         return size
 
-    def _multifile_download(
-        self,
-        remote_files: List[RemoteModelFile],
-        dest: Path,
-        subfolder: Optional[Path] = None,
-        access_token: Optional[str] = None,
-        submit_job: bool = True,
-    ) -> MultiFileDownloadJob:
-        # HuggingFace repo subfolders are a little tricky. If the name of the model is "sdxl-turbo", and
-        # we are installing the "vae" subfolder, we do not want to create an additional folder level, such
-        # as "sdxl-turbo/vae", nor do we want to put the contents of the vae folder directly into "sdxl-turbo".
-        # So what we do is to synthesize a folder named "sdxl-turbo_vae" here.
-        if subfolder:
-            top = Path(remote_files[0].path.parts[0])  # e.g. "sdxl-turbo/"
-            path_to_remove = top / subfolder.parts[-1]  # sdxl-turbo/vae/
-            path_to_add = Path(f"{top}_{subfolder}")
-        else:
-            path_to_remove = Path(".")
-            path_to_add = Path(".")
-
-        parts: List[RemoteModelFile] = []
-        for model_file in remote_files:
-            assert model_file.size is not None
-            parts.append(
-                RemoteModelFile(
-                    url=model_file.url,  # if a subfolder, then sdxl-turbo_vae/config.json
-                    path=path_to_add / model_file.path.relative_to(path_to_remove),
-                )
-            )
-
-        return self._download_queue.multifile_download(
-            parts=parts,
-            dest=dest,
-            access_token=access_token,
-            submit_job=submit_job,
-            on_start=self._download_started_callback,
-            on_progress=self._download_progress_callback,
-            on_complete=self._download_complete_callback,
-            on_error=self._download_error_callback,
-            on_cancelled=self._download_cancelled_callback,
-        )
-
     # ------------------------------------------------------------------
     # Callbacks are executed by the download queue in a separate thread
     # ------------------------------------------------------------------
-    def _download_started_callback(self, download_job: MultiFileDownloadJob) -> None:
+    def _download_started_callback(self, download_job: DownloadJob) -> None:
+        self._logger.info(f"Model download started: {download_job.source}")
         with self._lock:
-            if install_job := self._download_cache.get(download_job.id, None):
-                install_job.status = InstallStatus.DOWNLOADING
+            install_job = self._download_cache[download_job.source]
+            install_job.status = InstallStatus.DOWNLOADING
 
-                if install_job.local_path == install_job._install_tmpdir:  # first time
-                    assert download_job.download_path
-                    install_job.local_path = download_job.download_path
-                install_job.download_parts = download_job.download_parts
-                install_job.bytes = sum(x.bytes for x in download_job.download_parts)
-                install_job.total_bytes = download_job.total_bytes
+            assert download_job.download_path
+            if install_job.local_path == install_job._install_tmpdir:
+                partial_path = download_job.download_path.relative_to(install_job._install_tmpdir)
+                dest_name = partial_path.parts[0]
+                install_job.local_path = install_job._install_tmpdir / dest_name
+
+            # Update the total bytes count for remote sources.
+            if not install_job.total_bytes:
+                install_job.total_bytes = sum(x.total_bytes for x in install_job.download_parts)
+
+    def _download_progress_callback(self, download_job: DownloadJob) -> None:
+        with self._lock:
+            install_job = self._download_cache[download_job.source]
+            if install_job.cancelled:  # This catches the case in which the caller directly calls job.cancel()
+                self._cancel_download_parts(install_job)
+            else:
+                # update sizes
+                install_job.bytes = sum(x.bytes for x in install_job.download_parts)
                 self._signal_job_downloading(install_job)
 
-    def _download_progress_callback(self, download_job: MultiFileDownloadJob) -> None:
+    def _download_complete_callback(self, download_job: DownloadJob) -> None:
+        self._logger.info(f"Model download complete: {download_job.source}")
         with self._lock:
-            if install_job := self._download_cache.get(download_job.id, None):
-                if install_job.cancelled:  # This catches the case in which the caller directly calls job.cancel()
-                    self._download_queue.cancel_job(download_job)
-                else:
-                    # update sizes
-                    install_job.bytes = sum(x.bytes for x in download_job.download_parts)
-                    install_job.total_bytes = sum(x.total_bytes for x in download_job.download_parts)
-                    self._signal_job_downloading(install_job)
+            install_job = self._download_cache[download_job.source]
 
-    def _download_complete_callback(self, download_job: MultiFileDownloadJob) -> None:
-        with self._lock:
-            if install_job := self._download_cache.pop(download_job.id, None):
+            # are there any more active jobs left in this task?
+            if install_job.downloading and all(x.complete for x in install_job.download_parts):
                 self._signal_job_downloads_done(install_job)
-                self._put_in_queue(install_job)  # this starts the installation and registration
+                self._put_in_queue(install_job)
 
-                # Let other threads know that the number of downloads has changed
-                self._downloads_changed_event.set()
+            # Let other threads know that the number of downloads has changed
+            self._download_cache.pop(download_job.source, None)
+            self._downloads_changed_event.set()
 
-    def _download_error_callback(self, download_job: MultiFileDownloadJob, excp: Optional[Exception] = None) -> None:
+    def _download_error_callback(self, download_job: DownloadJob, excp: Optional[Exception] = None) -> None:
         with self._lock:
-            if install_job := self._download_cache.pop(download_job.id, None):
-                assert excp is not None
-                install_job.set_error(excp)
-                self._download_queue.cancel_job(download_job)
+            install_job = self._download_cache.pop(download_job.source, None)
+            assert install_job is not None
+            assert excp is not None
+            install_job.set_error(excp)
+            self._logger.error(
+                f"Cancelling {install_job.source} due to an error while downloading {download_job.source}: {str(excp)}"
+            )
+            self._cancel_download_parts(install_job)
 
-                # Let other threads know that the number of downloads has changed
-                self._downloads_changed_event.set()
+            # Let other threads know that the number of downloads has changed
+            self._downloads_changed_event.set()
 
-    def _download_cancelled_callback(self, download_job: MultiFileDownloadJob) -> None:
+    def _download_cancelled_callback(self, download_job: DownloadJob) -> None:
         with self._lock:
-            if install_job := self._download_cache.pop(download_job.id, None):
-                self._downloads_changed_event.set()
-                # if install job has already registered an error, then do not replace its status with cancelled
-                if not install_job.errored:
-                    install_job.cancel()
+            install_job = self._download_cache.pop(download_job.source, None)
+            if not install_job:
+                return
+            self._downloads_changed_event.set()
+            self._logger.warning(f"Model download canceled: {download_job.source}")
+            # if install job has already registered an error, then do not replace its status with cancelled
+            if not install_job.errored:
+                install_job.cancel()
+            self._cancel_download_parts(install_job)
 
-                # Let other threads know that the number of downloads has changed
-                self._downloads_changed_event.set()
+            # Let other threads know that the number of downloads has changed
+            self._downloads_changed_event.set()
+
+    def _cancel_download_parts(self, install_job: ModelInstallJob) -> None:
+        # on multipart downloads, _cancel_components() will get called repeatedly from the download callbacks
+        # do not lock here because it gets called within a locked context
+        for s in install_job.download_parts:
+            self._download_queue.cancel_job(s)
+
+        if all(x.in_terminal_state for x in install_job.download_parts):
+            # When all parts have reached their terminal state, we finalize the job to clean up the temporary directory and other resources
+            self._put_in_queue(install_job)
 
     # ------------------------------------------------------------------------------------------------
     # Internal methods that put events on the event bus
@@ -876,9 +861,6 @@ class ModelInstallService(ModelInstallServiceBase):
 
     def _signal_job_downloading(self, job: ModelInstallJob) -> None:
         if self._event_bus:
-            assert job._multifile_job is not None
-            assert job.bytes is not None
-            assert job.total_bytes is not None
             self._event_bus.emit_model_install_download_progress(job)
 
     def _signal_job_downloads_done(self, job: ModelInstallJob) -> None:
@@ -893,8 +875,6 @@ class ModelInstallService(ModelInstallServiceBase):
         self._logger.info(f"Model install complete: {job.source}")
         self._logger.debug(f"{job.local_path} registered key {job.config_out.key}")
         if self._event_bus:
-            assert job.local_path is not None
-            assert job.config_out is not None
             self._event_bus.emit_model_install_complete(job)
 
     def _signal_job_errored(self, job: ModelInstallJob) -> None:
@@ -910,13 +890,7 @@ class ModelInstallService(ModelInstallServiceBase):
             self._event_bus.emit_model_install_cancelled(job)
 
     @staticmethod
-    def get_fetcher_from_url(url: str) -> Type[ModelMetadataFetchBase]:
-        """
-        Return a metadata fetcher appropriate for provided url.
-
-        This used to be more useful, but the number of supported model
-        sources has been reduced to HuggingFace alone.
-        """
+    def get_fetcher_from_url(url: str) -> ModelMetadataFetchBase:
         if re.match(r"^https?://huggingface.co/[^/]+/[^/]+$", url.lower()):
             return HuggingFaceMetadataFetch
         raise ValueError(f"Unsupported model source: '{url}'")

invokeai/app/services/model_load/model_load_base.py

@@ -2,11 +2,10 @@
 """Base class for model loader."""
 
 from abc import ABC, abstractmethod
-from pathlib import Path
-from typing import Callable, Optional
+from typing import Optional
 
 from invokeai.backend.model_manager import AnyModel, AnyModelConfig, SubModelType
-from invokeai.backend.model_manager.load import LoadedModel, LoadedModelWithoutConfig
+from invokeai.backend.model_manager.load import LoadedModel
 from invokeai.backend.model_manager.load.convert_cache import ModelConvertCacheBase
 from invokeai.backend.model_manager.load.model_cache.model_cache_base import ModelCacheBase
 
@@ -32,26 +31,3 @@ class ModelLoadServiceBase(ABC):
     @abstractmethod
     def convert_cache(self) -> ModelConvertCacheBase:
         """Return the checkpoint convert cache used by this loader."""
-
-    @abstractmethod
-    def load_model_from_path(
-        self, model_path: Path, loader: Optional[Callable[[Path], AnyModel]] = None
-    ) -> LoadedModelWithoutConfig:
-        """
-        Load the model file or directory located at the indicated Path.
-
-        This will load an arbitrary model file into the RAM cache. If the optional loader
-        argument is provided, the loader will be invoked to load the model into
-        memory. Otherwise the method will call safetensors.torch.load_file() or
-        torch.load() as appropriate to the file suffix.
-
-        Be aware that this returns a LoadedModelWithoutConfig object, which is the same as
-        LoadedModel, but without the config attribute.
-
-        Args:
-          model_path: A pathlib.Path to a checkpoint-style models file
-          loader: A Callable that expects a Path and returns a Dict[str, Tensor]
-
-        Returns:
-          A LoadedModel object.
-        """

invokeai/app/services/model_load/model_load_default.py

@@ -1,26 +1,18 @@
 # Copyright (c) 2024 Lincoln D. Stein and the InvokeAI Team
 """Implementation of model loader service."""
 
-from pathlib import Path
-from typing import Callable, Optional, Type
-
-from picklescan.scanner import scan_file_path
-from safetensors.torch import load_file as safetensors_load_file
-from torch import load as torch_load
+from typing import Optional, Type
 
 from invokeai.app.services.config import InvokeAIAppConfig
 from invokeai.app.services.invoker import Invoker
 from invokeai.backend.model_manager import AnyModel, AnyModelConfig, SubModelType
 from invokeai.backend.model_manager.load import (
     LoadedModel,
-    LoadedModelWithoutConfig,
     ModelLoaderRegistry,
     ModelLoaderRegistryBase,
 )
 from invokeai.backend.model_manager.load.convert_cache import ModelConvertCacheBase
 from invokeai.backend.model_manager.load.model_cache.model_cache_base import ModelCacheBase
-from invokeai.backend.model_manager.load.model_loaders.generic_diffusers import GenericDiffusersLoader
-from invokeai.backend.util.devices import TorchDevice
 from invokeai.backend.util.logging import InvokeAILogger
 
 from .model_load_base import ModelLoadServiceBase
@@ -83,41 +75,3 @@ class ModelLoadService(ModelLoadServiceBase):
             self._invoker.services.events.emit_model_load_complete(model_config, submodel_type)
 
         return loaded_model
-
-    def load_model_from_path(
-        self, model_path: Path, loader: Optional[Callable[[Path], AnyModel]] = None
-    ) -> LoadedModelWithoutConfig:
-        cache_key = str(model_path)
-        ram_cache = self.ram_cache
-        try:
-            return LoadedModelWithoutConfig(_locker=ram_cache.get(key=cache_key))
-        except IndexError:
-            pass
-
-        def torch_load_file(checkpoint: Path) -> AnyModel:
-            scan_result = scan_file_path(checkpoint)
-            if scan_result.infected_files != 0:
-                raise Exception("The model at {checkpoint} is potentially infected by malware. Aborting load.")
-            result = torch_load(checkpoint, map_location="cpu")
-            return result
-
-        def diffusers_load_directory(directory: Path) -> AnyModel:
-            load_class = GenericDiffusersLoader(
-                app_config=self._app_config,
-                logger=self._logger,
-                ram_cache=self._ram_cache,
-                convert_cache=self.convert_cache,
-            ).get_hf_load_class(directory)
-            return load_class.from_pretrained(model_path, torch_dtype=TorchDevice.choose_torch_dtype())
-
-        loader = loader or (
-            diffusers_load_directory
-            if model_path.is_dir()
-            else torch_load_file
-            if model_path.suffix.endswith((".ckpt", ".pt", ".pth", ".bin"))
-            else lambda path: safetensors_load_file(path, device="cpu")
-        )
-        assert loader is not None
-        raw_model = loader(model_path)
-        ram_cache.put(key=cache_key, model=raw_model)
-        return LoadedModelWithoutConfig(_locker=ram_cache.get(key=cache_key))

invokeai/app/services/model_records/model_records_base.py

@@ -12,13 +12,15 @@ from pydantic import BaseModel, Field
 
 from invokeai.app.services.shared.pagination import PaginatedResults
 from invokeai.app.util.model_exclude_null import BaseModelExcludeNull
-from invokeai.backend.model_manager.config import (
+from invokeai.backend.model_manager import (
     AnyModelConfig,
     BaseModelType,
-    ControlAdapterDefaultSettings,
-    MainModelDefaultSettings,
     ModelFormat,
     ModelType,
+)
+from invokeai.backend.model_manager.config import (
+    ControlAdapterDefaultSettings,
+    MainModelDefaultSettings,
     ModelVariantType,
     SchedulerPredictionType,
 )

invokeai/app/services/shared/invocation_context.py

@@ -3,7 +3,6 @@ from pathlib import Path
 from typing import TYPE_CHECKING, Callable, Optional, Union
 
 from PIL.Image import Image
-from pydantic.networks import AnyHttpUrl
 from torch import Tensor
 
 from invokeai.app.invocations.constants import IMAGE_MODES
@@ -15,15 +14,8 @@ from invokeai.app.services.images.images_common import ImageDTO
 from invokeai.app.services.invocation_services import InvocationServices
 from invokeai.app.services.model_records.model_records_base import UnknownModelException
 from invokeai.app.util.step_callback import stable_diffusion_step_callback
-from invokeai.backend.model_manager.config import (
-    AnyModel,
-    AnyModelConfig,
-    BaseModelType,
-    ModelFormat,
-    ModelType,
-    SubModelType,
-)
-from invokeai.backend.model_manager.load.load_base import LoadedModel, LoadedModelWithoutConfig
+from invokeai.backend.model_manager.config import AnyModelConfig, BaseModelType, ModelFormat, ModelType, SubModelType
+from invokeai.backend.model_manager.load.load_base import LoadedModel
 from invokeai.backend.stable_diffusion.diffusers_pipeline import PipelineIntermediateState
 from invokeai.backend.stable_diffusion.diffusion.conditioning_data import ConditioningFieldData
 
@@ -328,10 +320,8 @@ class ConditioningInterface(InvocationContextInterface):
 
 
 class ModelsInterface(InvocationContextInterface):
-    """Common API for loading, downloading and managing models."""
-
     def exists(self, identifier: Union[str, "ModelIdentifierField"]) -> bool:
-        """Check if a model exists.
+        """Checks if a model exists.
 
         Args:
             identifier: The key or ModelField representing the model.
@@ -341,13 +331,13 @@ class ModelsInterface(InvocationContextInterface):
         """
         if isinstance(identifier, str):
             return self._services.model_manager.store.exists(identifier)
-        else:
-            return self._services.model_manager.store.exists(identifier.key)
+
+        return self._services.model_manager.store.exists(identifier.key)
 
     def load(
         self, identifier: Union[str, "ModelIdentifierField"], submodel_type: Optional[SubModelType] = None
     ) -> LoadedModel:
-        """Load a model.
+        """Loads a model.
 
         Args:
             identifier: The key or ModelField representing the model.
@@ -371,7 +361,7 @@ class ModelsInterface(InvocationContextInterface):
     def load_by_attrs(
         self, name: str, base: BaseModelType, type: ModelType, submodel_type: Optional[SubModelType] = None
     ) -> LoadedModel:
-        """Load a model by its attributes.
+        """Loads a model by its attributes.
 
         Args:
             name: Name of the model.
@@ -394,7 +384,7 @@ class ModelsInterface(InvocationContextInterface):
         return self._services.model_manager.load.load_model(configs[0], submodel_type)
 
     def get_config(self, identifier: Union[str, "ModelIdentifierField"]) -> AnyModelConfig:
-        """Get a model's config.
+        """Gets a model's config.
 
         Args:
             identifier: The key or ModelField representing the model.
@@ -404,11 +394,11 @@ class ModelsInterface(InvocationContextInterface):
         """
         if isinstance(identifier, str):
             return self._services.model_manager.store.get_model(identifier)
-        else:
-            return self._services.model_manager.store.get_model(identifier.key)
+
+        return self._services.model_manager.store.get_model(identifier.key)
 
     def search_by_path(self, path: Path) -> list[AnyModelConfig]:
-        """Search for models by path.
+        """Searches for models by path.
 
         Args:
             path: The path to search for.
@@ -425,7 +415,7 @@ class ModelsInterface(InvocationContextInterface):
         type: Optional[ModelType] = None,
         format: Optional[ModelFormat] = None,
     ) -> list[AnyModelConfig]:
-        """Search for models by attributes.
+        """Searches for models by attributes.
 
         Args:
             name: The name to search for (exact match).
@@ -444,72 +434,6 @@ class ModelsInterface(InvocationContextInterface):
             model_format=format,
         )
 
-    def download_and_cache_model(
-        self,
-        source: str | AnyHttpUrl,
-    ) -> Path:
-        """
-        Download the model file located at source to the models cache and return its Path.
-
-        This can be used to single-file install models and other resources of arbitrary types
-        which should not get registered with the database. If the model is already
-        installed, the cached path will be returned. Otherwise it will be downloaded.
-
-        Args:
-            source: A URL that points to the model, or a huggingface repo_id.
-
-        Returns:
-            Path to the downloaded model
-        """
-        return self._services.model_manager.install.download_and_cache_model(source=source)
-
-    def load_local_model(
-        self,
-        model_path: Path,
-        loader: Optional[Callable[[Path], AnyModel]] = None,
-    ) -> LoadedModelWithoutConfig:
-        """
-        Load the model file located at the indicated path
-
-        If a loader callable is provided, it will be invoked to load the model. Otherwise,
-        `safetensors.torch.load_file()` or `torch.load()` will be called to load the model.
-
-        Be aware that the LoadedModelWithoutConfig object has no `config` attribute
-
-        Args:
-            path: A model Path
-            loader: A Callable that expects a Path and returns a dict[str|int, Any]
-
-        Returns:
-            A LoadedModelWithoutConfig object.
-        """
-        return self._services.model_manager.load.load_model_from_path(model_path=model_path, loader=loader)
-
-    def load_remote_model(
-        self,
-        source: str | AnyHttpUrl,
-        loader: Optional[Callable[[Path], AnyModel]] = None,
-    ) -> LoadedModelWithoutConfig:
-        """
-        Download, cache, and load the model file located at the indicated URL or repo_id.
-
-        If the model is already downloaded, it will be loaded from the cache.
-
-        If the a loader callable is provided, it will be invoked to load the model. Otherwise,
-        `safetensors.torch.load_file()` or `torch.load()` will be called to load the model.
-
-        Be aware that the LoadedModelWithoutConfig object has no `config` attribute
-
-        Args:
-            source: A URL or huggingface repoid.
-            loader: A Callable that expects a Path and returns a dict[str|int, Any]
-
-        Returns:
-            A LoadedModelWithoutConfig object.
-        """
-        model_path = self._services.model_manager.install.download_and_cache_model(source=str(source))
-        return self._services.model_manager.load.load_model_from_path(model_path=model_path, loader=loader)
-
 
 class ConfigInterface(InvocationContextInterface):
     def get(self) -> InvokeAIAppConfig:

invokeai/app/services/shared/sqlite/sqlite_util.py

@@ -13,7 +13,6 @@ from invokeai.app.services.shared.sqlite_migrator.migrations.migration_7 import
 from invokeai.app.services.shared.sqlite_migrator.migrations.migration_8 import build_migration_8
 from invokeai.app.services.shared.sqlite_migrator.migrations.migration_9 import build_migration_9
 from invokeai.app.services.shared.sqlite_migrator.migrations.migration_10 import build_migration_10
-from invokeai.app.services.shared.sqlite_migrator.migrations.migration_11 import build_migration_11
 from invokeai.app.services.shared.sqlite_migrator.sqlite_migrator_impl import SqliteMigrator
 
 
@@ -44,7 +43,6 @@ def init_db(config: InvokeAIAppConfig, logger: Logger, image_files: ImageFileSto
     migrator.register_migration(build_migration_8(app_config=config))
     migrator.register_migration(build_migration_9())
     migrator.register_migration(build_migration_10())
-    migrator.register_migration(build_migration_11(app_config=config, logger=logger))
     migrator.run_migrations()
 
     return db

invokeai/app/services/shared/sqlite_migrator/migrations/migration_11.py

@@ -1,75 +0,0 @@
-import shutil
-import sqlite3
-from logging import Logger
-
-from invokeai.app.services.config import InvokeAIAppConfig
-from invokeai.app.services.shared.sqlite_migrator.sqlite_migrator_common import Migration
-
-LEGACY_CORE_MODELS = [
-    # OpenPose
-    "any/annotators/dwpose/yolox_l.onnx",
-    "any/annotators/dwpose/dw-ll_ucoco_384.onnx",
-    # DepthAnything
-    "any/annotators/depth_anything/depth_anything_vitl14.pth",
-    "any/annotators/depth_anything/depth_anything_vitb14.pth",
-    "any/annotators/depth_anything/depth_anything_vits14.pth",
-    # Lama inpaint
-    "core/misc/lama/lama.pt",
-    # RealESRGAN upscale
-    "core/upscaling/realesrgan/RealESRGAN_x4plus.pth",
-    "core/upscaling/realesrgan/RealESRGAN_x4plus_anime_6B.pth",
-    "core/upscaling/realesrgan/ESRGAN_SRx4_DF2KOST_official-ff704c30.pth",
-    "core/upscaling/realesrgan/RealESRGAN_x2plus.pth",
-]
-
-
-class Migration11Callback:
-    def __init__(self, app_config: InvokeAIAppConfig, logger: Logger) -> None:
-        self._app_config = app_config
-        self._logger = logger
-
-    def __call__(self, cursor: sqlite3.Cursor) -> None:
-        self._remove_convert_cache()
-        self._remove_downloaded_models()
-        self._remove_unused_core_models()
-
-    def _remove_convert_cache(self) -> None:
-        """Rename models/.cache to models/.convert_cache."""
-        self._logger.info("Removing .cache directory. Converted models will now be cached in .convert_cache.")
-        legacy_convert_path = self._app_config.root_path / "models" / ".cache"
-        shutil.rmtree(legacy_convert_path, ignore_errors=True)
-
-    def _remove_downloaded_models(self) -> None:
-        """Remove models from their old locations; they will re-download when needed."""
-        self._logger.info(
-            "Removing legacy just-in-time models. Downloaded models will now be cached in .download_cache."
-        )
-        for model_path in LEGACY_CORE_MODELS:
-            legacy_dest_path = self._app_config.models_path / model_path
-            legacy_dest_path.unlink(missing_ok=True)
-
-    def _remove_unused_core_models(self) -> None:
-        """Remove unused core models and their directories."""
-        self._logger.info("Removing defunct core models.")
-        for dir in ["face_restoration", "misc", "upscaling"]:
-            path_to_remove = self._app_config.models_path / "core" / dir
-            shutil.rmtree(path_to_remove, ignore_errors=True)
-        shutil.rmtree(self._app_config.models_path / "any" / "annotators", ignore_errors=True)
-
-
-def build_migration_11(app_config: InvokeAIAppConfig, logger: Logger) -> Migration:
-    """
-    Build the migration from database version 10 to 11.
-
-    This migration does the following:
-    - Moves "core" models previously downloaded with download_with_progress_bar() into new
-      "models/.download_cache" directory.
-    - Renames "models/.cache" to "models/.convert_cache".
-    """
-    migration_11 = Migration(
-        from_version=10,
-        to_version=11,
-        callback=Migration11Callback(app_config=app_config, logger=logger),
-    )
-
-    return migration_11

invokeai/app/util/controlnet_utils.py

@@ -289,7 +289,7 @@ def prepare_control_image(
     width: int,
     height: int,
     num_channels: int = 3,
-    device: str = "cuda",
+    device: str | torch.device = "cuda",
     dtype: torch.dtype = torch.float16,
     control_mode: CONTROLNET_MODE_VALUES = "balanced",
     resize_mode: CONTROLNET_RESIZE_VALUES = "just_resize_simple",
@@ -304,7 +304,7 @@ def prepare_control_image(
         num_channels (int, optional): The target number of image channels. This is achieved by converting the input
             image to RGB, then naively taking the first `num_channels` channels. The primary use case is converting a
             RGB image to a single-channel grayscale image. Raises if `num_channels` cannot be achieved. Defaults to 3.
-        device (str, optional): The target device for the output image. Defaults to "cuda".
+        device (str | torch.Device, optional): The target device for the output image. Defaults to "cuda".
         dtype (_type_, optional): The dtype for the output image. Defaults to torch.float16.
         do_classifier_free_guidance (bool, optional): If True, repeat the output image along the batch dimension.
             Defaults to True.

invokeai/app/util/download_with_progress.py

@@ -0,0 +1,51 @@
+from pathlib import Path
+from urllib import request
+
+from tqdm import tqdm
+
+from invokeai.backend.util.logging import InvokeAILogger
+
+
+class ProgressBar:
+    """Simple progress bar for urllib.request.urlretrieve using tqdm."""
+
+    def __init__(self, model_name: str = "file"):
+        self.pbar = None
+        self.name = model_name
+
+    def __call__(self, block_num: int, block_size: int, total_size: int):
+        if not self.pbar:
+            self.pbar = tqdm(
+                desc=self.name,
+                initial=0,
+                unit="iB",
+                unit_scale=True,
+                unit_divisor=1000,
+                total=total_size,
+            )
+        self.pbar.update(block_size)
+
+
+def download_with_progress_bar(name: str, url: str, dest_path: Path) -> bool:
+    """Download a file from a URL to a destination path, with a progress bar.
+    If the file already exists, it will not be downloaded again.
+
+    Exceptions are not caught.
+
+    Args:
+        name (str): Name of the file being downloaded.
+        url (str): URL to download the file from.
+        dest_path (Path): Destination path to save the file to.
+
+    Returns:
+        bool: True if the file was downloaded, False if it already existed.
+    """
+    if dest_path.exists():
+        return False  # already downloaded
+
+    InvokeAILogger.get_logger().info(f"Downloading {name}...")
+
+    dest_path.parent.mkdir(parents=True, exist_ok=True)
+    request.urlretrieve(url, dest_path, ProgressBar(name))
+
+    return True

invokeai/backend/image_util/depth_anything/__init__.py

@@ -1,5 +1,5 @@
-from pathlib import Path
-from typing import Literal
+import pathlib
+from typing import Literal, Union
 
 import cv2
 import numpy as np
@@ -10,17 +10,28 @@ from PIL import Image
 from torchvision.transforms import Compose
 
 from invokeai.app.services.config.config_default import get_config
+from invokeai.app.util.download_with_progress import download_with_progress_bar
 from invokeai.backend.image_util.depth_anything.model.dpt import DPT_DINOv2
 from invokeai.backend.image_util.depth_anything.utilities.util import NormalizeImage, PrepareForNet, Resize
+from invokeai.backend.util.devices import TorchDevice
 from invokeai.backend.util.logging import InvokeAILogger
 
 config = get_config()
 logger = InvokeAILogger.get_logger(config=config)
 
 DEPTH_ANYTHING_MODELS = {
-    "large": "https://huggingface.co/spaces/LiheYoung/Depth-Anything/resolve/main/checkpoints/depth_anything_vitl14.pth?download=true",
-    "base": "https://huggingface.co/spaces/LiheYoung/Depth-Anything/resolve/main/checkpoints/depth_anything_vitb14.pth?download=true",
-    "small": "https://huggingface.co/spaces/LiheYoung/Depth-Anything/resolve/main/checkpoints/depth_anything_vits14.pth?download=true",
+    "large": {
+        "url": "https://huggingface.co/spaces/LiheYoung/Depth-Anything/resolve/main/checkpoints/depth_anything_vitl14.pth?download=true",
+        "local": "any/annotators/depth_anything/depth_anything_vitl14.pth",
+    },
+    "base": {
+        "url": "https://huggingface.co/spaces/LiheYoung/Depth-Anything/resolve/main/checkpoints/depth_anything_vitb14.pth?download=true",
+        "local": "any/annotators/depth_anything/depth_anything_vitb14.pth",
+    },
+    "small": {
+        "url": "https://huggingface.co/spaces/LiheYoung/Depth-Anything/resolve/main/checkpoints/depth_anything_vits14.pth?download=true",
+        "local": "any/annotators/depth_anything/depth_anything_vits14.pth",
+    },
 }
 
 
@@ -42,27 +53,36 @@ transform = Compose(
 
 
 class DepthAnythingDetector:
-    def __init__(self, model: DPT_DINOv2, device: torch.device) -> None:
-        self.model = model
-        self.device = device
+    def __init__(self) -> None:
+        self.model = None
+        self.model_size: Union[Literal["large", "base", "small"], None] = None
+        self.device = TorchDevice.choose_torch_device()
 
-    @staticmethod
-    def load_model(
-        model_path: Path, device: torch.device, model_size: Literal["large", "base", "small"] = "small"
-    ) -> DPT_DINOv2:
-        match model_size:
-            case "small":
-                model = DPT_DINOv2(encoder="vits", features=64, out_channels=[48, 96, 192, 384])
-            case "base":
-                model = DPT_DINOv2(encoder="vitb", features=128, out_channels=[96, 192, 384, 768])
-            case "large":
-                model = DPT_DINOv2(encoder="vitl", features=256, out_channels=[256, 512, 1024, 1024])
+    def load_model(self, model_size: Literal["large", "base", "small"] = "small"):
+        DEPTH_ANYTHING_MODEL_PATH = config.models_path / DEPTH_ANYTHING_MODELS[model_size]["local"]
+        download_with_progress_bar(
+            pathlib.Path(DEPTH_ANYTHING_MODELS[model_size]["url"]).name,
+            DEPTH_ANYTHING_MODELS[model_size]["url"],
+            DEPTH_ANYTHING_MODEL_PATH,
+        )
 
-        model.load_state_dict(torch.load(model_path.as_posix(), map_location="cpu"))
-        model.eval()
+        if not self.model or model_size != self.model_size:
+            del self.model
+            self.model_size = model_size
 
-        model.to(device)
-        return model
+            match self.model_size:
+                case "small":
+                    self.model = DPT_DINOv2(encoder="vits", features=64, out_channels=[48, 96, 192, 384])
+                case "base":
+                    self.model = DPT_DINOv2(encoder="vitb", features=128, out_channels=[96, 192, 384, 768])
+                case "large":
+                    self.model = DPT_DINOv2(encoder="vitl", features=256, out_channels=[256, 512, 1024, 1024])
+
+            self.model.load_state_dict(torch.load(DEPTH_ANYTHING_MODEL_PATH.as_posix(), map_location="cpu"))
+            self.model.eval()
+
+        self.model.to(self.device)
+        return self.model
 
     def __call__(self, image: Image.Image, resolution: int = 512) -> Image.Image:
         if not self.model:

invokeai/backend/image_util/dw_openpose/__init__.py

@@ -1,53 +1,30 @@
-from pathlib import Path
-from typing import Dict
-
 import numpy as np
 import torch
 from controlnet_aux.util import resize_image
 from PIL import Image
 
-from invokeai.backend.image_util.dw_openpose.utils import NDArrayInt, draw_bodypose, draw_facepose, draw_handpose
+from invokeai.backend.image_util.dw_openpose.utils import draw_bodypose, draw_facepose, draw_handpose
 from invokeai.backend.image_util.dw_openpose.wholebody import Wholebody
 
-DWPOSE_MODELS = {
-    "yolox_l.onnx": "https://huggingface.co/yzd-v/DWPose/resolve/main/yolox_l.onnx?download=true",
-    "dw-ll_ucoco_384.onnx": "https://huggingface.co/yzd-v/DWPose/resolve/main/dw-ll_ucoco_384.onnx?download=true",
-}
 
-
-def draw_pose(
-    pose: Dict[str, NDArrayInt | Dict[str, NDArrayInt]],
-    H: int,
-    W: int,
-    draw_face: bool = True,
-    draw_body: bool = True,
-    draw_hands: bool = True,
-    resolution: int = 512,
-) -> Image.Image:
+def draw_pose(pose, H, W, draw_face=True, draw_body=True, draw_hands=True, resolution=512):
     bodies = pose["bodies"]
     faces = pose["faces"]
     hands = pose["hands"]
-
-    assert isinstance(bodies, dict)
     candidate = bodies["candidate"]
-
-    assert isinstance(bodies, dict)
     subset = bodies["subset"]
-
     canvas = np.zeros(shape=(H, W, 3), dtype=np.uint8)
 
     if draw_body:
         canvas = draw_bodypose(canvas, candidate, subset)
 
     if draw_hands:
-        assert isinstance(hands, np.ndarray)
         canvas = draw_handpose(canvas, hands)
 
     if draw_face:
-        assert isinstance(hands, np.ndarray)
-        canvas = draw_facepose(canvas, faces)  # type: ignore
+        canvas = draw_facepose(canvas, faces)
 
-    dwpose_image: Image.Image = resize_image(
+    dwpose_image = resize_image(
         canvas,
         resolution,
     )
@@ -62,16 +39,11 @@ class DWOpenposeDetector:
     Credits: https://github.com/IDEA-Research/DWPose
     """
 
-    def __init__(self, onnx_det: Path, onnx_pose: Path) -> None:
-        self.pose_estimation = Wholebody(onnx_det=onnx_det, onnx_pose=onnx_pose)
+    def __init__(self) -> None:
+        self.pose_estimation = Wholebody()
 
     def __call__(
-        self,
-        image: Image.Image,
-        draw_face: bool = False,
-        draw_body: bool = True,
-        draw_hands: bool = False,
-        resolution: int = 512,
+        self, image: Image.Image, draw_face=False, draw_body=True, draw_hands=False, resolution=512
     ) -> Image.Image:
         np_image = np.array(image)
         H, W, C = np_image.shape
@@ -107,6 +79,3 @@ class DWOpenposeDetector:
             return draw_pose(
                 pose, H, W, draw_face=draw_face, draw_hands=draw_hands, draw_body=draw_body, resolution=resolution
             )
-
-
-__all__ = ["DWPOSE_MODELS", "DWOpenposeDetector"]

invokeai/backend/image_util/dw_openpose/utils.py

@@ -5,13 +5,11 @@ import math
 import cv2
 import matplotlib
 import numpy as np
-import numpy.typing as npt
 
 eps = 0.01
-NDArrayInt = npt.NDArray[np.uint8]
 
 
-def draw_bodypose(canvas: NDArrayInt, candidate: NDArrayInt, subset: NDArrayInt) -> NDArrayInt:
+def draw_bodypose(canvas, candidate, subset):
     H, W, C = canvas.shape
     candidate = np.array(candidate)
     subset = np.array(subset)
@@ -90,7 +88,7 @@ def draw_bodypose(canvas: NDArrayInt, candidate: NDArrayInt, subset: NDArrayInt)
     return canvas
 
 
-def draw_handpose(canvas: NDArrayInt, all_hand_peaks: NDArrayInt) -> NDArrayInt:
+def draw_handpose(canvas, all_hand_peaks):
     H, W, C = canvas.shape
 
     edges = [
@@ -144,7 +142,7 @@ def draw_handpose(canvas: NDArrayInt, all_hand_peaks: NDArrayInt) -> NDArrayInt:
     return canvas
 
 
-def draw_facepose(canvas: NDArrayInt, all_lmks: NDArrayInt) -> NDArrayInt:
+def draw_facepose(canvas, all_lmks):
     H, W, C = canvas.shape
     for lmks in all_lmks:
         lmks = np.array(lmks)

invokeai/backend/image_util/dw_openpose/wholebody.py

@@ -2,26 +2,47 @@
 # Modified pathing to suit Invoke
 
 
-from pathlib import Path
-
 import numpy as np
 import onnxruntime as ort
 
 from invokeai.app.services.config.config_default import get_config
+from invokeai.app.util.download_with_progress import download_with_progress_bar
 from invokeai.backend.util.devices import TorchDevice
 
 from .onnxdet import inference_detector
 from .onnxpose import inference_pose
 
+DWPOSE_MODELS = {
+    "yolox_l.onnx": {
+        "local": "any/annotators/dwpose/yolox_l.onnx",
+        "url": "https://huggingface.co/yzd-v/DWPose/resolve/main/yolox_l.onnx?download=true",
+    },
+    "dw-ll_ucoco_384.onnx": {
+        "local": "any/annotators/dwpose/dw-ll_ucoco_384.onnx",
+        "url": "https://huggingface.co/yzd-v/DWPose/resolve/main/dw-ll_ucoco_384.onnx?download=true",
+    },
+}
+
 config = get_config()
 
 
 class Wholebody:
-    def __init__(self, onnx_det: Path, onnx_pose: Path):
+    def __init__(self):
         device = TorchDevice.choose_torch_device()
 
         providers = ["CUDAExecutionProvider"] if device.type == "cuda" else ["CPUExecutionProvider"]
 
+        DET_MODEL_PATH = config.models_path / DWPOSE_MODELS["yolox_l.onnx"]["local"]
+        download_with_progress_bar("yolox_l.onnx", DWPOSE_MODELS["yolox_l.onnx"]["url"], DET_MODEL_PATH)
+
+        POSE_MODEL_PATH = config.models_path / DWPOSE_MODELS["dw-ll_ucoco_384.onnx"]["local"]
+        download_with_progress_bar(
+            "dw-ll_ucoco_384.onnx", DWPOSE_MODELS["dw-ll_ucoco_384.onnx"]["url"], POSE_MODEL_PATH
+        )
+
+        onnx_det = DET_MODEL_PATH
+        onnx_pose = POSE_MODEL_PATH
+
         self.session_det = ort.InferenceSession(path_or_bytes=onnx_det, providers=providers)
         self.session_pose = ort.InferenceSession(path_or_bytes=onnx_pose, providers=providers)
 

invokeai/backend/image_util/infill_methods/lama.py

@@ -1,4 +1,4 @@
-from pathlib import Path
+import gc
 from typing import Any
 
 import numpy as np
@@ -6,7 +6,9 @@ import torch
 from PIL import Image
 
 import invokeai.backend.util.logging as logger
-from invokeai.backend.model_manager.config import AnyModel
+from invokeai.app.services.config.config_default import get_config
+from invokeai.app.util.download_with_progress import download_with_progress_bar
+from invokeai.backend.util.devices import TorchDevice
 
 
 def norm_img(np_img):
@@ -17,11 +19,28 @@ def norm_img(np_img):
     return np_img
 
 
-class LaMA:
-    def __init__(self, model: AnyModel):
-        self._model = model
+def load_jit_model(url_or_path, device):
+    model_path = url_or_path
+    logger.info(f"Loading model from: {model_path}")
+    model = torch.jit.load(model_path, map_location="cpu").to(device)
+    model.eval()
+    return model
 
+
+class LaMA:
     def __call__(self, input_image: Image.Image, *args: Any, **kwds: Any) -> Any:
+        device = TorchDevice.choose_torch_device()
+        model_location = get_config().models_path / "core/misc/lama/lama.pt"
+
+        if not model_location.exists():
+            download_with_progress_bar(
+                name="LaMa Inpainting Model",
+                url="https://github.com/Sanster/models/releases/download/add_big_lama/big-lama.pt",
+                dest_path=model_location,
+            )
+
+        model = load_jit_model(model_location, device)
+
         image = np.asarray(input_image.convert("RGB"))
         image = norm_img(image)
 
@@ -29,25 +48,20 @@ class LaMA:
         mask = np.asarray(mask)
         mask = np.invert(mask)
         mask = norm_img(mask)
-        mask = (mask > 0) * 1
 
-        device = next(self._model.buffers()).device
+        mask = (mask > 0) * 1
         image = torch.from_numpy(image).unsqueeze(0).to(device)
         mask = torch.from_numpy(mask).unsqueeze(0).to(device)
 
         with torch.inference_mode():
-            infilled_image = self._model(image, mask)
+            infilled_image = model(image, mask)
 
         infilled_image = infilled_image[0].permute(1, 2, 0).detach().cpu().numpy()
         infilled_image = np.clip(infilled_image * 255, 0, 255).astype("uint8")
         infilled_image = Image.fromarray(infilled_image)
 
-        return infilled_image
+        del model
+        gc.collect()
+        torch.cuda.empty_cache()
 
-    @staticmethod
-    def load_jit_model(url_or_path: str | Path, device: torch.device | str = "cpu") -> torch.nn.Module:
-        model_path = url_or_path
-        logger.info(f"Loading model from: {model_path}")
-        model: torch.nn.Module = torch.jit.load(model_path, map_location="cpu").to(device)  # type: ignore
-        model.eval()
-        return model
+        return infilled_image

invokeai/backend/image_util/realesrgan/realesrgan.py

@@ -1,5 +1,6 @@
 import math
 from enum import Enum
+from pathlib import Path
 from typing import Any, Optional
 
 import cv2
@@ -10,7 +11,6 @@ from cv2.typing import MatLike
 from tqdm import tqdm
 
 from invokeai.backend.image_util.basicsr.rrdbnet_arch import RRDBNet
-from invokeai.backend.model_manager.config import AnyModel
 from invokeai.backend.util.devices import TorchDevice
 
 """
@@ -52,7 +52,7 @@ class RealESRGAN:
     def __init__(
         self,
         scale: int,
-        loadnet: AnyModel,
+        model_path: Path,
         model: RRDBNet,
         tile: int = 0,
         tile_pad: int = 10,
@@ -67,6 +67,8 @@ class RealESRGAN:
         self.half = half
         self.device = TorchDevice.choose_torch_device()
 
+        loadnet = torch.load(model_path, map_location=torch.device("cpu"))
+
         # prefer to use params_ema
         if "params_ema" in loadnet:
             keyname = "params_ema"

invokeai/backend/model_manager/config.py

@@ -36,7 +36,7 @@ from ..raw_model import RawModel
 
 # ModelMixin is the base class for all diffusers and transformers models
 # RawModel is the InvokeAI wrapper class for ip_adapters, loras, textual_inversion and onnx runtime
-AnyModel = Union[ModelMixin, RawModel, torch.nn.Module, Dict[str, torch.Tensor]]
+AnyModel = Union[ModelMixin, RawModel, torch.nn.Module]
 
 
 class InvalidModelConfigException(Exception):
@@ -115,7 +115,7 @@ class SchedulerPredictionType(str, Enum):
 class ModelRepoVariant(str, Enum):
     """Various hugging face variants on the diffusers format."""
 
-    Default = ""  # model files without "fp16" or other qualifier
+    Default = ""  # model files without "fp16" or other qualifier - empty str
     FP16 = "fp16"
     FP32 = "fp32"
     ONNX = "onnx"

invokeai/backend/model_manager/load/__init__.py

@@ -7,7 +7,7 @@ from importlib import import_module
 from pathlib import Path
 
 from .convert_cache.convert_cache_default import ModelConvertCache
-from .load_base import LoadedModel, LoadedModelWithoutConfig, ModelLoaderBase
+from .load_base import LoadedModel, ModelLoaderBase
 from .load_default import ModelLoader
 from .model_cache.model_cache_default import ModelCache
 from .model_loader_registry import ModelLoaderRegistry, ModelLoaderRegistryBase
@@ -19,7 +19,6 @@ for module in loaders:
 
 __all__ = [
     "LoadedModel",
-    "LoadedModelWithoutConfig",
     "ModelCache",
     "ModelConvertCache",
     "ModelLoaderBase",

invokeai/backend/model_manager/load/convert_cache/convert_cache_default.py

@@ -7,7 +7,6 @@ from pathlib import Path
 
 from invokeai.backend.util import GIG, directory_size
 from invokeai.backend.util.logging import InvokeAILogger
-from invokeai.backend.util.util import safe_filename
 
 from .convert_cache_base import ModelConvertCacheBase
 
@@ -36,7 +35,6 @@ class ModelConvertCache(ModelConvertCacheBase):
 
     def cache_path(self, key: str) -> Path:
         """Return the path for a model with the indicated key."""
-        key = safe_filename(self._cache_path, key)
         return self._cache_path / key
 
     def make_room(self, size: float) -> None:

invokeai/backend/model_manager/load/load_base.py

@@ -4,13 +4,10 @@ Base class for model loading in InvokeAI.
 """
 
 from abc import ABC, abstractmethod
-from contextlib import contextmanager
 from dataclasses import dataclass
 from logging import Logger
 from pathlib import Path
-from typing import Any, Dict, Generator, Optional, Tuple
-
-import torch
+from typing import Any, Optional
 
 from invokeai.app.services.config import InvokeAIAppConfig
 from invokeai.backend.model_manager.config import (
@@ -23,44 +20,10 @@ from invokeai.backend.model_manager.load.model_cache.model_cache_base import Mod
 
 
 @dataclass
-class LoadedModelWithoutConfig:
-    """
-    Context manager object that mediates transfer from RAM<->VRAM.
-
-    This is a context manager object that has two distinct APIs:
-
-    1. Older API (deprecated):
-    Use the LoadedModel object directly as a context manager.
-    It will move the model into VRAM (on CUDA devices), and
-    return the model in a form suitable for passing to torch.
-    Example:
-    ```
-    loaded_model_= loader.get_model_by_key('f13dd932', SubModelType('vae'))
-    with loaded_model as vae:
-      image = vae.decode(latents)[0]
-    ```
-
-    2. Newer API (recommended):
-    Call the LoadedModel's `model_on_device()` method in a
-    context. It returns a tuple consisting of a copy of
-    the model's state dict in CPU RAM followed by a copy
-    of the model in VRAM. The state dict is provided to allow
-    LoRAs and other model patchers to return the model to
-    its unpatched state without expensive copy and restore
-    operations.
-
-    Example:
-    ```
-    loaded_model_= loader.get_model_by_key('f13dd932', SubModelType('vae'))
-    with loaded_model.model_on_device() as (state_dict, vae):
-        image = vae.decode(latents)[0]
-    ```
-
-    The state_dict should be treated as a read-only object and
-    never modified. Also be aware that some loadable models do
-    not have a state_dict, in which case this value will be None.
-    """
+class LoadedModel:
+    """Context manager object that mediates transfer from RAM<->VRAM."""
 
+    config: AnyModelConfig
     _locker: ModelLockerBase
 
     def __enter__(self) -> AnyModel:
@@ -72,29 +35,12 @@ class LoadedModelWithoutConfig:
         """Context exit."""
         self._locker.unlock()
 
-    @contextmanager
-    def model_on_device(self) -> Generator[Tuple[Optional[Dict[str, torch.Tensor]], AnyModel], None, None]:
-        """Return a tuple consisting of the model's state dict (if it exists) and the locked model on execution device."""
-        locked_model = self._locker.lock()
-        try:
-            state_dict = self._locker.get_state_dict()
-            yield (state_dict, locked_model)
-        finally:
-            self._locker.unlock()
-
     @property
     def model(self) -> AnyModel:
         """Return the model without locking it."""
         return self._locker.model
 
 
-@dataclass
-class LoadedModel(LoadedModelWithoutConfig):
-    """Context manager object that mediates transfer from RAM<->VRAM."""
-
-    config: Optional[AnyModelConfig] = None
-
-
 # TODO(MM2):
 # Some "intermediary" subclasses in the ModelLoaderBase class hierarchy define methods that their subclasses don't
 # know about. I think the problem may be related to this class being an ABC.

invokeai/backend/model_manager/load/load_default.py

@@ -16,7 +16,7 @@ from invokeai.backend.model_manager.config import DiffusersConfigBase, ModelType
 from invokeai.backend.model_manager.load.convert_cache import ModelConvertCacheBase
 from invokeai.backend.model_manager.load.load_base import LoadedModel, ModelLoaderBase
 from invokeai.backend.model_manager.load.model_cache.model_cache_base import ModelCacheBase, ModelLockerBase
-from invokeai.backend.model_manager.load.model_util import calc_model_size_by_fs
+from invokeai.backend.model_manager.load.model_util import calc_model_size_by_data, calc_model_size_by_fs
 from invokeai.backend.model_manager.load.optimizations import skip_torch_weight_init
 from invokeai.backend.util.devices import TorchDevice
 
@@ -84,7 +84,7 @@ class ModelLoader(ModelLoaderBase):
         except IndexError:
             pass
 
-        cache_path: Path = self._convert_cache.cache_path(str(model_path))
+        cache_path: Path = self._convert_cache.cache_path(config.key)
         if self._needs_conversion(config, model_path, cache_path):
             loaded_model = self._do_convert(config, model_path, cache_path, submodel_type)
         else:
@@ -95,6 +95,7 @@ class ModelLoader(ModelLoaderBase):
             config.key,
             submodel_type=submodel_type,
             model=loaded_model,
+            size=calc_model_size_by_data(loaded_model),
         )
 
         return self._ram_cache.get(
@@ -125,7 +126,9 @@ class ModelLoader(ModelLoaderBase):
                 if subtype == submodel_type:
                     continue
                 if submodel := getattr(pipeline, subtype.value, None):
-                    self._ram_cache.put(config.key, submodel_type=subtype, model=submodel)
+                    self._ram_cache.put(
+                        config.key, submodel_type=subtype, model=submodel, size=calc_model_size_by_data(submodel)
+                    )
         return getattr(pipeline, submodel_type.value) if submodel_type else pipeline
 
     def _needs_conversion(self, config: AnyModelConfig, model_path: Path, dest_path: Path) -> bool:

invokeai/backend/model_manager/load/model_cache/model_cache_base.py

@@ -30,11 +30,6 @@ class ModelLockerBase(ABC):
         """Unlock the contained model, and remove it from VRAM."""
         pass
 
-    @abstractmethod
-    def get_state_dict(self) -> Optional[Dict[str, torch.Tensor]]:
-        """Return the state dict (if any) for the cached model."""
-        pass
-
     @property
     @abstractmethod
     def model(self) -> AnyModel:
@@ -61,11 +56,6 @@ class CacheRecord(Generic[T]):
     and then injected into the model. When the model is finished, the VRAM
     copy of the state dict is deleted, and the RAM version is reinjected
     into the model.
-
-    The state_dict should be treated as a read-only attribute. Do not attempt
-    to patch or otherwise modify it. Instead, patch the copy of the state_dict
-    after it is loaded into the execution device (e.g. CUDA) using the `LoadedModel`
-    context manager call `model_on_device()`.
     """
 
     key: str
@@ -169,6 +159,7 @@ class ModelCacheBase(ABC, Generic[T]):
         self,
         key: str,
         model: T,
+        size: int,
         submodel_type: Optional[SubModelType] = None,
     ) -> None:
         """Store model under key and optional submodel_type."""

invokeai/backend/model_manager/load/model_cache/model_cache_default.py

@@ -29,7 +29,6 @@ import torch
 
 from invokeai.backend.model_manager import AnyModel, SubModelType
 from invokeai.backend.model_manager.load.memory_snapshot import MemorySnapshot, get_pretty_snapshot_diff
-from invokeai.backend.model_manager.load.model_util import calc_model_size_by_data
 from invokeai.backend.util.devices import TorchDevice
 from invokeai.backend.util.logging import InvokeAILogger
 
@@ -154,13 +153,13 @@ class ModelCache(ModelCacheBase[AnyModel]):
         self,
         key: str,
         model: AnyModel,
+        size: int,
         submodel_type: Optional[SubModelType] = None,
     ) -> None:
         """Store model under key and optional submodel_type."""
         key = self._make_cache_key(key, submodel_type)
         if key in self._cached_models:
             return
-        size = calc_model_size_by_data(model)
         self.make_room(size)
 
         state_dict = model.state_dict() if isinstance(model, torch.nn.Module) else None
@@ -253,7 +252,12 @@ class ModelCache(ModelCacheBase[AnyModel]):
 
         May raise a torch.cuda.OutOfMemoryError
         """
+        # These attributes are not in the base ModelMixin class but in various derived classes.
+        # Some models don't have these attributes, in which case they run in RAM/CPU.
         self.logger.debug(f"Called to move {cache_entry.key} to {target_device}")
+        if not (hasattr(cache_entry.model, "device") and hasattr(cache_entry.model, "to")):
+            return
+
         source_device = cache_entry.device
 
         # Note: We compare device types only so that 'cuda' == 'cuda:0'.
@@ -261,10 +265,6 @@ class ModelCache(ModelCacheBase[AnyModel]):
         if torch.device(source_device).type == torch.device(target_device).type:
             return
 
-        # Some models don't have a `to` method, in which case they run in RAM/CPU.
-        if not hasattr(cache_entry.model, "to"):
-            return
-
         # This roundabout method for moving the model around is done to avoid
         # the cost of moving the model from RAM to VRAM and then back from VRAM to RAM.
         # When moving to VRAM, we copy (not move) each element of the state dict from

invokeai/backend/model_manager/load/model_cache/model_locker.py

@@ -2,8 +2,6 @@
 Base class and implementation of a class that moves models in and out of VRAM.
 """
 
-from typing import Dict, Optional
-
 import torch
 
 from invokeai.backend.model_manager import AnyModel
@@ -29,18 +27,20 @@ class ModelLocker(ModelLockerBase):
         """Return the model without moving it around."""
         return self._cache_entry.model
 
-    def get_state_dict(self) -> Optional[Dict[str, torch.Tensor]]:
-        """Return the state dict (if any) for the cached model."""
-        return self._cache_entry.state_dict
-
     def lock(self) -> AnyModel:
         """Move the model into the execution device (GPU) and lock it."""
+        if not hasattr(self.model, "to"):
+            return self.model
+
+        # NOTE that the model has to have the to() method in order for this code to move it into GPU!
         self._cache_entry.lock()
         try:
             if self._cache.lazy_offloading:
                 self._cache.offload_unlocked_models(self._cache_entry.size)
+
             self._cache.move_model_to_device(self._cache_entry, self._cache.execution_device)
             self._cache_entry.loaded = True
+
             self._cache.logger.debug(f"Locking {self._cache_entry.key} in {self._cache.execution_device}")
             self._cache.print_cuda_stats()
         except torch.cuda.OutOfMemoryError:
@@ -55,6 +55,9 @@ class ModelLocker(ModelLockerBase):
 
     def unlock(self) -> None:
         """Call upon exit from context."""
+        if not hasattr(self.model, "to"):
+            return
+
         self._cache_entry.unlock()
         if not self._cache.lazy_offloading:
             self._cache.offload_unlocked_models(0)

invokeai/backend/model_manager/load/model_loaders/generic_diffusers.py

@@ -65,11 +65,14 @@ class GenericDiffusersLoader(ModelLoader):
         else:
             try:
                 config = self._load_diffusers_config(model_path, config_name="config.json")
-                if class_name := config.get("_class_name"):
+                class_name = config.get("_class_name", None)
+                if class_name:
                     result = self._hf_definition_to_type(module="diffusers", class_name=class_name)
-                elif class_name := config.get("architectures"):
+                if config.get("model_type", None) == "clip_vision_model":
+                    class_name = config.get("architectures")
+                    assert class_name is not None
                     result = self._hf_definition_to_type(module="transformers", class_name=class_name[0])
-                else:
+                if not class_name:
                     raise InvalidModelConfigException("Unable to decipher Load Class based on given config.json")
             except KeyError as e:
                 raise InvalidModelConfigException("An expected config.json file is missing from this model.") from e

invokeai/backend/model_manager/metadata/fetch/huggingface.py

@@ -83,7 +83,7 @@ class HuggingFaceMetadataFetch(ModelMetadataFetchBase):
             assert s.size is not None
             files.append(
                 RemoteModelFile(
-                    url=hf_hub_url(id, s.rfilename, revision=variant or "main"),
+                    url=hf_hub_url(id, s.rfilename, revision=variant),
                     path=Path(name, s.rfilename),
                     size=s.size,
                     sha256=s.lfs.get("sha256") if s.lfs else None,

invokeai/backend/model_manager/metadata/metadata_base.py

@@ -37,12 +37,9 @@ class RemoteModelFile(BaseModel):
 
     url: AnyHttpUrl = Field(description="The url to download this model file")
     path: Path = Field(description="The path to the file, relative to the model root")
-    size: Optional[int] = Field(description="The size of this file, in bytes", default=0)
+    size: int = Field(description="The size of this file, in bytes")
     sha256: Optional[str] = Field(description="SHA256 hash of this model (not always available)", default=None)
 
-    def __hash__(self) -> int:
-        return hash(str(self))
-
 
 class ModelMetadataBase(BaseModel):
     """Base class for model metadata information."""

invokeai/backend/model_manager/probe.py

@@ -10,7 +10,7 @@ from picklescan.scanner import scan_file_path
 import invokeai.backend.util.logging as logger
 from invokeai.app.util.misc import uuid_string
 from invokeai.backend.model_hash.model_hash import HASHING_ALGORITHMS, ModelHash
-from invokeai.backend.util.util import SilenceWarnings
+from invokeai.backend.util.silence_warnings import SilenceWarnings
 
 from .config import (
     AnyModelConfig,

invokeai/backend/model_patcher.py

@@ -5,7 +5,7 @@ from __future__ import annotations
 
 import pickle
 from contextlib import contextmanager
-from typing import Any, Dict, Generator, Iterator, List, Optional, Tuple, Union
+from typing import Any, Dict, Iterator, List, Optional, Tuple, Union
 
 import numpy as np
 import torch
@@ -66,14 +66,8 @@ class ModelPatcher:
         cls,
         unet: UNet2DConditionModel,
         loras: Iterator[Tuple[LoRAModelRaw, float]],
-        model_state_dict: Optional[Dict[str, torch.Tensor]] = None,
     ) -> None:
-        with cls.apply_lora(
-            unet,
-            loras=loras,
-            prefix="lora_unet_",
-            model_state_dict=model_state_dict,
-        ):
+        with cls.apply_lora(unet, loras, "lora_unet_"):
             yield
 
     @classmethod
@@ -82,9 +76,28 @@ class ModelPatcher:
         cls,
         text_encoder: CLIPTextModel,
         loras: Iterator[Tuple[LoRAModelRaw, float]],
-        model_state_dict: Optional[Dict[str, torch.Tensor]] = None,
     ) -> None:
-        with cls.apply_lora(text_encoder, loras=loras, prefix="lora_te_", model_state_dict=model_state_dict):
+        with cls.apply_lora(text_encoder, loras, "lora_te_"):
+            yield
+
+    @classmethod
+    @contextmanager
+    def apply_sdxl_lora_text_encoder(
+        cls,
+        text_encoder: CLIPTextModel,
+        loras: List[Tuple[LoRAModelRaw, float]],
+    ) -> None:
+        with cls.apply_lora(text_encoder, loras, "lora_te1_"):
+            yield
+
+    @classmethod
+    @contextmanager
+    def apply_sdxl_lora_text_encoder2(
+        cls,
+        text_encoder: CLIPTextModel,
+        loras: List[Tuple[LoRAModelRaw, float]],
+    ) -> None:
+        with cls.apply_lora(text_encoder, loras, "lora_te2_"):
             yield
 
     @classmethod
@@ -94,16 +107,7 @@ class ModelPatcher:
         model: AnyModel,
         loras: Iterator[Tuple[LoRAModelRaw, float]],
         prefix: str,
-        model_state_dict: Optional[Dict[str, torch.Tensor]] = None,
-    ) -> Generator[Any, None, None]:
-        """
-        Apply one or more LoRAs to a model.
-
-        :param model: The model to patch.
-        :param loras: An iterator that returns the LoRA to patch in and its patch weight.
-        :param prefix: A string prefix that precedes keys used in the LoRAs weight layers.
-        :model_state_dict: Read-only copy of the model's state dict in CPU, for unpatching purposes.
-        """
+    ) -> None:
         original_weights = {}
         try:
             with torch.no_grad():
@@ -129,10 +133,7 @@ class ModelPatcher:
                         dtype = module.weight.dtype
 
                         if module_key not in original_weights:
-                            if model_state_dict is not None:  # we were provided with the CPU copy of the state dict
-                                original_weights[module_key] = model_state_dict[module_key + ".weight"]
-                            else:
-                                original_weights[module_key] = module.weight.detach().to(device="cpu", copy=True)
+                            original_weights[module_key] = module.weight.detach().to(device="cpu", copy=True)
 
                         layer_scale = layer.alpha / layer.rank if (layer.alpha and layer.rank) else 1.0
 

invokeai/backend/stable_diffusion/diffusers_pipeline.py

@@ -11,7 +11,6 @@ import psutil
 import torch
 import torchvision.transforms as T
 from diffusers.models import AutoencoderKL, UNet2DConditionModel
-from diffusers.models.controlnet import ControlNetModel
 from diffusers.pipelines.stable_diffusion.pipeline_stable_diffusion import StableDiffusionPipeline
 from diffusers.pipelines.stable_diffusion.safety_checker import StableDiffusionSafetyChecker
 from diffusers.schedulers import KarrasDiffusionSchedulers
@@ -26,6 +25,7 @@ from invokeai.backend.stable_diffusion.diffusion.shared_invokeai_diffusion impor
 from invokeai.backend.stable_diffusion.diffusion.unet_attention_patcher import UNetAttentionPatcher, UNetIPAdapterData
 from invokeai.backend.util.attention import auto_detect_slice_size
 from invokeai.backend.util.devices import TorchDevice
+from invokeai.backend.util.hotfixes import ControlNetModel
 
 
 @dataclass

invokeai/backend/util/silence_warnings.py

@@ -1,29 +1,36 @@
-"""Context class to silence transformers and diffusers warnings."""
-
 import warnings
-from typing import Any
+from contextlib import ContextDecorator
 
-from diffusers import logging as diffusers_logging
+from diffusers.utils import logging as diffusers_logging
 from transformers import logging as transformers_logging
 
 
-class SilenceWarnings(object):
-    """Use in context to temporarily turn off warnings from transformers & diffusers modules.
+# Inherit from ContextDecorator to allow using SilenceWarnings as both a context manager and a decorator.
+class SilenceWarnings(ContextDecorator):
+    """A context manager that disables warnings from transformers & diffusers modules while active.
 
+    As context manager:
+    ```
     with SilenceWarnings():
         # do something
+    ```
+
+    As decorator:
+    ```
+    @SilenceWarnings()
+    def some_function():
+        # do something
+    ```
     """
 
-    def __init__(self) -> None:
-        self.transformers_verbosity = transformers_logging.get_verbosity()
-        self.diffusers_verbosity = diffusers_logging.get_verbosity()
-
     def __enter__(self) -> None:
+        self._transformers_verbosity = transformers_logging.get_verbosity()
+        self._diffusers_verbosity = diffusers_logging.get_verbosity()
         transformers_logging.set_verbosity_error()
         diffusers_logging.set_verbosity_error()
         warnings.simplefilter("ignore")
 
-    def __exit__(self, *args: Any) -> None:
-        transformers_logging.set_verbosity(self.transformers_verbosity)
-        diffusers_logging.set_verbosity(self.diffusers_verbosity)
+    def __exit__(self, *args) -> None:
+        transformers_logging.set_verbosity(self._transformers_verbosity)
+        diffusers_logging.set_verbosity(self._diffusers_verbosity)
         warnings.simplefilter("default")

invokeai/backend/util/util.py

@@ -1,46 +1,14 @@
 import base64
 import io
 import os
-import re
-import unicodedata
-import warnings
 from pathlib import Path
 
-from diffusers import logging as diffusers_logging
 from PIL import Image
-from transformers import logging as transformers_logging
 
 # actual size of a gig
 GIG = 1073741824
 
 
-def slugify(value: str, allow_unicode: bool = False) -> str:
-    """
-    Convert to ASCII if 'allow_unicode' is False. Convert spaces or repeated
-    dashes to single dashes. Remove characters that aren't alphanumerics,
-    underscores, or hyphens. Replace slashes with underscores.
-    Convert to lowercase. Also strip leading and
-    trailing whitespace, dashes, and underscores.
-
-    Adapted from Django: https://github.com/django/django/blob/main/django/utils/text.py
-    """
-    value = str(value)
-    if allow_unicode:
-        value = unicodedata.normalize("NFKC", value)
-    else:
-        value = unicodedata.normalize("NFKD", value).encode("ascii", "ignore").decode("ascii")
-    value = re.sub(r"[/]", "_", value.lower())
-    value = re.sub(r"[^.\w\s-]", "", value.lower())
-    return re.sub(r"[-\s]+", "-", value).strip("-_")
-
-
-def safe_filename(directory: Path, value: str) -> str:
-    """Make a string safe to use as a filename."""
-    escaped_string = slugify(value)
-    max_name_length = os.pathconf(directory, "PC_NAME_MAX") if hasattr(os, "pathconf") else 256
-    return escaped_string[len(escaped_string) - max_name_length :]
-
-
 def directory_size(directory: Path) -> int:
     """
     Return the aggregate size of all files in a directory (bytes).
@@ -80,21 +48,3 @@ class Chdir(object):
 
     def __exit__(self, *args):
         os.chdir(self.original)
-
-
-class SilenceWarnings(object):
-    """Context manager to temporarily lower verbosity of diffusers & transformers warning messages."""
-
-    def __enter__(self):
-        """Set verbosity to error."""
-        self.transformers_verbosity = transformers_logging.get_verbosity()
-        self.diffusers_verbosity = diffusers_logging.get_verbosity()
-        transformers_logging.set_verbosity_error()
-        diffusers_logging.set_verbosity_error()
-        warnings.simplefilter("ignore")
-
-    def __exit__(self, type, value, traceback):
-        """Restore logger verbosity to state before context was entered."""
-        transformers_logging.set_verbosity(self.transformers_verbosity)
-        diffusers_logging.set_verbosity(self.diffusers_verbosity)
-        warnings.simplefilter("default")

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/controlAdapterPreprocessor.ts

@@ -22,13 +22,7 @@ import type { BatchConfig } from 'services/api/types';
 import { socketInvocationComplete } from 'services/events/actions';
 import { assert } from 'tsafe';
 
-const matcher = isAnyOf(
-  caLayerImageChanged,
-  caLayerProcessedImageChanged,
-  caLayerProcessorConfigChanged,
-  caLayerModelChanged,
-  caLayerRecalled
-);
+const matcher = isAnyOf(caLayerImageChanged, caLayerProcessorConfigChanged, caLayerModelChanged, caLayerRecalled);
 
 const DEBOUNCE_MS = 300;
 const log = logger('session');
@@ -79,10 +73,9 @@ export const addControlAdapterPreprocessor = (startAppListening: AppStartListeni
       const originalConfig = originalLayer?.controlAdapter.processorConfig;
 
       const image = layer.controlAdapter.image;
-      const processedImage = layer.controlAdapter.processedImage;
       const config = layer.controlAdapter.processorConfig;
 
-      if (isEqual(config, originalConfig) && isEqual(image, originalImage) && processedImage) {
+      if (isEqual(config, originalConfig) && isEqual(image, originalImage)) {
         // Neither config nor image have changed, we can bail
         return;
       }

invokeai/frontend/web/src/features/controlLayers/components/CALayer/CALayerControlAdapterWrapper.tsx

@@ -4,7 +4,6 @@ import {
   caLayerControlModeChanged,
   caLayerImageChanged,
   caLayerModelChanged,
-  caLayerProcessedImageChanged,
   caLayerProcessorConfigChanged,
   caOrIPALayerBeginEndStepPctChanged,
   caOrIPALayerWeightChanged,
@@ -85,14 +84,6 @@ export const CALayerControlAdapterWrapper = memo(({ layerId }: Props) => {
     [dispatch, layerId]
   );
 
-  const onErrorLoadingImage = useCallback(() => {
-    dispatch(caLayerImageChanged({ layerId, imageDTO: null }));
-  }, [dispatch, layerId]);
-
-  const onErrorLoadingProcessedImage = useCallback(() => {
-    dispatch(caLayerProcessedImageChanged({ layerId, imageDTO: null }));
-  }, [dispatch, layerId]);
-
   const droppableData = useMemo<CALayerImageDropData>(
     () => ({
       actionType: 'SET_CA_LAYER_IMAGE',
@@ -123,8 +114,6 @@ export const CALayerControlAdapterWrapper = memo(({ layerId }: Props) => {
       onChangeImage={onChangeImage}
       droppableData={droppableData}
       postUploadAction={postUploadAction}
-      onErrorLoadingImage={onErrorLoadingImage}
-      onErrorLoadingProcessedImage={onErrorLoadingProcessedImage}
     />
   );
 });

invokeai/frontend/web/src/features/controlLayers/components/ControlAndIPAdapter/ControlAdapter.tsx

@@ -28,8 +28,6 @@ type Props = {
   onChangeProcessorConfig: (processorConfig: ProcessorConfig | null) => void;
   onChangeModel: (modelConfig: ControlNetModelConfig | T2IAdapterModelConfig) => void;
   onChangeImage: (imageDTO: ImageDTO | null) => void;
-  onErrorLoadingImage: () => void;
-  onErrorLoadingProcessedImage: () => void;
   droppableData: TypesafeDroppableData;
   postUploadAction: PostUploadAction;
 };
@@ -43,8 +41,6 @@ export const ControlAdapter = memo(
     onChangeProcessorConfig,
     onChangeModel,
     onChangeImage,
-    onErrorLoadingImage,
-    onErrorLoadingProcessedImage,
     droppableData,
     postUploadAction,
   }: Props) => {
@@ -95,8 +91,6 @@ export const ControlAdapter = memo(
               onChangeImage={onChangeImage}
               droppableData={droppableData}
               postUploadAction={postUploadAction}
-              onErrorLoadingImage={onErrorLoadingImage}
-              onErrorLoadingProcessedImage={onErrorLoadingProcessedImage}
             />
           </Flex>
         </Flex>

invokeai/frontend/web/src/features/controlLayers/components/ControlAndIPAdapter/ControlAdapterImagePreview.tsx

@@ -27,19 +27,10 @@ type Props = {
   onChangeImage: (imageDTO: ImageDTO | null) => void;
   droppableData: TypesafeDroppableData;
   postUploadAction: PostUploadAction;
-  onErrorLoadingImage: () => void;
-  onErrorLoadingProcessedImage: () => void;
 };
 
 export const ControlAdapterImagePreview = memo(
-  ({
-    controlAdapter,
-    onChangeImage,
-    droppableData,
-    postUploadAction,
-    onErrorLoadingImage,
-    onErrorLoadingProcessedImage,
-  }: Props) => {
+  ({ controlAdapter, onChangeImage, droppableData, postUploadAction }: Props) => {
     const { t } = useTranslation();
     const dispatch = useAppDispatch();
     const autoAddBoardId = useAppSelector((s) => s.gallery.autoAddBoardId);
@@ -137,23 +128,10 @@ export const ControlAdapterImagePreview = memo(
       controlAdapter.processorConfig !== null;
 
     useEffect(() => {
-      if (!isConnected) {
-        return;
+      if (isConnected && (isErrorControlImage || isErrorProcessedControlImage)) {
+        handleResetControlImage();
       }
-      if (isErrorControlImage) {
-        onErrorLoadingImage();
-      }
-      if (isErrorProcessedControlImage) {
-        onErrorLoadingProcessedImage();
-      }
-    }, [
-      handleResetControlImage,
-      isConnected,
-      isErrorControlImage,
-      isErrorProcessedControlImage,
-      onErrorLoadingImage,
-      onErrorLoadingProcessedImage,
-    ]);
+    }, [handleResetControlImage, isConnected, isErrorControlImage, isErrorProcessedControlImage]);
 
     return (
       <Flex
@@ -189,7 +167,6 @@ export const ControlAdapterImagePreview = memo(
             droppableData={droppableData}
             imageDTO={processedControlImage}
             isUploadDisabled={true}
-            onError={handleResetControlImage}
           />
         </Box>
 

invokeai/frontend/web/src/features/controlLayers/components/StageComponent.tsx

@@ -4,35 +4,20 @@ import { createSelector } from '@reduxjs/toolkit';
 import { logger } from 'app/logging/logger';
 import { createMemoizedSelector } from 'app/store/createMemoizedSelector';
 import { useAppDispatch, useAppSelector } from 'app/store/storeHooks';
-import { BRUSH_SPACING_PCT, MAX_BRUSH_SPACING_PX, MIN_BRUSH_SPACING_PX } from 'features/controlLayers/konva/constants';
-import { setStageEventHandlers } from 'features/controlLayers/konva/events';
-import { debouncedRenderers, renderers as normalRenderers } from 'features/controlLayers/konva/renderers';
+import { useMouseEvents } from 'features/controlLayers/hooks/mouseEventHooks';
 import {
-  $brushSize,
-  $brushSpacingPx,
-  $isDrawing,
-  $lastAddedPoint,
   $lastCursorPos,
   $lastMouseDownPos,
-  $selectedLayerId,
-  $selectedLayerType,
-  $shouldInvertBrushSizeScrollDirection,
   $tool,
-  brushSizeChanged,
   isRegionalGuidanceLayer,
   layerBboxChanged,
   layerTranslated,
-  rgLayerLineAdded,
-  rgLayerPointsAdded,
-  rgLayerRectAdded,
   selectControlLayersSlice,
 } from 'features/controlLayers/store/controlLayersSlice';
-import type { AddLineArg, AddPointToLineArg, AddRectArg } from 'features/controlLayers/store/types';
+import { debouncedRenderers, renderers as normalRenderers } from 'features/controlLayers/util/renderers';
 import Konva from 'konva';
 import type { IRect } from 'konva/lib/types';
-import { clamp } from 'lodash-es';
 import { memo, useCallback, useLayoutEffect, useMemo, useState } from 'react';
-import { getImageDTO } from 'services/api/endpoints/images';
 import { useDevicePixelRatio } from 'use-device-pixel-ratio';
 import { v4 as uuidv4 } from 'uuid';
 
@@ -62,6 +47,7 @@ const useStageRenderer = (
   const dispatch = useAppDispatch();
   const state = useAppSelector((s) => s.controlLayers.present);
   const tool = useStore($tool);
+  const mouseEventHandlers = useMouseEvents();
   const lastCursorPos = useStore($lastCursorPos);
   const lastMouseDownPos = useStore($lastMouseDownPos);
   const selectedLayerIdColor = useAppSelector(selectSelectedLayerColor);
@@ -70,26 +56,6 @@ const useStageRenderer = (
   const layerCount = useMemo(() => state.layers.length, [state.layers]);
   const renderers = useMemo(() => (asPreview ? debouncedRenderers : normalRenderers), [asPreview]);
   const dpr = useDevicePixelRatio({ round: false });
-  const shouldInvertBrushSizeScrollDirection = useAppSelector((s) => s.canvas.shouldInvertBrushSizeScrollDirection);
-  const brushSpacingPx = useMemo(
-    () => clamp(state.brushSize / BRUSH_SPACING_PCT, MIN_BRUSH_SPACING_PX, MAX_BRUSH_SPACING_PX),
-    [state.brushSize]
-  );
-
-  useLayoutEffect(() => {
-    $brushSize.set(state.brushSize);
-    $brushSpacingPx.set(brushSpacingPx);
-    $selectedLayerId.set(state.selectedLayerId);
-    $selectedLayerType.set(selectedLayerType);
-    $shouldInvertBrushSizeScrollDirection.set(shouldInvertBrushSizeScrollDirection);
-  }, [
-    brushSpacingPx,
-    selectedLayerIdColor,
-    selectedLayerType,
-    shouldInvertBrushSizeScrollDirection,
-    state.brushSize,
-    state.selectedLayerId,
-  ]);
 
   const onLayerPosChanged = useCallback(
     (layerId: string, x: number, y: number) => {
@@ -105,31 +71,6 @@ const useStageRenderer = (
     [dispatch]
   );
 
-  const onRGLayerLineAdded = useCallback(
-    (arg: AddLineArg) => {
-      dispatch(rgLayerLineAdded(arg));
-    },
-    [dispatch]
-  );
-  const onRGLayerPointAddedToLine = useCallback(
-    (arg: AddPointToLineArg) => {
-      dispatch(rgLayerPointsAdded(arg));
-    },
-    [dispatch]
-  );
-  const onRGLayerRectAdded = useCallback(
-    (arg: AddRectArg) => {
-      dispatch(rgLayerRectAdded(arg));
-    },
-    [dispatch]
-  );
-  const onBrushSizeChanged = useCallback(
-    (size: number) => {
-      dispatch(brushSizeChanged(size));
-    },
-    [dispatch]
-  );
-
   useLayoutEffect(() => {
     log.trace('Initializing stage');
     if (!container) {
@@ -147,29 +88,21 @@ const useStageRenderer = (
     if (asPreview) {
       return;
     }
-    const cleanup = setStageEventHandlers({
-      stage,
-      $tool,
-      $isDrawing,
-      $lastMouseDownPos,
-      $lastCursorPos,
-      $lastAddedPoint,
-      $brushSize,
-      $brushSpacingPx,
-      $selectedLayerId,
-      $selectedLayerType,
-      $shouldInvertBrushSizeScrollDirection,
-      onRGLayerLineAdded,
-      onRGLayerPointAddedToLine,
-      onRGLayerRectAdded,
-      onBrushSizeChanged,
-    });
+    stage.on('mousedown', mouseEventHandlers.onMouseDown);
+    stage.on('mouseup', mouseEventHandlers.onMouseUp);
+    stage.on('mousemove', mouseEventHandlers.onMouseMove);
+    stage.on('mouseleave', mouseEventHandlers.onMouseLeave);
+    stage.on('wheel', mouseEventHandlers.onMouseWheel);
 
     return () => {
-      log.trace('Removing stage listeners');
-      cleanup();
+      log.trace('Cleaning up stage listeners');
+      stage.off('mousedown', mouseEventHandlers.onMouseDown);
+      stage.off('mouseup', mouseEventHandlers.onMouseUp);
+      stage.off('mousemove', mouseEventHandlers.onMouseMove);
+      stage.off('mouseleave', mouseEventHandlers.onMouseLeave);
+      stage.off('wheel', mouseEventHandlers.onMouseWheel);
     };
-  }, [asPreview, onBrushSizeChanged, onRGLayerLineAdded, onRGLayerPointAddedToLine, onRGLayerRectAdded, stage]);
+  }, [stage, asPreview, mouseEventHandlers]);
 
   useLayoutEffect(() => {
     log.trace('Updating stage dimensions');
@@ -227,7 +160,7 @@ const useStageRenderer = (
 
   useLayoutEffect(() => {
     log.trace('Rendering layers');
-    renderers.renderLayers(stage, state.layers, state.globalMaskLayerOpacity, tool, getImageDTO, onLayerPosChanged);
+    renderers.renderLayers(stage, state.layers, state.globalMaskLayerOpacity, tool, onLayerPosChanged);
   }, [
     stage,
     state.layers,

invokeai/frontend/web/src/features/controlLayers/hooks/mouseEventHooks.ts

@@ -0,0 +1,233 @@
+import { $ctrl, $meta } from '@invoke-ai/ui-library';
+import { useStore } from '@nanostores/react';
+import { useAppDispatch, useAppSelector } from 'app/store/storeHooks';
+import { calculateNewBrushSize } from 'features/canvas/hooks/useCanvasZoom';
+import {
+  $isDrawing,
+  $lastCursorPos,
+  $lastMouseDownPos,
+  $tool,
+  brushSizeChanged,
+  rgLayerLineAdded,
+  rgLayerPointsAdded,
+  rgLayerRectAdded,
+} from 'features/controlLayers/store/controlLayersSlice';
+import type Konva from 'konva';
+import type { KonvaEventObject } from 'konva/lib/Node';
+import type { Vector2d } from 'konva/lib/types';
+import { clamp } from 'lodash-es';
+import { useCallback, useMemo, useRef } from 'react';
+
+const getIsFocused = (stage: Konva.Stage) => {
+  return stage.container().contains(document.activeElement);
+};
+const getIsMouseDown = (e: KonvaEventObject<MouseEvent>) => e.evt.buttons === 1;
+
+const SNAP_PX = 10;
+
+export const snapPosToStage = (pos: Vector2d, stage: Konva.Stage) => {
+  const snappedPos = { ...pos };
+  // Get the normalized threshold for snapping to the edge of the stage
+  const thresholdX = SNAP_PX / stage.scaleX();
+  const thresholdY = SNAP_PX / stage.scaleY();
+  const stageWidth = stage.width() / stage.scaleX();
+  const stageHeight = stage.height() / stage.scaleY();
+  // Snap to the edge of the stage if within threshold
+  if (pos.x - thresholdX < 0) {
+    snappedPos.x = 0;
+  } else if (pos.x + thresholdX > stageWidth) {
+    snappedPos.x = Math.floor(stageWidth);
+  }
+  if (pos.y - thresholdY < 0) {
+    snappedPos.y = 0;
+  } else if (pos.y + thresholdY > stageHeight) {
+    snappedPos.y = Math.floor(stageHeight);
+  }
+  return snappedPos;
+};
+
+export const getScaledFlooredCursorPosition = (stage: Konva.Stage) => {
+  const pointerPosition = stage.getPointerPosition();
+  const stageTransform = stage.getAbsoluteTransform().copy();
+  if (!pointerPosition) {
+    return;
+  }
+  const scaledCursorPosition = stageTransform.invert().point(pointerPosition);
+  return {
+    x: Math.floor(scaledCursorPosition.x),
+    y: Math.floor(scaledCursorPosition.y),
+  };
+};
+
+const syncCursorPos = (stage: Konva.Stage): Vector2d | null => {
+  const pos = getScaledFlooredCursorPosition(stage);
+  if (!pos) {
+    return null;
+  }
+  $lastCursorPos.set(pos);
+  return pos;
+};
+
+const BRUSH_SPACING_PCT = 10;
+const MIN_BRUSH_SPACING_PX = 5;
+const MAX_BRUSH_SPACING_PX = 15;
+
+export const useMouseEvents = () => {
+  const dispatch = useAppDispatch();
+  const selectedLayerId = useAppSelector((s) => s.controlLayers.present.selectedLayerId);
+  const selectedLayerType = useAppSelector((s) => {
+    const selectedLayer = s.controlLayers.present.layers.find((l) => l.id === s.controlLayers.present.selectedLayerId);
+    if (!selectedLayer) {
+      return null;
+    }
+    return selectedLayer.type;
+  });
+  const tool = useStore($tool);
+  const lastCursorPosRef = useRef<[number, number] | null>(null);
+  const shouldInvertBrushSizeScrollDirection = useAppSelector((s) => s.canvas.shouldInvertBrushSizeScrollDirection);
+  const brushSize = useAppSelector((s) => s.controlLayers.present.brushSize);
+  const brushSpacingPx = useMemo(
+    () => clamp(brushSize / BRUSH_SPACING_PCT, MIN_BRUSH_SPACING_PX, MAX_BRUSH_SPACING_PX),
+    [brushSize]
+  );
+
+  const onMouseDown = useCallback(
+    (e: KonvaEventObject<MouseEvent>) => {
+      const stage = e.target.getStage();
+      if (!stage) {
+        return;
+      }
+      const pos = syncCursorPos(stage);
+      if (!pos || !selectedLayerId || selectedLayerType !== 'regional_guidance_layer') {
+        return;
+      }
+      if (tool === 'brush' || tool === 'eraser') {
+        dispatch(
+          rgLayerLineAdded({
+            layerId: selectedLayerId,
+            points: [pos.x, pos.y, pos.x, pos.y],
+            tool,
+          })
+        );
+        $isDrawing.set(true);
+        $lastMouseDownPos.set(pos);
+      } else if (tool === 'rect') {
+        $lastMouseDownPos.set(snapPosToStage(pos, stage));
+      }
+    },
+    [dispatch, selectedLayerId, selectedLayerType, tool]
+  );
+
+  const onMouseUp = useCallback(
+    (e: KonvaEventObject<MouseEvent>) => {
+      const stage = e.target.getStage();
+      if (!stage) {
+        return;
+      }
+      const pos = $lastCursorPos.get();
+      if (!pos || !selectedLayerId || selectedLayerType !== 'regional_guidance_layer') {
+        return;
+      }
+      const lastPos = $lastMouseDownPos.get();
+      const tool = $tool.get();
+      if (lastPos && selectedLayerId && tool === 'rect') {
+        const snappedPos = snapPosToStage(pos, stage);
+        dispatch(
+          rgLayerRectAdded({
+            layerId: selectedLayerId,
+            rect: {
+              x: Math.min(snappedPos.x, lastPos.x),
+              y: Math.min(snappedPos.y, lastPos.y),
+              width: Math.abs(snappedPos.x - lastPos.x),
+              height: Math.abs(snappedPos.y - lastPos.y),
+            },
+          })
+        );
+      }
+      $isDrawing.set(false);
+      $lastMouseDownPos.set(null);
+    },
+    [dispatch, selectedLayerId, selectedLayerType]
+  );
+
+  const onMouseMove = useCallback(
+    (e: KonvaEventObject<MouseEvent>) => {
+      const stage = e.target.getStage();
+      if (!stage) {
+        return;
+      }
+      const pos = syncCursorPos(stage);
+      if (!pos || !selectedLayerId || selectedLayerType !== 'regional_guidance_layer') {
+        return;
+      }
+      if (getIsFocused(stage) && getIsMouseDown(e) && (tool === 'brush' || tool === 'eraser')) {
+        if ($isDrawing.get()) {
+          // Continue the last line
+          if (lastCursorPosRef.current) {
+            // Dispatching redux events impacts perf substantially - using brush spacing keeps dispatches to a reasonable number
+            if (Math.hypot(lastCursorPosRef.current[0] - pos.x, lastCursorPosRef.current[1] - pos.y) < brushSpacingPx) {
+              return;
+            }
+          }
+          lastCursorPosRef.current = [pos.x, pos.y];
+          dispatch(rgLayerPointsAdded({ layerId: selectedLayerId, point: lastCursorPosRef.current }));
+        } else {
+          // Start a new line
+          dispatch(rgLayerLineAdded({ layerId: selectedLayerId, points: [pos.x, pos.y, pos.x, pos.y], tool }));
+        }
+        $isDrawing.set(true);
+      }
+    },
+    [brushSpacingPx, dispatch, selectedLayerId, selectedLayerType, tool]
+  );
+
+  const onMouseLeave = useCallback(
+    (e: KonvaEventObject<MouseEvent>) => {
+      const stage = e.target.getStage();
+      if (!stage) {
+        return;
+      }
+      const pos = syncCursorPos(stage);
+      $isDrawing.set(false);
+      $lastCursorPos.set(null);
+      $lastMouseDownPos.set(null);
+      if (!pos || !selectedLayerId || selectedLayerType !== 'regional_guidance_layer') {
+        return;
+      }
+      if (getIsFocused(stage) && getIsMouseDown(e) && (tool === 'brush' || tool === 'eraser')) {
+        dispatch(rgLayerPointsAdded({ layerId: selectedLayerId, point: [pos.x, pos.y] }));
+      }
+    },
+    [selectedLayerId, selectedLayerType, tool, dispatch]
+  );
+
+  const onMouseWheel = useCallback(
+    (e: KonvaEventObject<WheelEvent>) => {
+      e.evt.preventDefault();
+
+      if (selectedLayerType !== 'regional_guidance_layer' || (tool !== 'brush' && tool !== 'eraser')) {
+        return;
+      }
+      // checking for ctrl key is pressed or not,
+      // so that brush size can be controlled using ctrl + scroll up/down
+
+      // Invert the delta if the property is set to true
+      let delta = e.evt.deltaY;
+      if (shouldInvertBrushSizeScrollDirection) {
+        delta = -delta;
+      }
+
+      if ($ctrl.get() || $meta.get()) {
+        dispatch(brushSizeChanged(calculateNewBrushSize(brushSize, delta)));
+      }
+    },
+    [selectedLayerType, tool, shouldInvertBrushSizeScrollDirection, dispatch, brushSize]
+  );
+
+  const handlers = useMemo(
+    () => ({ onMouseDown, onMouseUp, onMouseMove, onMouseLeave, onMouseWheel }),
+    [onMouseDown, onMouseUp, onMouseMove, onMouseLeave, onMouseWheel]
+  );
+
+  return handlers;
+};

invokeai/frontend/web/src/features/controlLayers/konva/constants.ts

@@ -1,36 +0,0 @@
-/**
- * A transparency checker pattern image.
- * This is invokeai/frontend/web/public/assets/images/transparent_bg.png as a dataURL
- */
-export const TRANSPARENCY_CHECKER_PATTERN =
-  'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAIAAAAC64paAAAEsmlUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPD94cGFja2V0IGJlZ2luPSLvu78iIGlkPSJXNU0wTXBDZWhpSHpyZVN6TlRjemtjOWQiPz4KPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNS41LjAiPgogPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgeG1sbnM6ZXhpZj0iaHR0cDovL25zLmFkb2JlLmNvbS9leGlmLzEuMC8iCiAgICB4bWxuczp0aWZmPSJodHRwOi8vbnMuYWRvYmUuY29tL3RpZmYvMS4wLyIKICAgIHhtbG5zOnBob3Rvc2hvcD0iaHR0cDovL25zLmFkb2JlLmNvbS9waG90b3Nob3AvMS4wLyIKICAgIHhtbG5zOnhtcD0iaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wLyIKICAgIHhtbG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIgogICAgeG1sbnM6c3RFdnQ9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9zVHlwZS9SZXNvdXJjZUV2ZW50IyIKICAgZXhpZjpQaXhlbFhEaW1lbnNpb249IjIwIgogICBleGlmOlBpeGVsWURpbWVuc2lvbj0iMjAiCiAgIGV4aWY6Q29sb3JTcGFjZT0iMSIKICAgdGlmZjpJbWFnZVdpZHRoPSIyMCIKICAgdGlmZjpJbWFnZUxlbmd0aD0iMjAiCiAgIHRpZmY6UmVzb2x1dGlvblVuaXQ9IjIiCiAgIHRpZmY6WFJlc29sdXRpb249IjMwMC8xIgogICB0aWZmOllSZXNvbHV0aW9uPSIzMDAvMSIKICAgcGhvdG9zaG9wOkNvbG9yTW9kZT0iMyIKICAgcGhvdG9zaG9wOklDQ1Byb2ZpbGU9InNSR0IgSUVDNjE5NjYtMi4xIgogICB4bXA6TW9kaWZ5RGF0ZT0iMjAyNC0wNC0yM1QwODoyMDo0NysxMDowMCIKICAgeG1wOk1ldGFkYXRhRGF0ZT0iMjAyNC0wNC0yM1QwODoyMDo0NysxMDowMCI+CiAgIDx4bXBNTTpIaXN0b3J5PgogICAgPHJkZjpTZXE+CiAgICAgPHJkZjpsaQogICAgICBzdEV2dDphY3Rpb249InByb2R1Y2VkIgogICAgICBzdEV2dDpzb2Z0d2FyZUFnZW50PSJBZmZpbml0eSBQaG90byAxLjEwLjgiCiAgICAgIHN0RXZ0OndoZW49IjIwMjQtMDQtMjNUMDg6MjA6NDcrMTA6MDAiLz4KICAgIDwvcmRmOlNlcT4KICAgPC94bXBNTTpIaXN0b3J5PgogIDwvcmRmOkRlc2NyaXB0aW9uPgogPC9yZGY6UkRGPgo8L3g6eG1wbWV0YT4KPD94cGFja2V0IGVuZD0iciI/Pn9pdVgAAAGBaUNDUHNSR0IgSUVDNjE5NjYtMi4xAAAokXWR3yuDURjHP5uJmKghFy6WxpVpqMWNMgm1tGbKr5vt3S+1d3t73y3JrXKrKHHj1wV/AbfKtVJESq53TdywXs9rakv2nJ7zfM73nOfpnOeAPZJRVMPhAzWb18NTAffC4pK7oYiDTjpw4YgqhjYeCgWpaR8P2Kx457Vq1T73rzXHE4YCtkbhMUXT88LTwsG1vGbxrnC7ko7Ghc+F+3W5oPC9pcfKXLQ4VeYvi/VIeALsbcLuVBXHqlhJ66qwvByPmikov/exXuJMZOfnJPaId2MQZooAbmaYZAI/g4zK7MfLEAOyoka+7yd/lpzkKjJrrKOzSoo0efpFLUj1hMSk6AkZGdat/v/tq5EcHipXdwag/sU033qhYQdK26b5eWyapROoe4arbCU/dwQj76JvVzTPIbRuwsV1RYvtweUWdD1pUT36I9WJ25NJeD2DlkVw3ULTcrlnv/ucPkJkQ77qBvYPoE/Ot658AxagZ8FoS/a7AAAACXBIWXMAAC4jAAAuIwF4pT92AAAAL0lEQVQ4jWM8ffo0A25gYmKCR5YJjxxBMKp5ZGhm/P//Px7pM2fO0MrmUc0jQzMAB2EIhZC3pUYAAAAASUVORK5CYII=';
-
-/**
- * The color of a bounding box stroke when its object is selected.
- */
-export const BBOX_SELECTED_STROKE = 'rgba(78, 190, 255, 1)';
-
-/**
- * The inner border color for the brush preview.
- */
-export const BRUSH_BORDER_INNER_COLOR = 'rgba(0,0,0,1)';
-
-/**
- * The outer border color for the brush preview.
- */
-export const BRUSH_BORDER_OUTER_COLOR = 'rgba(255,255,255,0.8)';
-
-/**
- * The target spacing of individual points of brush strokes, as a percentage of the brush size.
- */
-export const BRUSH_SPACING_PCT = 10;
-
-/**
- * The minimum brush spacing in pixels.
- */
-export const MIN_BRUSH_SPACING_PX = 5;
-
-/**
- * The maximum brush spacing in pixels.
- */
-export const MAX_BRUSH_SPACING_PX = 15;

invokeai/frontend/web/src/features/controlLayers/konva/events.ts

@@ -1,201 +0,0 @@
-import { calculateNewBrushSize } from 'features/canvas/hooks/useCanvasZoom';
-import {
-  getIsFocused,
-  getIsMouseDown,
-  getScaledFlooredCursorPosition,
-  snapPosToStage,
-} from 'features/controlLayers/konva/util';
-import type { AddLineArg, AddPointToLineArg, AddRectArg, Layer, Tool } from 'features/controlLayers/store/types';
-import type Konva from 'konva';
-import type { Vector2d } from 'konva/lib/types';
-import type { WritableAtom } from 'nanostores';
-
-import { TOOL_PREVIEW_LAYER_ID } from './naming';
-
-type SetStageEventHandlersArg = {
-  stage: Konva.Stage;
-  $tool: WritableAtom<Tool>;
-  $isDrawing: WritableAtom<boolean>;
-  $lastMouseDownPos: WritableAtom<Vector2d | null>;
-  $lastCursorPos: WritableAtom<Vector2d | null>;
-  $lastAddedPoint: WritableAtom<Vector2d | null>;
-  $brushSize: WritableAtom<number>;
-  $brushSpacingPx: WritableAtom<number>;
-  $selectedLayerId: WritableAtom<string | null>;
-  $selectedLayerType: WritableAtom<Layer['type'] | null>;
-  $shouldInvertBrushSizeScrollDirection: WritableAtom<boolean>;
-  onRGLayerLineAdded: (arg: AddLineArg) => void;
-  onRGLayerPointAddedToLine: (arg: AddPointToLineArg) => void;
-  onRGLayerRectAdded: (arg: AddRectArg) => void;
-  onBrushSizeChanged: (size: number) => void;
-};
-
-const syncCursorPos = (stage: Konva.Stage, $lastCursorPos: WritableAtom<Vector2d | null>) => {
-  const pos = getScaledFlooredCursorPosition(stage);
-  if (!pos) {
-    return null;
-  }
-  $lastCursorPos.set(pos);
-  return pos;
-};
-
-export const setStageEventHandlers = ({
-  stage,
-  $tool,
-  $isDrawing,
-  $lastMouseDownPos,
-  $lastCursorPos,
-  $lastAddedPoint,
-  $brushSize,
-  $brushSpacingPx,
-  $selectedLayerId,
-  $selectedLayerType,
-  $shouldInvertBrushSizeScrollDirection,
-  onRGLayerLineAdded,
-  onRGLayerPointAddedToLine,
-  onRGLayerRectAdded,
-  onBrushSizeChanged,
-}: SetStageEventHandlersArg): (() => void) => {
-  stage.on('mouseenter', (e) => {
-    const stage = e.target.getStage();
-    if (!stage) {
-      return;
-    }
-    const tool = $tool.get();
-    stage.findOne<Konva.Layer>(`#${TOOL_PREVIEW_LAYER_ID}`)?.visible(tool === 'brush' || tool === 'eraser');
-  });
-
-  stage.on('mousedown', (e) => {
-    const stage = e.target.getStage();
-    if (!stage) {
-      return;
-    }
-    const tool = $tool.get();
-    const pos = syncCursorPos(stage, $lastCursorPos);
-    const selectedLayerId = $selectedLayerId.get();
-    const selectedLayerType = $selectedLayerType.get();
-    if (!pos || !selectedLayerId || selectedLayerType !== 'regional_guidance_layer') {
-      return;
-    }
-    if (tool === 'brush' || tool === 'eraser') {
-      onRGLayerLineAdded({
-        layerId: selectedLayerId,
-        points: [pos.x, pos.y, pos.x, pos.y],
-        tool,
-      });
-      $isDrawing.set(true);
-      $lastMouseDownPos.set(pos);
-    } else if (tool === 'rect') {
-      $lastMouseDownPos.set(snapPosToStage(pos, stage));
-    }
-  });
-
-  stage.on('mouseup', (e) => {
-    const stage = e.target.getStage();
-    if (!stage) {
-      return;
-    }
-    const pos = $lastCursorPos.get();
-    const selectedLayerId = $selectedLayerId.get();
-    const selectedLayerType = $selectedLayerType.get();
-
-    if (!pos || !selectedLayerId || selectedLayerType !== 'regional_guidance_layer') {
-      return;
-    }
-    const lastPos = $lastMouseDownPos.get();
-    const tool = $tool.get();
-    if (lastPos && selectedLayerId && tool === 'rect') {
-      const snappedPos = snapPosToStage(pos, stage);
-      onRGLayerRectAdded({
-        layerId: selectedLayerId,
-        rect: {
-          x: Math.min(snappedPos.x, lastPos.x),
-          y: Math.min(snappedPos.y, lastPos.y),
-          width: Math.abs(snappedPos.x - lastPos.x),
-          height: Math.abs(snappedPos.y - lastPos.y),
-        },
-      });
-    }
-    $isDrawing.set(false);
-    $lastMouseDownPos.set(null);
-  });
-
-  stage.on('mousemove', (e) => {
-    const stage = e.target.getStage();
-    if (!stage) {
-      return;
-    }
-    const tool = $tool.get();
-    const pos = syncCursorPos(stage, $lastCursorPos);
-    const selectedLayerId = $selectedLayerId.get();
-    const selectedLayerType = $selectedLayerType.get();
-
-    stage.findOne<Konva.Layer>(`#${TOOL_PREVIEW_LAYER_ID}`)?.visible(tool === 'brush' || tool === 'eraser');
-
-    if (!pos || !selectedLayerId || selectedLayerType !== 'regional_guidance_layer') {
-      return;
-    }
-    if (getIsFocused(stage) && getIsMouseDown(e) && (tool === 'brush' || tool === 'eraser')) {
-      if ($isDrawing.get()) {
-        // Continue the last line
-        const lastAddedPoint = $lastAddedPoint.get();
-        if (lastAddedPoint) {
-          // Dispatching redux events impacts perf substantially - using brush spacing keeps dispatches to a reasonable number
-          if (Math.hypot(lastAddedPoint.x - pos.x, lastAddedPoint.y - pos.y) < $brushSpacingPx.get()) {
-            return;
-          }
-        }
-        $lastAddedPoint.set({ x: pos.x, y: pos.y });
-        onRGLayerPointAddedToLine({ layerId: selectedLayerId, point: [pos.x, pos.y] });
-      } else {
-        // Start a new line
-        onRGLayerLineAdded({ layerId: selectedLayerId, points: [pos.x, pos.y, pos.x, pos.y], tool });
-      }
-      $isDrawing.set(true);
-    }
-  });
-
-  stage.on('mouseleave', (e) => {
-    const stage = e.target.getStage();
-    if (!stage) {
-      return;
-    }
-    const pos = syncCursorPos(stage, $lastCursorPos);
-    $isDrawing.set(false);
-    $lastCursorPos.set(null);
-    $lastMouseDownPos.set(null);
-    const selectedLayerId = $selectedLayerId.get();
-    const selectedLayerType = $selectedLayerType.get();
-    const tool = $tool.get();
-
-    stage.findOne<Konva.Layer>(`#${TOOL_PREVIEW_LAYER_ID}`)?.visible(false);
-
-    if (!pos || !selectedLayerId || selectedLayerType !== 'regional_guidance_layer') {
-      return;
-    }
-    if (getIsFocused(stage) && getIsMouseDown(e) && (tool === 'brush' || tool === 'eraser')) {
-      onRGLayerPointAddedToLine({ layerId: selectedLayerId, point: [pos.x, pos.y] });
-    }
-  });
-
-  stage.on('wheel', (e) => {
-    e.evt.preventDefault();
-    const selectedLayerType = $selectedLayerType.get();
-    const tool = $tool.get();
-    if (selectedLayerType !== 'regional_guidance_layer' || (tool !== 'brush' && tool !== 'eraser')) {
-      return;
-    }
-
-    // Invert the delta if the property is set to true
-    let delta = e.evt.deltaY;
-    if ($shouldInvertBrushSizeScrollDirection.get()) {
-      delta = -delta;
-    }
-
-    if (e.evt.ctrlKey || e.evt.metaKey) {
-      onBrushSizeChanged(calculateNewBrushSize($brushSize.get(), delta));
-    }
-  });
-
-  return () => stage.off('mousedown mouseup mousemove mouseenter mouseleave wheel');
-};

invokeai/frontend/web/src/features/controlLayers/konva/filters.ts

@@ -1,21 +0,0 @@
-/**
- * Konva filters
- * https://konvajs.org/docs/filters/Custom_Filter.html
- */
-
-/**
- * Calculates the lightness (HSL) of a given pixel and sets the alpha channel to that value.
- * This is useful for edge maps and other masks, to make the black areas transparent.
- * @param imageData The image data to apply the filter to
- */
-export const LightnessToAlphaFilter = (imageData: ImageData): void => {
-  const len = imageData.data.length / 4;
-  for (let i = 0; i < len; i++) {
-    const r = imageData.data[i * 4 + 0] as number;
-    const g = imageData.data[i * 4 + 1] as number;
-    const b = imageData.data[i * 4 + 2] as number;
-    const cMin = Math.min(r, g, b);
-    const cMax = Math.max(r, g, b);
-    imageData.data[i * 4 + 3] = (cMin + cMax) / 2;
-  }
-};

invokeai/frontend/web/src/features/controlLayers/konva/naming.ts

@@ -1,38 +0,0 @@
-/**
- * This file contains IDs, names, and ID getters for konva layers and objects.
- */
-
-// IDs for singleton Konva layers and objects
-export const TOOL_PREVIEW_LAYER_ID = 'tool_preview_layer';
-export const TOOL_PREVIEW_BRUSH_GROUP_ID = 'tool_preview_layer.brush_group';
-export const TOOL_PREVIEW_BRUSH_FILL_ID = 'tool_preview_layer.brush_fill';
-export const TOOL_PREVIEW_BRUSH_BORDER_INNER_ID = 'tool_preview_layer.brush_border_inner';
-export const TOOL_PREVIEW_BRUSH_BORDER_OUTER_ID = 'tool_preview_layer.brush_border_outer';
-export const TOOL_PREVIEW_RECT_ID = 'tool_preview_layer.rect';
-export const BACKGROUND_LAYER_ID = 'background_layer';
-export const BACKGROUND_RECT_ID = 'background_layer.rect';
-export const NO_LAYERS_MESSAGE_LAYER_ID = 'no_layers_message';
-
-// Names for Konva layers and objects (comparable to CSS classes)
-export const CA_LAYER_NAME = 'control_adapter_layer';
-export const CA_LAYER_IMAGE_NAME = 'control_adapter_layer.image';
-export const RG_LAYER_NAME = 'regional_guidance_layer';
-export const RG_LAYER_LINE_NAME = 'regional_guidance_layer.line';
-export const RG_LAYER_OBJECT_GROUP_NAME = 'regional_guidance_layer.object_group';
-export const RG_LAYER_RECT_NAME = 'regional_guidance_layer.rect';
-export const INITIAL_IMAGE_LAYER_ID = 'singleton_initial_image_layer';
-export const INITIAL_IMAGE_LAYER_NAME = 'initial_image_layer';
-export const INITIAL_IMAGE_LAYER_IMAGE_NAME = 'initial_image_layer.image';
-export const LAYER_BBOX_NAME = 'layer.bbox';
-export const COMPOSITING_RECT_NAME = 'compositing-rect';
-
-// Getters for non-singleton layer and object IDs
-export const getRGLayerId = (layerId: string) => `${RG_LAYER_NAME}_${layerId}`;
-export const getRGLayerLineId = (layerId: string, lineId: string) => `${layerId}.line_${lineId}`;
-export const getRGLayerRectId = (layerId: string, lineId: string) => `${layerId}.rect_${lineId}`;
-export const getRGLayerObjectGroupId = (layerId: string, groupId: string) => `${layerId}.objectGroup_${groupId}`;
-export const getLayerBboxId = (layerId: string) => `${layerId}.bbox`;
-export const getCALayerId = (layerId: string) => `control_adapter_layer_${layerId}`;
-export const getCALayerImageId = (layerId: string, imageName: string) => `${layerId}.image_${imageName}`;
-export const getIILayerImageId = (layerId: string, imageName: string) => `${layerId}.image_${imageName}`;
-export const getIPALayerId = (layerId: string) => `ip_adapter_layer_${layerId}`;

invokeai/frontend/web/src/features/controlLayers/konva/util.ts

@@ -1,67 +0,0 @@
-import type Konva from 'konva';
-import type { KonvaEventObject } from 'konva/lib/Node';
-import type { Vector2d } from 'konva/lib/types';
-
-//#region getScaledFlooredCursorPosition
-/**
- * Gets the scaled and floored cursor position on the stage. If the cursor is not currently over the stage, returns null.
- * @param stage The konva stage
- */
-export const getScaledFlooredCursorPosition = (stage: Konva.Stage): Vector2d | null => {
-  const pointerPosition = stage.getPointerPosition();
-  const stageTransform = stage.getAbsoluteTransform().copy();
-  if (!pointerPosition) {
-    return null;
-  }
-  const scaledCursorPosition = stageTransform.invert().point(pointerPosition);
-  return {
-    x: Math.floor(scaledCursorPosition.x),
-    y: Math.floor(scaledCursorPosition.y),
-  };
-};
-//#endregion
-
-//#region snapPosToStage
-/**
- * Snaps a position to the edge of the stage if within a threshold of the edge
- * @param pos The position to snap
- * @param stage The konva stage
- * @param snapPx The snap threshold in pixels
- */
-export const snapPosToStage = (pos: Vector2d, stage: Konva.Stage, snapPx = 10): Vector2d => {
-  const snappedPos = { ...pos };
-  // Get the normalized threshold for snapping to the edge of the stage
-  const thresholdX = snapPx / stage.scaleX();
-  const thresholdY = snapPx / stage.scaleY();
-  const stageWidth = stage.width() / stage.scaleX();
-  const stageHeight = stage.height() / stage.scaleY();
-  // Snap to the edge of the stage if within threshold
-  if (pos.x - thresholdX < 0) {
-    snappedPos.x = 0;
-  } else if (pos.x + thresholdX > stageWidth) {
-    snappedPos.x = Math.floor(stageWidth);
-  }
-  if (pos.y - thresholdY < 0) {
-    snappedPos.y = 0;
-  } else if (pos.y + thresholdY > stageHeight) {
-    snappedPos.y = Math.floor(stageHeight);
-  }
-  return snappedPos;
-};
-//#endregion
-
-//#region getIsMouseDown
-/**
- * Checks if the left mouse button is currently pressed
- * @param e The konva event
- */
-export const getIsMouseDown = (e: KonvaEventObject<MouseEvent>): boolean => e.evt.buttons === 1;
-//#endregion
-
-//#region getIsFocused
-/**
- * Checks if the stage is currently focused
- * @param stage The konva stage
- */
-export const getIsFocused = (stage: Konva.Stage): boolean => stage.container().contains(document.activeElement);
-//#endregion

invokeai/frontend/web/src/features/controlLayers/store/controlLayersSlice.ts

@@ -4,14 +4,6 @@ import type { PersistConfig, RootState } from 'app/store/store';
 import { moveBackward, moveForward, moveToBack, moveToFront } from 'common/util/arrayUtils';
 import { deepClone } from 'common/util/deepClone';
 import { roundDownToMultiple } from 'common/util/roundDownToMultiple';
-import {
-  getCALayerId,
-  getIPALayerId,
-  getRGLayerId,
-  getRGLayerLineId,
-  getRGLayerRectId,
-  INITIAL_IMAGE_LAYER_ID,
-} from 'features/controlLayers/konva/naming';
 import type {
   CLIPVisionModelV2,
   ControlModeV2,
@@ -44,9 +36,6 @@ import { assert } from 'tsafe';
 import { v4 as uuidv4 } from 'uuid';
 
 import type {
-  AddLineArg,
-  AddPointToLineArg,
-  AddRectArg,
   ControlAdapterLayer,
   ControlLayersState,
   DrawingTool,
@@ -503,11 +492,11 @@ export const controlLayersSlice = createSlice({
         layer.bboxNeedsUpdate = true;
         layer.uploadedMaskImage = null;
       },
-      prepare: (payload: AddLineArg) => ({
+      prepare: (payload: { layerId: string; points: [number, number, number, number]; tool: DrawingTool }) => ({
         payload: { ...payload, lineUuid: uuidv4() },
       }),
     },
-    rgLayerPointsAdded: (state, action: PayloadAction<AddPointToLineArg>) => {
+    rgLayerPointsAdded: (state, action: PayloadAction<{ layerId: string; point: [number, number] }>) => {
       const { layerId, point } = action.payload;
       const layer = selectRGLayerOrThrow(state, layerId);
       const lastLine = layer.maskObjects.findLast(isLine);
@@ -540,7 +529,7 @@ export const controlLayersSlice = createSlice({
         layer.bboxNeedsUpdate = true;
         layer.uploadedMaskImage = null;
       },
-      prepare: (payload: AddRectArg) => ({ payload: { ...payload, rectUuid: uuidv4() } }),
+      prepare: (payload: { layerId: string; rect: IRect }) => ({ payload: { ...payload, rectUuid: uuidv4() } }),
     },
     rgLayerMaskImageUploaded: (state, action: PayloadAction<{ layerId: string; imageDTO: ImageDTO }>) => {
       const { layerId, imageDTO } = action.payload;
@@ -894,21 +883,45 @@ const migrateControlLayersState = (state: any): any => {
   return state;
 };
 
-// Ephemeral interaction state
 export const $isDrawing = atom(false);
 export const $lastMouseDownPos = atom<Vector2d | null>(null);
 export const $tool = atom<Tool>('brush');
 export const $lastCursorPos = atom<Vector2d | null>(null);
-export const $isPreviewVisible = atom(true);
-export const $lastAddedPoint = atom<Vector2d | null>(null);
 
-// Some nanostores that are manually synced to redux state to provide imperative access
-// TODO(psyche): This is a hack, figure out another way to handle this...
-export const $brushSize = atom<number>(0);
-export const $brushSpacingPx = atom<number>(0);
-export const $selectedLayerId = atom<string | null>(null);
-export const $selectedLayerType = atom<Layer['type'] | null>(null);
-export const $shouldInvertBrushSizeScrollDirection = atom(false);
+// IDs for singleton Konva layers and objects
+export const TOOL_PREVIEW_LAYER_ID = 'tool_preview_layer';
+export const TOOL_PREVIEW_BRUSH_GROUP_ID = 'tool_preview_layer.brush_group';
+export const TOOL_PREVIEW_BRUSH_FILL_ID = 'tool_preview_layer.brush_fill';
+export const TOOL_PREVIEW_BRUSH_BORDER_INNER_ID = 'tool_preview_layer.brush_border_inner';
+export const TOOL_PREVIEW_BRUSH_BORDER_OUTER_ID = 'tool_preview_layer.brush_border_outer';
+export const TOOL_PREVIEW_RECT_ID = 'tool_preview_layer.rect';
+export const BACKGROUND_LAYER_ID = 'background_layer';
+export const BACKGROUND_RECT_ID = 'background_layer.rect';
+export const NO_LAYERS_MESSAGE_LAYER_ID = 'no_layers_message';
+
+// Names (aka classes) for Konva layers and objects
+export const CA_LAYER_NAME = 'control_adapter_layer';
+export const CA_LAYER_IMAGE_NAME = 'control_adapter_layer.image';
+export const RG_LAYER_NAME = 'regional_guidance_layer';
+export const RG_LAYER_LINE_NAME = 'regional_guidance_layer.line';
+export const RG_LAYER_OBJECT_GROUP_NAME = 'regional_guidance_layer.object_group';
+export const RG_LAYER_RECT_NAME = 'regional_guidance_layer.rect';
+export const INITIAL_IMAGE_LAYER_ID = 'singleton_initial_image_layer';
+export const INITIAL_IMAGE_LAYER_NAME = 'initial_image_layer';
+export const INITIAL_IMAGE_LAYER_IMAGE_NAME = 'initial_image_layer.image';
+export const LAYER_BBOX_NAME = 'layer.bbox';
+export const COMPOSITING_RECT_NAME = 'compositing-rect';
+
+// Getters for non-singleton layer and object IDs
+export const getRGLayerId = (layerId: string) => `${RG_LAYER_NAME}_${layerId}`;
+const getRGLayerLineId = (layerId: string, lineId: string) => `${layerId}.line_${lineId}`;
+const getRGLayerRectId = (layerId: string, lineId: string) => `${layerId}.rect_${lineId}`;
+export const getRGLayerObjectGroupId = (layerId: string, groupId: string) => `${layerId}.objectGroup_${groupId}`;
+export const getLayerBboxId = (layerId: string) => `${layerId}.bbox`;
+export const getCALayerId = (layerId: string) => `control_adapter_layer_${layerId}`;
+export const getCALayerImageId = (layerId: string, imageName: string) => `${layerId}.image_${imageName}`;
+export const getIILayerImageId = (layerId: string, imageName: string) => `${layerId}.image_${imageName}`;
+export const getIPALayerId = (layerId: string) => `ip_adapter_layer_${layerId}`;
 
 export const controlLayersPersistConfig: PersistConfig<ControlLayersState> = {
   name: controlLayersSlice.name,

invokeai/frontend/web/src/features/controlLayers/store/types.ts

@@ -17,7 +17,6 @@ import {
   zParameterPositivePrompt,
   zParameterStrength,
 } from 'features/parameters/types/parameterSchemas';
-import type { IRect } from 'konva/lib/types';
 import { z } from 'zod';
 
 const zTool = z.enum(['brush', 'eraser', 'move', 'rect']);
@@ -130,7 +129,3 @@ export type ControlLayersState = {
     aspectRatio: AspectRatioState;
   };
 };
-
-export type AddLineArg = { layerId: string; points: [number, number, number, number]; tool: DrawingTool };
-export type AddPointToLineArg = { layerId: string; point: [number, number] };
-export type AddRectArg = { layerId: string; rect: IRect };

invokeai/frontend/web/src/features/controlLayers/util/bbox.ts

@@ -1,10 +1,11 @@
 import openBase64ImageInTab from 'common/util/openBase64ImageInTab';
 import { imageDataToDataURL } from 'features/canvas/util/blobToDataURL';
+import { RG_LAYER_OBJECT_GROUP_NAME } from 'features/controlLayers/store/controlLayersSlice';
 import Konva from 'konva';
 import type { IRect } from 'konva/lib/types';
 import { assert } from 'tsafe';
 
-import { RG_LAYER_OBJECT_GROUP_NAME } from './naming';
+const GET_CLIENT_RECT_CONFIG = { skipTransform: true };
 
 type Extents = {
   minX: number;
@@ -13,13 +14,10 @@ type Extents = {
   maxY: number;
 };
 
-const GET_CLIENT_RECT_CONFIG = { skipTransform: true };
-
-//#region getImageDataBbox
 /**
  * Get the bounding box of an image.
  * @param imageData The ImageData object to get the bounding box of.
- * @returns The minimum and maximum x and y values of the image's bounding box, or null if the image has no pixels.
+ * @returns The minimum and maximum x and y values of the image's bounding box.
  */
 const getImageDataBbox = (imageData: ImageData): Extents | null => {
   const { data, width, height } = imageData;
@@ -53,9 +51,7 @@ const getImageDataBbox = (imageData: ImageData): Extents | null => {
 
   return isEmpty ? null : { minX, minY, maxX, maxY };
 };
-//#endregion
 
-//#region getIsolatedRGLayerClone
 /**
  * Clones a regional guidance konva layer onto an offscreen stage/canvas. This allows the pixel data for a given layer
  * to be captured, manipulated or analyzed without interference from other layers.
@@ -92,9 +88,7 @@ const getIsolatedRGLayerClone = (layer: Konva.Layer): { stageClone: Konva.Stage;
 
   return { stageClone, layerClone };
 };
-//#endregion
 
-//#region getLayerBboxPixels
 /**
  * Get the bounding box of a regional prompt konva layer. This function has special handling for regional prompt layers.
  * @param layer The konva layer to get the bounding box of.
@@ -143,9 +137,7 @@ export const getLayerBboxPixels = (layer: Konva.Layer, preview: boolean = false)
 
   return correctedLayerBbox;
 };
-//#endregion
 
-//#region getLayerBboxFast
 /**
  * Get the bounding box of a konva layer. This function is faster than `getLayerBboxPixels` but less accurate. It
  * should only be used when there are no eraser strokes or shapes in the layer.
@@ -161,4 +153,3 @@ export const getLayerBboxFast = (layer: Konva.Layer): IRect => {
     height: Math.floor(bbox.height),
   };
 };
-//#endregion

invokeai/frontend/web/src/features/controlLayers/util/getLayerBlobs.ts

@@ -0,0 +1,66 @@
+import { getStore } from 'app/store/nanostores/store';
+import openBase64ImageInTab from 'common/util/openBase64ImageInTab';
+import { blobToDataURL } from 'features/canvas/util/blobToDataURL';
+import { isRegionalGuidanceLayer, RG_LAYER_NAME } from 'features/controlLayers/store/controlLayersSlice';
+import { renderers } from 'features/controlLayers/util/renderers';
+import Konva from 'konva';
+import { assert } from 'tsafe';
+
+/**
+ * Get the blobs of all regional prompt layers. Only visible layers are returned.
+ * @param layerIds The IDs of the layers to get blobs for. If not provided, all regional prompt layers are used.
+ * @param preview Whether to open a new tab displaying each layer.
+ * @returns A map of layer IDs to blobs.
+ */
+export const getRegionalPromptLayerBlobs = async (
+  layerIds?: string[],
+  preview: boolean = false
+): Promise<Record<string, Blob>> => {
+  const state = getStore().getState();
+  const { layers } = state.controlLayers.present;
+  const { width, height } = state.controlLayers.present.size;
+  const reduxLayers = layers.filter(isRegionalGuidanceLayer);
+  const container = document.createElement('div');
+  const stage = new Konva.Stage({ container, width, height });
+  renderers.renderLayers(stage, reduxLayers, 1, 'brush');
+
+  const konvaLayers = stage.find<Konva.Layer>(`.${RG_LAYER_NAME}`);
+  const blobs: Record<string, Blob> = {};
+
+  // First remove all layers
+  for (const layer of konvaLayers) {
+    layer.remove();
+  }
+
+  // Next render each layer to a blob
+  for (const layer of konvaLayers) {
+    if (layerIds && !layerIds.includes(layer.id())) {
+      continue;
+    }
+    const reduxLayer = reduxLayers.find((l) => l.id === layer.id());
+    assert(reduxLayer, `Redux layer ${layer.id()} not found`);
+    stage.add(layer);
+    const blob = await new Promise<Blob>((resolve) => {
+      stage.toBlob({
+        callback: (blob) => {
+          assert(blob, 'Blob is null');
+          resolve(blob);
+        },
+      });
+    });
+
+    if (preview) {
+      const base64 = await blobToDataURL(blob);
+      openBase64ImageInTab([
+        {
+          base64,
+          caption: `${reduxLayer.id}: ${reduxLayer.positivePrompt} / ${reduxLayer.negativePrompt}`,
+        },
+      ]);
+    }
+    layer.remove();
+    blobs[layer.id()] = blob;
+  }
+
+  return blobs;
+};

invokeai/frontend/web/src/features/controlLayers/util/renderers.ts

@@ -1,7 +1,8 @@
+import { getStore } from 'app/store/nanostores/store';
 import { rgbaColorToString, rgbColorToString } from 'features/canvas/util/colorToString';
-import { getLayerBboxFast, getLayerBboxPixels } from 'features/controlLayers/konva/bbox';
-import { LightnessToAlphaFilter } from 'features/controlLayers/konva/filters';
+import { getScaledFlooredCursorPosition, snapPosToStage } from 'features/controlLayers/hooks/mouseEventHooks';
 import {
+  $tool,
   BACKGROUND_LAYER_ID,
   BACKGROUND_RECT_ID,
   CA_LAYER_IMAGE_NAME,
@@ -13,6 +14,10 @@ import {
   getRGLayerObjectGroupId,
   INITIAL_IMAGE_LAYER_IMAGE_NAME,
   INITIAL_IMAGE_LAYER_NAME,
+  isControlAdapterLayer,
+  isInitialImageLayer,
+  isRegionalGuidanceLayer,
+  isRenderableLayer,
   LAYER_BBOX_NAME,
   NO_LAYERS_MESSAGE_LAYER_ID,
   RG_LAYER_LINE_NAME,
@@ -25,13 +30,6 @@ import {
   TOOL_PREVIEW_BRUSH_GROUP_ID,
   TOOL_PREVIEW_LAYER_ID,
   TOOL_PREVIEW_RECT_ID,
-} from 'features/controlLayers/konva/naming';
-import { getScaledFlooredCursorPosition, snapPosToStage } from 'features/controlLayers/konva/util';
-import {
-  isControlAdapterLayer,
-  isInitialImageLayer,
-  isRegionalGuidanceLayer,
-  isRenderableLayer,
 } from 'features/controlLayers/store/controlLayersSlice';
 import type {
   ControlAdapterLayer,
@@ -42,46 +40,61 @@ import type {
   VectorMaskLine,
   VectorMaskRect,
 } from 'features/controlLayers/store/types';
+import { getLayerBboxFast, getLayerBboxPixels } from 'features/controlLayers/util/bbox';
 import { t } from 'i18next';
 import Konva from 'konva';
 import type { IRect, Vector2d } from 'konva/lib/types';
 import { debounce } from 'lodash-es';
 import type { RgbColor } from 'react-colorful';
-import type { ImageDTO } from 'services/api/types';
+import { imagesApi } from 'services/api/endpoints/images';
 import { assert } from 'tsafe';
 import { v4 as uuidv4 } from 'uuid';
 
-import {
-  BBOX_SELECTED_STROKE,
-  BRUSH_BORDER_INNER_COLOR,
-  BRUSH_BORDER_OUTER_COLOR,
-  TRANSPARENCY_CHECKER_PATTERN,
-} from './constants';
+const BBOX_SELECTED_STROKE = 'rgba(78, 190, 255, 1)';
+const BRUSH_BORDER_INNER_COLOR = 'rgba(0,0,0,1)';
+const BRUSH_BORDER_OUTER_COLOR = 'rgba(255,255,255,0.8)';
+// This is invokeai/frontend/web/public/assets/images/transparent_bg.png as a dataURL
+export const STAGE_BG_DATAURL =
+  'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAIAAAAC64paAAAEsmlUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPD94cGFja2V0IGJlZ2luPSLvu78iIGlkPSJXNU0wTXBDZWhpSHpyZVN6TlRjemtjOWQiPz4KPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNS41LjAiPgogPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgeG1sbnM6ZXhpZj0iaHR0cDovL25zLmFkb2JlLmNvbS9leGlmLzEuMC8iCiAgICB4bWxuczp0aWZmPSJodHRwOi8vbnMuYWRvYmUuY29tL3RpZmYvMS4wLyIKICAgIHhtbG5zOnBob3Rvc2hvcD0iaHR0cDovL25zLmFkb2JlLmNvbS9waG90b3Nob3AvMS4wLyIKICAgIHhtbG5zOnhtcD0iaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wLyIKICAgIHhtbG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIgogICAgeG1sbnM6c3RFdnQ9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9zVHlwZS9SZXNvdXJjZUV2ZW50IyIKICAgZXhpZjpQaXhlbFhEaW1lbnNpb249IjIwIgogICBleGlmOlBpeGVsWURpbWVuc2lvbj0iMjAiCiAgIGV4aWY6Q29sb3JTcGFjZT0iMSIKICAgdGlmZjpJbWFnZVdpZHRoPSIyMCIKICAgdGlmZjpJbWFnZUxlbmd0aD0iMjAiCiAgIHRpZmY6UmVzb2x1dGlvblVuaXQ9IjIiCiAgIHRpZmY6WFJlc29sdXRpb249IjMwMC8xIgogICB0aWZmOllSZXNvbHV0aW9uPSIzMDAvMSIKICAgcGhvdG9zaG9wOkNvbG9yTW9kZT0iMyIKICAgcGhvdG9zaG9wOklDQ1Byb2ZpbGU9InNSR0IgSUVDNjE5NjYtMi4xIgogICB4bXA6TW9kaWZ5RGF0ZT0iMjAyNC0wNC0yM1QwODoyMDo0NysxMDowMCIKICAgeG1wOk1ldGFkYXRhRGF0ZT0iMjAyNC0wNC0yM1QwODoyMDo0NysxMDowMCI+CiAgIDx4bXBNTTpIaXN0b3J5PgogICAgPHJkZjpTZXE+CiAgICAgPHJkZjpsaQogICAgICBzdEV2dDphY3Rpb249InByb2R1Y2VkIgogICAgICBzdEV2dDpzb2Z0d2FyZUFnZW50PSJBZmZpbml0eSBQaG90byAxLjEwLjgiCiAgICAgIHN0RXZ0OndoZW49IjIwMjQtMDQtMjNUMDg6MjA6NDcrMTA6MDAiLz4KICAgIDwvcmRmOlNlcT4KICAgPC94bXBNTTpIaXN0b3J5PgogIDwvcmRmOkRlc2NyaXB0aW9uPgogPC9yZGY6UkRGPgo8L3g6eG1wbWV0YT4KPD94cGFja2V0IGVuZD0iciI/Pn9pdVgAAAGBaUNDUHNSR0IgSUVDNjE5NjYtMi4xAAAokXWR3yuDURjHP5uJmKghFy6WxpVpqMWNMgm1tGbKr5vt3S+1d3t73y3JrXKrKHHj1wV/AbfKtVJESq53TdywXs9rakv2nJ7zfM73nOfpnOeAPZJRVMPhAzWb18NTAffC4pK7oYiDTjpw4YgqhjYeCgWpaR8P2Kx457Vq1T73rzXHE4YCtkbhMUXT88LTwsG1vGbxrnC7ko7Ghc+F+3W5oPC9pcfKXLQ4VeYvi/VIeALsbcLuVBXHqlhJ66qwvByPmikov/exXuJMZOfnJPaId2MQZooAbmaYZAI/g4zK7MfLEAOyoka+7yd/lpzkKjJrrKOzSoo0efpFLUj1hMSk6AkZGdat/v/tq5EcHipXdwag/sU033qhYQdK26b5eWyapROoe4arbCU/dwQj76JvVzTPIbRuwsV1RYvtweUWdD1pUT36I9WJ25NJeD2DlkVw3ULTcrlnv/ucPkJkQ77qBvYPoE/Ot658AxagZ8FoS/a7AAAACXBIWXMAAC4jAAAuIwF4pT92AAAAL0lEQVQ4jWM8ffo0A25gYmKCR5YJjxxBMKp5ZGhm/P//Px7pM2fO0MrmUc0jQzMAB2EIhZC3pUYAAAAASUVORK5CYII=';
 
-const mapId = (object: { id: string }): string => object.id;
+const mapId = (object: { id: string }) => object.id;
 
-/**
- * Konva selection callback to select all renderable layers. This includes RG, CA and II layers.
- */
-const selectRenderableLayers = (n: Konva.Node): boolean =>
+const selectRenderableLayers = (n: Konva.Node) =>
   n.name() === RG_LAYER_NAME || n.name() === CA_LAYER_NAME || n.name() === INITIAL_IMAGE_LAYER_NAME;
 
-/**
- * Konva selection callback to select RG mask objects. This includes lines and rects.
- */
-const selectVectorMaskObjects = (node: Konva.Node): boolean => {
+const selectVectorMaskObjects = (node: Konva.Node) => {
   return node.name() === RG_LAYER_LINE_NAME || node.name() === RG_LAYER_RECT_NAME;
 };
 
 /**
- * Creates the singleton tool preview layer and all its objects.
- * @param stage The konva stage
+ * Creates the brush preview layer.
+ * @param stage The konva stage to render on.
+ * @returns The brush preview layer.
  */
-const createToolPreviewLayer = (stage: Konva.Stage): Konva.Layer => {
+const createToolPreviewLayer = (stage: Konva.Stage) => {
   // Initialize the brush preview layer & add to the stage
   const toolPreviewLayer = new Konva.Layer({ id: TOOL_PREVIEW_LAYER_ID, visible: false, listening: false });
   stage.add(toolPreviewLayer);
 
+  // Add handlers to show/hide the brush preview layer
+  stage.on('mousemove', (e) => {
+    const tool = $tool.get();
+    e.target
+      .getStage()
+      ?.findOne<Konva.Layer>(`#${TOOL_PREVIEW_LAYER_ID}`)
+      ?.visible(tool === 'brush' || tool === 'eraser');
+  });
+  stage.on('mouseleave', (e) => {
+    e.target.getStage()?.findOne<Konva.Layer>(`#${TOOL_PREVIEW_LAYER_ID}`)?.visible(false);
+  });
+  stage.on('mouseenter', (e) => {
+    const tool = $tool.get();
+    e.target
+      .getStage()
+      ?.findOne<Konva.Layer>(`#${TOOL_PREVIEW_LAYER_ID}`)
+      ?.visible(tool === 'brush' || tool === 'eraser');
+  });
+
   // Create the brush preview group & circles
   const brushPreviewGroup = new Konva.Group({ id: TOOL_PREVIEW_BRUSH_GROUP_ID });
   const brushPreviewFill = new Konva.Circle({
@@ -108,7 +121,7 @@ const createToolPreviewLayer = (stage: Konva.Stage): Konva.Layer => {
   brushPreviewGroup.add(brushPreviewBorderOuter);
   toolPreviewLayer.add(brushPreviewGroup);
 
-  // Create the rect preview - this is a rectangle drawn from the last mouse down position to the current cursor position
+  // Create the rect preview
   const rectPreview = new Konva.Rect({ id: TOOL_PREVIEW_RECT_ID, listening: false, stroke: 'white', strokeWidth: 1 });
   toolPreviewLayer.add(rectPreview);
 
@@ -117,14 +130,12 @@ const createToolPreviewLayer = (stage: Konva.Stage): Konva.Layer => {
 
 /**
  * Renders the brush preview for the selected tool.
- * @param stage The konva stage
- * @param tool The selected tool
- * @param color The selected layer's color
- * @param selectedLayerType The selected layer's type
- * @param globalMaskLayerOpacity The global mask layer opacity
- * @param cursorPos The cursor position
- * @param lastMouseDownPos The position of the last mouse down event - used for the rect tool
- * @param brushSize The brush size
+ * @param stage The konva stage to render on.
+ * @param tool The selected tool.
+ * @param color The selected layer's color.
+ * @param cursorPos The cursor position.
+ * @param lastMouseDownPos The position of the last mouse down event - used for the rect tool.
+ * @param brushSize The brush size.
  */
 const renderToolPreview = (
   stage: Konva.Stage,
@@ -135,7 +146,7 @@ const renderToolPreview = (
   cursorPos: Vector2d | null,
   lastMouseDownPos: Vector2d | null,
   brushSize: number
-): void => {
+) => {
   const layerCount = stage.find(selectRenderableLayers).length;
   // Update the stage's pointer style
   if (layerCount === 0) {
@@ -151,7 +162,7 @@ const renderToolPreview = (
     // Move rect gets a crosshair
     stage.container().style.cursor = 'crosshair';
   } else {
-    // Else we hide the native cursor and use the konva-rendered brush preview
+    // Else we use the brush preview
     stage.container().style.cursor = 'none';
   }
 
@@ -216,29 +227,28 @@ const renderToolPreview = (
 };
 
 /**
- * Creates a regional guidance layer.
- * @param stage The konva stage
- * @param layerState The regional guidance layer state
- * @param onLayerPosChanged Callback for when the layer's position changes
+ * Creates a vector mask layer.
+ * @param stage The konva stage to attach the layer to.
+ * @param reduxLayer The redux layer to create the konva layer from.
+ * @param onLayerPosChanged Callback for when the layer's position changes.
  */
-const createRGLayer = (
+const createRegionalGuidanceLayer = (
   stage: Konva.Stage,
-  layerState: RegionalGuidanceLayer,
+  reduxLayer: RegionalGuidanceLayer,
   onLayerPosChanged?: (layerId: string, x: number, y: number) => void
-): Konva.Layer => {
+) => {
   // This layer hasn't been added to the konva state yet
   const konvaLayer = new Konva.Layer({
-    id: layerState.id,
+    id: reduxLayer.id,
     name: RG_LAYER_NAME,
     draggable: true,
     dragDistance: 0,
   });
 
-  // When a drag on the layer finishes, update the layer's position in state. During the drag, konva handles changing
-  // the position - we do not need to call this on the `dragmove` event.
+  // Create a `dragmove` listener for this layer
   if (onLayerPosChanged) {
     konvaLayer.on('dragend', function (e) {
-      onLayerPosChanged(layerState.id, Math.floor(e.target.x()), Math.floor(e.target.y()));
+      onLayerPosChanged(reduxLayer.id, Math.floor(e.target.x()), Math.floor(e.target.y()));
     });
   }
 
@@ -248,7 +258,7 @@ const createRGLayer = (
     if (!cursorPos) {
       return this.getAbsolutePosition();
     }
-    // Prevent the user from dragging the layer out of the stage bounds by constaining the cursor position to the stage bounds
+    // Prevent the user from dragging the layer out of the stage bounds.
     if (
       cursorPos.x < 0 ||
       cursorPos.x > stage.width() / stage.scaleX() ||
@@ -262,7 +272,7 @@ const createRGLayer = (
 
   // The object group holds all of the layer's objects (e.g. lines and rects)
   const konvaObjectGroup = new Konva.Group({
-    id: getRGLayerObjectGroupId(layerState.id, uuidv4()),
+    id: getRGLayerObjectGroupId(reduxLayer.id, uuidv4()),
     name: RG_LAYER_OBJECT_GROUP_NAME,
     listening: false,
   });
@@ -274,51 +284,47 @@ const createRGLayer = (
 };
 
 /**
- * Creates a konva line from a vector mask line.
- * @param vectorMaskLine The vector mask line state
- * @param layerObjectGroup The konva layer's object group to add the line to
+ * Creates a konva line from a redux vector mask line.
+ * @param reduxObject The redux object to create the konva line from.
+ * @param konvaGroup The konva group to add the line to.
  */
-const createVectorMaskLine = (vectorMaskLine: VectorMaskLine, layerObjectGroup: Konva.Group): Konva.Line => {
-  const konvaLine = new Konva.Line({
-    id: vectorMaskLine.id,
-    key: vectorMaskLine.id,
+const createVectorMaskLine = (reduxObject: VectorMaskLine, konvaGroup: Konva.Group): Konva.Line => {
+  const vectorMaskLine = new Konva.Line({
+    id: reduxObject.id,
+    key: reduxObject.id,
     name: RG_LAYER_LINE_NAME,
-    strokeWidth: vectorMaskLine.strokeWidth,
+    strokeWidth: reduxObject.strokeWidth,
     tension: 0,
     lineCap: 'round',
     lineJoin: 'round',
     shadowForStrokeEnabled: false,
-    globalCompositeOperation: vectorMaskLine.tool === 'brush' ? 'source-over' : 'destination-out',
+    globalCompositeOperation: reduxObject.tool === 'brush' ? 'source-over' : 'destination-out',
     listening: false,
   });
-  layerObjectGroup.add(konvaLine);
-  return konvaLine;
+  konvaGroup.add(vectorMaskLine);
+  return vectorMaskLine;
 };
 
 /**
- * Creates a konva rect from a vector mask rect.
- * @param vectorMaskRect The vector mask rect state
- * @param layerObjectGroup The konva layer's object group to add the line to
+ * Creates a konva rect from a redux vector mask rect.
+ * @param reduxObject The redux object to create the konva rect from.
+ * @param konvaGroup The konva group to add the rect to.
  */
-const createVectorMaskRect = (vectorMaskRect: VectorMaskRect, layerObjectGroup: Konva.Group): Konva.Rect => {
-  const konvaRect = new Konva.Rect({
-    id: vectorMaskRect.id,
-    key: vectorMaskRect.id,
+const createVectorMaskRect = (reduxObject: VectorMaskRect, konvaGroup: Konva.Group): Konva.Rect => {
+  const vectorMaskRect = new Konva.Rect({
+    id: reduxObject.id,
+    key: reduxObject.id,
     name: RG_LAYER_RECT_NAME,
-    x: vectorMaskRect.x,
-    y: vectorMaskRect.y,
-    width: vectorMaskRect.width,
-    height: vectorMaskRect.height,
+    x: reduxObject.x,
+    y: reduxObject.y,
+    width: reduxObject.width,
+    height: reduxObject.height,
     listening: false,
   });
-  layerObjectGroup.add(konvaRect);
-  return konvaRect;
+  konvaGroup.add(vectorMaskRect);
+  return vectorMaskRect;
 };
 
-/**
- * Creates the "compositing rect" for a layer.
- * @param konvaLayer The konva layer
- */
 const createCompositingRect = (konvaLayer: Konva.Layer): Konva.Rect => {
   const compositingRect = new Konva.Rect({ name: COMPOSITING_RECT_NAME, listening: false });
   konvaLayer.add(compositingRect);
@@ -326,41 +332,41 @@ const createCompositingRect = (konvaLayer: Konva.Layer): Konva.Rect => {
 };
 
 /**
- * Renders a regional guidance layer.
- * @param stage The konva stage
- * @param layerState The regional guidance layer state
- * @param globalMaskLayerOpacity The global mask layer opacity
- * @param tool The current tool
- * @param onLayerPosChanged Callback for when the layer's position changes
+ * Renders a vector mask layer.
+ * @param stage The konva stage to render on.
+ * @param reduxLayer The redux vector mask layer to render.
+ * @param reduxLayerIndex The index of the layer in the redux store.
+ * @param globalMaskLayerOpacity The opacity of the global mask layer.
+ * @param tool The current tool.
  */
-const renderRGLayer = (
+const renderRegionalGuidanceLayer = (
   stage: Konva.Stage,
-  layerState: RegionalGuidanceLayer,
+  reduxLayer: RegionalGuidanceLayer,
   globalMaskLayerOpacity: number,
   tool: Tool,
   onLayerPosChanged?: (layerId: string, x: number, y: number) => void
 ): void => {
   const konvaLayer =
-    stage.findOne<Konva.Layer>(`#${layerState.id}`) ?? createRGLayer(stage, layerState, onLayerPosChanged);
+    stage.findOne<Konva.Layer>(`#${reduxLayer.id}`) ??
+    createRegionalGuidanceLayer(stage, reduxLayer, onLayerPosChanged);
 
   // Update the layer's position and listening state
   konvaLayer.setAttrs({
     listening: tool === 'move', // The layer only listens when using the move tool - otherwise the stage is handling mouse events
-    x: Math.floor(layerState.x),
-    y: Math.floor(layerState.y),
+    x: Math.floor(reduxLayer.x),
+    y: Math.floor(reduxLayer.y),
   });
 
   // Convert the color to a string, stripping the alpha - the object group will handle opacity.
-  const rgbColor = rgbColorToString(layerState.previewColor);
+  const rgbColor = rgbColorToString(reduxLayer.previewColor);
 
   const konvaObjectGroup = konvaLayer.findOne<Konva.Group>(`.${RG_LAYER_OBJECT_GROUP_NAME}`);
-  assert(konvaObjectGroup, `Object group not found for layer ${layerState.id}`);
+  assert(konvaObjectGroup, `Object group not found for layer ${reduxLayer.id}`);
 
   // We use caching to handle "global" layer opacity, but caching is expensive and we should only do it when required.
   let groupNeedsCache = false;
 
-  const objectIds = layerState.maskObjects.map(mapId);
-  // Destroy any objects that are no longer in the redux state
+  const objectIds = reduxLayer.maskObjects.map(mapId);
   for (const objectNode of konvaObjectGroup.find(selectVectorMaskObjects)) {
     if (!objectIds.includes(objectNode.id())) {
       objectNode.destroy();
@@ -368,15 +374,15 @@ const renderRGLayer = (
     }
   }
 
-  for (const maskObject of layerState.maskObjects) {
-    if (maskObject.type === 'vector_mask_line') {
+  for (const reduxObject of reduxLayer.maskObjects) {
+    if (reduxObject.type === 'vector_mask_line') {
       const vectorMaskLine =
-        stage.findOne<Konva.Line>(`#${maskObject.id}`) ?? createVectorMaskLine(maskObject, konvaObjectGroup);
+        stage.findOne<Konva.Line>(`#${reduxObject.id}`) ?? createVectorMaskLine(reduxObject, konvaObjectGroup);
 
       // Only update the points if they have changed. The point values are never mutated, they are only added to the
       // array, so checking the length is sufficient to determine if we need to re-cache.
-      if (vectorMaskLine.points().length !== maskObject.points.length) {
-        vectorMaskLine.points(maskObject.points);
+      if (vectorMaskLine.points().length !== reduxObject.points.length) {
+        vectorMaskLine.points(reduxObject.points);
         groupNeedsCache = true;
       }
       // Only update the color if it has changed.
@@ -384,9 +390,9 @@ const renderRGLayer = (
         vectorMaskLine.stroke(rgbColor);
         groupNeedsCache = true;
       }
-    } else if (maskObject.type === 'vector_mask_rect') {
+    } else if (reduxObject.type === 'vector_mask_rect') {
       const konvaObject =
-        stage.findOne<Konva.Rect>(`#${maskObject.id}`) ?? createVectorMaskRect(maskObject, konvaObjectGroup);
+        stage.findOne<Konva.Rect>(`#${reduxObject.id}`) ?? createVectorMaskRect(reduxObject, konvaObjectGroup);
 
       // Only update the color if it has changed.
       if (konvaObject.fill() !== rgbColor) {
@@ -397,8 +403,8 @@ const renderRGLayer = (
   }
 
   // Only update layer visibility if it has changed.
-  if (konvaLayer.visible() !== layerState.isEnabled) {
-    konvaLayer.visible(layerState.isEnabled);
+  if (konvaLayer.visible() !== reduxLayer.isEnabled) {
+    konvaLayer.visible(reduxLayer.isEnabled);
     groupNeedsCache = true;
   }
 
@@ -422,7 +428,7 @@ const renderRGLayer = (
    * Instead, with the special handling, the effect is as if you drew all the shapes at 100% opacity, flattened them to
    * a single raster image, and _then_ applied the 50% opacity.
    */
-  if (layerState.isSelected && tool !== 'move') {
+  if (reduxLayer.isSelected && tool !== 'move') {
     // We must clear the cache first so Konva will re-draw the group with the new compositing rect
     if (konvaObjectGroup.isCached()) {
       konvaObjectGroup.clearCache();
@@ -432,7 +438,7 @@ const renderRGLayer = (
 
     compositingRect.setAttrs({
       // The rect should be the size of the layer - use the fast method if we don't have a pixel-perfect bbox already
-      ...(!layerState.bboxNeedsUpdate && layerState.bbox ? layerState.bbox : getLayerBboxFast(konvaLayer)),
+      ...(!reduxLayer.bboxNeedsUpdate && reduxLayer.bbox ? reduxLayer.bbox : getLayerBboxFast(konvaLayer)),
       fill: rgbColor,
       opacity: globalMaskLayerOpacity,
       // Draw this rect only where there are non-transparent pixels under it (e.g. the mask shapes)
@@ -453,14 +459,9 @@ const renderRGLayer = (
   }
 };
 
-/**
- * Creates an initial image konva layer.
- * @param stage The konva stage
- * @param layerState The initial image layer state
- */
-const createIILayer = (stage: Konva.Stage, layerState: InitialImageLayer): Konva.Layer => {
+const createInitialImageLayer = (stage: Konva.Stage, reduxLayer: InitialImageLayer): Konva.Layer => {
   const konvaLayer = new Konva.Layer({
-    id: layerState.id,
+    id: reduxLayer.id,
     name: INITIAL_IMAGE_LAYER_NAME,
     imageSmoothingEnabled: true,
     listening: false,
@@ -469,27 +470,20 @@ const createIILayer = (stage: Konva.Stage, layerState: InitialImageLayer): Konva
   return konvaLayer;
 };
 
-/**
- * Creates the konva image for an initial image layer.
- * @param konvaLayer The konva layer
- * @param imageEl The image element
- */
-const createIILayerImage = (konvaLayer: Konva.Layer, imageEl: HTMLImageElement): Konva.Image => {
+const createInitialImageLayerImage = (konvaLayer: Konva.Layer, image: HTMLImageElement): Konva.Image => {
   const konvaImage = new Konva.Image({
     name: INITIAL_IMAGE_LAYER_IMAGE_NAME,
-    image: imageEl,
+    image,
   });
   konvaLayer.add(konvaImage);
   return konvaImage;
 };
 
-/**
- * Updates an initial image layer's attributes (width, height, opacity, visibility).
- * @param stage The konva stage
- * @param konvaImage The konva image
- * @param layerState The initial image layer state
- */
-const updateIILayerImageAttrs = (stage: Konva.Stage, konvaImage: Konva.Image, layerState: InitialImageLayer): void => {
+const updateInitialImageLayerImageAttrs = (
+  stage: Konva.Stage,
+  konvaImage: Konva.Image,
+  reduxLayer: InitialImageLayer
+) => {
   // Konva erroneously reports NaN for width and height when the stage is hidden. This causes errors when caching,
   // but it doesn't seem to break anything.
   // TODO(psyche): Investigate and report upstream.
@@ -498,55 +492,46 @@ const updateIILayerImageAttrs = (stage: Konva.Stage, konvaImage: Konva.Image, la
   if (
     konvaImage.width() !== newWidth ||
     konvaImage.height() !== newHeight ||
-    konvaImage.visible() !== layerState.isEnabled
+    konvaImage.visible() !== reduxLayer.isEnabled
   ) {
     konvaImage.setAttrs({
-      opacity: layerState.opacity,
+      opacity: reduxLayer.opacity,
       scaleX: 1,
       scaleY: 1,
       width: stage.width() / stage.scaleX(),
       height: stage.height() / stage.scaleY(),
-      visible: layerState.isEnabled,
+      visible: reduxLayer.isEnabled,
     });
   }
-  if (konvaImage.opacity() !== layerState.opacity) {
-    konvaImage.opacity(layerState.opacity);
+  if (konvaImage.opacity() !== reduxLayer.opacity) {
+    konvaImage.opacity(reduxLayer.opacity);
   }
 };
 
-/**
- * Update an initial image layer's image source when the image changes.
- * @param stage The konva stage
- * @param konvaLayer The konva layer
- * @param layerState The initial image layer state
- * @param getImageDTO A function to retrieve an image DTO from the server, used to update the image source
- */
-const updateIILayerImageSource = async (
+const updateInitialImageLayerImageSource = async (
   stage: Konva.Stage,
   konvaLayer: Konva.Layer,
-  layerState: InitialImageLayer,
-  getImageDTO: (imageName: string) => Promise<ImageDTO | null>
-): Promise<void> => {
-  if (layerState.image) {
-    const imageName = layerState.image.name;
-    const imageDTO = await getImageDTO(imageName);
-    if (!imageDTO) {
-      return;
-    }
+  reduxLayer: InitialImageLayer
+) => {
+  if (reduxLayer.image) {
+    const imageName = reduxLayer.image.name;
+    const req = getStore().dispatch(imagesApi.endpoints.getImageDTO.initiate(imageName));
+    const imageDTO = await req.unwrap();
+    req.unsubscribe();
     const imageEl = new Image();
-    const imageId = getIILayerImageId(layerState.id, imageName);
+    const imageId = getIILayerImageId(reduxLayer.id, imageName);
     imageEl.onload = () => {
       // Find the existing image or create a new one - must find using the name, bc the id may have just changed
       const konvaImage =
         konvaLayer.findOne<Konva.Image>(`.${INITIAL_IMAGE_LAYER_IMAGE_NAME}`) ??
-        createIILayerImage(konvaLayer, imageEl);
+        createInitialImageLayerImage(konvaLayer, imageEl);
 
       // Update the image's attributes
       konvaImage.setAttrs({
         id: imageId,
         image: imageEl,
       });
-      updateIILayerImageAttrs(stage, konvaImage, layerState);
+      updateInitialImageLayerImageAttrs(stage, konvaImage, reduxLayer);
       imageEl.id = imageId;
     };
     imageEl.src = imageDTO.image_url;
@@ -555,24 +540,14 @@ const updateIILayerImageSource = async (
   }
 };
 
-/**
- * Renders an initial image layer.
- * @param stage The konva stage
- * @param layerState The initial image layer state
- * @param getImageDTO A function to retrieve an image DTO from the server, used to update the image source
- */
-const renderIILayer = (
-  stage: Konva.Stage,
-  layerState: InitialImageLayer,
-  getImageDTO: (imageName: string) => Promise<ImageDTO | null>
-): void => {
-  const konvaLayer = stage.findOne<Konva.Layer>(`#${layerState.id}`) ?? createIILayer(stage, layerState);
+const renderInitialImageLayer = (stage: Konva.Stage, reduxLayer: InitialImageLayer) => {
+  const konvaLayer = stage.findOne<Konva.Layer>(`#${reduxLayer.id}`) ?? createInitialImageLayer(stage, reduxLayer);
   const konvaImage = konvaLayer.findOne<Konva.Image>(`.${INITIAL_IMAGE_LAYER_IMAGE_NAME}`);
   const canvasImageSource = konvaImage?.image();
   let imageSourceNeedsUpdate = false;
   if (canvasImageSource instanceof HTMLImageElement) {
-    const image = layerState.image;
-    if (image && canvasImageSource.id !== getCALayerImageId(layerState.id, image.name)) {
+    const image = reduxLayer.image;
+    if (image && canvasImageSource.id !== getCALayerImageId(reduxLayer.id, image.name)) {
       imageSourceNeedsUpdate = true;
     } else if (!image) {
       imageSourceNeedsUpdate = true;
@@ -582,20 +557,15 @@ const renderIILayer = (
   }
 
   if (imageSourceNeedsUpdate) {
-    updateIILayerImageSource(stage, konvaLayer, layerState, getImageDTO);
+    updateInitialImageLayerImageSource(stage, konvaLayer, reduxLayer);
   } else if (konvaImage) {
-    updateIILayerImageAttrs(stage, konvaImage, layerState);
+    updateInitialImageLayerImageAttrs(stage, konvaImage, reduxLayer);
   }
 };
 
-/**
- * Creates a control adapter layer.
- * @param stage The konva stage
- * @param layerState The control adapter layer state
- */
-const createCALayer = (stage: Konva.Stage, layerState: ControlAdapterLayer): Konva.Layer => {
+const createControlNetLayer = (stage: Konva.Stage, reduxLayer: ControlAdapterLayer): Konva.Layer => {
   const konvaLayer = new Konva.Layer({
-    id: layerState.id,
+    id: reduxLayer.id,
     name: CA_LAYER_NAME,
     imageSmoothingEnabled: true,
     listening: false,
@@ -604,53 +574,39 @@ const createCALayer = (stage: Konva.Stage, layerState: ControlAdapterLayer): Kon
   return konvaLayer;
 };
 
-/**
- * Creates a control adapter layer image.
- * @param konvaLayer The konva layer
- * @param imageEl The image element
- */
-const createCALayerImage = (konvaLayer: Konva.Layer, imageEl: HTMLImageElement): Konva.Image => {
+const createControlNetLayerImage = (konvaLayer: Konva.Layer, image: HTMLImageElement): Konva.Image => {
   const konvaImage = new Konva.Image({
     name: CA_LAYER_IMAGE_NAME,
-    image: imageEl,
+    image,
   });
   konvaLayer.add(konvaImage);
   return konvaImage;
 };
 
-/**
- * Updates the image source for a control adapter layer. This includes loading the image from the server and updating the konva image.
- * @param stage The konva stage
- * @param konvaLayer The konva layer
- * @param layerState The control adapter layer state
- * @param getImageDTO A function to retrieve an image DTO from the server, used to update the image source
- */
-const updateCALayerImageSource = async (
+const updateControlNetLayerImageSource = async (
   stage: Konva.Stage,
   konvaLayer: Konva.Layer,
-  layerState: ControlAdapterLayer,
-  getImageDTO: (imageName: string) => Promise<ImageDTO | null>
-): Promise<void> => {
-  const image = layerState.controlAdapter.processedImage ?? layerState.controlAdapter.image;
+  reduxLayer: ControlAdapterLayer
+) => {
+  const image = reduxLayer.controlAdapter.processedImage ?? reduxLayer.controlAdapter.image;
   if (image) {
     const imageName = image.name;
-    const imageDTO = await getImageDTO(imageName);
-    if (!imageDTO) {
-      return;
-    }
+    const req = getStore().dispatch(imagesApi.endpoints.getImageDTO.initiate(imageName));
+    const imageDTO = await req.unwrap();
+    req.unsubscribe();
     const imageEl = new Image();
-    const imageId = getCALayerImageId(layerState.id, imageName);
+    const imageId = getCALayerImageId(reduxLayer.id, imageName);
     imageEl.onload = () => {
       // Find the existing image or create a new one - must find using the name, bc the id may have just changed
       const konvaImage =
-        konvaLayer.findOne<Konva.Image>(`.${CA_LAYER_IMAGE_NAME}`) ?? createCALayerImage(konvaLayer, imageEl);
+        konvaLayer.findOne<Konva.Image>(`.${CA_LAYER_IMAGE_NAME}`) ?? createControlNetLayerImage(konvaLayer, imageEl);
 
       // Update the image's attributes
       konvaImage.setAttrs({
         id: imageId,
         image: imageEl,
       });
-      updateCALayerImageAttrs(stage, konvaImage, layerState);
+      updateControlNetLayerImageAttrs(stage, konvaImage, reduxLayer);
       // Must cache after this to apply the filters
       konvaImage.cache();
       imageEl.id = imageId;
@@ -661,17 +617,11 @@ const updateCALayerImageSource = async (
   }
 };
 
-/**
- * Updates the image attributes for a control adapter layer's image (width, height, visibility, opacity, filters).
- * @param stage The konva stage
- * @param konvaImage The konva image
- * @param layerState The control adapter layer state
- */
-const updateCALayerImageAttrs = (
+const updateControlNetLayerImageAttrs = (
   stage: Konva.Stage,
   konvaImage: Konva.Image,
-  layerState: ControlAdapterLayer
-): void => {
+  reduxLayer: ControlAdapterLayer
+) => {
   let needsCache = false;
   // Konva erroneously reports NaN for width and height when the stage is hidden. This causes errors when caching,
   // but it doesn't seem to break anything.
@@ -682,47 +632,36 @@ const updateCALayerImageAttrs = (
   if (
     konvaImage.width() !== newWidth ||
     konvaImage.height() !== newHeight ||
-    konvaImage.visible() !== layerState.isEnabled ||
-    hasFilter !== layerState.isFilterEnabled
+    konvaImage.visible() !== reduxLayer.isEnabled ||
+    hasFilter !== reduxLayer.isFilterEnabled
   ) {
     konvaImage.setAttrs({
-      opacity: layerState.opacity,
+      opacity: reduxLayer.opacity,
       scaleX: 1,
       scaleY: 1,
       width: stage.width() / stage.scaleX(),
       height: stage.height() / stage.scaleY(),
-      visible: layerState.isEnabled,
-      filters: layerState.isFilterEnabled ? [LightnessToAlphaFilter] : [],
+      visible: reduxLayer.isEnabled,
+      filters: reduxLayer.isFilterEnabled ? [LightnessToAlphaFilter] : [],
     });
     needsCache = true;
   }
-  if (konvaImage.opacity() !== layerState.opacity) {
-    konvaImage.opacity(layerState.opacity);
+  if (konvaImage.opacity() !== reduxLayer.opacity) {
+    konvaImage.opacity(reduxLayer.opacity);
   }
   if (needsCache) {
     konvaImage.cache();
   }
 };
 
-/**
- * Renders a control adapter layer. If the layer doesn't already exist, it is created. Otherwise, the layer is updated
- * with the current image source and attributes.
- * @param stage The konva stage
- * @param layerState The control adapter layer state
- * @param getImageDTO A function to retrieve an image DTO from the server, used to update the image source
- */
-const renderCALayer = (
-  stage: Konva.Stage,
-  layerState: ControlAdapterLayer,
-  getImageDTO: (imageName: string) => Promise<ImageDTO | null>
-): void => {
-  const konvaLayer = stage.findOne<Konva.Layer>(`#${layerState.id}`) ?? createCALayer(stage, layerState);
+const renderControlNetLayer = (stage: Konva.Stage, reduxLayer: ControlAdapterLayer) => {
+  const konvaLayer = stage.findOne<Konva.Layer>(`#${reduxLayer.id}`) ?? createControlNetLayer(stage, reduxLayer);
   const konvaImage = konvaLayer.findOne<Konva.Image>(`.${CA_LAYER_IMAGE_NAME}`);
   const canvasImageSource = konvaImage?.image();
   let imageSourceNeedsUpdate = false;
   if (canvasImageSource instanceof HTMLImageElement) {
-    const image = layerState.controlAdapter.processedImage ?? layerState.controlAdapter.image;
-    if (image && canvasImageSource.id !== getCALayerImageId(layerState.id, image.name)) {
+    const image = reduxLayer.controlAdapter.processedImage ?? reduxLayer.controlAdapter.image;
+    if (image && canvasImageSource.id !== getCALayerImageId(reduxLayer.id, image.name)) {
       imageSourceNeedsUpdate = true;
     } else if (!image) {
       imageSourceNeedsUpdate = true;
@@ -732,46 +671,44 @@ const renderCALayer = (
   }
 
   if (imageSourceNeedsUpdate) {
-    updateCALayerImageSource(stage, konvaLayer, layerState, getImageDTO);
+    updateControlNetLayerImageSource(stage, konvaLayer, reduxLayer);
   } else if (konvaImage) {
-    updateCALayerImageAttrs(stage, konvaImage, layerState);
+    updateControlNetLayerImageAttrs(stage, konvaImage, reduxLayer);
   }
 };
 
 /**
  * Renders the layers on the stage.
- * @param stage The konva stage
- * @param layerStates Array of all layer states
- * @param globalMaskLayerOpacity The global mask layer opacity
- * @param tool The current tool
- * @param getImageDTO A function to retrieve an image DTO from the server, used to update the image source
- * @param onLayerPosChanged Callback for when the layer's position changes
+ * @param stage The konva stage to render on.
+ * @param reduxLayers Array of the layers from the redux store.
+ * @param layerOpacity The opacity of the layer.
+ * @param onLayerPosChanged Callback for when the layer's position changes. This is optional to allow for offscreen rendering.
+ * @returns
  */
 const renderLayers = (
   stage: Konva.Stage,
-  layerStates: Layer[],
+  reduxLayers: Layer[],
   globalMaskLayerOpacity: number,
   tool: Tool,
-  getImageDTO: (imageName: string) => Promise<ImageDTO | null>,
   onLayerPosChanged?: (layerId: string, x: number, y: number) => void
-): void => {
-  const layerIds = layerStates.filter(isRenderableLayer).map(mapId);
+) => {
+  const reduxLayerIds = reduxLayers.filter(isRenderableLayer).map(mapId);
   // Remove un-rendered layers
   for (const konvaLayer of stage.find<Konva.Layer>(selectRenderableLayers)) {
-    if (!layerIds.includes(konvaLayer.id())) {
+    if (!reduxLayerIds.includes(konvaLayer.id())) {
       konvaLayer.destroy();
     }
   }
 
-  for (const layer of layerStates) {
-    if (isRegionalGuidanceLayer(layer)) {
-      renderRGLayer(stage, layer, globalMaskLayerOpacity, tool, onLayerPosChanged);
+  for (const reduxLayer of reduxLayers) {
+    if (isRegionalGuidanceLayer(reduxLayer)) {
+      renderRegionalGuidanceLayer(stage, reduxLayer, globalMaskLayerOpacity, tool, onLayerPosChanged);
     }
-    if (isControlAdapterLayer(layer)) {
-      renderCALayer(stage, layer, getImageDTO);
+    if (isControlAdapterLayer(reduxLayer)) {
+      renderControlNetLayer(stage, reduxLayer);
     }
-    if (isInitialImageLayer(layer)) {
-      renderIILayer(stage, layer, getImageDTO);
+    if (isInitialImageLayer(reduxLayer)) {
+      renderInitialImageLayer(stage, reduxLayer);
     }
     // IP Adapter layers are not rendered
   }
@@ -779,12 +716,13 @@ const renderLayers = (
 
 /**
  * Creates a bounding box rect for a layer.
- * @param layerState The layer state for the layer to create the bounding box for
- * @param konvaLayer The konva layer to attach the bounding box to
+ * @param reduxLayer The redux layer to create the bounding box for.
+ * @param konvaLayer The konva layer to attach the bounding box to.
+ * @param onBboxMouseDown Callback for when the bounding box is clicked.
  */
-const createBboxRect = (layerState: Layer, konvaLayer: Konva.Layer): Konva.Rect => {
+const createBboxRect = (reduxLayer: Layer, konvaLayer: Konva.Layer) => {
   const rect = new Konva.Rect({
-    id: getLayerBboxId(layerState.id),
+    id: getLayerBboxId(reduxLayer.id),
     name: LAYER_BBOX_NAME,
     strokeWidth: 1,
     visible: false,
@@ -795,12 +733,12 @@ const createBboxRect = (layerState: Layer, konvaLayer: Konva.Layer): Konva.Rect
 
 /**
  * Renders the bounding boxes for the layers.
- * @param stage The konva stage
- * @param layerStates An array of layers to draw bboxes for
+ * @param stage The konva stage to render on
+ * @param reduxLayers An array of all redux layers to draw bboxes for
  * @param tool The current tool
  * @returns
  */
-const renderBboxes = (stage: Konva.Stage, layerStates: Layer[], tool: Tool): void => {
+const renderBboxes = (stage: Konva.Stage, reduxLayers: Layer[], tool: Tool) => {
   // Hide all bboxes so they don't interfere with getClientRect
   for (const bboxRect of stage.find<Konva.Rect>(`.${LAYER_BBOX_NAME}`)) {
     bboxRect.visible(false);
@@ -811,39 +749,39 @@ const renderBboxes = (stage: Konva.Stage, layerStates: Layer[], tool: Tool): voi
     return;
   }
 
-  for (const layer of layerStates.filter(isRegionalGuidanceLayer)) {
-    if (!layer.bbox) {
+  for (const reduxLayer of reduxLayers.filter(isRegionalGuidanceLayer)) {
+    if (!reduxLayer.bbox) {
       continue;
     }
-    const konvaLayer = stage.findOne<Konva.Layer>(`#${layer.id}`);
-    assert(konvaLayer, `Layer ${layer.id} not found in stage`);
+    const konvaLayer = stage.findOne<Konva.Layer>(`#${reduxLayer.id}`);
+    assert(konvaLayer, `Layer ${reduxLayer.id} not found in stage`);
 
-    const bboxRect = konvaLayer.findOne<Konva.Rect>(`.${LAYER_BBOX_NAME}`) ?? createBboxRect(layer, konvaLayer);
+    const bboxRect = konvaLayer.findOne<Konva.Rect>(`.${LAYER_BBOX_NAME}`) ?? createBboxRect(reduxLayer, konvaLayer);
 
     bboxRect.setAttrs({
-      visible: !layer.bboxNeedsUpdate,
-      listening: layer.isSelected,
-      x: layer.bbox.x,
-      y: layer.bbox.y,
-      width: layer.bbox.width,
-      height: layer.bbox.height,
-      stroke: layer.isSelected ? BBOX_SELECTED_STROKE : '',
+      visible: !reduxLayer.bboxNeedsUpdate,
+      listening: reduxLayer.isSelected,
+      x: reduxLayer.bbox.x,
+      y: reduxLayer.bbox.y,
+      width: reduxLayer.bbox.width,
+      height: reduxLayer.bbox.height,
+      stroke: reduxLayer.isSelected ? BBOX_SELECTED_STROKE : '',
     });
   }
 };
 
 /**
  * Calculates the bbox of each regional guidance layer. Only calculates if the mask has changed.
- * @param stage The konva stage
- * @param layerStates An array of layers to calculate bboxes for
+ * @param stage The konva stage to render on.
+ * @param reduxLayers An array of redux layers to calculate bboxes for
  * @param onBboxChanged Callback for when the bounding box changes
  */
 const updateBboxes = (
   stage: Konva.Stage,
-  layerStates: Layer[],
+  reduxLayers: Layer[],
   onBboxChanged: (layerId: string, bbox: IRect | null) => void
-): void => {
-  for (const rgLayer of layerStates.filter(isRegionalGuidanceLayer)) {
+) => {
+  for (const rgLayer of reduxLayers.filter(isRegionalGuidanceLayer)) {
     const konvaLayer = stage.findOne<Konva.Layer>(`#${rgLayer.id}`);
     assert(konvaLayer, `Layer ${rgLayer.id} not found in stage`);
     // We only need to recalculate the bbox if the layer has changed
@@ -870,7 +808,7 @@ const updateBboxes = (
 
 /**
  * Creates the background layer for the stage.
- * @param stage The konva stage
+ * @param stage The konva stage to render on
  */
 const createBackgroundLayer = (stage: Konva.Stage): Konva.Layer => {
   const layer = new Konva.Layer({
@@ -891,17 +829,17 @@ const createBackgroundLayer = (stage: Konva.Stage): Konva.Layer => {
   image.onload = () => {
     background.fillPatternImage(image);
   };
-  image.src = TRANSPARENCY_CHECKER_PATTERN;
+  image.src = STAGE_BG_DATAURL;
   return layer;
 };
 
 /**
  * Renders the background layer for the stage.
- * @param stage The konva stage
+ * @param stage The konva stage to render on
  * @param width The unscaled width of the canvas
  * @param height The unscaled height of the canvas
  */
-const renderBackground = (stage: Konva.Stage, width: number, height: number): void => {
+const renderBackground = (stage: Konva.Stage, width: number, height: number) => {
   const layer = stage.findOne<Konva.Layer>(`#${BACKGROUND_LAYER_ID}`) ?? createBackgroundLayer(stage);
 
   const background = layer.findOne<Konva.Rect>(`#${BACKGROUND_RECT_ID}`);
@@ -942,10 +880,6 @@ const arrangeLayers = (stage: Konva.Stage, layerIds: string[]): void => {
   stage.findOne<Konva.Layer>(`#${TOOL_PREVIEW_LAYER_ID}`)?.zIndex(nextZIndex++);
 };
 
-/**
- * Creates the "no layers" fallback layer
- * @param stage The konva stage
- */
 const createNoLayersMessageLayer = (stage: Konva.Stage): Konva.Layer => {
   const noLayersMessageLayer = new Konva.Layer({
     id: NO_LAYERS_MESSAGE_LAYER_ID,
@@ -957,7 +891,7 @@ const createNoLayersMessageLayer = (stage: Konva.Stage): Konva.Layer => {
     y: 0,
     align: 'center',
     verticalAlign: 'middle',
-    text: t('controlLayers.noLayersAdded', 'No Layers Added'),
+    text: t('controlLayers.noLayersAdded'),
     fontFamily: '"Inter Variable", sans-serif',
     fontStyle: '600',
     fill: 'white',
@@ -967,14 +901,7 @@ const createNoLayersMessageLayer = (stage: Konva.Stage): Konva.Layer => {
   return noLayersMessageLayer;
 };
 
-/**
- * Renders the "no layers" message when there are no layers to render
- * @param stage The konva stage
- * @param layerCount The current number of layers
- * @param width The target width of the text
- * @param height The target height of the text
- */
-const renderNoLayersMessage = (stage: Konva.Stage, layerCount: number, width: number, height: number): void => {
+const renderNoLayersMessage = (stage: Konva.Stage, layerCount: number, width: number, height: number) => {
   const noLayersMessageLayer =
     stage.findOne<Konva.Layer>(`#${NO_LAYERS_MESSAGE_LAYER_ID}`) ?? createNoLayersMessageLayer(stage);
   if (layerCount === 0) {
@@ -1009,3 +936,20 @@ export const debouncedRenderers = {
   arrangeLayers: debounce(arrangeLayers, DEBOUNCE_MS),
   updateBboxes: debounce(updateBboxes, DEBOUNCE_MS),
 };
+
+/**
+ * Calculates the lightness (HSL) of a given pixel and sets the alpha channel to that value.
+ * This is useful for edge maps and other masks, to make the black areas transparent.
+ * @param imageData The image data to apply the filter to
+ */
+const LightnessToAlphaFilter = (imageData: ImageData) => {
+  const len = imageData.data.length / 4;
+  for (let i = 0; i < len; i++) {
+    const r = imageData.data[i * 4 + 0] as number;
+    const g = imageData.data[i * 4 + 1] as number;
+    const b = imageData.data[i * 4 + 2] as number;
+    const cMin = Math.min(r, g, b);
+    const cMax = Math.max(r, g, b);
+    imageData.data[i * 4 + 3] = (cMin + cMax) / 2;
+  }
+};

invokeai/frontend/web/src/features/gallery/components/ImageViewer/ImageComparisonHover.tsx

@@ -3,7 +3,7 @@ import { useAppSelector } from 'app/store/storeHooks';
 import { useBoolean } from 'common/hooks/useBoolean';
 import { preventDefault } from 'common/util/stopPropagation';
 import type { Dimensions } from 'features/canvas/store/canvasTypes';
-import { TRANSPARENCY_CHECKER_PATTERN } from 'features/controlLayers/konva/constants';
+import { STAGE_BG_DATAURL } from 'features/controlLayers/util/renderers';
 import { ImageComparisonLabel } from 'features/gallery/components/ImageViewer/ImageComparisonLabel';
 import { memo, useMemo, useRef } from 'react';
 
@@ -78,7 +78,7 @@ export const ImageComparisonHover = memo(({ firstImage, secondImage, containerDi
               left={0}
               right={0}
               bottom={0}
-              backgroundImage={TRANSPARENCY_CHECKER_PATTERN}
+              backgroundImage={STAGE_BG_DATAURL}
               backgroundRepeat="repeat"
               opacity={0.2}
             />

invokeai/frontend/web/src/features/gallery/components/ImageViewer/ImageComparisonSlider.tsx

@@ -2,7 +2,7 @@ import { Box, Flex, Icon, Image } from '@invoke-ai/ui-library';
 import { useAppSelector } from 'app/store/storeHooks';
 import { preventDefault } from 'common/util/stopPropagation';
 import type { Dimensions } from 'features/canvas/store/canvasTypes';
-import { TRANSPARENCY_CHECKER_PATTERN } from 'features/controlLayers/konva/constants';
+import { STAGE_BG_DATAURL } from 'features/controlLayers/util/renderers';
 import { ImageComparisonLabel } from 'features/gallery/components/ImageViewer/ImageComparisonLabel';
 import { memo, useCallback, useEffect, useMemo, useRef, useState } from 'react';
 import { PiCaretLeftBold, PiCaretRightBold } from 'react-icons/pi';
@@ -120,7 +120,7 @@ export const ImageComparisonSlider = memo(({ firstImage, secondImage, containerD
             left={0}
             right={0}
             bottom={0}
-            backgroundImage={TRANSPARENCY_CHECKER_PATTERN}
+            backgroundImage={STAGE_BG_DATAURL}
             backgroundRepeat="repeat"
             opacity={0.2}
           />

invokeai/frontend/web/src/features/metadata/util/parsers.ts

@@ -4,7 +4,7 @@ import {
   initialT2IAdapter,
 } from 'features/controlAdapters/util/buildControlAdapter';
 import { buildControlAdapterProcessor } from 'features/controlAdapters/util/buildControlAdapterProcessor';
-import { getCALayerId, getIPALayerId, INITIAL_IMAGE_LAYER_ID } from 'features/controlLayers/konva/naming';
+import { getCALayerId, getIPALayerId, INITIAL_IMAGE_LAYER_ID } from 'features/controlLayers/store/controlLayersSlice';
 import type { ControlAdapterLayer, InitialImageLayer, IPAdapterLayer, Layer } from 'features/controlLayers/store/types';
 import { zLayer } from 'features/controlLayers/store/types';
 import {

invokeai/frontend/web/src/features/metadata/util/recallers.ts

@@ -6,10 +6,12 @@ import {
   ipAdaptersReset,
   t2iAdaptersReset,
 } from 'features/controlAdapters/store/controlAdaptersSlice';
-import { getCALayerId, getIPALayerId, getRGLayerId } from 'features/controlLayers/konva/naming';
 import {
   allLayersDeleted,
   caLayerRecalled,
+  getCALayerId,
+  getIPALayerId,
+  getRGLayerId,
   heightChanged,
   iiLayerRecalled,
   ipaLayerRecalled,

invokeai/frontend/web/src/features/nodes/util/graph/generation/addControlLayers.ts

@@ -1,10 +1,6 @@
 import { getStore } from 'app/store/nanostores/store';
 import type { RootState } from 'app/store/store';
 import { deepClone } from 'common/util/deepClone';
-import openBase64ImageInTab from 'common/util/openBase64ImageInTab';
-import { blobToDataURL } from 'features/canvas/util/blobToDataURL';
-import { RG_LAYER_NAME } from 'features/controlLayers/konva/naming';
-import { renderers } from 'features/controlLayers/konva/renderers';
 import {
   isControlAdapterLayer,
   isInitialImageLayer,
@@ -20,6 +16,7 @@ import type {
   ProcessorConfig,
   T2IAdapterConfigV2,
 } from 'features/controlLayers/util/controlAdapters';
+import { getRegionalPromptLayerBlobs } from 'features/controlLayers/util/getLayerBlobs';
 import type { ImageField } from 'features/nodes/types/common';
 import {
   CONTROL_NET_COLLECT,
@@ -34,13 +31,11 @@ import {
   T2I_ADAPTER_COLLECT,
 } from 'features/nodes/util/graph/constants';
 import type { Graph } from 'features/nodes/util/graph/generation/Graph';
-import Konva from 'konva';
 import { size } from 'lodash-es';
 import { getImageDTO, imagesApi } from 'services/api/endpoints/images';
 import type { BaseModelType, ImageDTO, Invocation } from 'services/api/types';
 import { assert } from 'tsafe';
 
-//#region addControlLayers
 /**
  * Adds the control layers to the graph
  * @param state The app root state
@@ -95,7 +90,7 @@ export const addControlLayers = async (
 
   const validRGLayers = validLayers.filter(isRegionalGuidanceLayer);
   const layerIds = validRGLayers.map((l) => l.id);
-  const blobs = await getRGLayerBlobs(layerIds);
+  const blobs = await getRegionalPromptLayerBlobs(layerIds);
   assert(size(blobs) === size(layerIds), 'Mismatch between layer IDs and blobs');
 
   for (const layer of validRGLayers) {
@@ -262,7 +257,6 @@ export const addControlLayers = async (
   g.upsertMetadata({ control_layers: { layers: validLayers, version: state.controlLayers.present._version } });
   return validLayers;
 };
-//#endregion
 
 //#region Control Adapters
 const addGlobalControlAdapterToGraph = (
@@ -515,7 +509,7 @@ const isValidLayer = (layer: Layer, base: BaseModelType) => {
 };
 //#endregion
 
-//#region getMaskImage
+//#region Helpers
 const getMaskImage = async (layer: RegionalGuidanceLayer, blob: Blob): Promise<ImageDTO> => {
   if (layer.uploadedMaskImage) {
     const imageDTO = await getImageDTO(layer.uploadedMaskImage.name);
@@ -535,9 +529,7 @@ const getMaskImage = async (layer: RegionalGuidanceLayer, blob: Blob): Promise<I
   dispatch(rgLayerMaskImageUploaded({ layerId: layer.id, imageDTO }));
   return imageDTO;
 };
-//#endregion
 
-//#region buildControlImage
 const buildControlImage = (
   image: ImageWithDims | null,
   processedImage: ImageWithDims | null,
@@ -557,61 +549,3 @@ const buildControlImage = (
   assert(false, 'Attempted to add unprocessed control image');
 };
 //#endregion
-
-//#region getRGLayerBlobs
-/**
- * Get the blobs of all regional prompt layers. Only visible layers are returned.
- * @param layerIds The IDs of the layers to get blobs for. If not provided, all regional prompt layers are used.
- * @param preview Whether to open a new tab displaying each layer.
- * @returns A map of layer IDs to blobs.
- */
-const getRGLayerBlobs = async (layerIds?: string[], preview: boolean = false): Promise<Record<string, Blob>> => {
-  const state = getStore().getState();
-  const { layers } = state.controlLayers.present;
-  const { width, height } = state.controlLayers.present.size;
-  const reduxLayers = layers.filter(isRegionalGuidanceLayer);
-  const container = document.createElement('div');
-  const stage = new Konva.Stage({ container, width, height });
-  renderers.renderLayers(stage, reduxLayers, 1, 'brush', getImageDTO);
-
-  const konvaLayers = stage.find<Konva.Layer>(`.${RG_LAYER_NAME}`);
-  const blobs: Record<string, Blob> = {};
-
-  // First remove all layers
-  for (const layer of konvaLayers) {
-    layer.remove();
-  }
-
-  // Next render each layer to a blob
-  for (const layer of konvaLayers) {
-    if (layerIds && !layerIds.includes(layer.id())) {
-      continue;
-    }
-    const reduxLayer = reduxLayers.find((l) => l.id === layer.id());
-    assert(reduxLayer, `Redux layer ${layer.id()} not found`);
-    stage.add(layer);
-    const blob = await new Promise<Blob>((resolve) => {
-      stage.toBlob({
-        callback: (blob) => {
-          assert(blob, 'Blob is null');
-          resolve(blob);
-        },
-      });
-    });
-
-    if (preview) {
-      const base64 = await blobToDataURL(blob);
-      openBase64ImageInTab([
-        {
-          base64,
-          caption: `${reduxLayer.id}: ${reduxLayer.positivePrompt} / ${reduxLayer.negativePrompt}`,
-        },
-      ]);
-    }
-    layer.remove();
-    blobs[layer.id()] = blob;
-  }
-
-  return blobs;
-};
-//#endregion

invokeai/frontend/web/src/features/parameters/components/ImageSize/AspectRatioCanvasPreview.tsx

@@ -1,17 +1,8 @@
 import { Flex } from '@invoke-ai/ui-library';
-import { useStore } from '@nanostores/react';
 import { StageComponent } from 'features/controlLayers/components/StageComponent';
-import { $isPreviewVisible } from 'features/controlLayers/store/controlLayersSlice';
-import { AspectRatioIconPreview } from 'features/parameters/components/ImageSize/AspectRatioIconPreview';
 import { memo } from 'react';
 
 export const AspectRatioCanvasPreview = memo(() => {
-  const isPreviewVisible = useStore($isPreviewVisible);
-
-  if (!isPreviewVisible) {
-    return <AspectRatioIconPreview />;
-  }
-
   return (
     <Flex w="full" h="full" alignItems="center" justifyContent="center" position="relative">
       <StageComponent asPreview />

invokeai/frontend/web/src/features/settingsAccordions/components/ImageSettingsAccordion/ImageSizeLinear.tsx

@@ -3,12 +3,15 @@ import { aspectRatioChanged, heightChanged, widthChanged } from 'features/contro
 import { ParamHeight } from 'features/parameters/components/Core/ParamHeight';
 import { ParamWidth } from 'features/parameters/components/Core/ParamWidth';
 import { AspectRatioCanvasPreview } from 'features/parameters/components/ImageSize/AspectRatioCanvasPreview';
+import { AspectRatioIconPreview } from 'features/parameters/components/ImageSize/AspectRatioIconPreview';
 import { ImageSize } from 'features/parameters/components/ImageSize/ImageSize';
 import type { AspectRatioState } from 'features/parameters/components/ImageSize/types';
+import { activeTabNameSelector } from 'features/ui/store/uiSelectors';
 import { memo, useCallback } from 'react';
 
 export const ImageSizeLinear = memo(() => {
   const dispatch = useAppDispatch();
+  const tab = useAppSelector(activeTabNameSelector);
   const width = useAppSelector((s) => s.controlLayers.present.size.width);
   const height = useAppSelector((s) => s.controlLayers.present.size.height);
   const aspectRatioState = useAppSelector((s) => s.controlLayers.present.size.aspectRatio);
@@ -47,7 +50,7 @@ export const ImageSizeLinear = memo(() => {
       aspectRatioState={aspectRatioState}
       heightComponent={<ParamHeight />}
       widthComponent={<ParamWidth />}
-      previewComponent={<AspectRatioCanvasPreview />}
+      previewComponent={tab === 'generation' ? <AspectRatioCanvasPreview /> : <AspectRatioIconPreview />}
       onChangeAspectRatioState={onChangeAspectRatioState}
       onChangeWidth={onChangeWidth}
       onChangeHeight={onChangeHeight}

invokeai/frontend/web/src/features/ui/components/ParametersPanelTextToImage.tsx

@@ -3,7 +3,6 @@ import { Box, Flex, Tab, TabList, TabPanel, TabPanels, Tabs } from '@invoke-ai/u
 import { useAppDispatch, useAppSelector } from 'app/store/storeHooks';
 import { overlayScrollbarsParams } from 'common/components/OverlayScrollbars/constants';
 import { ControlLayersPanelContent } from 'features/controlLayers/components/ControlLayersPanelContent';
-import { $isPreviewVisible } from 'features/controlLayers/store/controlLayersSlice';
 import { isImageViewerOpenChanged } from 'features/gallery/store/gallerySlice';
 import { Prompts } from 'features/parameters/components/Prompts/Prompts';
 import QueueControls from 'features/queue/components/QueueControls';
@@ -54,7 +53,6 @@ const ParametersPanelTextToImage = () => {
       if (i === 1) {
         dispatch(isImageViewerOpenChanged(false));
       }
-      $isPreviewVisible.set(i === 0);
     },
     [dispatch]
   );
@@ -68,7 +66,6 @@ const ParametersPanelTextToImage = () => {
             <Flex gap={2} flexDirection="column" h="full" w="full">
               {isSDXL ? <SDXLPrompts /> : <Prompts />}
               <Tabs
-                defaultIndex={0}
                 variant="enclosed"
                 display="flex"
                 flexDir="column"

tests/app/services/download/test_download_queue.py

@@ -2,18 +2,14 @@
 
 import re
 import time
-from contextlib import contextmanager
 from pathlib import Path
-from typing import Any, Generator, Optional
 
 import pytest
 from pydantic.networks import AnyHttpUrl
 from requests.sessions import Session
-from requests_testadapter import TestAdapter
+from requests_testadapter import TestAdapter, TestSession
 
-from invokeai.app.services.config import get_config
-from invokeai.app.services.config.config_default import URLRegexTokenPair
-from invokeai.app.services.download import DownloadJob, DownloadJobStatus, DownloadQueueService, MultiFileDownloadJob
+from invokeai.app.services.download import DownloadJob, DownloadJobStatus, DownloadQueueService
 from invokeai.app.services.events.events_common import (
     DownloadCancelledEvent,
     DownloadCompleteEvent,
@@ -21,23 +17,56 @@ from invokeai.app.services.events.events_common import (
     DownloadProgressEvent,
     DownloadStartedEvent,
 )
-from invokeai.backend.model_manager.metadata import HuggingFaceMetadataFetch, ModelMetadataWithFiles, RemoteModelFile
-from tests.backend.model_manager.model_manager_fixtures import *  # noqa F403
 from tests.test_nodes import TestEventService
 
 # Prevent pytest deprecation warnings
-TestAdapter.__test__ = False
+TestAdapter.__test__ = False  # type: ignore
 
 
-@pytest.mark.timeout(timeout=10, method="thread")
-def test_basic_queue_download(tmp_path: Path, mm2_session: Session) -> None:
+@pytest.fixture
+def session() -> Session:
+    sess = TestSession()
+    for i in ["12345", "9999", "54321"]:
+        content = (
+            b"I am a safetensors file " + bytearray(i, "utf-8") + bytearray(32_000)
+        )  # for pause tests, must make content large
+        sess.mount(
+            f"http://www.civitai.com/models/{i}",
+            TestAdapter(
+                content,
+                headers={
+                    "Content-Length": len(content),
+                    "Content-Disposition": f'filename="mock{i}.safetensors"',
+                },
+            ),
+        )
+
+    # here are some malformed URLs to test
+    # missing the content length
+    sess.mount(
+        "http://www.civitai.com/models/missing",
+        TestAdapter(
+            b"Missing content length",
+            headers={
+                "Content-Disposition": 'filename="missing.txt"',
+            },
+        ),
+    )
+    # not found test
+    sess.mount("http://www.civitai.com/models/broken", TestAdapter(b"Not found", status=404))
+
+    return sess
+
+
+@pytest.mark.timeout(timeout=20, method="thread")
+def test_basic_queue_download(tmp_path: Path, session: Session) -> None:
     events = set()
 
-    def event_handler(job: DownloadJob, excp: Optional[Exception] = None) -> None:
+    def event_handler(job: DownloadJob) -> None:
         events.add(job.status)
 
     queue = DownloadQueueService(
-        requests_session=mm2_session,
+        requests_session=session,
     )
     queue.start()
     job = queue.download(
@@ -53,17 +82,16 @@ def test_basic_queue_download(tmp_path: Path, mm2_session: Session) -> None:
     queue.join()
 
     assert job.status == DownloadJobStatus("completed"), "expected job status to be completed"
-    assert job.download_path == tmp_path / "mock12345.safetensors"
     assert Path(tmp_path, "mock12345.safetensors").exists(), f"expected {tmp_path}/mock12345.safetensors to exist"
 
     assert events == {DownloadJobStatus.RUNNING, DownloadJobStatus.COMPLETED}
     queue.stop()
 
 
-@pytest.mark.timeout(timeout=10, method="thread")
-def test_errors(tmp_path: Path, mm2_session: Session) -> None:
+@pytest.mark.timeout(timeout=20, method="thread")
+def test_errors(tmp_path: Path, session: Session) -> None:
     queue = DownloadQueueService(
-        requests_session=mm2_session,
+        requests_session=session,
     )
     queue.start()
 
@@ -82,11 +110,11 @@ def test_errors(tmp_path: Path, mm2_session: Session) -> None:
     queue.stop()
 
 
-@pytest.mark.timeout(timeout=10, method="thread")
-def test_event_bus(tmp_path: Path, mm2_session: Session) -> None:
+@pytest.mark.timeout(timeout=20, method="thread")
+def test_event_bus(tmp_path: Path, session: Session) -> None:
     event_bus = TestEventService()
 
-    queue = DownloadQueueService(requests_session=mm2_session, event_bus=event_bus)
+    queue = DownloadQueueService(requests_session=session, event_bus=event_bus)
     queue.start()
     queue.download(
         source=AnyHttpUrl("http://www.civitai.com/models/12345"),
@@ -118,10 +146,10 @@ def test_event_bus(tmp_path: Path, mm2_session: Session) -> None:
     queue.stop()
 
 
-@pytest.mark.timeout(timeout=10, method="thread")
-def test_broken_callbacks(tmp_path: Path, mm2_session: Session, capsys) -> None:
+@pytest.mark.timeout(timeout=20, method="thread")
+def test_broken_callbacks(tmp_path: Path, session: Session, capsys) -> None:
     queue = DownloadQueueService(
-        requests_session=mm2_session,
+        requests_session=session,
     )
     queue.start()
 
@@ -150,11 +178,11 @@ def test_broken_callbacks(tmp_path: Path, mm2_session: Session, capsys) -> None:
     queue.stop()
 
 
-@pytest.mark.timeout(timeout=10, method="thread")
-def test_cancel(tmp_path: Path, mm2_session: Session) -> None:
+@pytest.mark.timeout(timeout=15, method="thread")
+def test_cancel(tmp_path: Path, session: Session) -> None:
     event_bus = TestEventService()
 
-    queue = DownloadQueueService(requests_session=mm2_session, event_bus=event_bus)
+    queue = DownloadQueueService(requests_session=session, event_bus=event_bus)
     queue.start()
 
     cancelled = False
@@ -166,6 +194,9 @@ def test_cancel(tmp_path: Path, mm2_session: Session) -> None:
         nonlocal cancelled
         cancelled = True
 
+    def handler(signum, frame):
+        raise TimeoutError("Join took too long to return")
+
     job = queue.download(
         source=AnyHttpUrl("http://www.civitai.com/models/12345"),
         dest=tmp_path,
@@ -181,178 +212,3 @@ def test_cancel(tmp_path: Path, mm2_session: Session) -> None:
     assert isinstance(events[-1], DownloadCancelledEvent)
     assert events[-1].source == "http://www.civitai.com/models/12345"
     queue.stop()
-
-
-@pytest.mark.timeout(timeout=10, method="thread")
-def test_multifile_download(tmp_path: Path, mm2_session: Session) -> None:
-    fetcher = HuggingFaceMetadataFetch(mm2_session)
-    metadata = fetcher.from_id("stabilityai/sdxl-turbo")
-    assert isinstance(metadata, ModelMetadataWithFiles)
-    events = set()
-
-    def event_handler(job: DownloadJob | MultiFileDownloadJob, excp: Optional[Exception] = None) -> None:
-        events.add(job.status)
-
-    queue = DownloadQueueService(
-        requests_session=mm2_session,
-    )
-    queue.start()
-    job = queue.multifile_download(
-        parts=metadata.download_urls(session=mm2_session),
-        dest=tmp_path,
-        on_start=event_handler,
-        on_progress=event_handler,
-        on_complete=event_handler,
-        on_error=event_handler,
-    )
-    assert isinstance(job, MultiFileDownloadJob), "expected the job to be of type MultiFileDownloadJobBase"
-    queue.join()
-
-    assert job.status == DownloadJobStatus("completed"), "expected job status to be completed"
-    assert job.bytes > 0, "expected download bytes to be positive"
-    assert job.bytes == job.total_bytes, "expected download bytes to equal total bytes"
-    assert job.download_path == tmp_path / "sdxl-turbo"
-    assert Path(
-        tmp_path, "sdxl-turbo/model_index.json"
-    ).exists(), f"expected {tmp_path}/sdxl-turbo/model_inded.json to exist"
-    assert Path(
-        tmp_path, "sdxl-turbo/text_encoder/config.json"
-    ).exists(), f"expected {tmp_path}/sdxl-turbo/text_encoder/config.json to exist"
-
-    assert events == {DownloadJobStatus.RUNNING, DownloadJobStatus.COMPLETED}
-    queue.stop()
-
-
-@pytest.mark.timeout(timeout=10, method="thread")
-def test_multifile_download_error(tmp_path: Path, mm2_session: Session) -> None:
-    fetcher = HuggingFaceMetadataFetch(mm2_session)
-    metadata = fetcher.from_id("stabilityai/sdxl-turbo")
-    assert isinstance(metadata, ModelMetadataWithFiles)
-    events = set()
-
-    def event_handler(job: DownloadJob | MultiFileDownloadJob, excp: Optional[Exception] = None) -> None:
-        events.add(job.status)
-
-    queue = DownloadQueueService(
-        requests_session=mm2_session,
-    )
-    queue.start()
-    files = metadata.download_urls(session=mm2_session)
-    # this will give a 404 error
-    files.append(RemoteModelFile(url="https://test.com/missing_model.safetensors", path=Path("sdxl-turbo/broken")))
-    job = queue.multifile_download(
-        parts=files,
-        dest=tmp_path,
-        on_start=event_handler,
-        on_progress=event_handler,
-        on_complete=event_handler,
-        on_error=event_handler,
-    )
-    queue.join()
-
-    assert job.status == DownloadJobStatus("error"), "expected job status to be errored"
-    assert job.error_type is not None
-    assert "HTTPError(NOT FOUND)" in job.error_type
-    assert DownloadJobStatus.ERROR in events
-    queue.stop()
-
-
-@pytest.mark.timeout(timeout=10, method="thread")
-def test_multifile_cancel(tmp_path: Path, mm2_session: Session, monkeypatch: Any) -> None:
-    event_bus = TestEventService()
-
-    queue = DownloadQueueService(requests_session=mm2_session, event_bus=event_bus)
-    queue.start()
-
-    cancelled = False
-
-    def cancelled_callback(job: DownloadJob) -> None:
-        nonlocal cancelled
-        cancelled = True
-
-    fetcher = HuggingFaceMetadataFetch(mm2_session)
-    metadata = fetcher.from_id("stabilityai/sdxl-turbo")
-    assert isinstance(metadata, ModelMetadataWithFiles)
-
-    job = queue.multifile_download(
-        parts=metadata.download_urls(session=mm2_session),
-        dest=tmp_path,
-        on_cancelled=cancelled_callback,
-    )
-    queue.cancel_job(job)
-    queue.join()
-
-    assert job.status == DownloadJobStatus.CANCELLED
-    assert cancelled
-    events = event_bus.events
-    assert DownloadCancelledEvent in [type(x) for x in events]
-    queue.stop()
-
-
-def test_multifile_onefile(tmp_path: Path, mm2_session: Session) -> None:
-    queue = DownloadQueueService(
-        requests_session=mm2_session,
-    )
-    queue.start()
-    job = queue.multifile_download(
-        parts=[
-            RemoteModelFile(url=AnyHttpUrl("http://www.civitai.com/models/12345"), path=Path("mock12345.safetensors"))
-        ],
-        dest=tmp_path,
-    )
-    assert isinstance(job, MultiFileDownloadJob), "expected the job to be of type MultiFileDownloadJobBase"
-    queue.join()
-
-    assert job.status == DownloadJobStatus("completed"), "expected job status to be completed"
-    assert job.bytes > 0, "expected download bytes to be positive"
-    assert job.bytes == job.total_bytes, "expected download bytes to equal total bytes"
-    assert job.download_path == tmp_path / "mock12345.safetensors"
-    assert Path(tmp_path, "mock12345.safetensors").exists(), f"expected {tmp_path}/mock12345.safetensors to exist"
-    queue.stop()
-
-
-def test_multifile_no_rel_paths(tmp_path: Path, mm2_session: Session) -> None:
-    queue = DownloadQueueService(
-        requests_session=mm2_session,
-    )
-
-    with pytest.raises(AssertionError) as error:
-        queue.multifile_download(
-            parts=[RemoteModelFile(url=AnyHttpUrl("http://www.civitai.com/models/12345"), path=Path("/etc/passwd"))],
-            dest=tmp_path,
-        )
-    assert str(error.value) == "only relative download paths accepted"
-
-
-@contextmanager
-def clear_config() -> Generator[None, None, None]:
-    try:
-        yield None
-    finally:
-        get_config.cache_clear()
-
-
-def test_tokens(tmp_path: Path, mm2_session: Session):
-    with clear_config():
-        config = get_config()
-        config.remote_api_tokens = [URLRegexTokenPair(url_regex="civitai", token="cv_12345")]
-        queue = DownloadQueueService(requests_session=mm2_session)
-        queue.start()
-        # this one has an access token assigned
-        job1 = queue.download(
-            source=AnyHttpUrl("http://www.civitai.com/models/12345"),
-            dest=tmp_path,
-        )
-        # this one doesn't
-        job2 = queue.download(
-            source=AnyHttpUrl(
-                "http://www.huggingface.co/foo.txt",
-            ),
-            dest=tmp_path,
-        )
-        queue.join()
-        # this token is defined in the temporary root invokeai.yaml
-        # see tests/backend/model_manager/data/invokeai_root/invokeai.yaml
-        assert job1.access_token == "cv_12345"
-        assert job2.access_token is None
-        queue.stop()

tests/app/services/model_install/test_model_install.py

@@ -20,7 +20,6 @@ from invokeai.app.services.events.events_common import (
     ModelInstallStartedEvent,
 )
 from invokeai.app.services.model_install import (
-    HFModelSource,
     ModelInstallServiceBase,
 )
 from invokeai.app.services.model_install.model_install_common import (
@@ -30,14 +29,7 @@ from invokeai.app.services.model_install.model_install_common import (
     URLModelSource,
 )
 from invokeai.app.services.model_records import ModelRecordChanges, UnknownModelException
-from invokeai.backend.model_manager.config import (
-    BaseModelType,
-    InvalidModelConfigException,
-    ModelFormat,
-    ModelRepoVariant,
-    ModelType,
-)
-from tests.backend.model_manager.model_manager_fixtures import *  # noqa F403
+from invokeai.backend.model_manager.config import BaseModelType, InvalidModelConfigException, ModelFormat, ModelType
 from tests.test_nodes import TestEventService
 
 OS = platform.uname().system
@@ -230,7 +222,7 @@ def test_delete_register(
         store.get_model(key)
 
 
-@pytest.mark.timeout(timeout=10, method="thread")
+@pytest.mark.timeout(timeout=20, method="thread")
 def test_simple_download(mm2_installer: ModelInstallServiceBase, mm2_app_config: InvokeAIAppConfig) -> None:
     source = URLModelSource(url=Url("https://www.test.foo/download/test_embedding.safetensors"))
 
@@ -251,16 +243,15 @@ def test_simple_download(mm2_installer: ModelInstallServiceBase, mm2_app_config:
     model_record = store.get_model(key)
     assert (mm2_app_config.models_path / model_record.path).exists()
 
-    assert len(bus.events) == 5
-    assert isinstance(bus.events[0], ModelInstallDownloadProgressEvent)  # download starts
-    assert isinstance(bus.events[1], ModelInstallDownloadProgressEvent)  # download progresses
-    assert isinstance(bus.events[2], ModelInstallDownloadsCompleteEvent)  # download completed
-    assert isinstance(bus.events[3], ModelInstallStartedEvent)  # install started
-    assert isinstance(bus.events[4], ModelInstallCompleteEvent)  # install completed
+    assert len(bus.events) == 4
+    assert isinstance(bus.events[0], ModelInstallDownloadProgressEvent)
+    assert isinstance(bus.events[1], ModelInstallDownloadsCompleteEvent)
+    assert isinstance(bus.events[2], ModelInstallStartedEvent)
+    assert isinstance(bus.events[3], ModelInstallCompleteEvent)
 
 
-@pytest.mark.timeout(timeout=10, method="thread")
-def test_huggingface_install(mm2_installer: ModelInstallServiceBase, mm2_app_config: InvokeAIAppConfig) -> None:
+@pytest.mark.timeout(timeout=20, method="thread")
+def test_huggingface_download(mm2_installer: ModelInstallServiceBase, mm2_app_config: InvokeAIAppConfig) -> None:
     source = URLModelSource(url=Url("https://huggingface.co/stabilityai/sdxl-turbo"))
 
     bus: TestEventService = mm2_installer.event_bus
@@ -286,49 +277,6 @@ def test_huggingface_install(mm2_installer: ModelInstallServiceBase, mm2_app_con
     assert len(bus.events) >= 3
 
 
-@pytest.mark.timeout(timeout=10, method="thread")
-def test_huggingface_repo_id(mm2_installer: ModelInstallServiceBase, mm2_app_config: InvokeAIAppConfig) -> None:
-    source = HFModelSource(repo_id="stabilityai/sdxl-turbo", variant=ModelRepoVariant.Default)
-
-    bus = mm2_installer.event_bus
-    store = mm2_installer.record_store
-    assert isinstance(bus, EventServiceBase)
-    assert store is not None
-
-    job = mm2_installer.import_model(source)
-    job_list = mm2_installer.wait_for_installs(timeout=10)
-    assert len(job_list) == 1
-    assert job.complete
-    assert job.config_out
-
-    key = job.config_out.key
-    model_record = store.get_model(key)
-    assert (mm2_app_config.models_path / model_record.path).exists()
-    assert model_record.type == ModelType.Main
-    assert model_record.format == ModelFormat.Diffusers
-
-    assert hasattr(bus, "events")  # the dummyeventservice has this
-    assert len(bus.events) >= 3
-    event_types = [type(x) for x in bus.events]
-    assert all(
-        x in event_types
-        for x in [
-            ModelInstallDownloadProgressEvent,
-            ModelInstallDownloadsCompleteEvent,
-            ModelInstallStartedEvent,
-            ModelInstallCompleteEvent,
-        ]
-    )
-
-    completed_events = [x for x in bus.events if isinstance(x, ModelInstallCompleteEvent)]
-    downloading_events = [x for x in bus.events if isinstance(x, ModelInstallDownloadProgressEvent)]
-    assert completed_events[0].total_bytes == downloading_events[-1].bytes
-    assert job.total_bytes == completed_events[0].total_bytes
-    print(downloading_events[-1])
-    print(job.download_parts)
-    assert job.total_bytes == sum(x["total_bytes"] for x in downloading_events[-1].parts)
-
-
 def test_404_download(mm2_installer: ModelInstallServiceBase, mm2_app_config: InvokeAIAppConfig) -> None:
     source = URLModelSource(url=Url("https://test.com/missing_model.safetensors"))
     job = mm2_installer.import_model(source)
@@ -360,6 +308,7 @@ def test_other_error_during_install(
     assert job.error == "Test error"
 
 
+# TODO: Fix bug in model install causing jobs to get installed multiple times then uncomment this test
 @pytest.mark.parametrize(
     "model_params",
     [
@@ -377,7 +326,7 @@ def test_other_error_during_install(
         },
     ],
 )
-@pytest.mark.timeout(timeout=10, method="thread")
+@pytest.mark.timeout(timeout=40, method="thread")
 def test_heuristic_import_with_type(mm2_installer: ModelInstallServiceBase, model_params: Dict[str, str]):
     """Test whether or not type is respected on configs when passed to heuristic import."""
     assert "name" in model_params and "type" in model_params
@@ -393,7 +342,7 @@ def test_heuristic_import_with_type(mm2_installer: ModelInstallServiceBase, mode
     }
     assert "repo_id" in model_params
     install_job1 = mm2_installer.heuristic_import(source=model_params["repo_id"], config=config1)
-    mm2_installer.wait_for_job(install_job1, timeout=10)
+    mm2_installer.wait_for_job(install_job1, timeout=20)
     if model_params["type"] != "embedding":
         assert install_job1.errored
         assert install_job1.error_type == "InvalidModelConfigException"
@@ -402,6 +351,6 @@ def test_heuristic_import_with_type(mm2_installer: ModelInstallServiceBase, mode
     assert install_job1.config_out if model_params["type"] == "embedding" else not install_job1.config_out
 
     install_job2 = mm2_installer.heuristic_import(source=model_params["repo_id"], config=config2)
-    mm2_installer.wait_for_job(install_job2, timeout=10)
+    mm2_installer.wait_for_job(install_job2, timeout=20)
     assert install_job2.complete
     assert install_job2.config_out if model_params["type"] == "embedding" else not install_job2.config_out

tests/app/services/model_load/test_load_api.py

@@ -1,88 +0,0 @@
-from pathlib import Path
-
-import pytest
-import torch
-from diffusers import AutoencoderTiny
-
-from invokeai.app.services.invocation_services import InvocationServices
-from invokeai.app.services.model_manager import ModelManagerServiceBase
-from invokeai.app.services.shared.invocation_context import InvocationContext, build_invocation_context
-from invokeai.backend.model_manager.load.load_base import LoadedModelWithoutConfig
-from tests.backend.model_manager.model_manager_fixtures import *  # noqa F403
-
-
-@pytest.fixture()
-def mock_context(
-    mock_services: InvocationServices,
-    mm2_model_manager: ModelManagerServiceBase,
-) -> InvocationContext:
-    mock_services.model_manager = mm2_model_manager
-    return build_invocation_context(
-        services=mock_services,
-        data=None,  # type: ignore
-        is_canceled=None,  # type: ignore
-    )
-
-
-def test_download_and_cache(mock_context: InvocationContext, mm2_root_dir: Path) -> None:
-    downloaded_path = mock_context.models.download_and_cache_model(
-        "https://www.test.foo/download/test_embedding.safetensors"
-    )
-    assert downloaded_path.is_file()
-    assert downloaded_path.exists()
-    assert downloaded_path.name == "test_embedding.safetensors"
-    assert downloaded_path.parent.parent == mm2_root_dir / "models/.download_cache"
-
-    downloaded_path_2 = mock_context.models.download_and_cache_model(
-        "https://www.test.foo/download/test_embedding.safetensors"
-    )
-    assert downloaded_path == downloaded_path_2
-
-
-def test_load_from_path(mock_context: InvocationContext, embedding_file: Path) -> None:
-    downloaded_path = mock_context.models.download_and_cache_model(
-        "https://www.test.foo/download/test_embedding.safetensors"
-    )
-    loaded_model_1 = mock_context.models.load_local_model(downloaded_path)
-    assert isinstance(loaded_model_1, LoadedModelWithoutConfig)
-
-    loaded_model_2 = mock_context.models.load_local_model(downloaded_path)
-    assert isinstance(loaded_model_2, LoadedModelWithoutConfig)
-    assert loaded_model_1.model is loaded_model_2.model
-
-    loaded_model_3 = mock_context.models.load_local_model(embedding_file)
-    assert isinstance(loaded_model_3, LoadedModelWithoutConfig)
-    assert loaded_model_1.model is not loaded_model_3.model
-    assert isinstance(loaded_model_1.model, dict)
-    assert isinstance(loaded_model_3.model, dict)
-    assert torch.equal(loaded_model_1.model["emb_params"], loaded_model_3.model["emb_params"])
-
-
-@pytest.mark.skip(reason="This requires a test model to load")
-def test_load_from_dir(mock_context: InvocationContext, vae_directory: Path) -> None:
-    loaded_model = mock_context.models.load_local_model(vae_directory)
-    assert isinstance(loaded_model, LoadedModelWithoutConfig)
-    assert isinstance(loaded_model.model, AutoencoderTiny)
-
-
-def test_download_and_load(mock_context: InvocationContext) -> None:
-    loaded_model_1 = mock_context.models.load_remote_model("https://www.test.foo/download/test_embedding.safetensors")
-    assert isinstance(loaded_model_1, LoadedModelWithoutConfig)
-
-    loaded_model_2 = mock_context.models.load_remote_model("https://www.test.foo/download/test_embedding.safetensors")
-    assert isinstance(loaded_model_2, LoadedModelWithoutConfig)
-    assert loaded_model_1.model is loaded_model_2.model  # should be cached copy
-
-
-def test_download_diffusers(mock_context: InvocationContext) -> None:
-    model_path = mock_context.models.download_and_cache_model("stabilityai/sdxl-turbo")
-    assert (model_path / "model_index.json").exists()
-    assert (model_path / "vae").is_dir()
-
-
-def test_download_diffusers_subfolder(mock_context: InvocationContext) -> None:
-    model_path = mock_context.models.download_and_cache_model("stabilityai/sdxl-turbo::vae")
-    assert model_path.is_dir()
-    assert (model_path / "diffusion_pytorch_model.fp16.safetensors").exists() or (
-        model_path / "diffusion_pytorch_model.safetensors"
-    ).exists()

tests/backend/model_manager/model_manager_fixtures.py

@@ -61,13 +61,6 @@ def embedding_file(mm2_model_files: Path) -> Path:
     return mm2_model_files / "test_embedding.safetensors"
 
 
-# Can be used to test diffusers model directory loading, but
-# the test file adds ~10MB of space.
-# @pytest.fixture
-# def vae_directory(mm2_model_files: Path) -> Path:
-#     return mm2_model_files / "taesdxl"
-
-
 @pytest.fixture
 def diffusers_dir(mm2_model_files: Path) -> Path:
     return mm2_model_files / "test-diffusers-main"
@@ -301,45 +294,4 @@ def mm2_session(embedding_file: Path, diffusers_dir: Path) -> Session:
                     },
                 ),
             )
-
-    for i in ["12345", "9999", "54321"]:
-        content = (
-            b"I am a safetensors file " + bytearray(i, "utf-8") + bytearray(32_000)
-        )  # for pause tests, must make content large
-        sess.mount(
-            f"http://www.civitai.com/models/{i}",
-            TestAdapter(
-                content,
-                headers={
-                    "Content-Length": len(content),
-                    "Content-Disposition": f'filename="mock{i}.safetensors"',
-                },
-            ),
-        )
-
-    sess.mount(
-        "http://www.huggingface.co/foo.txt",
-        TestAdapter(
-            content,
-            headers={
-                "Content-Length": len(content),
-                "Content-Disposition": 'filename="foo.safetensors"',
-            },
-        ),
-    )
-
-    # here are some malformed URLs to test
-    # missing the content length
-    sess.mount(
-        "http://www.civitai.com/models/missing",
-        TestAdapter(
-            b"Missing content length",
-            headers={
-                "Content-Disposition": 'filename="missing.txt"',
-            },
-        ),
-    )
-    # not found test
-    sess.mount("http://www.civitai.com/models/broken", TestAdapter(b"Not found", status=404))
-
     return sess