Corrected version

## What type of PR is this? (check all applicable)

- [ ] Refactor
- [ ] Feature
- [ ] Bug Fix
- [x] Optimization
- [ ] Documentation Update
- [ ] Community Node Submission
      
## Have you updated all relevant documentation?
- [x] Yes
- [ ] No


## Description

This PR optimizes the time to load models from disk.
In my local testing, SDXL text_encoder_2 models saw the greatest
improvement:
- Before change, load time (disk to cpu): 14 secs
- After change, load time (disk to cpu): 4 secs

See the in-code documentation for an explanation of how this speedup is
achieved.

## Related Tickets & Documents

This change was previously proposed on the HF transformers repo, but did
not get any traction:
https://github.com/huggingface/transformers/issues/18505#issue-1330728188

## QA Instructions, Screenshots, Recordings

I don't expect any adverse effects, but the new context manager is
applied while loading **all** models, so it would make sense to exercise
everything.

## Added/updated tests?

- [x] Yes
- [ ] No

docs/contributing/LOCAL_DEVELOPMENT.md

@@ -47,34 +47,9 @@ pip install ".[dev,test]"
 These are optional groups of packages which are defined within the `pyproject.toml`
 and will be required for testing the changes you make to the code.
 
-### Running Tests
-
-We use [pytest](https://docs.pytest.org/en/7.2.x/) for our test suite. Tests can
-be found under the `./tests` folder and can be run with a single `pytest`
-command. Optionally, to review test coverage you can append `--cov`.
-
-```zsh
-pytest --cov
-```
-
-Test outcomes and coverage will be reported in the terminal. In addition a more
-detailed report is created in both XML and HTML format in the `./coverage`
-folder. The HTML one in particular can help identify missing statements
-requiring tests to ensure coverage. This can be run by opening
-`./coverage/html/index.html`.
-
-For example.
-
-```zsh
-pytest --cov; open ./coverage/html/index.html
-```
-
-??? info "HTML coverage report output"
-
-    ![html-overview](../assets/contributing/html-overview.png)
-
-    ![html-detail](../assets/contributing/html-detail.png)
+### Tests
 
+See the [tests documentation](./TESTS.md) for information about running and writing tests.
 ### Reloading Changes
 
 Experimenting with changes to the Python source code is a drag if you have to re-start the server —
@@ -167,6 +142,23 @@ and so you'll have access to the same python environment as the InvokeAI app.
 
 This is _super_ handy.
 
+#### Enabling Type-Checking with Pylance
+
+We use python's typing system in InvokeAI. PR reviews will include checking that types are present and correct. We don't enforce types with `mypy` at this time, but that is on the horizon.
+
+Using a code analysis tool to automatically type check your code (and types) is very important when writing with types. These tools provide immediate feedback in your editor when types are incorrect, and following their suggestions lead to fewer runtime bugs.
+
+Pylance, installed at the beginning of this guide, is the de-facto python LSP (language server protocol). It provides type checking in the editor (among many other features). Once installed, you do need to enable type checking manually:
+
+- Open a python file
+- Look along the status bar in VSCode for `{ } Python`
+- Click the `{ }`
+- Turn type checking on - basic is fine
+
+You'll now see red squiggly lines where type issues are detected. Hover your cursor over the indicated symbols to see what's wrong.
+
+In 99% of cases when the type checker says there is a problem, there really is a problem, and you should take some time to understand and resolve what it is pointing out.
+
 #### Debugging configs with `launch.json`
 
 Debugging configs are managed in a `launch.json` file. Like most VSCode configs,

docs/contributing/TESTS.md

@@ -0,0 +1,89 @@
+# InvokeAI Backend Tests
+
+We use `pytest` to run the backend python tests. (See [pyproject.toml](/pyproject.toml) for the default `pytest` options.)
+
+## Fast vs. Slow
+All tests are categorized as either 'fast' (no test annotation) or 'slow' (annotated with the `@pytest.mark.slow` decorator).
+
+'Fast' tests are run to validate every PR, and are fast enough that they can be run routinely during development.
+
+'Slow' tests are currently only run manually on an ad-hoc basis. In the future, they may be automated to run nightly. Most developers are only expected to run the 'slow' tests that directly relate to the feature(s) that they are working on.
+
+As a rule of thumb, tests should be marked as 'slow' if there is a chance that they take >1s (e.g. on a CPU-only machine with slow internet connection). Common examples of slow tests are tests that depend on downloading a model, or running model inference.
+
+## Running Tests
+
+Below are some common test commands:
+```bash
+# Run the fast tests. (This implicitly uses the configured default option: `-m "not slow"`.)
+pytest tests/
+
+# Equivalent command to run the fast tests.
+pytest tests/ -m "not slow"
+
+# Run the slow tests.
+pytest tests/ -m "slow"
+
+# Run the slow tests from a specific file.
+pytest tests/path/to/slow_test.py -m "slow"
+
+# Run all tests (fast and slow).
+pytest tests -m ""
+```
+
+## Test Organization
+
+All backend tests are in the [`tests/`](/tests/) directory. This directory mirrors the organization of the `invokeai/` directory. For example, tests for `invokeai/model_management/model_manager.py` would be found in `tests/model_management/test_model_manager.py`.
+
+TODO: The above statement is aspirational. A re-organization of legacy tests is required to make it true.
+
+## Tests that depend on models
+
+There are a few things to keep in mind when adding tests that depend on models.
+
+1. If a required model is not already present, it should automatically be downloaded as part of the test setup.
+2. If a model is already downloaded, it should not be re-downloaded unnecessarily.
+3. Take reasonable care to keep the total number of models required for the tests low. Whenever possible, re-use models that are already required for other tests. If you are adding a new model, consider including a comment to explain why it is required/unique.
+
+There are several utilities to help with model setup for tests. Here is a sample test that depends on a model:
+```python
+import pytest
+import torch
+
+from invokeai.backend.model_management.models.base import BaseModelType, ModelType
+from invokeai.backend.util.test_utils import install_and_load_model
+
+@pytest.mark.slow
+def test_model(model_installer, torch_device):
+    model_info = install_and_load_model(
+        model_installer=model_installer,
+        model_path_id_or_url="HF/dummy_model_id",
+        model_name="dummy_model",
+        base_model=BaseModelType.StableDiffusion1,
+        model_type=ModelType.Dummy,
+    )
+
+    dummy_input = build_dummy_input(torch_device)
+
+    with torch.no_grad(), model_info as model:
+        model.to(torch_device, dtype=torch.float32)
+        output = model(dummy_input)
+
+    # Validate output...
+
+```
+
+## Test Coverage
+
+To review test coverage, append `--cov` to your pytest command:
+```bash
+pytest tests/ --cov
+```
+
+Test outcomes and coverage will be reported in the terminal. In addition, a more detailed report is created in both XML and HTML format in the `./coverage` folder. The HTML output is particularly helpful in identifying untested statements where coverage should be improved. The HTML report can be viewed by opening `./coverage/html/index.html`.
+
+??? info "HTML coverage report output"
+
+    ![html-overview](../assets/contributing/html-overview.png)
+
+    ![html-detail](../assets/contributing/html-detail.png)

docs/contributing/contribution_guides/development.md

@@ -12,7 +12,7 @@ To get started, take a look at our [new contributors checklist](newContributorCh
 Once you're setup, for more information, you can review the documentation specific to your area of interest:
 
 * #### [InvokeAI Architecure](../ARCHITECTURE.md)
-* #### [Frontend Documentation](development_guides/contributingToFrontend.md)
+* #### [Frontend Documentation](./contributingToFrontend.md)
 * #### [Node Documentation](../INVOCATIONS.md)
 * #### [Local Development](../LOCAL_DEVELOPMENT.md)
 
@@ -38,9 +38,9 @@ There are two paths to making a development contribution:
 
 If you need help, you can ask questions in the [#dev-chat](https://discord.com/channels/1020123559063990373/1049495067846524939) channel of the Discord.
 
-For frontend related work, **@pyschedelicious** is the best person to reach out to. 
+For frontend related work, **@psychedelicious** is the best person to reach out to. 
 
-For backend related work, please reach out to **@blessedcoolant**, **@lstein**, **@StAlKeR7779** or **@pyschedelicious**.
+For backend related work, please reach out to **@blessedcoolant**, **@lstein**, **@StAlKeR7779** or **@psychedelicious**.
 
 
 ## **What does the Code of Conduct mean for me?**

docs/contributing/contribution_guides/documentation.md

@@ -10,4 +10,4 @@ When updating or creating documentation, please keep in mind InvokeAI is a tool
 
 ## Help & Questions
 
-Please ping @imic1 or @hipsterusername in the [Discord](https://discord.com/channels/1020123559063990373/1049495067846524939) if you have any questions.
+Please ping @imic or @hipsterusername in the [Discord](https://discord.com/channels/1020123559063990373/1049495067846524939) if you have any questions.

docs/features/CONTROLNET.md

@@ -1,13 +1,11 @@
 ---
-title: ControlNet
+title: Control Adapters
 ---
 
-# :material-loupe: ControlNet
+# :material-loupe: Control Adapters
 
 ## ControlNet
 
-ControlNet
-
 ControlNet is a powerful set of features developed by the open-source
 community (notably, Stanford researcher
 [**@ilyasviel**](https://github.com/lllyasviel)) that allows you to
@@ -20,7 +18,7 @@ towards generating images that better fit your desired style or
 outcome.
 
 
-### How it works
+#### How it works
 
 ControlNet works by analyzing an input image, pre-processing that
 image to identify relevant information that can be interpreted by each
@@ -30,7 +28,7 @@ composition, or other aspects of the image to better achieve a
 specific result.
 
 
-### Models
+#### Models
 
 InvokeAI provides access to a series of ControlNet models that provide
 different effects or styles in your generated images.  Currently
@@ -96,6 +94,8 @@ A model that generates normal maps from input images, allowing for more realisti
 **Image Segmentation**: 
 A model that divides input images into segments or regions, each of which corresponds to a different object or part of the image. (More details coming soon)
 
+**QR Code Monster**:
+A model that helps generate creative QR codes that still scan. Can also be used to create images with text, logos or shapes within them. 
 
 **Openpose**: 
 The OpenPose control model allows for the identification of the general pose of a character by pre-processing an existing image with a clear human structure. With advanced options, Openpose can also detect the face or hands in the image. 
@@ -120,7 +120,7 @@ With Pix2Pix, you can input an image into the controlnet, and then "instruct" th
 Each of these models can be adjusted and combined with other ControlNet models to achieve different results, giving you even more control over your image generation process.
 
 
-## Using ControlNet
+### Using ControlNet
 
 To use ControlNet, you can simply select the desired model and adjust both the ControlNet and Pre-processor settings to achieve the desired result. You can also use multiple ControlNet models at the same time, allowing you to achieve even more complex effects or styles in your generated images.
 
@@ -132,3 +132,31 @@ Weight - Strength of the Controlnet model applied to the generation for the sect
 Start/End  - 0 represents the start of the generation, 1 represents the end. The Start/end setting controls what steps during the generation process have the ControlNet applied.
 
 Additionally, each ControlNet section can be expanded in order to manipulate settings for the image pre-processor that adjusts your uploaded image before using it in when you Invoke.
+
+
+## IP-Adapter
+
+[IP-Adapter](https://ip-adapter.github.io) is a tooling that allows for image prompt capabilities with text-to-image diffusion models. IP-Adapter works by analyzing the given image prompt to extract features, then passing those features to the UNet along with any other conditioning provided. 
+
+![IP-Adapter + T2I](https://github.com/tencent-ailab/IP-Adapter/raw/main/assets/demo/ip_adpter_plus_multi.jpg)
+
+![IP-Adapter + IMG2IMG](https://github.com/tencent-ailab/IP-Adapter/blob/main/assets/demo/image-to-image.jpg)
+
+#### Installation
+There are several ways to install IP-Adapter models with an existing InvokeAI installation:
+
+1. Through the command line interface launched from the invoke.sh / invoke.bat scripts, option [5] to download models.
+2. Through the Model Manager UI with models from the *Tools* section of [www.models.invoke.ai](www.models.invoke.ai). To do this, copy the repo ID from the desired model page, and paste it in the Add Model field of the model manager. **Note** Both the IP-Adapter and the Image Encoder must be installed for IP-Adapter to work. For example, the [SD 1.5 IP-Adapter](https://models.invoke.ai/InvokeAI/ip_adapter_plus_sd15) and [SD1.5 Image Encoder](https://models.invoke.ai/InvokeAI/ip_adapter_sd_image_encoder) must be installed to use IP-Adapter with SD1.5 based models.  
+3. **Advanced -- Not recommended ** Manually downloading the IP-Adapter and Image Encoder files - Image Encoder folders shouid be placed in the `models\any\clip_vision` folders. IP Adapter Model folders should be placed in the relevant `ip-adapter` folder of relevant base model folder of Invoke root directory. For example, for the SDXL IP-Adapter, files should be added to the `model/sdxl/ip_adapter/` folder. 
+
+#### Using IP-Adapter
+
+IP-Adapter can be used by navigating to the *Control Adapters* options and enabling IP-Adapter. 
+
+IP-Adapter requires an image to be used as the Image Prompt. It can also be used in conjunction with text prompts, Image-to-Image, Inpainting, Outpainting, ControlNets and LoRAs.
+
+
+Each IP-Adapter has two settings that are applied to the IP-Adapter:
+
+* Weight - Strength of the IP-Adapter model applied to the generation for the section, defined by start/end
+* Start/End  - 0 represents the start of the generation, 1 represents the end. The Start/end setting controls what steps during the generation process have the IP-Adapter applied.

docs/installation/020_INSTALL_MANUAL.md

@@ -256,6 +256,10 @@ manager, please follow these steps:
         *highly recommended** if your virtual environment is located outside of
         your runtime directory.
 
+    !!! tip
+
+        On linux, it is recommended to run invokeai with the following env var: `MALLOC_MMAP_THRESHOLD_=1048576`. For example: `MALLOC_MMAP_THRESHOLD_=1048576 invokeai --web`. This helps to prevent memory fragmentation that can lead to memory accumulation over time. This env var is set automatically when running via `invoke.sh`.
+
 10.  Render away!
 
     Browse the [features](../features/index.md) section to learn about all the

docs/installation/050_INSTALLING_MODELS.md

@@ -171,3 +171,16 @@ subfolders and organize them as you wish.
 
 The location of the autoimport directories are controlled by settings
 in `invokeai.yaml`. See [Configuration](../features/CONFIGURATION.md).
+
+### Installing models that live in HuggingFace subfolders
+
+On rare occasions you may need to install a diffusers-style model that
+lives in a subfolder of a HuggingFace repo id. In this event, simply
+add ":_subfolder-name_" to the end of the repo id. For example, if the
+repo id is "monster-labs/control_v1p_sd15_qrcode_monster" and the model
+you wish to fetch lives in a subfolder named "v2", then the repo id to
+pass to the various model installers should be 
+
+```
+monster-labs/control_v1p_sd15_qrcode_monster:v2
+```

docs/nodes/NODES.md

@@ -4,12 +4,12 @@ The workflow editor is a blank canvas allowing for the use of individual functio
 
 If you're not familiar with Diffusion, take a look at our [Diffusion Overview.](../help/diffusion.md) Understanding how diffusion works will enable you to more easily use the Workflow Editor and build workflows to suit your needs.
 
-## UI Features
+## Features
 
 ### Linear View
 The Workflow Editor allows you to create a UI for your workflow, to make it easier to iterate on your generations. 
 
-To add an input to the Linear UI, right click on the input and select "Add to Linear View".
+To add an input to the Linear UI, right click on the input label and select "Add to Linear View".
 
 The Linear UI View will also be part of the saved workflow, allowing you share workflows and enable other to use them, regardless of complexity. 
 
@@ -25,6 +25,10 @@ Any node or input field can be renamed in the workflow editor. If the input fiel
 * Backspace/Delete to delete a node
 * Shift+Click to drag and select multiple nodes 
 
+### Node Caching 
+
+Nodes have a "Use Cache" option in their footer. This allows for performance improvements by using the previously cached values during the workflow processing. 
+
 
 ## Important Concepts 
 

docs/nodes/communityNodes.md

@@ -8,19 +8,21 @@ To download a node, simply download the `.py` node file from the link and add it
 
 To use a community workflow, download the the `.json` node graph file and load it into Invoke AI via the **Load Workflow** button in the Workflow Editor. 
 
-## Community Nodes
+--------------------------------
 
-### FaceTools
+--------------------------------
+### Make 3D
 
-**Description:** FaceTools is a collection of nodes created to manipulate faces as you would in Unified Canvas. It includes FaceMask, FaceOff, and FacePlace. FaceMask autodetects a face in the image using MediaPipe and creates a mask from it. FaceOff similarly detects a face, then takes the face off of the image by adding a square bounding box around it and cropping/scaling it. FacePlace puts the bounded face image from FaceOff back onto the original image. Using these nodes with other inpainting node(s), you can put new faces on existing things, put new things around existing faces, and work closer with a face as a bounded image. Additionally, you can supply X and Y offset values to scale/change the shape of the mask for finer control on FaceMask and FaceOff. See GitHub repository below for usage examples.
+**Description:** Create compelling 3D stereo images from 2D originals.
 
-**Node Link:** https://github.com/ymgenesis/FaceTools/
+**Node Link:** [https://gitlab.com/srcrr/shift3d/-/raw/main/make3d.py](https://gitlab.com/srcrr/shift3d)
 
-**FaceMask Output Examples** 
+**Example Node Graph:**  https://gitlab.com/srcrr/shift3d/-/raw/main/example-workflow.json?ref_type=heads&inline=false
 
-![5cc8abce-53b0-487a-b891-3bf94dcc8960](https://github.com/invoke-ai/InvokeAI/assets/25252829/43f36d24-1429-4ab1-bd06-a4bedfe0955e)
-![b920b710-1882-49a0-8d02-82dff2cca907](https://github.com/invoke-ai/InvokeAI/assets/25252829/7660c1ed-bf7d-4d0a-947f-1fc1679557ba)
-![71a91805-fda5-481c-b380-264665703133](https://github.com/invoke-ai/InvokeAI/assets/25252829/f8f6a2ee-2b68-4482-87da-b90221d5c3e2)
+**Output Examples** 
+
+![Painting of a cozy delapidated house](https://gitlab.com/srcrr/shift3d/-/raw/main/example-1.png){: style="height:512px;width:512px"}
+![Photo of cute puppies](https://gitlab.com/srcrr/shift3d/-/raw/main/example-2.png){: style="height:512px;width:512px"}
 
 --------------------------------
 ### Ideal Size
@@ -43,6 +45,52 @@ To use a community workflow, download the the `.json` node graph file and load i
 
 **Node Link:** https://github.com/JPPhoto/image-picker-node
 
+--------------------------------
+### Thresholding
+
+**Description:** This node generates masks for highlights, midtones, and shadows given an input image. You can optionally specify a blur for the lookup table used in making those masks from the source image.
+
+**Node Link:** https://github.com/JPPhoto/thresholding-node
+
+**Examples**
+
+Input:
+
+![image](https://github.com/invoke-ai/InvokeAI/assets/34005131/c88ada13-fb3d-484c-a4fe-947b44712632){: style="height:512px;width:512px"}
+
+Highlights/Midtones/Shadows:
+
+<img src="https://github.com/invoke-ai/InvokeAI/assets/34005131/727021c1-36ff-4ec8-90c8-105e00de986d" style="width: 30%" />
+<img src="https://github.com/invoke-ai/InvokeAI/assets/34005131/0b721bfc-f051-404e-b905-2f16b824ddfe" style="width: 30%" />
+<img src="https://github.com/invoke-ai/InvokeAI/assets/34005131/04c1297f-1c88-42b6-a7df-dd090b976286" style="width: 30%" />
+
+Highlights/Midtones/Shadows (with LUT blur enabled):
+
+<img src="https://github.com/invoke-ai/InvokeAI/assets/34005131/19aa718a-70c1-4668-8169-d68f4bd13771" style="width: 30%" />
+<img src="https://github.com/invoke-ai/InvokeAI/assets/34005131/0a440e43-697f-4d17-82ee-f287467df0a5" style="width: 30%" />
+<img src="https://github.com/invoke-ai/InvokeAI/assets/34005131/0701fd0f-2ca7-4fe2-8613-2b52547bafce" style="width: 30%" />
+
+--------------------------------
+### Halftone
+
+**Description**: Halftone converts the source image to grayscale and then performs halftoning. CMYK Halftone converts the image to CMYK and applies a per-channel halftoning to make the source image look like a magazine or newspaper. For both nodes, you can specify angles and halftone dot spacing.
+
+**Node Link:** https://github.com/JPPhoto/halftone-node
+
+**Example**
+
+Input:
+
+![image](https://github.com/invoke-ai/InvokeAI/assets/34005131/fd5efb9f-4355-4409-a1c2-c1ca99e0cab4){: style="height:512px;width:512px"}
+
+Halftone Output:
+
+![image](https://github.com/invoke-ai/InvokeAI/assets/34005131/7e606f29-e68f-4d46-b3d5-97f799a4ec2f){: style="height:512px;width:512px"}
+
+CMYK Halftone Output:
+
+![image](https://github.com/invoke-ai/InvokeAI/assets/34005131/c59c578f-db8e-4d66-8c66-2851752d75ea){: style="height:512px;width:512px"}
+
 --------------------------------
 ### Retroize
 
@@ -77,7 +125,7 @@ Generated Prompt: An enchanted weapon will be usable by any character regardless
 **Example Node Graph:**  https://github.com/helix4u/load_video_frame/blob/main/Example_Workflow.json
 
 **Output Example:** 
-=======
+
 ![Example animation](https://github.com/helix4u/load_video_frame/blob/main/testmp4_embed_converted.gif)
 [Full mp4 of Example Output test.mp4](https://github.com/helix4u/load_video_frame/blob/main/test.mp4)
 
@@ -141,7 +189,8 @@ This includes 3 Nodes:
 
 **Description:** This is a pack of nodes for composing masks and images, including a simple text mask creator and both image and latent offset nodes. The offsets wrap around, so these can be used in conjunction with the Seamless node to progressively generate centered on different parts of the seamless tiling.
 
-This includes 14 Nodes:
+This includes 15 Nodes:
+
 - *Adjust Image Hue Plus* - Rotate the hue of an image in one of several different color spaces.
 - *Blend Latents/Noise (Masked)* - Use a mask to blend part of one latents tensor [including Noise outputs] into another. Can be used to "renoise" sections during a multi-stage [masked] denoising process.
 - *Enhance Image* - Boost or reduce color saturation, contrast, brightness, sharpness, or invert colors of any image at any stage with this simple wrapper for pillow [PIL]'s ImageEnhance module.
@@ -154,6 +203,7 @@ This includes 14 Nodes:
 - *Image Value Thresholds* - Clip an image to pure black/white beyond specified thresholds.
 - *Offset Latents* - Offset a latents tensor in the vertical and/or horizontal dimensions, wrapping it around.
 - *Offset Image* - Offset an image in the vertical and/or horizontal dimensions, wrapping it around.
+- *Rotate/Flip Image* - Rotate an image in degrees clockwise/counterclockwise about its center, optionally resizing the image boundaries to fit, or flipping it about the vertical and/or horizontal axes.
 - *Shadows/Highlights/Midtones* - Extract three masks (with adjustable hard or soft thresholds) representing shadows, midtones, and highlights regions of an image.
 - *Text Mask (simple 2D)* - create and position a white on black (or black on white) line of text using any font locally available to Invoke.
 
@@ -228,6 +278,36 @@ See full docs here: https://github.com/skunkworxdark/XYGrid_nodes/edit/main/READ
 
 --------------------------------
 
+### Image to Character Art Image Node's
+
+**Description:** Group of nodes to convert an input image into ascii/unicode art Image
+
+**Node Link:** https://github.com/mickr777/imagetoasciiimage
+
+**Output Examples**
+
+<img src="https://github.com/invoke-ai/InvokeAI/assets/115216705/8e061fcc-9a2c-4fa9-bcc7-c0f7b01e9056" width="300" />
+<img src="https://github.com/mickr777/imagetoasciiimage/assets/115216705/3c4990eb-2f42-46b9-90f9-0088b939dc6a" width="300" /></br>
+<img src="https://github.com/mickr777/imagetoasciiimage/assets/115216705/fee7f800-a4a8-41e2-a66b-c66e4343307e" width="300" />
+<img src="https://github.com/mickr777/imagetoasciiimage/assets/115216705/1d9c1003-a45f-45c2-aac7-46470bb89330" width="300" />
+
+--------------------------------
+
+### Grid to Gif
+
+**Description:** One node that turns a grid image into an image colletion, one node that turns an image collection into a gif
+
+**Node Link:** https://github.com/mildmisery/invokeai-GridToGifNode/blob/main/GridToGif.py
+
+**Example Node Graph:**  https://github.com/mildmisery/invokeai-GridToGifNode/blob/main/Grid%20to%20Gif%20Example%20Workflow.json
+
+**Output Examples** 
+
+<img src="https://raw.githubusercontent.com/mildmisery/invokeai-GridToGifNode/main/input.png" width="300" />
+<img src="https://raw.githubusercontent.com/mildmisery/invokeai-GridToGifNode/main/output.gif" width="300" />
+
+--------------------------------
+
 ### Example Node Template
 
 **Description:** This node allows you to do super cool things with InvokeAI.

docs/nodes/defaultNodes.md

@@ -1,6 +1,6 @@
 # List of Default Nodes
 
-The table below contains a list of the default nodes shipped with InvokeAI and their descriptions. 
+The table below contains a list of the default nodes shipped with InvokeAI and their descriptions.
 
 | Node <img width=160 align="right"> | Function                                                                              |
 |: ---------------------------------- | :--------------------------------------------------------------------------------------|
@@ -17,11 +17,12 @@ The table below contains a list of the default nodes shipped with InvokeAI and t
 |Conditioning Primitive 			| A conditioning tensor primitive value|
 |Content Shuffle Processor 			| Applies content shuffle processing to image|
 |ControlNet 			| Collects ControlNet info to pass to other nodes|
-|OpenCV Inpaint 			| Simple inpaint using opencv.|
 |Denoise Latents 			| Denoises noisy latents to decodable images|
 |Divide Integers 			| Divides two numbers|
 |Dynamic Prompt 			| Parses a prompt using adieyal/dynamicprompts' random or combinatorial generator|
-|Upscale (RealESRGAN) 			| Upscales an image using RealESRGAN.|
+|[FaceMask](./detailedNodes/faceTools.md#facemask) | Generates masks for faces in an image to use with Inpainting|
+|[FaceIdentifier](./detailedNodes/faceTools.md#faceidentifier)             | Identifies and labels faces in an image|
+|[FaceOff](./detailedNodes/faceTools.md#faceoff)             | Creates a new image that is a scaled bounding box with a mask on the face for Inpainting|
 |Float Math             | Perform basic math operations on two floats|
 |Float Primitive Collection 			| A collection of float primitive values|
 |Float Primitive 			| A float primitive value|
@@ -76,6 +77,7 @@ The table below contains a list of the default nodes shipped with InvokeAI and t
 |ONNX Prompt (Raw) 			| A node to process inputs and produce outputs. May use dependency injection in __init__ to receive providers.|
 |ONNX Text to Latents 			| Generates latents from conditionings.|
 |ONNX Model Loader 			| Loads a main model, outputting its submodels.|
+|OpenCV Inpaint 			| Simple inpaint using opencv.|
 |Openpose Processor 			| Applies Openpose processing to image|
 |PIDI Processor 			| Applies PIDI processing to image|
 |Prompts from File 			| Loads prompts from a text file|
@@ -97,5 +99,6 @@ The table below contains a list of the default nodes shipped with InvokeAI and t
 |String Primitive 			| A string primitive value|
 |Subtract Integers 			| Subtracts two numbers|
 |Tile Resample Processor 			| Tile resampler processor|
+|Upscale (RealESRGAN) 			| Upscales an image using RealESRGAN.|
 |VAE Loader 			| Loads a VAE model, outputting a VaeLoaderOutput|
 |Zoe (Depth) Processor 			| Applies Zoe depth processing to image|

docs/nodes/detailedNodes/faceTools.md

@@ -0,0 +1,154 @@
+# Face Nodes
+
+## FaceOff
+
+FaceOff mimics a user finding a face in an image and resizing the bounding box
+around the head in Canvas.
+
+Enter a face ID (found with FaceIdentifier) to choose which face to mask.
+
+Just as you would add more context inside the bounding box by making it larger
+in Canvas, the node gives you a padding input (in pixels) which will
+simultaneously add more context, and increase the resolution of the bounding box
+so the face remains the same size inside it.
+
+The "Minimum Confidence" input defaults to 0.5 (50%), and represents a pass/fail
+threshold a detected face must reach for it to be processed. Lowering this value
+may help if detection is failing. If the detected masks are imperfect and stray
+too far outside/inside of faces, the node gives you X & Y offsets to shrink/grow
+the masks by a multiplier.
+
+FaceOff will output the face in a bounded image, taking the face off of the
+original image for input into any node that accepts image inputs. The node also
+outputs a face mask with the dimensions of the bounded image. The X & Y outputs
+are for connecting to the X & Y inputs of the Paste Image node, which will place
+the bounded image back on the original image using these coordinates.
+
+###### Inputs/Outputs
+
+| Input              | Description                                                                                                                                                                              |
+| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Image              | Image for face detection                                                                                                                                                                 |
+| Face ID            | The face ID to process, numbered from 0. Multiple faces not supported. Find a face's ID with FaceIdentifier node.                                                                        |
+| Minimum Confidence | Minimum confidence for face detection (lower if detection is failing)                                                                                                                    |
+| X Offset           | X-axis offset of the mask                                                                                                                                                                |
+| Y Offset           | Y-axis offset of the mask                                                                                                                                                                |
+| Padding            | All-axis padding around the mask in pixels                                                                                                                                               |
+| Chunk              | Chunk (or divide) the image into sections to greatly improve face detection success. Defaults to off, but will activate if no faces are detected normally. Activate to chunk by default. |
+
+| Output        | Description                                      |
+| ------------- | ------------------------------------------------ |
+| Bounded Image | Original image bound, cropped, and resized       |
+| Width         | The width of the bounded image in pixels         |
+| Height        | The height of the bounded image in pixels        |
+| Mask          | The output mask                                  |
+| X             | The x coordinate of the bounding box's left side |
+| Y             | The y coordinate of the bounding box's top side  |
+
+## FaceMask
+
+FaceMask mimics a user drawing masks on faces in an image in Canvas.
+
+The "Face IDs" input allows the user to select specific faces to be masked.
+Leave empty to detect and mask all faces, or a comma-separated list for a
+specific combination of faces (ex: `1,2,4`). A single integer will detect and
+mask that specific face. Find face IDs with the FaceIdentifier node.
+
+The "Minimum Confidence" input defaults to 0.5 (50%), and represents a pass/fail
+threshold a detected face must reach for it to be processed. Lowering this value
+may help if detection is failing.
+
+If the detected masks are imperfect and stray too far outside/inside of faces,
+the node gives you X & Y offsets to shrink/grow the masks by a multiplier. All
+masks shrink/grow together by the X & Y offset values.
+
+By default, masks are created to change faces. When masks are inverted, they
+change surrounding areas, protecting faces.
+
+###### Inputs/Outputs
+
+| Input              | Description                                                                                                                                                                              |
+| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Image              | Image for face detection                                                                                                                                                                 |
+| Face IDs           | Comma-separated list of face ids to mask eg '0,2,7'. Numbered from 0. Leave empty to mask all. Find face IDs with FaceIdentifier node.                                                   |
+| Minimum Confidence | Minimum confidence for face detection (lower if detection is failing)                                                                                                                    |
+| X Offset           | X-axis offset of the mask                                                                                                                                                                |
+| Y Offset           | Y-axis offset of the mask                                                                                                                                                                |
+| Chunk              | Chunk (or divide) the image into sections to greatly improve face detection success. Defaults to off, but will activate if no faces are detected normally. Activate to chunk by default. |
+| Invert Mask        | Toggle to invert the face mask                                                                                                                                                           |
+
+| Output | Description                       |
+| ------ | --------------------------------- |
+| Image  | The original image                |
+| Width  | The width of the image in pixels  |
+| Height | The height of the image in pixels |
+| Mask   | The output face mask              |
+
+## FaceIdentifier
+
+FaceIdentifier outputs an image with detected face IDs printed in white numbers
+onto each face.
+
+Face IDs can then be used in FaceMask and FaceOff to selectively mask all, a
+specific combination, or single faces.
+
+The FaceIdentifier output image is generated for user reference, and isn't meant
+to be passed on to other image-processing nodes.
+
+The "Minimum Confidence" input defaults to 0.5 (50%), and represents a pass/fail
+threshold a detected face must reach for it to be processed. Lowering this value
+may help if detection is failing. If an image is changed in the slightest, run
+it through FaceIdentifier again to get updated FaceIDs.
+
+###### Inputs/Outputs
+
+| Input              | Description                                                                                                                                                                              |
+| ------------------ | ---------------------------------------------------------------------------------------------------------------------------------------------------------------------------------------- |
+| Image              | Image for face detection                                                                                                                                                                 |
+| Minimum Confidence | Minimum confidence for face detection (lower if detection is failing)                                                                                                                    |
+| Chunk              | Chunk (or divide) the image into sections to greatly improve face detection success. Defaults to off, but will activate if no faces are detected normally. Activate to chunk by default. |
+
+| Output | Description                                                                                      |
+| ------ | ------------------------------------------------------------------------------------------------ |
+| Image  | The original image with small face ID numbers printed in white onto each face for user reference |
+| Width  | The width of the original image in pixels                                                        |
+| Height | The height of the original image in pixels                                                       |
+
+## Tips
+
+- If not all target faces are being detected, activate Chunk to bypass full
+  image face detection and greatly improve detection success.
+- Final results will vary between full-image detection and chunking for faces
+  that are detectable by both due to the nature of the process. Try either to
+  your taste.
+- Be sure Minimum Confidence is set the same when using FaceIdentifier with
+  FaceOff/FaceMask.
+- For FaceOff, use the color correction node before faceplace to correct edges
+  being noticeable in the final image (see example screenshot).
+- Non-inpainting models may struggle to paint/generate correctly around faces.
+- If your face won't change the way you want it to no matter what you change,
+  consider that the change you're trying to make is too much at that resolution.
+  For example, if an image is only 512x768 total, the face might only be 128x128
+  or 256x256, much smaller than the 512x512 your SD1.5 model was probably
+  trained on. Try increasing the resolution of the image by upscaling or
+  resizing, add padding to increase the bounding box's resolution, or use an
+  image where the face takes up more pixels.
+- If the resulting face seems out of place pasted back on the original image
+  (ie. too large, not proportional), add more padding on the FaceOff node to
+  give inpainting more context. Context and good prompting are important to
+  keeping things proportional.
+- If you find the mask is too big/small and going too far outside/inside the
+  area you want to affect, adjust the x & y offsets to shrink/grow the mask area
+- Use a higher denoise start value to resemble aspects of the original face or
+  surroundings. Denoise start = 0 & denoise end = 1 will make something new,
+  while denoise start = 0.50 & denoise end = 1 will be 50% old and 50% new.
+- mediapipe isn't good at detecting faces with lots of face paint, hair covering
+  the face, etc. Anything that obstructs the face will likely result in no faces
+  being detected.
+- If you find your face isn't being detected, try lowering the minimum
+  confidence value from 0.5. This could result in false positives, however
+  (random areas being detected as faces and masked).
+- After altering an image and wanting to process a different face in the newly
+  altered image, run the altered image through FaceIdentifier again to see the
+  new Face IDs. MediaPipe will most likely detect faces in a different order
+  after an image has been changed in the slightest.

docs/nodes/exampleWorkflows.md

@@ -9,5 +9,6 @@ If you're interested in finding more workflows, checkout the [#share-your-workfl
 * [SD1.5 / SD2 Text to Image](https://github.com/invoke-ai/InvokeAI/blob/main/docs/workflows/Text_to_Image.json)
 * [SDXL Text to Image](https://github.com/invoke-ai/InvokeAI/blob/main/docs/workflows/SDXL_Text_to_Image.json)
 * [SDXL (with Refiner) Text to Image](https://github.com/invoke-ai/InvokeAI/blob/main/docs/workflows/SDXL_Text_to_Image.json) 
-* [Tiled Upscaling with ControlNet](https://github.com/invoke-ai/InvokeAI/blob/main/docs/workflows/ESRGAN_img2img_upscale w_Canny_ControlNet.json)ß
-
+* [Tiled Upscaling with ControlNet](https://github.com/invoke-ai/InvokeAI/blob/main/docs/workflows/ESRGAN_img2img_upscale w_Canny_ControlNet.json)
+* [FaceMask](https://github.com/invoke-ai/InvokeAI/blob/main/docs/workflows/FaceMask.json)
+* [FaceOff with 2x Face Scaling](https://github.com/invoke-ai/InvokeAI/blob/main/docs/workflows/FaceOff_FaceScale2x.json)

installer/lib/installer.py

@@ -332,6 +332,7 @@ class InvokeAiInstance:
         Configure the InvokeAI runtime directory
         """
 
+        auto_install = False
         # set sys.argv to a consistent state
         new_argv = [sys.argv[0]]
         for i in range(1, len(sys.argv)):
@@ -340,13 +341,17 @@ class InvokeAiInstance:
                 new_argv.append(el)
                 new_argv.append(sys.argv[i + 1])
             elif el in ["-y", "--yes", "--yes-to-all"]:
-                new_argv.append(el)
+                auto_install = True
         sys.argv = new_argv
 
+        import messages
         import requests  # to catch download exceptions
-        from messages import introduction
 
-        introduction()
+        auto_install = auto_install or messages.user_wants_auto_configuration()
+        if auto_install:
+            sys.argv.append("--yes")
+        else:
+            messages.introduction()
 
         from invokeai.frontend.install.invokeai_configure import invokeai_configure
 

installer/lib/messages.py

@@ -7,7 +7,7 @@ import os
 import platform
 from pathlib import Path
 
-from prompt_toolkit import prompt
+from prompt_toolkit import HTML, prompt
 from prompt_toolkit.completion import PathCompleter
 from prompt_toolkit.validation import Validator
 from rich import box, print
@@ -65,17 +65,50 @@ def confirm_install(dest: Path) -> bool:
     if dest.exists():
         print(f":exclamation: Directory {dest} already exists :exclamation:")
         dest_confirmed = Confirm.ask(
-            ":stop_sign: Are you sure you want to (re)install in this location?",
+            ":stop_sign: (re)install in this location?",
             default=False,
         )
     else:
         print(f"InvokeAI will be installed in {dest}")
-        dest_confirmed = not Confirm.ask("Would you like to pick a different location?", default=False)
+        dest_confirmed = Confirm.ask("Use this location?", default=True)
     console.line()
 
     return dest_confirmed
 
 
+def user_wants_auto_configuration() -> bool:
+    """Prompt the user to choose between manual and auto configuration."""
+    console.rule("InvokeAI Configuration Section")
+    console.print(
+        Panel(
+            Group(
+                "\n".join(
+                    [
+                        "Libraries are installed and InvokeAI will now set up its root directory and configuration. Choose between:",
+                        "",
+                        "  * AUTOMATIC configuration:  install reasonable defaults and a minimal set of starter models.",
+                        "  * MANUAL configuration: manually inspect and adjust configuration options and pick from a larger set of starter models.",
+                        "",
+                        "Later you can fine tune your configuration by selecting option [6] 'Change InvokeAI startup options' from the invoke.bat/invoke.sh launcher script.",
+                    ]
+                ),
+            ),
+            box=box.MINIMAL,
+            padding=(1, 1),
+        )
+    )
+    choice = (
+        prompt(
+            HTML("Choose <b>&lt;a&gt;</b>utomatic or <b>&lt;m&gt;</b>anual configuration [a/m] (a): "),
+            validator=Validator.from_callable(
+                lambda n: n == "" or n.startswith(("a", "A", "m", "M")), error_message="Please select 'a' or 'm'"
+            ),
+        )
+        or "a"
+    )
+    return choice.lower().startswith("a")
+
+
 def dest_path(dest=None) -> Path:
     """
     Prompt the user for the destination path and create the path

installer/templates/invoke.sh.in

@@ -46,6 +46,9 @@ if [ "$(uname -s)" == "Darwin" ]; then
     export PYTORCH_ENABLE_MPS_FALLBACK=1
 fi
 
+# Avoid glibc memory fragmentation. See invokeai/backend/model_management/README.md for details.
+export MALLOC_MMAP_THRESHOLD_=1048576
+
 # Primary function for the case statement to determine user input
 do_choice() {
     case $1 in

invokeai/app/invocations/baseinvocation.py

@@ -68,6 +68,7 @@ class FieldDescriptions:
     height = "Height of output (px)"
     control = "ControlNet(s) to apply"
     ip_adapter = "IP-Adapter to apply"
+    t2i_adapter = "T2I-Adapter(s) to apply"
     denoised_latents = "Denoised latents tensor"
     latents = "Latents tensor"
     strength = "Strength of denoising (proportional to steps)"
@@ -91,6 +92,9 @@ class FieldDescriptions:
     board = "The board to save the image to"
     image = "The image to process"
     tile_size = "Tile size"
+    inclusive_low = "The inclusive low value"
+    exclusive_high = "The exclusive high value"
+    decimal_places = "The number of decimal places to round to"
 
 
 class Input(str, Enum):

invokeai/app/invocations/facetools.py

@@ -0,0 +1,692 @@
+import math
+import re
+from pathlib import Path
+from typing import Optional, TypedDict
+
+import cv2
+import numpy as np
+from mediapipe.python.solutions.face_mesh import FaceMesh  # type: ignore[import]
+from PIL import Image, ImageDraw, ImageFilter, ImageFont, ImageOps
+from PIL.Image import Image as ImageType
+from pydantic import validator
+
+import invokeai.assets.fonts as font_assets
+from invokeai.app.invocations.baseinvocation import (
+    BaseInvocation,
+    InputField,
+    InvocationContext,
+    OutputField,
+    invocation,
+    invocation_output,
+)
+from invokeai.app.invocations.primitives import ImageField, ImageOutput
+from invokeai.app.models.image import ImageCategory, ResourceOrigin
+
+
+@invocation_output("face_mask_output")
+class FaceMaskOutput(ImageOutput):
+    """Base class for FaceMask output"""
+
+    mask: ImageField = OutputField(description="The output mask")
+
+
+@invocation_output("face_off_output")
+class FaceOffOutput(ImageOutput):
+    """Base class for FaceOff Output"""
+
+    mask: ImageField = OutputField(description="The output mask")
+    x: int = OutputField(description="The x coordinate of the bounding box's left side")
+    y: int = OutputField(description="The y coordinate of the bounding box's top side")
+
+
+class FaceResultData(TypedDict):
+    image: ImageType
+    mask: ImageType
+    x_center: float
+    y_center: float
+    mesh_width: int
+    mesh_height: int
+
+
+class FaceResultDataWithId(FaceResultData):
+    face_id: int
+
+
+class ExtractFaceData(TypedDict):
+    bounded_image: ImageType
+    bounded_mask: ImageType
+    x_min: int
+    y_min: int
+    x_max: int
+    y_max: int
+
+
+class FaceMaskResult(TypedDict):
+    image: ImageType
+    mask: ImageType
+
+
+def create_white_image(w: int, h: int) -> ImageType:
+    return Image.new("L", (w, h), color=255)
+
+
+def create_black_image(w: int, h: int) -> ImageType:
+    return Image.new("L", (w, h), color=0)
+
+
+FONT_SIZE = 32
+FONT_STROKE_WIDTH = 4
+
+
+def prepare_faces_list(
+    face_result_list: list[FaceResultData],
+) -> list[FaceResultDataWithId]:
+    """Deduplicates a list of faces, adding IDs to them."""
+    deduped_faces: list[FaceResultData] = []
+
+    if len(face_result_list) == 0:
+        return list()
+
+    for candidate in face_result_list:
+        should_add = True
+        candidate_x_center = candidate["x_center"]
+        candidate_y_center = candidate["y_center"]
+        for face in deduped_faces:
+            face_center_x = face["x_center"]
+            face_center_y = face["y_center"]
+            face_radius_w = face["mesh_width"] / 2
+            face_radius_h = face["mesh_height"] / 2
+            # Determine if the center of the candidate_face is inside the ellipse of the added face
+            # p < 1 -> Inside
+            # p = 1 -> Exactly on the ellipse
+            # p > 1 -> Outside
+            p = (math.pow((candidate_x_center - face_center_x), 2) / math.pow(face_radius_w, 2)) + (
+                math.pow((candidate_y_center - face_center_y), 2) / math.pow(face_radius_h, 2)
+            )
+
+            if p < 1:  # Inside of the already-added face's radius
+                should_add = False
+                break
+
+        if should_add is True:
+            deduped_faces.append(candidate)
+
+    sorted_faces = sorted(deduped_faces, key=lambda x: x["y_center"])
+    sorted_faces = sorted(sorted_faces, key=lambda x: x["x_center"])
+
+    # add face_id for reference
+    sorted_faces_with_ids: list[FaceResultDataWithId] = []
+    face_id_counter = 0
+    for face in sorted_faces:
+        sorted_faces_with_ids.append(
+            FaceResultDataWithId(
+                **face,
+                face_id=face_id_counter,
+            )
+        )
+        face_id_counter += 1
+
+    return sorted_faces_with_ids
+
+
+def generate_face_box_mask(
+    context: InvocationContext,
+    minimum_confidence: float,
+    x_offset: float,
+    y_offset: float,
+    pil_image: ImageType,
+    chunk_x_offset: int = 0,
+    chunk_y_offset: int = 0,
+    draw_mesh: bool = True,
+    check_bounds: bool = True,
+) -> list[FaceResultData]:
+    result = []
+    mask_pil = None
+
+    # Convert the PIL image to a NumPy array.
+    np_image = np.array(pil_image, dtype=np.uint8)
+
+    # Check if the input image has four channels (RGBA).
+    if np_image.shape[2] == 4:
+        # Convert RGBA to RGB by removing the alpha channel.
+        np_image = np_image[:, :, :3]
+
+    # Create a FaceMesh object for face landmark detection and mesh generation.
+    face_mesh = FaceMesh(
+        max_num_faces=999,
+        min_detection_confidence=minimum_confidence,
+        min_tracking_confidence=minimum_confidence,
+    )
+
+    # Detect the face landmarks and mesh in the input image.
+    results = face_mesh.process(np_image)
+
+    # Check if any face is detected.
+    if results.multi_face_landmarks:  # type: ignore # this are via protobuf and not typed
+        # Search for the face_id in the detected faces.
+        for face_id, face_landmarks in enumerate(results.multi_face_landmarks):  # type: ignore #this are via protobuf and not typed
+            # Get the bounding box of the face mesh.
+            x_coordinates = [landmark.x for landmark in face_landmarks.landmark]
+            y_coordinates = [landmark.y for landmark in face_landmarks.landmark]
+            x_min, x_max = min(x_coordinates), max(x_coordinates)
+            y_min, y_max = min(y_coordinates), max(y_coordinates)
+
+            # Calculate the width and height of the face mesh.
+            mesh_width = int((x_max - x_min) * np_image.shape[1])
+            mesh_height = int((y_max - y_min) * np_image.shape[0])
+
+            # Get the center of the face.
+            x_center = np.mean([landmark.x * np_image.shape[1] for landmark in face_landmarks.landmark])
+            y_center = np.mean([landmark.y * np_image.shape[0] for landmark in face_landmarks.landmark])
+
+            face_landmark_points = np.array(
+                [
+                    [landmark.x * np_image.shape[1], landmark.y * np_image.shape[0]]
+                    for landmark in face_landmarks.landmark
+                ]
+            )
+
+            # Apply the scaling offsets to the face landmark points with a multiplier.
+            scale_multiplier = 0.2
+            x_center = np.mean(face_landmark_points[:, 0])
+            y_center = np.mean(face_landmark_points[:, 1])
+
+            if draw_mesh:
+                x_scaled = face_landmark_points[:, 0] + scale_multiplier * x_offset * (
+                    face_landmark_points[:, 0] - x_center
+                )
+                y_scaled = face_landmark_points[:, 1] + scale_multiplier * y_offset * (
+                    face_landmark_points[:, 1] - y_center
+                )
+
+                convex_hull = cv2.convexHull(np.column_stack((x_scaled, y_scaled)).astype(np.int32))
+
+                # Generate a binary face mask using the face mesh.
+                mask_image = np.ones(np_image.shape[:2], dtype=np.uint8) * 255
+                cv2.fillConvexPoly(mask_image, convex_hull, 0)
+
+                # Convert the binary mask image to a PIL Image.
+                init_mask_pil = Image.fromarray(mask_image, mode="L")
+                w, h = init_mask_pil.size
+                mask_pil = create_white_image(w + chunk_x_offset, h + chunk_y_offset)
+                mask_pil.paste(init_mask_pil, (chunk_x_offset, chunk_y_offset))
+
+            left_side = x_center - mesh_width
+            right_side = x_center + mesh_width
+            top_side = y_center - mesh_height
+            bottom_side = y_center + mesh_height
+            im_width, im_height = pil_image.size
+            over_w = im_width * 0.1
+            over_h = im_height * 0.1
+            if not check_bounds or (
+                (left_side >= -over_w)
+                and (right_side < im_width + over_w)
+                and (top_side >= -over_h)
+                and (bottom_side < im_height + over_h)
+            ):
+                x_center = float(x_center)
+                y_center = float(y_center)
+                face = FaceResultData(
+                    image=pil_image,
+                    mask=mask_pil or create_white_image(*pil_image.size),
+                    x_center=x_center + chunk_x_offset,
+                    y_center=y_center + chunk_y_offset,
+                    mesh_width=mesh_width,
+                    mesh_height=mesh_height,
+                )
+
+                result.append(face)
+            else:
+                context.services.logger.info("FaceTools --> Face out of bounds, ignoring.")
+
+    return result
+
+
+def extract_face(
+    context: InvocationContext,
+    image: ImageType,
+    face: FaceResultData,
+    padding: int,
+) -> ExtractFaceData:
+    mask = face["mask"]
+    center_x = face["x_center"]
+    center_y = face["y_center"]
+    mesh_width = face["mesh_width"]
+    mesh_height = face["mesh_height"]
+
+    # Determine the minimum size of the square crop
+    min_size = min(mask.width, mask.height)
+
+    # Calculate the crop boundaries for the output image and mask.
+    mesh_width += 128 + padding  # add pixels to account for mask variance
+    mesh_height += 128 + padding  # add pixels to account for mask variance
+    crop_size = min(
+        max(mesh_width, mesh_height, 128), min_size
+    )  # Choose the smaller of the two (given value or face mask size)
+    if crop_size > 128:
+        crop_size = (crop_size + 7) // 8 * 8  # Ensure crop side is multiple of 8
+
+    # Calculate the actual crop boundaries within the bounds of the original image.
+    x_min = int(center_x - crop_size / 2)
+    y_min = int(center_y - crop_size / 2)
+    x_max = int(center_x + crop_size / 2)
+    y_max = int(center_y + crop_size / 2)
+
+    # Adjust the crop boundaries to stay within the original image's dimensions
+    if x_min < 0:
+        context.services.logger.warning("FaceTools --> -X-axis padding reached image edge.")
+        x_max -= x_min
+        x_min = 0
+    elif x_max > mask.width:
+        context.services.logger.warning("FaceTools --> +X-axis padding reached image edge.")
+        x_min -= x_max - mask.width
+        x_max = mask.width
+
+    if y_min < 0:
+        context.services.logger.warning("FaceTools --> +Y-axis padding reached image edge.")
+        y_max -= y_min
+        y_min = 0
+    elif y_max > mask.height:
+        context.services.logger.warning("FaceTools --> -Y-axis padding reached image edge.")
+        y_min -= y_max - mask.height
+        y_max = mask.height
+
+    # Ensure the crop is square and adjust the boundaries if needed
+    if x_max - x_min != crop_size:
+        context.services.logger.warning("FaceTools --> Limiting x-axis padding to constrain bounding box to a square.")
+        diff = crop_size - (x_max - x_min)
+        x_min -= diff // 2
+        x_max += diff - diff // 2
+
+    if y_max - y_min != crop_size:
+        context.services.logger.warning("FaceTools --> Limiting y-axis padding to constrain bounding box to a square.")
+        diff = crop_size - (y_max - y_min)
+        y_min -= diff // 2
+        y_max += diff - diff // 2
+
+    context.services.logger.info(f"FaceTools --> Calculated bounding box (8 multiple): {crop_size}")
+
+    # Crop the output image to the specified size with the center of the face mesh as the center.
+    mask = mask.crop((x_min, y_min, x_max, y_max))
+    bounded_image = image.crop((x_min, y_min, x_max, y_max))
+
+    # blur mask edge by small radius
+    mask = mask.filter(ImageFilter.GaussianBlur(radius=2))
+
+    return ExtractFaceData(
+        bounded_image=bounded_image,
+        bounded_mask=mask,
+        x_min=x_min,
+        y_min=y_min,
+        x_max=x_max,
+        y_max=y_max,
+    )
+
+
+def get_faces_list(
+    context: InvocationContext,
+    image: ImageType,
+    should_chunk: bool,
+    minimum_confidence: float,
+    x_offset: float,
+    y_offset: float,
+    draw_mesh: bool = True,
+) -> list[FaceResultDataWithId]:
+    result = []
+
+    # Generate the face box mask and get the center of the face.
+    if not should_chunk:
+        context.services.logger.info("FaceTools --> Attempting full image face detection.")
+        result = generate_face_box_mask(
+            context=context,
+            minimum_confidence=minimum_confidence,
+            x_offset=x_offset,
+            y_offset=y_offset,
+            pil_image=image,
+            chunk_x_offset=0,
+            chunk_y_offset=0,
+            draw_mesh=draw_mesh,
+            check_bounds=False,
+        )
+    if should_chunk or len(result) == 0:
+        context.services.logger.info("FaceTools --> Chunking image (chunk toggled on, or no face found in full image).")
+        width, height = image.size
+        image_chunks = []
+        x_offsets = []
+        y_offsets = []
+        result = []
+
+        # If width == height, there's nothing more we can do... otherwise...
+        if width > height:
+            # Landscape - slice the image horizontally
+            fx = 0.0
+            steps = int(width * 2 / height)
+            while fx <= (width - height):
+                x = int(fx)
+                image_chunks.append(image.crop((x, 0, x + height - 1, height - 1)))
+                x_offsets.append(x)
+                y_offsets.append(0)
+                fx += (width - height) / steps
+                context.services.logger.info(f"FaceTools --> Chunk starting at x = {x}")
+        elif height > width:
+            # Portrait - slice the image vertically
+            fy = 0.0
+            steps = int(height * 2 / width)
+            while fy <= (height - width):
+                y = int(fy)
+                image_chunks.append(image.crop((0, y, width - 1, y + width - 1)))
+                x_offsets.append(0)
+                y_offsets.append(y)
+                fy += (height - width) / steps
+                context.services.logger.info(f"FaceTools --> Chunk starting at y = {y}")
+
+        for idx in range(len(image_chunks)):
+            context.services.logger.info(f"FaceTools --> Evaluating faces in chunk {idx}")
+            result = result + generate_face_box_mask(
+                context=context,
+                minimum_confidence=minimum_confidence,
+                x_offset=x_offset,
+                y_offset=y_offset,
+                pil_image=image_chunks[idx],
+                chunk_x_offset=x_offsets[idx],
+                chunk_y_offset=y_offsets[idx],
+                draw_mesh=draw_mesh,
+            )
+
+        if len(result) == 0:
+            # Give up
+            context.services.logger.warning(
+                "FaceTools --> No face detected in chunked input image. Passing through original image."
+            )
+
+    all_faces = prepare_faces_list(result)
+
+    return all_faces
+
+
+@invocation("face_off", title="FaceOff", tags=["image", "faceoff", "face", "mask"], category="image", version="1.0.1")
+class FaceOffInvocation(BaseInvocation):
+    """Bound, extract, and mask a face from an image using MediaPipe detection"""
+
+    image: ImageField = InputField(description="Image for face detection")
+    face_id: int = InputField(
+        default=0,
+        ge=0,
+        description="The face ID to process, numbered from 0. Multiple faces not supported. Find a face's ID with FaceIdentifier node.",
+    )
+    minimum_confidence: float = InputField(
+        default=0.5, description="Minimum confidence for face detection (lower if detection is failing)"
+    )
+    x_offset: float = InputField(default=0.0, description="X-axis offset of the mask")
+    y_offset: float = InputField(default=0.0, description="Y-axis offset of the mask")
+    padding: int = InputField(default=0, description="All-axis padding around the mask in pixels")
+    chunk: bool = InputField(
+        default=False,
+        description="Whether to bypass full image face detection and default to image chunking. Chunking will occur if no faces are found in the full image.",
+    )
+
+    def faceoff(self, context: InvocationContext, image: ImageType) -> Optional[ExtractFaceData]:
+        all_faces = get_faces_list(
+            context=context,
+            image=image,
+            should_chunk=self.chunk,
+            minimum_confidence=self.minimum_confidence,
+            x_offset=self.x_offset,
+            y_offset=self.y_offset,
+            draw_mesh=True,
+        )
+
+        if len(all_faces) == 0:
+            context.services.logger.warning("FaceOff --> No faces detected. Passing through original image.")
+            return None
+
+        if self.face_id > len(all_faces) - 1:
+            context.services.logger.warning(
+                f"FaceOff --> Face ID {self.face_id} is outside of the number of faces detected ({len(all_faces)}). Passing through original image."
+            )
+            return None
+
+        face_data = extract_face(context=context, image=image, face=all_faces[self.face_id], padding=self.padding)
+        # Convert the input image to RGBA mode to ensure it has an alpha channel.
+        face_data["bounded_image"] = face_data["bounded_image"].convert("RGBA")
+
+        return face_data
+
+    def invoke(self, context: InvocationContext) -> FaceOffOutput:
+        image = context.services.images.get_pil_image(self.image.image_name)
+        result = self.faceoff(context=context, image=image)
+
+        if result is None:
+            result_image = image
+            result_mask = create_white_image(*image.size)
+            x = 0
+            y = 0
+        else:
+            result_image = result["bounded_image"]
+            result_mask = result["bounded_mask"]
+            x = result["x_min"]
+            y = result["y_min"]
+
+        image_dto = context.services.images.create(
+            image=result_image,
+            image_origin=ResourceOrigin.INTERNAL,
+            image_category=ImageCategory.GENERAL,
+            node_id=self.id,
+            session_id=context.graph_execution_state_id,
+            is_intermediate=self.is_intermediate,
+            workflow=self.workflow,
+        )
+
+        mask_dto = context.services.images.create(
+            image=result_mask,
+            image_origin=ResourceOrigin.INTERNAL,
+            image_category=ImageCategory.MASK,
+            node_id=self.id,
+            session_id=context.graph_execution_state_id,
+            is_intermediate=self.is_intermediate,
+        )
+
+        output = FaceOffOutput(
+            image=ImageField(image_name=image_dto.image_name),
+            width=image_dto.width,
+            height=image_dto.height,
+            mask=ImageField(image_name=mask_dto.image_name),
+            x=x,
+            y=y,
+        )
+
+        return output
+
+
+@invocation("face_mask_detection", title="FaceMask", tags=["image", "face", "mask"], category="image", version="1.0.1")
+class FaceMaskInvocation(BaseInvocation):
+    """Face mask creation using mediapipe face detection"""
+
+    image: ImageField = InputField(description="Image to face detect")
+    face_ids: str = InputField(
+        default="",
+        description="Comma-separated list of face ids to mask eg '0,2,7'. Numbered from 0. Leave empty to mask all. Find face IDs with FaceIdentifier node.",
+    )
+    minimum_confidence: float = InputField(
+        default=0.5, description="Minimum confidence for face detection (lower if detection is failing)"
+    )
+    x_offset: float = InputField(default=0.0, description="Offset for the X-axis of the face mask")
+    y_offset: float = InputField(default=0.0, description="Offset for the Y-axis of the face mask")
+    chunk: bool = InputField(
+        default=False,
+        description="Whether to bypass full image face detection and default to image chunking. Chunking will occur if no faces are found in the full image.",
+    )
+    invert_mask: bool = InputField(default=False, description="Toggle to invert the mask")
+
+    @validator("face_ids")
+    def validate_comma_separated_ints(cls, v) -> str:
+        comma_separated_ints_regex = re.compile(r"^\d*(,\d+)*$")
+        if comma_separated_ints_regex.match(v) is None:
+            raise ValueError('Face IDs must be a comma-separated list of integers (e.g. "1,2,3")')
+        return v
+
+    def facemask(self, context: InvocationContext, image: ImageType) -> FaceMaskResult:
+        all_faces = get_faces_list(
+            context=context,
+            image=image,
+            should_chunk=self.chunk,
+            minimum_confidence=self.minimum_confidence,
+            x_offset=self.x_offset,
+            y_offset=self.y_offset,
+            draw_mesh=True,
+        )
+
+        mask_pil = create_white_image(*image.size)
+
+        id_range = list(range(0, len(all_faces)))
+        ids_to_extract = id_range
+        if self.face_ids != "":
+            parsed_face_ids = [int(id) for id in self.face_ids.split(",")]
+            # get requested face_ids that are in range
+            intersected_face_ids = set(parsed_face_ids) & set(id_range)
+
+            if len(intersected_face_ids) == 0:
+                id_range_str = ",".join([str(id) for id in id_range])
+                context.services.logger.warning(
+                    f"Face IDs must be in range of detected faces - requested {self.face_ids}, detected {id_range_str}. Passing through original image."
+                )
+                return FaceMaskResult(
+                    image=image,  # original image
+                    mask=mask_pil,  # white mask
+                )
+
+            ids_to_extract = list(intersected_face_ids)
+
+        for face_id in ids_to_extract:
+            face_data = extract_face(context=context, image=image, face=all_faces[face_id], padding=0)
+            face_mask_pil = face_data["bounded_mask"]
+            x_min = face_data["x_min"]
+            y_min = face_data["y_min"]
+            x_max = face_data["x_max"]
+            y_max = face_data["y_max"]
+
+            mask_pil.paste(
+                create_black_image(x_max - x_min, y_max - y_min),
+                box=(x_min, y_min),
+                mask=ImageOps.invert(face_mask_pil),
+            )
+
+        if self.invert_mask:
+            mask_pil = ImageOps.invert(mask_pil)
+
+        # Create an RGBA image with transparency
+        image = image.convert("RGBA")
+
+        return FaceMaskResult(
+            image=image,
+            mask=mask_pil,
+        )
+
+    def invoke(self, context: InvocationContext) -> FaceMaskOutput:
+        image = context.services.images.get_pil_image(self.image.image_name)
+        result = self.facemask(context=context, image=image)
+
+        image_dto = context.services.images.create(
+            image=result["image"],
+            image_origin=ResourceOrigin.INTERNAL,
+            image_category=ImageCategory.GENERAL,
+            node_id=self.id,
+            session_id=context.graph_execution_state_id,
+            is_intermediate=self.is_intermediate,
+            workflow=self.workflow,
+        )
+
+        mask_dto = context.services.images.create(
+            image=result["mask"],
+            image_origin=ResourceOrigin.INTERNAL,
+            image_category=ImageCategory.MASK,
+            node_id=self.id,
+            session_id=context.graph_execution_state_id,
+            is_intermediate=self.is_intermediate,
+        )
+
+        output = FaceMaskOutput(
+            image=ImageField(image_name=image_dto.image_name),
+            width=image_dto.width,
+            height=image_dto.height,
+            mask=ImageField(image_name=mask_dto.image_name),
+        )
+
+        return output
+
+
+@invocation(
+    "face_identifier", title="FaceIdentifier", tags=["image", "face", "identifier"], category="image", version="1.0.1"
+)
+class FaceIdentifierInvocation(BaseInvocation):
+    """Outputs an image with detected face IDs printed on each face. For use with other FaceTools."""
+
+    image: ImageField = InputField(description="Image to face detect")
+    minimum_confidence: float = InputField(
+        default=0.5, description="Minimum confidence for face detection (lower if detection is failing)"
+    )
+    chunk: bool = InputField(
+        default=False,
+        description="Whether to bypass full image face detection and default to image chunking. Chunking will occur if no faces are found in the full image.",
+    )
+
+    def faceidentifier(self, context: InvocationContext, image: ImageType) -> ImageType:
+        image = image.copy()
+
+        all_faces = get_faces_list(
+            context=context,
+            image=image,
+            should_chunk=self.chunk,
+            minimum_confidence=self.minimum_confidence,
+            x_offset=0,
+            y_offset=0,
+            draw_mesh=False,
+        )
+
+        # Note - font may be found either in the repo if running an editable install, or in the venv if running a package install
+        font_path = [x for x in [Path(y, "inter/Inter-Regular.ttf") for y in font_assets.__path__] if x.exists()]
+        font = ImageFont.truetype(font_path[0].as_posix(), FONT_SIZE)
+
+        # Paste face IDs on the output image
+        draw = ImageDraw.Draw(image)
+        for face in all_faces:
+            x_coord = face["x_center"]
+            y_coord = face["y_center"]
+            text = str(face["face_id"])
+            # get bbox of the text so we can center the id on the face
+            _, _, bbox_w, bbox_h = draw.textbbox(xy=(0, 0), text=text, font=font, stroke_width=FONT_STROKE_WIDTH)
+            x = x_coord - bbox_w / 2
+            y = y_coord - bbox_h / 2
+            draw.text(
+                xy=(x, y),
+                text=str(text),
+                fill=(255, 255, 255, 255),
+                font=font,
+                stroke_width=FONT_STROKE_WIDTH,
+                stroke_fill=(0, 0, 0, 255),
+            )
+
+        # Create an RGBA image with transparency
+        image = image.convert("RGBA")
+
+        return image
+
+    def invoke(self, context: InvocationContext) -> ImageOutput:
+        image = context.services.images.get_pil_image(self.image.image_name)
+        result_image = self.faceidentifier(context=context, image=image)
+
+        image_dto = context.services.images.create(
+            image=result_image,
+            image_origin=ResourceOrigin.INTERNAL,
+            image_category=ImageCategory.GENERAL,
+            node_id=self.id,
+            session_id=context.graph_execution_state_id,
+            is_intermediate=self.is_intermediate,
+            workflow=self.workflow,
+        )
+
+        return ImageOutput(
+            image=ImageField(image_name=image_dto.image_name),
+            width=image_dto.width,
+            height=image_dto.height,
+        )

invokeai/app/invocations/infill.py

@@ -269,7 +269,7 @@ class LaMaInfillInvocation(BaseInvocation):
         )
 
 
-@invocation("infill_cv2", title="CV2 Infill", tags=["image", "inpaint"], category="inpaint")
+@invocation("infill_cv2", title="CV2 Infill", tags=["image", "inpaint"], category="inpaint", version="1.0.0")
 class CV2InfillInvocation(BaseInvocation):
     """Infills transparent areas of an image using OpenCV Inpainting"""
 

invokeai/app/invocations/latent.py

@@ -10,7 +10,7 @@ import torch
 import torchvision.transforms as T
 from diffusers import AutoencoderKL, AutoencoderTiny
 from diffusers.image_processor import VaeImageProcessor
-from diffusers.models import UNet2DConditionModel
+from diffusers.models.adapter import FullAdapterXL, T2IAdapter
 from diffusers.models.attention_processor import (
     AttnProcessor2_0,
     LoRAAttnProcessor2_0,
@@ -33,6 +33,7 @@ from invokeai.app.invocations.primitives import (
     LatentsOutput,
     build_latents_output,
 )
+from invokeai.app.invocations.t2i_adapter import T2IAdapterField
 from invokeai.app.util.controlnet_utils import prepare_control_image
 from invokeai.app.util.step_callback import stable_diffusion_step_callback
 from invokeai.backend.ip_adapter.ip_adapter import IPAdapter, IPAdapterPlus
@@ -47,6 +48,7 @@ from ...backend.stable_diffusion.diffusers_pipeline import (
     ControlNetData,
     IPAdapterData,
     StableDiffusionGeneratorPipeline,
+    T2IAdapterData,
     image_resized_to_grid_as_tensor,
 )
 from ...backend.stable_diffusion.diffusion.shared_invokeai_diffusion import PostprocessingSettings
@@ -196,7 +198,7 @@ def get_scheduler(
     title="Denoise Latents",
     tags=["latents", "denoise", "txt2img", "t2i", "t2l", "img2img", "i2i", "l2l"],
     category="latents",
-    version="1.1.0",
+    version="1.3.0",
 )
 class DenoiseLatentsInvocation(BaseInvocation):
     """Denoises noisy latents to decodable images"""
@@ -223,12 +225,15 @@ class DenoiseLatentsInvocation(BaseInvocation):
         input=Input.Connection,
         ui_order=5,
     )
-    ip_adapter: Optional[IPAdapterField] = InputField(
+    ip_adapter: Optional[Union[IPAdapterField, list[IPAdapterField]]] = InputField(
         description=FieldDescriptions.ip_adapter, title="IP-Adapter", default=None, input=Input.Connection, ui_order=6
     )
+    t2i_adapter: Union[T2IAdapterField, list[T2IAdapterField]] = InputField(
+        description=FieldDescriptions.t2i_adapter, title="T2I-Adapter", default=None, input=Input.Connection, ui_order=7
+    )
     latents: Optional[LatentsField] = InputField(description=FieldDescriptions.latents, input=Input.Connection)
     denoise_mask: Optional[DenoiseMaskField] = InputField(
-        default=None, description=FieldDescriptions.mask, input=Input.Connection, ui_order=7
+        default=None, description=FieldDescriptions.mask, input=Input.Connection, ui_order=8
     )
 
     @validator("cfg_scale")
@@ -404,52 +409,150 @@ class DenoiseLatentsInvocation(BaseInvocation):
     def prep_ip_adapter_data(
         self,
         context: InvocationContext,
-        ip_adapter: Optional[IPAdapterField],
+        ip_adapter: Optional[Union[IPAdapterField, list[IPAdapterField]]],
         conditioning_data: ConditioningData,
-        unet: UNet2DConditionModel,
         exit_stack: ExitStack,
-    ) -> Optional[IPAdapterData]:
+    ) -> Optional[list[IPAdapterData]]:
         """If IP-Adapter is enabled, then this function loads the requisite models, and adds the image prompt embeddings
         to the `conditioning_data` (in-place).
         """
         if ip_adapter is None:
             return None
 
-        image_encoder_model_info = context.services.model_manager.get_model(
-            model_name=ip_adapter.image_encoder_model.model_name,
-            model_type=ModelType.CLIPVision,
-            base_model=ip_adapter.image_encoder_model.base_model,
-            context=context,
-        )
+        # ip_adapter could be a list or a single IPAdapterField. Normalize to a list here.
+        if not isinstance(ip_adapter, list):
+            ip_adapter = [ip_adapter]
 
-        ip_adapter_model: Union[IPAdapter, IPAdapterPlus] = exit_stack.enter_context(
-            context.services.model_manager.get_model(
-                model_name=ip_adapter.ip_adapter_model.model_name,
-                model_type=ModelType.IPAdapter,
-                base_model=ip_adapter.ip_adapter_model.base_model,
+        if len(ip_adapter) == 0:
+            return None
+
+        ip_adapter_data_list = []
+        conditioning_data.ip_adapter_conditioning = []
+        for single_ip_adapter in ip_adapter:
+            ip_adapter_model: Union[IPAdapter, IPAdapterPlus] = exit_stack.enter_context(
+                context.services.model_manager.get_model(
+                    model_name=single_ip_adapter.ip_adapter_model.model_name,
+                    model_type=ModelType.IPAdapter,
+                    base_model=single_ip_adapter.ip_adapter_model.base_model,
+                    context=context,
+                )
+            )
+
+            image_encoder_model_info = context.services.model_manager.get_model(
+                model_name=single_ip_adapter.image_encoder_model.model_name,
+                model_type=ModelType.CLIPVision,
+                base_model=single_ip_adapter.image_encoder_model.base_model,
                 context=context,
             )
-        )
 
-        input_image = context.services.images.get_pil_image(ip_adapter.image.image_name)
+            input_image = context.services.images.get_pil_image(single_ip_adapter.image.image_name)
 
-        # TODO(ryand): With some effort, the step of running the CLIP Vision encoder could be done before any other
-        # models are needed in memory. This would help to reduce peak memory utilization in low-memory environments.
-        with image_encoder_model_info as image_encoder_model:
-            # Get image embeddings from CLIP and ImageProjModel.
-            image_prompt_embeds, uncond_image_prompt_embeds = ip_adapter_model.get_image_embeds(
-                input_image, image_encoder_model
-            )
-            conditioning_data.ip_adapter_conditioning = IPAdapterConditioningInfo(
-                image_prompt_embeds, uncond_image_prompt_embeds
+            # TODO(ryand): With some effort, the step of running the CLIP Vision encoder could be done before any other
+            # models are needed in memory. This would help to reduce peak memory utilization in low-memory environments.
+            with image_encoder_model_info as image_encoder_model:
+                # Get image embeddings from CLIP and ImageProjModel.
+                image_prompt_embeds, uncond_image_prompt_embeds = ip_adapter_model.get_image_embeds(
+                    input_image, image_encoder_model
+                )
+                conditioning_data.ip_adapter_conditioning.append(
+                    IPAdapterConditioningInfo(image_prompt_embeds, uncond_image_prompt_embeds)
+                )
+
+            ip_adapter_data_list.append(
+                IPAdapterData(
+                    ip_adapter_model=ip_adapter_model,
+                    weight=single_ip_adapter.weight,
+                    begin_step_percent=single_ip_adapter.begin_step_percent,
+                    end_step_percent=single_ip_adapter.end_step_percent,
+                )
             )
 
-        return IPAdapterData(
-            ip_adapter_model=ip_adapter_model,
-            weight=ip_adapter.weight,
-            begin_step_percent=ip_adapter.begin_step_percent,
-            end_step_percent=ip_adapter.end_step_percent,
-        )
+        return ip_adapter_data_list
+
+    def run_t2i_adapters(
+        self,
+        context: InvocationContext,
+        t2i_adapter: Optional[Union[T2IAdapterField, list[T2IAdapterField]]],
+        latents_shape: list[int],
+        do_classifier_free_guidance: bool,
+    ) -> Optional[list[T2IAdapterData]]:
+        if t2i_adapter is None:
+            return None
+
+        # Handle the possibility that t2i_adapter could be a list or a single T2IAdapterField.
+        if isinstance(t2i_adapter, T2IAdapterField):
+            t2i_adapter = [t2i_adapter]
+
+        if len(t2i_adapter) == 0:
+            return None
+
+        t2i_adapter_data = []
+        for t2i_adapter_field in t2i_adapter:
+            t2i_adapter_model_info = context.services.model_manager.get_model(
+                model_name=t2i_adapter_field.t2i_adapter_model.model_name,
+                model_type=ModelType.T2IAdapter,
+                base_model=t2i_adapter_field.t2i_adapter_model.base_model,
+                context=context,
+            )
+            image = context.services.images.get_pil_image(t2i_adapter_field.image.image_name)
+
+            # The max_unet_downscale is the maximum amount that the UNet model downscales the latent image internally.
+            if t2i_adapter_field.t2i_adapter_model.base_model == BaseModelType.StableDiffusion1:
+                max_unet_downscale = 8
+            elif t2i_adapter_field.t2i_adapter_model.base_model == BaseModelType.StableDiffusionXL:
+                max_unet_downscale = 4
+            else:
+                raise ValueError(
+                    f"Unexpected T2I-Adapter base model type: '{t2i_adapter_field.t2i_adapter_model.base_model}'."
+                )
+
+            t2i_adapter_model: T2IAdapter
+            with t2i_adapter_model_info as t2i_adapter_model:
+                total_downscale_factor = t2i_adapter_model.total_downscale_factor
+                if isinstance(t2i_adapter_model.adapter, FullAdapterXL):
+                    # HACK(ryand): Work around a bug in FullAdapterXL. This is being addressed upstream in diffusers by
+                    # this PR: https://github.com/huggingface/diffusers/pull/5134.
+                    total_downscale_factor = total_downscale_factor // 2
+
+                # Resize the T2I-Adapter input image.
+                # We select the resize dimensions so that after the T2I-Adapter's total_downscale_factor is applied, the
+                # result will match the latent image's dimensions after max_unet_downscale is applied.
+                t2i_input_height = latents_shape[2] // max_unet_downscale * total_downscale_factor
+                t2i_input_width = latents_shape[3] // max_unet_downscale * total_downscale_factor
+
+                # Note: We have hard-coded `do_classifier_free_guidance=False`. This is because we only want to prepare
+                # a single image. If CFG is enabled, we will duplicate the resultant tensor after applying the
+                # T2I-Adapter model.
+                #
+                # Note: We re-use the `prepare_control_image(...)` from ControlNet for T2I-Adapter, because it has many
+                # of the same requirements (e.g. preserving binary masks during resize).
+                t2i_image = prepare_control_image(
+                    image=image,
+                    do_classifier_free_guidance=False,
+                    width=t2i_input_width,
+                    height=t2i_input_height,
+                    num_channels=t2i_adapter_model.config.in_channels,
+                    device=t2i_adapter_model.device,
+                    dtype=t2i_adapter_model.dtype,
+                    resize_mode=t2i_adapter_field.resize_mode,
+                )
+
+                adapter_state = t2i_adapter_model(t2i_image)
+
+            if do_classifier_free_guidance:
+                for idx, value in enumerate(adapter_state):
+                    adapter_state[idx] = torch.cat([value] * 2, dim=0)
+
+            t2i_adapter_data.append(
+                T2IAdapterData(
+                    adapter_state=adapter_state,
+                    weight=t2i_adapter_field.weight,
+                    begin_step_percent=t2i_adapter_field.begin_step_percent,
+                    end_step_percent=t2i_adapter_field.end_step_percent,
+                )
+            )
+
+        return t2i_adapter_data
 
     # original idea by https://github.com/AmericanPresidentJimmyCarter
     # TODO: research more for second order schedulers timesteps
@@ -522,6 +625,12 @@ class DenoiseLatentsInvocation(BaseInvocation):
 
             mask, masked_latents = self.prep_inpaint_mask(context, latents)
 
+            # TODO(ryand): I have hard-coded `do_classifier_free_guidance=True` to mirror the behaviour of ControlNets,
+            # below. Investigate whether this is appropriate.
+            t2i_adapter_data = self.run_t2i_adapters(
+                context, self.t2i_adapter, latents.shape, do_classifier_free_guidance=True
+            )
+
             # Get the source node id (we are invoking the prepared node)
             graph_execution_state = context.services.graph_execution_manager.get(context.graph_execution_state_id)
             source_node_id = graph_execution_state.prepared_source_mapping[self.id]
@@ -580,7 +689,6 @@ class DenoiseLatentsInvocation(BaseInvocation):
                     context=context,
                     ip_adapter=self.ip_adapter,
                     conditioning_data=conditioning_data,
-                    unet=unet,
                     exit_stack=exit_stack,
                 )
 
@@ -602,8 +710,9 @@ class DenoiseLatentsInvocation(BaseInvocation):
                     masked_latents=masked_latents,
                     num_inference_steps=num_inference_steps,
                     conditioning_data=conditioning_data,
-                    control_data=controlnet_data,  # list[ControlNetData],
-                    ip_adapter_data=ip_adapter_data,  # IPAdapterData,
+                    control_data=controlnet_data,
+                    ip_adapter_data=ip_adapter_data,
+                    t2i_adapter_data=t2i_adapter_data,
                     callback=step_callback,
                 )
 

invokeai/app/invocations/math.py

@@ -65,13 +65,27 @@ class DivideInvocation(BaseInvocation):
 class RandomIntInvocation(BaseInvocation):
     """Outputs a single random integer."""
 
-    low: int = InputField(default=0, description="The inclusive low value")
-    high: int = InputField(default=np.iinfo(np.int32).max, description="The exclusive high value")
+    low: int = InputField(default=0, description=FieldDescriptions.inclusive_low)
+    high: int = InputField(default=np.iinfo(np.int32).max, description=FieldDescriptions.exclusive_high)
 
     def invoke(self, context: InvocationContext) -> IntegerOutput:
         return IntegerOutput(value=np.random.randint(self.low, self.high))
 
 
+@invocation("rand_float", title="Random Float", tags=["math", "float", "random"], category="math", version="1.0.0")
+class RandomFloatInvocation(BaseInvocation):
+    """Outputs a single random float"""
+
+    low: float = InputField(default=0.0, description=FieldDescriptions.inclusive_low)
+    high: float = InputField(default=1.0, description=FieldDescriptions.exclusive_high)
+    decimals: int = InputField(default=2, description=FieldDescriptions.decimal_places)
+
+    def invoke(self, context: InvocationContext) -> FloatOutput:
+        random_float = np.random.uniform(self.low, self.high)
+        rounded_float = round(random_float, self.decimals)
+        return FloatOutput(value=rounded_float)
+
+
 @invocation(
     "float_to_int",
     title="Float To Integer",

invokeai/app/invocations/metadata.py

@@ -12,7 +12,10 @@ from invokeai.app.invocations.baseinvocation import (
     invocation_output,
 )
 from invokeai.app.invocations.controlnet_image_processors import ControlField
+from invokeai.app.invocations.ip_adapter import IPAdapterModelField
 from invokeai.app.invocations.model import LoRAModelField, MainModelField, VAEModelField
+from invokeai.app.invocations.primitives import ImageField
+from invokeai.app.invocations.t2i_adapter import T2IAdapterField
 from invokeai.app.util.model_exclude_null import BaseModelExcludeNull
 
 from ...version import __version__
@@ -25,6 +28,18 @@ class LoRAMetadataField(BaseModelExcludeNull):
     weight: float = Field(description="The weight of the LoRA model")
 
 
+class IPAdapterMetadataField(BaseModelExcludeNull):
+    image: ImageField = Field(description="The IP-Adapter image prompt.")
+    ip_adapter_model: IPAdapterModelField = Field(description="The IP-Adapter model to use.")
+    weight: float = Field(description="The weight of the IP-Adapter model")
+    begin_step_percent: float = Field(
+        default=0, ge=0, le=1, description="When the IP-Adapter is first applied (% of total steps)"
+    )
+    end_step_percent: float = Field(
+        default=1, ge=0, le=1, description="When the IP-Adapter is last applied (% of total steps)"
+    )
+
+
 class CoreMetadata(BaseModelExcludeNull):
     """Core generation metadata for an image generated in InvokeAI."""
 
@@ -48,6 +63,8 @@ class CoreMetadata(BaseModelExcludeNull):
     )
     model: MainModelField = Field(description="The main model used for inference")
     controlnets: list[ControlField] = Field(description="The ControlNets used for inference")
+    ipAdapters: list[IPAdapterMetadataField] = Field(description="The IP Adapters used for inference")
+    t2iAdapters: list[T2IAdapterField] = Field(description="The IP Adapters used for inference")
     loras: list[LoRAMetadataField] = Field(description="The LoRAs used for inference")
     vae: Optional[VAEModelField] = Field(
         default=None,
@@ -123,6 +140,8 @@ class MetadataAccumulatorInvocation(BaseInvocation):
     )
     model: MainModelField = InputField(description="The main model used for inference")
     controlnets: list[ControlField] = InputField(description="The ControlNets used for inference")
+    ipAdapters: list[IPAdapterMetadataField] = InputField(description="The IP Adapters used for inference")
+    t2iAdapters: list[T2IAdapterField] = Field(description="The IP Adapters used for inference")
     loras: list[LoRAMetadataField] = InputField(description="The LoRAs used for inference")
     strength: Optional[float] = InputField(
         default=None,

invokeai/app/invocations/t2i_adapter.py

@@ -0,0 +1,83 @@
+from typing import Union
+
+from pydantic import BaseModel, Field
+
+from invokeai.app.invocations.baseinvocation import (
+    BaseInvocation,
+    BaseInvocationOutput,
+    FieldDescriptions,
+    Input,
+    InputField,
+    InvocationContext,
+    OutputField,
+    UIType,
+    invocation,
+    invocation_output,
+)
+from invokeai.app.invocations.controlnet_image_processors import CONTROLNET_RESIZE_VALUES
+from invokeai.app.invocations.primitives import ImageField
+from invokeai.backend.model_management.models.base import BaseModelType
+
+
+class T2IAdapterModelField(BaseModel):
+    model_name: str = Field(description="Name of the T2I-Adapter model")
+    base_model: BaseModelType = Field(description="Base model")
+
+
+class T2IAdapterField(BaseModel):
+    image: ImageField = Field(description="The T2I-Adapter image prompt.")
+    t2i_adapter_model: T2IAdapterModelField = Field(description="The T2I-Adapter model to use.")
+    weight: Union[float, list[float]] = Field(default=1, description="The weight given to the T2I-Adapter")
+    begin_step_percent: float = Field(
+        default=0, ge=0, le=1, description="When the T2I-Adapter is first applied (% of total steps)"
+    )
+    end_step_percent: float = Field(
+        default=1, ge=0, le=1, description="When the T2I-Adapter is last applied (% of total steps)"
+    )
+    resize_mode: CONTROLNET_RESIZE_VALUES = Field(default="just_resize", description="The resize mode to use")
+
+
+@invocation_output("t2i_adapter_output")
+class T2IAdapterOutput(BaseInvocationOutput):
+    t2i_adapter: T2IAdapterField = OutputField(description=FieldDescriptions.t2i_adapter, title="T2I Adapter")
+
+
+@invocation(
+    "t2i_adapter", title="T2I-Adapter", tags=["t2i_adapter", "control"], category="t2i_adapter", version="1.0.0"
+)
+class T2IAdapterInvocation(BaseInvocation):
+    """Collects T2I-Adapter info to pass to other nodes."""
+
+    # Inputs
+    image: ImageField = InputField(description="The IP-Adapter image prompt.")
+    t2i_adapter_model: T2IAdapterModelField = InputField(
+        description="The T2I-Adapter model.",
+        title="T2I-Adapter Model",
+        input=Input.Direct,
+        ui_order=-1,
+    )
+    weight: Union[float, list[float]] = InputField(
+        default=1, ge=0, description="The weight given to the T2I-Adapter", ui_type=UIType.Float, title="Weight"
+    )
+    begin_step_percent: float = InputField(
+        default=0, ge=-1, le=2, description="When the T2I-Adapter is first applied (% of total steps)"
+    )
+    end_step_percent: float = InputField(
+        default=1, ge=0, le=1, description="When the T2I-Adapter is last applied (% of total steps)"
+    )
+    resize_mode: CONTROLNET_RESIZE_VALUES = InputField(
+        default="just_resize",
+        description="The resize mode applied to the T2I-Adapter input image so that it matches the target output size.",
+    )
+
+    def invoke(self, context: InvocationContext) -> T2IAdapterOutput:
+        return T2IAdapterOutput(
+            t2i_adapter=T2IAdapterField(
+                image=self.image,
+                t2i_adapter_model=self.t2i_adapter_model,
+                weight=self.weight,
+                begin_step_percent=self.begin_step_percent,
+                end_step_percent=self.end_step_percent,
+                resize_mode=self.resize_mode,
+            )
+        )

invokeai/app/invocations/upscale.py

@@ -4,12 +4,14 @@ from typing import Literal
 
 import cv2 as cv
 import numpy as np
+import torch
 from basicsr.archs.rrdbnet_arch import RRDBNet
 from PIL import Image
 from realesrgan import RealESRGANer
 
 from invokeai.app.invocations.primitives import ImageField, ImageOutput
 from invokeai.app.models.image import ImageCategory, ResourceOrigin
+from invokeai.backend.util.devices import choose_torch_device
 
 from .baseinvocation import BaseInvocation, InputField, InvocationContext, invocation
 
@@ -22,13 +24,19 @@ ESRGAN_MODELS = Literal[
     "RealESRGAN_x2plus.pth",
 ]
 
+if choose_torch_device() == torch.device("mps"):
+    from torch import mps
 
-@invocation("esrgan", title="Upscale (RealESRGAN)", tags=["esrgan", "upscale"], category="esrgan", version="1.0.0")
+
+@invocation("esrgan", title="Upscale (RealESRGAN)", tags=["esrgan", "upscale"], category="esrgan", version="1.1.0")
 class ESRGANInvocation(BaseInvocation):
     """Upscales an image using RealESRGAN."""
 
     image: ImageField = InputField(description="The input image")
     model_name: ESRGAN_MODELS = InputField(default="RealESRGAN_x4plus.pth", description="The Real-ESRGAN model to use")
+    tile_size: int = InputField(
+        default=400, ge=0, description="Tile size for tiled ESRGAN upscaling (0=tiling disabled)"
+    )
 
     def invoke(self, context: InvocationContext) -> ImageOutput:
         image = context.services.images.get_pil_image(self.image.image_name)
@@ -86,9 +94,11 @@ class ESRGANInvocation(BaseInvocation):
             model_path=str(models_path / esrgan_model_path),
             model=rrdbnet_model,
             half=False,
+            tile=self.tile_size,
         )
 
         # prepare image - Real-ESRGAN uses cv2 internally, and cv2 uses BGR vs RGB for PIL
+        # TODO: This strips the alpha... is that okay?
         cv_image = cv.cvtColor(np.array(image.convert("RGB")), cv.COLOR_RGB2BGR)
 
         # We can pass an `outscale` value here, but it just resizes the image by that factor after
@@ -99,6 +109,10 @@ class ESRGANInvocation(BaseInvocation):
         # back to PIL
         pil_image = Image.fromarray(cv.cvtColor(upscaled_image, cv.COLOR_BGR2RGB)).convert("RGBA")
 
+        torch.cuda.empty_cache()
+        if choose_torch_device() == torch.device("mps"):
+            mps.empty_cache()
+
         image_dto = context.services.images.create(
             image=pil_image,
             image_origin=ResourceOrigin.INTERNAL,

invokeai/app/services/config/invokeai_config.py

@@ -241,7 +241,7 @@ class InvokeAIAppConfig(InvokeAISettings):
     version             : bool = Field(default=False, description="Show InvokeAI version and exit", category="Other")
 
     # CACHE
-    ram                 : Union[float, Literal["auto"]] = Field(default=6.0, gt=0, description="Maximum memory amount used by model cache for rapid switching (floating point number or 'auto')", category="Model Cache", )
+    ram                 : Union[float, Literal["auto"]] = Field(default=7.5, gt=0, description="Maximum memory amount used by model cache for rapid switching (floating point number or 'auto')", category="Model Cache", )
     vram                : Union[float, Literal["auto"]] = Field(default=0.25, ge=0, description="Amount of VRAM reserved for model storage (floating point number or 'auto')", category="Model Cache", )
     lazy_offload        : bool = Field(default=True, description="Keep models in VRAM until their space is needed", category="Model Cache", )
 
@@ -255,6 +255,7 @@ class InvokeAIAppConfig(InvokeAISettings):
     attention_slice_size: Literal["auto", "balanced", "max", 1, 2, 3, 4, 5, 6, 7, 8] = Field(default="auto", description='Slice size, valid when attention_type=="sliced"', category="Generation", )
     force_tiled_decode  : bool = Field(default=False, description="Whether to enable tiled VAE decode (reduces memory consumption with some performance penalty)", category="Generation",)
     force_tiled_decode: bool = Field(default=False, description="Whether to enable tiled VAE decode (reduces memory consumption with some performance penalty)", category="Generation",)
+    png_compress_level  : int = Field(default=6, description="The compress_level setting of PIL.Image.save(), used for PNG encoding. All settings are lossless. 0 = fastest, largest filesize, 9 = slowest, smallest filesize", category="Generation", )
 
     # QUEUE
     max_queue_size      : int = Field(default=10000, gt=0, description="Maximum number of items in the session queue", category="Queue", )

invokeai/app/services/events.py

@@ -4,7 +4,12 @@ from typing import Any, Optional
 
 from invokeai.app.models.image import ProgressImage
 from invokeai.app.services.model_manager_service import BaseModelType, ModelInfo, ModelType, SubModelType
-from invokeai.app.services.session_queue.session_queue_common import EnqueueBatchResult, SessionQueueItem
+from invokeai.app.services.session_queue.session_queue_common import (
+    BatchStatus,
+    EnqueueBatchResult,
+    SessionQueueItem,
+    SessionQueueStatus,
+)
 from invokeai.app.util.misc import get_timestamp
 
 
@@ -262,21 +267,31 @@ class EventServiceBase:
             ),
         )
 
-    def emit_queue_item_status_changed(self, session_queue_item: SessionQueueItem) -> None:
+    def emit_queue_item_status_changed(
+        self,
+        session_queue_item: SessionQueueItem,
+        batch_status: BatchStatus,
+        queue_status: SessionQueueStatus,
+    ) -> None:
         """Emitted when a queue item's status changes"""
         self.__emit_queue_event(
             event_name="queue_item_status_changed",
             payload=dict(
-                queue_id=session_queue_item.queue_id,
-                queue_item_id=session_queue_item.item_id,
-                status=session_queue_item.status,
-                batch_id=session_queue_item.batch_id,
-                session_id=session_queue_item.session_id,
-                error=session_queue_item.error,
-                created_at=str(session_queue_item.created_at) if session_queue_item.created_at else None,
-                updated_at=str(session_queue_item.updated_at) if session_queue_item.updated_at else None,
-                started_at=str(session_queue_item.started_at) if session_queue_item.started_at else None,
-                completed_at=str(session_queue_item.completed_at) if session_queue_item.completed_at else None,
+                queue_id=queue_status.queue_id,
+                queue_item=dict(
+                    queue_id=session_queue_item.queue_id,
+                    item_id=session_queue_item.item_id,
+                    status=session_queue_item.status,
+                    batch_id=session_queue_item.batch_id,
+                    session_id=session_queue_item.session_id,
+                    error=session_queue_item.error,
+                    created_at=str(session_queue_item.created_at) if session_queue_item.created_at else None,
+                    updated_at=str(session_queue_item.updated_at) if session_queue_item.updated_at else None,
+                    started_at=str(session_queue_item.started_at) if session_queue_item.started_at else None,
+                    completed_at=str(session_queue_item.completed_at) if session_queue_item.completed_at else None,
+                ),
+                batch_status=batch_status.dict(),
+                queue_status=queue_status.dict(),
             ),
         )
 

invokeai/app/services/graph.py

@@ -2,7 +2,7 @@
 
 import copy
 import itertools
-from typing import Annotated, Any, Optional, Union, cast, get_args, get_origin, get_type_hints
+from typing import Annotated, Any, Optional, Union, get_args, get_origin, get_type_hints
 
 import networkx as nx
 from pydantic import BaseModel, root_validator, validator
@@ -170,6 +170,18 @@ class NodeIdMismatchError(ValueError):
     pass
 
 
+class InvalidSubGraphError(ValueError):
+    pass
+
+
+class CyclicalGraphError(ValueError):
+    pass
+
+
+class UnknownGraphValidationError(ValueError):
+    pass
+
+
 # TODO: Create and use an Empty output?
 @invocation_output("graph_output")
 class GraphInvocationOutput(BaseInvocationOutput):
@@ -254,59 +266,6 @@ class Graph(BaseModel):
         default_factory=list,
     )
 
-    @root_validator
-    def validate_nodes_and_edges(cls, values):
-        """Validates that all edges match nodes in the graph"""
-        nodes = cast(Optional[dict[str, BaseInvocation]], values.get("nodes"))
-        edges = cast(Optional[list[Edge]], values.get("edges"))
-
-        if nodes is not None:
-            # Validate that all node ids are unique
-            node_ids = [n.id for n in nodes.values()]
-            duplicate_node_ids = set([node_id for node_id in node_ids if node_ids.count(node_id) >= 2])
-            if duplicate_node_ids:
-                raise DuplicateNodeIdError(f"Node ids must be unique, found duplicates {duplicate_node_ids}")
-
-            # Validate that all node ids match the keys in the nodes dict
-            for k, v in nodes.items():
-                if k != v.id:
-                    raise NodeIdMismatchError(f"Node ids must match, got {k} and {v.id}")
-
-        if edges is not None and nodes is not None:
-            # Validate that all edges match nodes in the graph
-            node_ids = set([e.source.node_id for e in edges] + [e.destination.node_id for e in edges])
-            missing_node_ids = [node_id for node_id in node_ids if node_id not in nodes]
-            if missing_node_ids:
-                raise NodeNotFoundError(
-                    f"All edges must reference nodes in the graph, missing nodes: {missing_node_ids}"
-                )
-
-            # Validate that all edge fields match node fields in the graph
-            for edge in edges:
-                source_node = nodes.get(edge.source.node_id, None)
-                if source_node is None:
-                    raise NodeFieldNotFoundError(f"Edge source node {edge.source.node_id} does not exist in the graph")
-
-                destination_node = nodes.get(edge.destination.node_id, None)
-                if destination_node is None:
-                    raise NodeFieldNotFoundError(
-                        f"Edge destination node {edge.destination.node_id} does not exist in the graph"
-                    )
-
-                # output fields are not on the node object directly, they are on the output type
-                if edge.source.field not in source_node.get_output_type().__fields__:
-                    raise NodeFieldNotFoundError(
-                        f"Edge source field {edge.source.field} does not exist in node {edge.source.node_id}"
-                    )
-
-                # input fields are on the node
-                if edge.destination.field not in destination_node.__fields__:
-                    raise NodeFieldNotFoundError(
-                        f"Edge destination field {edge.destination.field} does not exist in node {edge.destination.node_id}"
-                    )
-
-        return values
-
     def add_node(self, node: BaseInvocation) -> None:
         """Adds a node to a graph
 
@@ -377,53 +336,108 @@ class Graph(BaseModel):
         except KeyError:
             pass
 
-    def is_valid(self) -> bool:
-        """Validates the graph."""
+    def validate_self(self) -> None:
+        """
+        Validates the graph.
+
+        Raises an exception if the graph is invalid:
+        - `DuplicateNodeIdError`
+        - `NodeIdMismatchError`
+        - `InvalidSubGraphError`
+        - `NodeNotFoundError`
+        - `NodeFieldNotFoundError`
+        - `CyclicalGraphError`
+        - `InvalidEdgeError`
+        """
+
+        # Validate that all node ids are unique
+        node_ids = [n.id for n in self.nodes.values()]
+        duplicate_node_ids = set([node_id for node_id in node_ids if node_ids.count(node_id) >= 2])
+        if duplicate_node_ids:
+            raise DuplicateNodeIdError(f"Node ids must be unique, found duplicates {duplicate_node_ids}")
+
+        # Validate that all node ids match the keys in the nodes dict
+        for k, v in self.nodes.items():
+            if k != v.id:
+                raise NodeIdMismatchError(f"Node ids must match, got {k} and {v.id}")
 
         # Validate all subgraphs
         for gn in (n for n in self.nodes.values() if isinstance(n, GraphInvocation)):
-            if not gn.graph.is_valid():
-                return False
+            try:
+                gn.graph.validate_self()
+            except Exception as e:
+                raise InvalidSubGraphError(f"Subgraph {gn.id} is invalid") from e
 
-        # Validate all edges reference nodes in the graph
-        node_ids = set([e.source.node_id for e in self.edges] + [e.destination.node_id for e in self.edges])
-        if not all((self.has_node(node_id) for node_id in node_ids)):
-            return False
+        # Validate that all edges match nodes and fields in the graph
+        for edge in self.edges:
+            source_node = self.nodes.get(edge.source.node_id, None)
+            if source_node is None:
+                raise NodeNotFoundError(f"Edge source node {edge.source.node_id} does not exist in the graph")
+
+            destination_node = self.nodes.get(edge.destination.node_id, None)
+            if destination_node is None:
+                raise NodeNotFoundError(f"Edge destination node {edge.destination.node_id} does not exist in the graph")
+
+            # output fields are not on the node object directly, they are on the output type
+            if edge.source.field not in source_node.get_output_type().__fields__:
+                raise NodeFieldNotFoundError(
+                    f"Edge source field {edge.source.field} does not exist in node {edge.source.node_id}"
+                )
+
+            # input fields are on the node
+            if edge.destination.field not in destination_node.__fields__:
+                raise NodeFieldNotFoundError(
+                    f"Edge destination field {edge.destination.field} does not exist in node {edge.destination.node_id}"
+                )
 
         # Validate there are no cycles
         g = self.nx_graph_flat()
         if not nx.is_directed_acyclic_graph(g):
-            return False
+            raise CyclicalGraphError("Graph contains cycles")
 
         # Validate all edge connections are valid
-        if not all(
-            (
-                are_connections_compatible(
-                    self.get_node(e.source.node_id),
-                    e.source.field,
-                    self.get_node(e.destination.node_id),
-                    e.destination.field,
+        for e in self.edges:
+            if not are_connections_compatible(
+                self.get_node(e.source.node_id),
+                e.source.field,
+                self.get_node(e.destination.node_id),
+                e.destination.field,
+            ):
+                raise InvalidEdgeError(
+                    f"Invalid edge from {e.source.node_id}.{e.source.field} to {e.destination.node_id}.{e.destination.field}"
                 )
-                for e in self.edges
-            )
+
+        # Validate all iterators & collectors
+        # TODO: may need to validate all iterators & collectors in subgraphs so edge connections in parent graphs will be available
+        for n in self.nodes.values():
+            if isinstance(n, IterateInvocation) and not self._is_iterator_connection_valid(n.id):
+                raise InvalidEdgeError(f"Invalid iterator node {n.id}")
+            if isinstance(n, CollectInvocation) and not self._is_collector_connection_valid(n.id):
+                raise InvalidEdgeError(f"Invalid collector node {n.id}")
+
+        return None
+
+    def is_valid(self) -> bool:
+        """
+        Checks if the graph is valid.
+
+        Raises `UnknownGraphValidationError` if there is a problem validating the graph (not a validation error).
+        """
+        try:
+            self.validate_self()
+            return True
+        except (
+            DuplicateNodeIdError,
+            NodeIdMismatchError,
+            InvalidSubGraphError,
+            NodeNotFoundError,
+            NodeFieldNotFoundError,
+            CyclicalGraphError,
+            InvalidEdgeError,
         ):
             return False
-
-        # Validate all iterators
-        # TODO: may need to validate all iterators in subgraphs so edge connections in parent graphs will be available
-        if not all(
-            (self._is_iterator_connection_valid(n.id) for n in self.nodes.values() if isinstance(n, IterateInvocation))
-        ):
-            return False
-
-        # Validate all collectors
-        # TODO: may need to validate all collectors in subgraphs so edge connections in parent graphs will be available
-        if not all(
-            (self._is_collector_connection_valid(n.id) for n in self.nodes.values() if isinstance(n, CollectInvocation))
-        ):
-            return False
-
-        return True
+        except Exception as e:
+            raise UnknownGraphValidationError(f"Problem validating graph {e}") from e
 
     def _validate_edge(self, edge: Edge):
         """Validates that a new edge doesn't create a cycle in the graph"""
@@ -804,6 +818,12 @@ class GraphExecutionState(BaseModel):
         default_factory=dict,
     )
 
+    @validator("graph")
+    def graph_is_valid(cls, v: Graph):
+        """Validates that the graph is valid"""
+        v.validate_self()
+        return v
+
     class Config:
         schema_extra = {
             "required": [

invokeai/app/services/image_file_storage.py

@@ -9,6 +9,7 @@ from PIL import Image, PngImagePlugin
 from PIL.Image import Image as PILImageType
 from send2trash import send2trash
 
+from invokeai.app.services.config.invokeai_config import InvokeAIAppConfig
 from invokeai.app.util.thumbnails import get_thumbnail_name, make_thumbnail
 
 
@@ -79,6 +80,7 @@ class DiskImageFileStorage(ImageFileStorageBase):
     __cache_ids: Queue  # TODO: this is an incredibly naive cache
     __cache: Dict[Path, PILImageType]
     __max_cache_size: int
+    __compress_level: int
 
     def __init__(self, output_folder: Union[str, Path]):
         self.__cache = dict()
@@ -87,7 +89,7 @@ class DiskImageFileStorage(ImageFileStorageBase):
 
         self.__output_folder: Path = output_folder if isinstance(output_folder, Path) else Path(output_folder)
         self.__thumbnails_folder = self.__output_folder / "thumbnails"
-
+        self.__compress_level = InvokeAIAppConfig.get_config().png_compress_level
         # Validate required output folders at launch
         self.__validate_storage_folders()
 
@@ -134,7 +136,7 @@ class DiskImageFileStorage(ImageFileStorageBase):
                 if original_workflow is not None:
                     pnginfo.add_text("invokeai_workflow", original_workflow)
 
-            image.save(image_path, "PNG", pnginfo=pnginfo)
+            image.save(image_path, "PNG", pnginfo=pnginfo, compress_level=self.__compress_level)
 
             thumbnail_name = get_thumbnail_name(image_name)
             thumbnail_path = self.get_path(thumbnail_name, thumbnail=True)

invokeai/app/services/image_record_storage.py

@@ -584,7 +584,7 @@ class SqliteImageRecordStorage(ImageRecordStorageBase):
                 FROM images
                 JOIN board_images ON images.image_name = board_images.image_name
                 WHERE board_images.board_id = ?
-                ORDER BY images.created_at DESC
+                ORDER BY images.starred DESC, images.created_at DESC
                 LIMIT 1;
                 """,
                 (board_id,),

invokeai/app/services/invocation_cache/invocation_cache_memory.py

@@ -1,7 +1,6 @@
 from collections import OrderedDict
 from dataclasses import dataclass, field
 from threading import Lock
-from time import time
 from typing import Optional, Union
 
 from invokeai.app.invocations.baseinvocation import BaseInvocation, BaseInvocationOutput
@@ -59,7 +58,7 @@ class MemoryInvocationCache(InvocationCacheBase):
             # If the cache is full, we need to remove the least used
             number_to_delete = len(self._cache) + 1 - self._max_cache_size
             self._delete_oldest_access(number_to_delete)
-            self._cache[key] = CachedItem(time(), invocation_output, invocation_output.json())
+            self._cache[key] = CachedItem(invocation_output, invocation_output.json())
 
     def _delete_oldest_access(self, number_to_delete: int) -> None:
         number_to_delete = min(number_to_delete, len(self._cache))

invokeai/app/services/session_processor/session_processor_default.py

@@ -1,3 +1,4 @@
+import traceback
 from threading import BoundedSemaphore
 from threading import Event as ThreadEvent
 from threading import Thread
@@ -123,6 +124,10 @@ class DefaultSessionProcessor(SessionProcessorBase):
                         continue
                 except Exception as e:
                     self.__invoker.services.logger.error(f"Error in session processor: {e}")
+                    if queue_item is not None:
+                        self.__invoker.services.session_queue.cancel_queue_item(
+                            queue_item.item_id, error=traceback.format_exc()
+                        )
                     poll_now_event.wait(POLLING_INTERVAL)
                     continue
         except Exception as e:

invokeai/app/services/session_queue/session_queue_base.py

@@ -80,7 +80,7 @@ class SessionQueueBase(ABC):
         pass
 
     @abstractmethod
-    def cancel_queue_item(self, item_id: int) -> SessionQueueItem:
+    def cancel_queue_item(self, item_id: int, error: Optional[str] = None) -> SessionQueueItem:
         """Cancels a session queue item"""
         pass
 

invokeai/app/services/session_queue/session_queue_common.py

@@ -123,6 +123,11 @@ class Batch(BaseModel):
                     raise NodeNotFoundError(f"Field {batch_data.field_name} not found in node {batch_data.node_path}")
         return values
 
+    @validator("graph")
+    def validate_graph(cls, v: Graph):
+        v.validate_self()
+        return v
+
     class Config:
         schema_extra = {
             "required": [

invokeai/app/services/session_queue/session_queue_sqlite.py

@@ -427,7 +427,13 @@ class SqliteSessionQueue(SessionQueueBase):
         finally:
             self.__lock.release()
         queue_item = self.get_queue_item(item_id)
-        self.__invoker.services.events.emit_queue_item_status_changed(queue_item)
+        batch_status = self.get_batch_status(queue_id=queue_item.queue_id, batch_id=queue_item.batch_id)
+        queue_status = self.get_queue_status(queue_id=queue_item.queue_id)
+        self.__invoker.services.events.emit_queue_item_status_changed(
+            session_queue_item=queue_item,
+            batch_status=batch_status,
+            queue_status=queue_status,
+        )
         return queue_item
 
     def is_empty(self, queue_id: str) -> IsEmptyResult:
@@ -555,10 +561,11 @@ class SqliteSessionQueue(SessionQueueBase):
             self.__lock.release()
         return PruneResult(deleted=count)
 
-    def cancel_queue_item(self, item_id: int) -> SessionQueueItem:
+    def cancel_queue_item(self, item_id: int, error: Optional[str] = None) -> SessionQueueItem:
         queue_item = self.get_queue_item(item_id)
         if queue_item.status not in ["canceled", "failed", "completed"]:
-            queue_item = self._set_queue_item_status(item_id=item_id, status="canceled")
+            status = "failed" if error is not None else "canceled"
+            queue_item = self._set_queue_item_status(item_id=item_id, status=status, error=error)
             self.__invoker.services.queue.cancel(queue_item.session_id)
             self.__invoker.services.events.emit_session_canceled(
                 queue_item_id=queue_item.item_id,
@@ -608,7 +615,13 @@ class SqliteSessionQueue(SessionQueueBase):
                     queue_batch_id=current_queue_item.batch_id,
                     graph_execution_state_id=current_queue_item.session_id,
                 )
-                self.__invoker.services.events.emit_queue_item_status_changed(current_queue_item)
+                batch_status = self.get_batch_status(queue_id=queue_id, batch_id=current_queue_item.batch_id)
+                queue_status = self.get_queue_status(queue_id=queue_id)
+                self.__invoker.services.events.emit_queue_item_status_changed(
+                    session_queue_item=current_queue_item,
+                    batch_status=batch_status,
+                    queue_status=queue_status,
+                )
         except Exception:
             self.__conn.rollback()
             raise
@@ -654,7 +667,13 @@ class SqliteSessionQueue(SessionQueueBase):
                     queue_batch_id=current_queue_item.batch_id,
                     graph_execution_state_id=current_queue_item.session_id,
                 )
-                self.__invoker.services.events.emit_queue_item_status_changed(current_queue_item)
+                batch_status = self.get_batch_status(queue_id=queue_id, batch_id=current_queue_item.batch_id)
+                queue_status = self.get_queue_status(queue_id=queue_id)
+                self.__invoker.services.events.emit_queue_item_status_changed(
+                    session_queue_item=current_queue_item,
+                    batch_status=batch_status,
+                    queue_status=queue_status,
+                )
         except Exception:
             self.__conn.rollback()
             raise

invokeai/app/util/controlnet_utils.py

@@ -265,22 +265,41 @@ def np_img_resize(np_img: np.ndarray, resize_mode: str, h: int, w: int, device:
 
 
 def prepare_control_image(
-    # image used to be Union[PIL.Image.Image, List[PIL.Image.Image], torch.Tensor, List[torch.Tensor]]
-    # but now should be able to assume that image is a single PIL.Image, which simplifies things
     image: Image,
-    # FIXME: need to fix hardwiring of width and height, change to basing on latents dimensions?
-    # latents_to_match_resolution, # TorchTensor of shape (batch_size, 3, height, width)
-    width=512,  # should be 8 * latent.shape[3]
-    height=512,  # should be 8 * latent height[2]
-    # batch_size=1, # currently no batching
-    # num_images_per_prompt=1, # currently only single image
+    width: int,
+    height: int,
+    num_channels: int = 3,
     device="cuda",
     dtype=torch.float16,
     do_classifier_free_guidance=True,
     control_mode="balanced",
     resize_mode="just_resize_simple",
 ):
-    # FIXME: implement "crop_resize_simple" and "fill_resize_simple", or pull them out
+    """Pre-process images for ControlNets or T2I-Adapters.
+
+    Args:
+        image (Image): The PIL image to pre-process.
+        width (int): The target width in pixels.
+        height (int): The target height in pixels.
+        num_channels (int, optional): The target number of image channels. This is achieved by converting the input
+            image to RGB, then naively taking the first `num_channels` channels. The primary use case is converting a
+            RGB image to a single-channel grayscale image. Raises if `num_channels` cannot be achieved. Defaults to 3.
+        device (str, optional): The target device for the output image. Defaults to "cuda".
+        dtype (_type_, optional): The dtype for the output image. Defaults to torch.float16.
+        do_classifier_free_guidance (bool, optional): If True, repeat the output image along the batch dimension.
+            Defaults to True.
+        control_mode (str, optional): Defaults to "balanced".
+        resize_mode (str, optional): Defaults to "just_resize_simple".
+
+    Raises:
+        NotImplementedError: If resize_mode == "crop_resize_simple".
+        NotImplementedError: If resize_mode == "fill_resize_simple".
+        ValueError: If `resize_mode` is not recognized.
+        ValueError: If `num_channels` is out of range.
+
+    Returns:
+        torch.Tensor: The pre-processed input tensor.
+    """
     if (
         resize_mode == "just_resize_simple"
         or resize_mode == "crop_resize_simple"
@@ -289,10 +308,10 @@ def prepare_control_image(
         image = image.convert("RGB")
         if resize_mode == "just_resize_simple":
             image = image.resize((width, height), resample=PIL_INTERPOLATION["lanczos"])
-        elif resize_mode == "crop_resize_simple":  # not yet implemented
-            pass
-        elif resize_mode == "fill_resize_simple":  # not yet implemented
-            pass
+        elif resize_mode == "crop_resize_simple":
+            raise NotImplementedError(f"prepare_control_image is not implemented for resize_mode='{resize_mode}'.")
+        elif resize_mode == "fill_resize_simple":
+            raise NotImplementedError(f"prepare_control_image is not implemented for resize_mode='{resize_mode}'.")
         nimage = np.array(image)
         nimage = nimage[None, :]
         nimage = np.concatenate([nimage], axis=0)
@@ -313,9 +332,11 @@ def prepare_control_image(
             device=device,
         )
     else:
-        pass
-        print("ERROR: invalid resize_mode ==> ", resize_mode)
-        exit(1)
+        raise ValueError(f"Unsupported resize_mode: '{resize_mode}'.")
+
+    if timage.shape[1] < num_channels or num_channels <= 0:
+        raise ValueError(f"Cannot achieve the target of num_channels={num_channels}.")
+    timage = timage[:, :num_channels, :, :]
 
     timage = timage.to(device=device, dtype=dtype)
     cfg_injection = control_mode == "more_control" or control_mode == "unbalanced"

invokeai/assets/fonts/inter/LICENSE.txt

@@ -0,0 +1,94 @@
+Copyright (c) 2016-2020 The Inter Project Authors.
+"Inter" is trademark of Rasmus Andersson.
+https://github.com/rsms/inter
+
+This Font Software is licensed under the SIL Open Font License, Version 1.1.
+This license is copied below, and is also available with a FAQ at:
+http://scripts.sil.org/OFL
+
+-----------------------------------------------------------
+SIL OPEN FONT LICENSE Version 1.1 - 26 February 2007
+-----------------------------------------------------------
+
+PREAMBLE
+The goals of the Open Font License (OFL) are to stimulate worldwide
+development of collaborative font projects, to support the font creation
+efforts of academic and linguistic communities, and to provide a free and
+open framework in which fonts may be shared and improved in partnership
+with others.
+
+The OFL allows the licensed fonts to be used, studied, modified and
+redistributed freely as long as they are not sold by themselves. The
+fonts, including any derivative works, can be bundled, embedded,
+redistributed and/or sold with any software provided that any reserved
+names are not used by derivative works. The fonts and derivatives,
+however, cannot be released under any other type of license. The
+requirement for fonts to remain under this license does not apply
+to any document created using the fonts or their derivatives.
+
+DEFINITIONS
+"Font Software" refers to the set of files released by the Copyright
+Holder(s) under this license and clearly marked as such. This may
+include source files, build scripts and documentation.
+
+"Reserved Font Name" refers to any names specified as such after the
+copyright statement(s).
+
+"Original Version" refers to the collection of Font Software components as
+distributed by the Copyright Holder(s).
+
+"Modified Version" refers to any derivative made by adding to, deleting,
+or substituting -- in part or in whole -- any of the components of the
+Original Version, by changing formats or by porting the Font Software to a
+new environment.
+
+"Author" refers to any designer, engineer, programmer, technical
+writer or other person who contributed to the Font Software.
+
+PERMISSION AND CONDITIONS
+Permission is hereby granted, free of charge, to any person obtaining
+a copy of the Font Software, to use, study, copy, merge, embed, modify,
+redistribute, and sell modified and unmodified copies of the Font
+Software, subject to the following conditions:
+
+1) Neither the Font Software nor any of its individual components,
+in Original or Modified Versions, may be sold by itself.
+
+2) Original or Modified Versions of the Font Software may be bundled,
+redistributed and/or sold with any software, provided that each copy
+contains the above copyright notice and this license. These can be
+included either as stand-alone text files, human-readable headers or
+in the appropriate machine-readable metadata fields within text or
+binary files as long as those fields can be easily viewed by the user.
+
+3) No Modified Version of the Font Software may use the Reserved Font
+Name(s) unless explicit written permission is granted by the corresponding
+Copyright Holder. This restriction only applies to the primary font name as
+presented to the users.
+
+4) The name(s) of the Copyright Holder(s) or the Author(s) of the Font
+Software shall not be used to promote, endorse or advertise any
+Modified Version, except to acknowledge the contribution(s) of the
+Copyright Holder(s) and the Author(s) or with their explicit written
+permission.
+
+5) The Font Software, modified or unmodified, in part or in whole,
+must be distributed entirely under this license, and must not be
+distributed under any other license. The requirement for fonts to
+remain under this license does not apply to any document created
+using the Font Software.
+
+TERMINATION
+This license becomes null and void if any of the above conditions are
+not met.
+
+DISCLAIMER
+THE FONT SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND,
+EXPRESS OR IMPLIED, INCLUDING BUT NOT LIMITED TO ANY WARRANTIES OF
+MERCHANTABILITY, FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT
+OF COPYRIGHT, PATENT, TRADEMARK, OR OTHER RIGHT. IN NO EVENT SHALL THE
+COPYRIGHT HOLDER BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER LIABILITY,
+INCLUDING ANY GENERAL, SPECIAL, INDIRECT, INCIDENTAL, OR CONSEQUENTIAL
+DAMAGES, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING
+FROM, OUT OF THE USE OR INABILITY TO USE THE FONT SOFTWARE OR FROM
+OTHER DEALINGS IN THE FONT SOFTWARE.

invokeai/backend/install/invokeai_configure.py

@@ -70,7 +70,6 @@ def get_literal_fields(field) -> list[Any]:
 config = InvokeAIAppConfig.get_config()
 
 Model_dir = "models"
-
 Default_config_file = config.model_conf_path
 SD_Configs = config.legacy_conf_path
 
@@ -458,7 +457,7 @@ Use cursor arrows to make a checkbox selection, and space to toggle.
         )
         self.add_widget_intelligent(
             npyscreen.TitleFixedText,
-            name="Model RAM cache size (GB). Make this at least large enough to hold a single full model.",
+            name="Model RAM cache size (GB). Make this at least large enough to hold a single full model (2GB for SD-1, 6GB for SDXL).",
             begin_entry_at=0,
             editable=False,
             color="CONTROL",
@@ -651,8 +650,19 @@ def edit_opts(program_opts: Namespace, invokeai_opts: Namespace) -> argparse.Nam
     return editApp.new_opts()
 
 
+def default_ramcache() -> float:
+    """Run a heuristic for the default RAM cache based on installed RAM."""
+
+    # Note that on my 64 GB machine, psutil.virtual_memory().total gives 62 GB,
+    # So we adjust everthing down a bit.
+    return (
+        15.0 if MAX_RAM >= 60 else 7.5 if MAX_RAM >= 30 else 4 if MAX_RAM >= 14 else 2.1
+    )  # 2.1 is just large enough for sd 1.5 ;-)
+
+
 def default_startup_options(init_file: Path) -> Namespace:
     opts = InvokeAIAppConfig.get_config()
+    opts.ram = default_ramcache()
     return opts
 
 

invokeai/backend/install/model_install_backend.py

@@ -2,6 +2,7 @@
 Utility (backend) functions used by model_install.py
 """
 import os
+import re
 import shutil
 import warnings
 from dataclasses import dataclass, field
@@ -88,6 +89,7 @@ class ModelLoadInfo:
     base_type: BaseModelType
     path: Optional[Path] = None
     repo_id: Optional[str] = None
+    subfolder: Optional[str] = None
     description: str = ""
     installed: bool = False
     recommended: bool = False
@@ -126,7 +128,10 @@ class ModelInstall(object):
             value["name"] = name
             value["base_type"] = base
             value["model_type"] = model_type
-            model_dict[key] = ModelLoadInfo(**value)
+            model_info = ModelLoadInfo(**value)
+            if model_info.subfolder and model_info.repo_id:
+                model_info.repo_id += f":{model_info.subfolder}"
+            model_dict[key] = model_info
 
         # supplement with entries in models.yaml
         installed_models = [x for x in self.mgr.list_models()]
@@ -317,46 +322,63 @@ class ModelInstall(object):
         return self._install_path(Path(models_path), info)
 
     def _install_repo(self, repo_id: str) -> AddModelResult:
+        # hack to recover models stored in subfolders --
+        # Required to get the "v2" model of monster-labs/control_v1p_sd15_qrcode_monster
+        subfolder = None
+        if match := re.match(r"^([^/]+/[^/]+):(\w+)$", repo_id):
+            repo_id = match.group(1)
+            subfolder = match.group(2)
+
         hinfo = HfApi().model_info(repo_id)
 
         # we try to figure out how to download this most economically
         # list all the files in the repo
         files = [x.rfilename for x in hinfo.siblings]
+        if subfolder:
+            files = [x for x in files if x.startswith(f"{subfolder}/")]
+        prefix = f"{subfolder}/" if subfolder else ""
+
         location = None
 
         with TemporaryDirectory(dir=self.config.models_path) as staging:
             staging = Path(staging)
-            if "model_index.json" in files:
-                location = self._download_hf_pipeline(repo_id, staging)  # pipeline
-            elif "unet/model.onnx" in files:
+            if f"{prefix}model_index.json" in files:
+                location = self._download_hf_pipeline(repo_id, staging, subfolder=subfolder)  # pipeline
+            elif f"{prefix}unet/model.onnx" in files:
                 location = self._download_hf_model(repo_id, files, staging)
             else:
                 for suffix in ["safetensors", "bin"]:
-                    if f"pytorch_lora_weights.{suffix}" in files:
-                        location = self._download_hf_model(repo_id, ["pytorch_lora_weights.bin"], staging)  # LoRA
+                    if f"{prefix}pytorch_lora_weights.{suffix}" in files:
+                        location = self._download_hf_model(
+                            repo_id, ["pytorch_lora_weights.bin"], staging, subfolder=subfolder
+                        )  # LoRA
                         break
                     elif (
-                        self.config.precision == "float16" and f"diffusion_pytorch_model.fp16.{suffix}" in files
+                        self.config.precision == "float16" and f"{prefix}diffusion_pytorch_model.fp16.{suffix}" in files
                     ):  # vae, controlnet or some other standalone
                         files = ["config.json", f"diffusion_pytorch_model.fp16.{suffix}"]
-                        location = self._download_hf_model(repo_id, files, staging)
+                        location = self._download_hf_model(repo_id, files, staging, subfolder=subfolder)
                         break
-                    elif f"diffusion_pytorch_model.{suffix}" in files:
+                    elif f"{prefix}diffusion_pytorch_model.{suffix}" in files:
                         files = ["config.json", f"diffusion_pytorch_model.{suffix}"]
-                        location = self._download_hf_model(repo_id, files, staging)
+                        location = self._download_hf_model(repo_id, files, staging, subfolder=subfolder)
                         break
-                    elif f"learned_embeds.{suffix}" in files:
-                        location = self._download_hf_model(repo_id, [f"learned_embeds.{suffix}"], staging)
+                    elif f"{prefix}learned_embeds.{suffix}" in files:
+                        location = self._download_hf_model(
+                            repo_id, [f"learned_embeds.{suffix}"], staging, subfolder=subfolder
+                        )
                         break
-                    elif "image_encoder.txt" in files and f"ip_adapter.{suffix}" in files:  # IP-Adapter
+                    elif (
+                        f"{prefix}image_encoder.txt" in files and f"{prefix}ip_adapter.{suffix}" in files
+                    ):  # IP-Adapter
                         files = ["image_encoder.txt", f"ip_adapter.{suffix}"]
-                        location = self._download_hf_model(repo_id, files, staging)
+                        location = self._download_hf_model(repo_id, files, staging, subfolder=subfolder)
                         break
-                    elif f"model.{suffix}" in files and "config.json" in files:
+                    elif f"{prefix}model.{suffix}" in files and f"{prefix}config.json" in files:
                         # This elif-condition is pretty fragile, but it is intended to handle CLIP Vision models hosted
                         # by InvokeAI for use with IP-Adapters.
                         files = ["config.json", f"model.{suffix}"]
-                        location = self._download_hf_model(repo_id, files, staging)
+                        location = self._download_hf_model(repo_id, files, staging, subfolder=subfolder)
                         break
             if not location:
                 logger.warning(f"Could not determine type of repo {repo_id}. Skipping install.")
@@ -443,9 +465,9 @@ class ModelInstall(object):
         else:
             return path
 
-    def _download_hf_pipeline(self, repo_id: str, staging: Path) -> Path:
+    def _download_hf_pipeline(self, repo_id: str, staging: Path, subfolder: str = None) -> Path:
         """
-        This retrieves a StableDiffusion model from cache or remote and then
+        Retrieve a StableDiffusion model from cache or remote and then
         does a save_pretrained() to the indicated staging area.
         """
         _, name = repo_id.split("/")
@@ -460,6 +482,7 @@ class ModelInstall(object):
                     variant=variant,
                     torch_dtype=precision,
                     safety_checker=None,
+                    subfolder=subfolder,
                 )
             except Exception as e:  # most errors are due to fp16 not being present. Fix this to catch other errors
                 if "fp16" not in str(e):
@@ -474,7 +497,7 @@ class ModelInstall(object):
         model.save_pretrained(staging / name, safe_serialization=True)
         return staging / name
 
-    def _download_hf_model(self, repo_id: str, files: List[str], staging: Path) -> Path:
+    def _download_hf_model(self, repo_id: str, files: List[str], staging: Path, subfolder: None) -> Path:
         _, name = repo_id.split("/")
         location = staging / name
         paths = list()
@@ -485,7 +508,7 @@ class ModelInstall(object):
                 model_dir=location / filePath.parent,
                 model_name=filePath.name,
                 access_token=self.access_token,
-                subfolder=filePath.parent,
+                subfolder=filePath.parent / subfolder if subfolder else filePath.parent,
             )
             if p:
                 paths.append(p)

invokeai/backend/ip_adapter/attention_processor.py

@@ -8,6 +8,8 @@ import torch.nn as nn
 import torch.nn.functional as F
 from diffusers.models.attention_processor import AttnProcessor2_0 as DiffusersAttnProcessor2_0
 
+from invokeai.backend.ip_adapter.ip_attention_weights import IPAttentionProcessorWeights
+
 
 # Create a version of AttnProcessor2_0 that is a sub-class of nn.Module. This is required for IP-Adapter state_dict
 # loading.
@@ -45,18 +47,16 @@ class IPAttnProcessor2_0(torch.nn.Module):
             the weight scale of image prompt.
     """
 
-    def __init__(self, hidden_size, cross_attention_dim=None, scale=1.0):
+    def __init__(self, weights: list[IPAttentionProcessorWeights], scales: list[float]):
         super().__init__()
 
         if not hasattr(F, "scaled_dot_product_attention"):
             raise ImportError("AttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.")
 
-        self.hidden_size = hidden_size
-        self.cross_attention_dim = cross_attention_dim
-        self.scale = scale
+        assert len(weights) == len(scales)
 
-        self.to_k_ip = nn.Linear(cross_attention_dim or hidden_size, hidden_size, bias=False)
-        self.to_v_ip = nn.Linear(cross_attention_dim or hidden_size, hidden_size, bias=False)
+        self._weights = weights
+        self._scales = scales
 
     def __call__(
         self,
@@ -67,16 +67,6 @@ class IPAttnProcessor2_0(torch.nn.Module):
         temb=None,
         ip_adapter_image_prompt_embeds=None,
     ):
-        if encoder_hidden_states is not None:
-            # If encoder_hidden_states is not None, then we are doing cross-attention, not self-attention. In this case,
-            # we will apply IP-Adapter conditioning. We validate the inputs for IP-Adapter conditioning here.
-            assert ip_adapter_image_prompt_embeds is not None
-            # The batch dimensions should match.
-            assert ip_adapter_image_prompt_embeds.shape[0] == encoder_hidden_states.shape[0]
-            # The channel dimensions should match.
-            assert ip_adapter_image_prompt_embeds.shape[2] == encoder_hidden_states.shape[2]
-            ip_hidden_states = ip_adapter_image_prompt_embeds
-
         residual = hidden_states
 
         if attn.spatial_norm is not None:
@@ -128,23 +118,36 @@ class IPAttnProcessor2_0(torch.nn.Module):
         hidden_states = hidden_states.transpose(1, 2).reshape(batch_size, -1, attn.heads * head_dim)
         hidden_states = hidden_states.to(query.dtype)
 
-        if ip_hidden_states is not None:
-            ip_key = self.to_k_ip(ip_hidden_states)
-            ip_value = self.to_v_ip(ip_hidden_states)
+        if encoder_hidden_states is not None:
+            # If encoder_hidden_states is not None, then we are doing cross-attention, not self-attention. In this case,
+            # we will apply IP-Adapter conditioning. We validate the inputs for IP-Adapter conditioning here.
+            assert ip_adapter_image_prompt_embeds is not None
+            assert len(ip_adapter_image_prompt_embeds) == len(self._weights)
 
-            ip_key = ip_key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
-            ip_value = ip_value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
+            for ipa_embed, ipa_weights, scale in zip(ip_adapter_image_prompt_embeds, self._weights, self._scales):
+                # The batch dimensions should match.
+                assert ipa_embed.shape[0] == encoder_hidden_states.shape[0]
+                # The channel dimensions should match.
+                assert ipa_embed.shape[2] == encoder_hidden_states.shape[2]
 
-            # the output of sdp = (batch, num_heads, seq_len, head_dim)
-            # TODO: add support for attn.scale when we move to Torch 2.1
-            ip_hidden_states = F.scaled_dot_product_attention(
-                query, ip_key, ip_value, attn_mask=None, dropout_p=0.0, is_causal=False
-            )
+                ip_hidden_states = ipa_embed
 
-            ip_hidden_states = ip_hidden_states.transpose(1, 2).reshape(batch_size, -1, attn.heads * head_dim)
-            ip_hidden_states = ip_hidden_states.to(query.dtype)
+                ip_key = ipa_weights.to_k_ip(ip_hidden_states)
+                ip_value = ipa_weights.to_v_ip(ip_hidden_states)
 
-            hidden_states = hidden_states + self.scale * ip_hidden_states
+                ip_key = ip_key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
+                ip_value = ip_value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
+
+                # The output of sdpa has shape: (batch, num_heads, seq_len, head_dim)
+                # TODO: add support for attn.scale when we move to Torch 2.1
+                ip_hidden_states = F.scaled_dot_product_attention(
+                    query, ip_key, ip_value, attn_mask=None, dropout_p=0.0, is_causal=False
+                )
+
+                ip_hidden_states = ip_hidden_states.transpose(1, 2).reshape(batch_size, -1, attn.heads * head_dim)
+                ip_hidden_states = ip_hidden_states.to(query.dtype)
+
+                hidden_states = hidden_states + scale * ip_hidden_states
 
         # linear proj
         hidden_states = attn.to_out[0](hidden_states)

invokeai/backend/ip_adapter/ip_adapter.py

@@ -1,15 +1,15 @@
 # copied from https://github.com/tencent-ailab/IP-Adapter (Apache License 2.0)
 #   and modified as needed
 
-from contextlib import contextmanager
 from typing import Optional, Union
 
 import torch
-from diffusers.models import UNet2DConditionModel
 from PIL import Image
 from transformers import CLIPImageProcessor, CLIPVisionModelWithProjection
 
-from .attention_processor import AttnProcessor2_0, IPAttnProcessor2_0
+from invokeai.backend.ip_adapter.ip_attention_weights import IPAttentionWeights
+from invokeai.backend.model_management.models.base import calc_model_size_by_data
+
 from .resampler import Resampler
 
 
@@ -59,7 +59,7 @@ class IPAdapter:
 
     def __init__(
         self,
-        state_dict: dict[torch.Tensor],
+        state_dict: dict[str, torch.Tensor],
         device: torch.device,
         dtype: torch.dtype = torch.float16,
         num_tokens: int = 4,
@@ -71,12 +71,11 @@ class IPAdapter:
 
         self._clip_image_processor = CLIPImageProcessor()
 
-        self._state_dict = state_dict
+        self._image_proj_model = self._init_image_proj_model(state_dict["image_proj"])
 
-        self._image_proj_model = self._init_image_proj_model(self._state_dict["image_proj"])
-
-        # The _attn_processors will be initialized later when we have access to the UNet.
-        self._attn_processors = None
+        self.attn_weights = IPAttentionWeights.from_state_dict(state_dict["ip_adapter"]).to(
+            self.device, dtype=self.dtype
+        )
 
     def to(self, device: torch.device, dtype: Optional[torch.dtype] = None):
         self.device = device
@@ -84,85 +83,14 @@ class IPAdapter:
             self.dtype = dtype
 
         self._image_proj_model.to(device=self.device, dtype=self.dtype)
-        if self._attn_processors is not None:
-            torch.nn.ModuleList(self._attn_processors.values()).to(device=self.device, dtype=self.dtype)
+        self.attn_weights.to(device=self.device, dtype=self.dtype)
+
+    def calc_size(self):
+        return calc_model_size_by_data(self._image_proj_model) + calc_model_size_by_data(self.attn_weights)
 
     def _init_image_proj_model(self, state_dict):
         return ImageProjModel.from_state_dict(state_dict, self._num_tokens).to(self.device, dtype=self.dtype)
 
-    def _prepare_attention_processors(self, unet: UNet2DConditionModel):
-        """Prepare a dict of attention processors that can later be injected into a unet, and load the IP-Adapter
-        attention weights into them.
-
-        Note that the `unet` param is only used to determine attention block dimensions and naming.
-        TODO(ryand): As a future improvement, this could all be inferred from the state_dict when the IPAdapter is
-        intialized.
-        """
-        attn_procs = {}
-        for name in unet.attn_processors.keys():
-            cross_attention_dim = None if name.endswith("attn1.processor") else unet.config.cross_attention_dim
-            if name.startswith("mid_block"):
-                hidden_size = unet.config.block_out_channels[-1]
-            elif name.startswith("up_blocks"):
-                block_id = int(name[len("up_blocks.")])
-                hidden_size = list(reversed(unet.config.block_out_channels))[block_id]
-            elif name.startswith("down_blocks"):
-                block_id = int(name[len("down_blocks.")])
-                hidden_size = unet.config.block_out_channels[block_id]
-            if cross_attention_dim is None:
-                attn_procs[name] = AttnProcessor2_0()
-            else:
-                attn_procs[name] = IPAttnProcessor2_0(
-                    hidden_size=hidden_size,
-                    cross_attention_dim=cross_attention_dim,
-                    scale=1.0,
-                ).to(self.device, dtype=self.dtype)
-
-        ip_layers = torch.nn.ModuleList(attn_procs.values())
-        ip_layers.load_state_dict(self._state_dict["ip_adapter"])
-        self._attn_processors = attn_procs
-        self._state_dict = None
-
-    # @genomancer: pushed scaling back out into its own method (like original Tencent implementation)
-    #      which makes implementing begin_step_percent and end_step_percent easier
-    #  but based on self._attn_processors (ala @Ryan) instead of original Tencent unet.attn_processors,
-    #      which should make it easier to implement multiple IPAdapters
-    def set_scale(self, scale):
-        if self._attn_processors is not None:
-            for attn_processor in self._attn_processors.values():
-                if isinstance(attn_processor, IPAttnProcessor2_0):
-                    attn_processor.scale = scale
-
-    @contextmanager
-    def apply_ip_adapter_attention(self, unet: UNet2DConditionModel, scale: float):
-        """A context manager that patches `unet` with this IP-Adapter's attention processors while it is active.
-
-        Yields:
-            None
-        """
-        if self._attn_processors is None:
-            # We only have to call _prepare_attention_processors(...) once, and then the result is cached and can be
-            # used on any UNet model (with the same dimensions).
-            self._prepare_attention_processors(unet)
-
-        # Set scale
-        self.set_scale(scale)
-        # for attn_processor in self._attn_processors.values():
-        #     if isinstance(attn_processor, IPAttnProcessor2_0):
-        #         attn_processor.scale = scale
-
-        orig_attn_processors = unet.attn_processors
-
-        # Make a (moderately-) shallow copy of the self._attn_processors dict, because unet.set_attn_processor(...)
-        # actually pops elements from the passed dict.
-        ip_adapter_attn_processors = {k: v for k, v in self._attn_processors.items()}
-
-        try:
-            unet.set_attn_processor(ip_adapter_attn_processors)
-            yield None
-        finally:
-            unet.set_attn_processor(orig_attn_processors)
-
     @torch.inference_mode()
     def get_image_embeds(self, pil_image, image_encoder: CLIPVisionModelWithProjection):
         if isinstance(pil_image, Image.Image):
@@ -202,6 +130,20 @@ class IPAdapterPlus(IPAdapter):
         return image_prompt_embeds, uncond_image_prompt_embeds
 
 
+class IPAdapterPlusXL(IPAdapterPlus):
+    """IP-Adapter Plus for SDXL."""
+
+    def _init_image_proj_model(self, state_dict):
+        return Resampler.from_state_dict(
+            state_dict=state_dict,
+            depth=4,
+            dim_head=64,
+            heads=20,
+            num_queries=self._num_tokens,
+            ff_mult=4,
+        ).to(self.device, dtype=self.dtype)
+
+
 def build_ip_adapter(
     ip_adapter_ckpt_path: str, device: torch.device, dtype: torch.dtype = torch.float16
 ) -> Union[IPAdapter, IPAdapterPlus]:
@@ -212,6 +154,14 @@ def build_ip_adapter(
     is_plus = "proj.weight" not in state_dict["image_proj"]
 
     if is_plus:
-        return IPAdapterPlus(state_dict, device=device, dtype=dtype)
+        cross_attention_dim = state_dict["ip_adapter"]["1.to_k_ip.weight"].shape[-1]
+        if cross_attention_dim == 768:
+            # SD1 IP-Adapter Plus
+            return IPAdapterPlus(state_dict, device=device, dtype=dtype)
+        elif cross_attention_dim == 2048:
+            # SDXL IP-Adapter Plus
+            return IPAdapterPlusXL(state_dict, device=device, dtype=dtype)
+        else:
+            raise Exception(f"Unsupported IP-Adapter Plus cross-attention dimension: {cross_attention_dim}.")
     else:
         return IPAdapter(state_dict, device=device, dtype=dtype)

invokeai/backend/ip_adapter/ip_attention_weights.py

@@ -0,0 +1,46 @@
+import torch
+
+
+class IPAttentionProcessorWeights(torch.nn.Module):
+    """The IP-Adapter weights for a single attention processor.
+
+    This class is a torch.nn.Module sub-class to facilitate loading from a state_dict. It does not have a forward(...)
+    method.
+    """
+
+    def __init__(self, in_dim: int, out_dim: int):
+        super().__init__()
+        self.to_k_ip = torch.nn.Linear(in_dim, out_dim, bias=False)
+        self.to_v_ip = torch.nn.Linear(in_dim, out_dim, bias=False)
+
+
+class IPAttentionWeights(torch.nn.Module):
+    """A collection of all the `IPAttentionProcessorWeights` objects for an IP-Adapter model.
+
+    This class is a torch.nn.Module sub-class so that it inherits the `.to(...)` functionality. It does not have a
+    forward(...) method.
+    """
+
+    def __init__(self, weights: torch.nn.ModuleDict):
+        super().__init__()
+        self._weights = weights
+
+    def get_attention_processor_weights(self, idx: int) -> IPAttentionProcessorWeights:
+        """Get the `IPAttentionProcessorWeights` for the idx'th attention processor."""
+        # Cast to int first, because we expect the key to represent an int. Then cast back to str, because
+        # `torch.nn.ModuleDict` only supports str keys.
+        return self._weights[str(int(idx))]
+
+    @classmethod
+    def from_state_dict(cls, state_dict: dict[str, torch.Tensor]):
+        attn_proc_weights: dict[str, IPAttentionProcessorWeights] = {}
+
+        for tensor_name, tensor in state_dict.items():
+            if "to_k_ip.weight" in tensor_name:
+                index = str(int(tensor_name.split(".")[0]))
+                attn_proc_weights[index] = IPAttentionProcessorWeights(tensor.shape[1], tensor.shape[0])
+
+        attn_proc_weights_module = torch.nn.ModuleDict(attn_proc_weights)
+        attn_proc_weights_module.load_state_dict(state_dict)
+
+        return cls(attn_proc_weights_module)

invokeai/backend/ip_adapter/unet_patcher.py

@@ -0,0 +1,53 @@
+from contextlib import contextmanager
+
+from diffusers.models import UNet2DConditionModel
+
+from invokeai.backend.ip_adapter.attention_processor import AttnProcessor2_0, IPAttnProcessor2_0
+from invokeai.backend.ip_adapter.ip_adapter import IPAdapter
+
+
+class UNetPatcher:
+    """A class that contains multiple IP-Adapters and can apply them to a UNet."""
+
+    def __init__(self, ip_adapters: list[IPAdapter]):
+        self._ip_adapters = ip_adapters
+        self._scales = [1.0] * len(self._ip_adapters)
+
+    def set_scale(self, idx: int, value: float):
+        self._scales[idx] = value
+
+    def _prepare_attention_processors(self, unet: UNet2DConditionModel):
+        """Prepare a dict of attention processors that can be injected into a unet, and load the IP-Adapter attention
+        weights into them.
+
+        Note that the `unet` param is only used to determine attention block dimensions and naming.
+        """
+        # Construct a dict of attention processors based on the UNet's architecture.
+        attn_procs = {}
+        for idx, name in enumerate(unet.attn_processors.keys()):
+            if name.endswith("attn1.processor"):
+                attn_procs[name] = AttnProcessor2_0()
+            else:
+                # Collect the weights from each IP Adapter for the idx'th attention processor.
+                attn_procs[name] = IPAttnProcessor2_0(
+                    [ip_adapter.attn_weights.get_attention_processor_weights(idx) for ip_adapter in self._ip_adapters],
+                    self._scales,
+                )
+        return attn_procs
+
+    @contextmanager
+    def apply_ip_adapter_attention(self, unet: UNet2DConditionModel):
+        """A context manager that patches `unet` with IP-Adapter attention processors."""
+
+        attn_procs = self._prepare_attention_processors(unet)
+
+        orig_attn_processors = unet.attn_processors
+
+        try:
+            # Note to future devs: set_attn_processor(...) does something slightly unexpected - it pops elements from the
+            # passed dict. So, if you wanted to keep the dict for future use, you'd have to make a moderately-shallow copy
+            # of it. E.g. `attn_procs_copy = {k: v for k, v in attn_procs.items()}`.
+            unet.set_attn_processor(attn_procs)
+            yield None
+        finally:
+            unet.set_attn_processor(orig_attn_processors)

invokeai/backend/model_management/README.md

@@ -0,0 +1,27 @@
+# Model Cache
+
+## `glibc` Memory Allocator Fragmentation
+
+Python (and PyTorch) relies on the memory allocator from the C Standard Library (`libc`). On linux, with the GNU C Standard Library implementation (`glibc`), our memory access patterns have been observed to cause severe memory fragmentation. This fragmentation results in large amounts of memory that has been freed but can't be released back to the OS. Loading models from disk and moving them between CPU/CUDA seem to be the operations that contribute most to the fragmentation. This memory fragmentation issue can result in OOM crashes during frequent model switching, even if `max_cache_size` is set to a reasonable value (e.g. a OOM crash with `max_cache_size=16` on a system with 32GB of RAM).
+
+This problem may also exist on other OSes, and other `libc` implementations. But, at the time of writing, it has only been investigated on linux with `glibc`.
+
+To better understand how the `glibc` memory allocator works, see these references:
+- Basics: https://www.gnu.org/software/libc/manual/html_node/The-GNU-Allocator.html
+- Details: https://sourceware.org/glibc/wiki/MallocInternals
+
+Note the differences between memory allocated as chunks in an arena vs. memory allocated with `mmap`. Under `glibc`'s default configuration, most model tensors get allocated as chunks in an arena making them vulnerable to the problem of fragmentation.
+
+We can work around this memory fragmentation issue by setting the following env var:
+
+```bash
+# Force blocks >1MB to be allocated with `mmap` so that they are released to the system immediately when they are freed.
+MALLOC_MMAP_THRESHOLD_=1048576
+```
+
+See the following references for more information about the `malloc` tunable parameters:
+- https://www.gnu.org/software/libc/manual/html_node/Malloc-Tunable-Parameters.html
+- https://www.gnu.org/software/libc/manual/html_node/Memory-Allocation-Tunables.html
+- https://man7.org/linux/man-pages/man3/mallopt.3.html
+
+The model cache emits debug logs that provide visibility into the state of the `libc` memory allocator. See the `LibcUtil` class for more info on how these `libc` malloc stats are collected.

invokeai/backend/model_management/libc_util.py

@@ -0,0 +1,75 @@
+import ctypes
+
+
+class Struct_mallinfo2(ctypes.Structure):
+    """A ctypes Structure that matches the libc mallinfo2 struct.
+
+    Docs:
+    - https://man7.org/linux/man-pages/man3/mallinfo.3.html
+    - https://www.gnu.org/software/libc/manual/html_node/Statistics-of-Malloc.html
+
+    struct mallinfo2 {
+        size_t arena;     /* Non-mmapped space allocated (bytes) */
+        size_t ordblks;   /* Number of free chunks */
+        size_t smblks;    /* Number of free fastbin blocks */
+        size_t hblks;     /* Number of mmapped regions */
+        size_t hblkhd;    /* Space allocated in mmapped regions (bytes) */
+        size_t usmblks;   /* See below */
+        size_t fsmblks;   /* Space in freed fastbin blocks (bytes) */
+        size_t uordblks;  /* Total allocated space (bytes) */
+        size_t fordblks;  /* Total free space (bytes) */
+        size_t keepcost;  /* Top-most, releasable space (bytes) */
+    };
+    """
+
+    _fields_ = [
+        ("arena", ctypes.c_size_t),
+        ("ordblks", ctypes.c_size_t),
+        ("smblks", ctypes.c_size_t),
+        ("hblks", ctypes.c_size_t),
+        ("hblkhd", ctypes.c_size_t),
+        ("usmblks", ctypes.c_size_t),
+        ("fsmblks", ctypes.c_size_t),
+        ("uordblks", ctypes.c_size_t),
+        ("fordblks", ctypes.c_size_t),
+        ("keepcost", ctypes.c_size_t),
+    ]
+
+    def __str__(self):
+        s = ""
+        s += f"{'arena': <10}= {(self.arena/2**30):15.5f}   # Non-mmapped space allocated (GB) (uordblks + fordblks)\n"
+        s += f"{'ordblks': <10}= {(self.ordblks): >15}   # Number of free chunks\n"
+        s += f"{'smblks': <10}= {(self.smblks): >15}   # Number of free fastbin blocks \n"
+        s += f"{'hblks': <10}= {(self.hblks): >15}   # Number of mmapped regions \n"
+        s += f"{'hblkhd': <10}= {(self.hblkhd/2**30):15.5f}   # Space allocated in mmapped regions (GB)\n"
+        s += f"{'usmblks': <10}= {(self.usmblks): >15}   # Unused\n"
+        s += f"{'fsmblks': <10}= {(self.fsmblks/2**30):15.5f}   # Space in freed fastbin blocks (GB)\n"
+        s += (
+            f"{'uordblks': <10}= {(self.uordblks/2**30):15.5f}   # Space used by in-use allocations (non-mmapped)"
+            " (GB)\n"
+        )
+        s += f"{'fordblks': <10}= {(self.fordblks/2**30):15.5f}   # Space in free blocks (non-mmapped) (GB)\n"
+        s += f"{'keepcost': <10}= {(self.keepcost/2**30):15.5f}   # Top-most, releasable space (GB)\n"
+        return s
+
+
+class LibcUtil:
+    """A utility class for interacting with the C Standard Library (`libc`) via ctypes.
+
+    Note that this class will raise on __init__() if 'libc.so.6' can't be found. Take care to handle environments where
+    this shared library is not available.
+
+    TODO: Improve cross-OS compatibility of this class.
+    """
+
+    def __init__(self):
+        self._libc = ctypes.cdll.LoadLibrary("libc.so.6")
+
+    def mallinfo2(self) -> Struct_mallinfo2:
+        """Calls `libc` `mallinfo2`.
+
+        Docs: https://man7.org/linux/man-pages/man3/mallinfo.3.html
+        """
+        mallinfo2 = self._libc.mallinfo2
+        mallinfo2.restype = Struct_mallinfo2
+        return mallinfo2()

invokeai/backend/model_management/memory_snapshot.py

@@ -0,0 +1,94 @@
+import gc
+from typing import Optional
+
+import psutil
+import torch
+
+from invokeai.backend.model_management.libc_util import LibcUtil, Struct_mallinfo2
+
+GB = 2**30  # 1 GB
+
+
+class MemorySnapshot:
+    """A snapshot of RAM and VRAM usage. All values are in bytes."""
+
+    def __init__(self, process_ram: int, vram: Optional[int], malloc_info: Optional[Struct_mallinfo2]):
+        """Initialize a MemorySnapshot.
+
+        Most of the time, `MemorySnapshot` will be constructed with `MemorySnapshot.capture()`.
+
+        Args:
+            process_ram (int): CPU RAM used by the current process.
+            vram (Optional[int]): VRAM used by torch.
+            malloc_info (Optional[Struct_mallinfo2]): Malloc info obtained from LibcUtil.
+        """
+        self.process_ram = process_ram
+        self.vram = vram
+        self.malloc_info = malloc_info
+
+    @classmethod
+    def capture(cls, run_garbage_collector: bool = True):
+        """Capture and return a MemorySnapshot.
+
+        Note: This function has significant overhead, particularly if `run_garbage_collector == True`.
+
+        Args:
+            run_garbage_collector (bool, optional): If true, gc.collect() will be run before checking the process RAM
+                usage. Defaults to True.
+
+        Returns:
+            MemorySnapshot
+        """
+        if run_garbage_collector:
+            gc.collect()
+
+        # According to the psutil docs (https://psutil.readthedocs.io/en/latest/#psutil.Process.memory_info), rss is
+        # supported on all platforms.
+        process_ram = psutil.Process().memory_info().rss
+
+        if torch.cuda.is_available():
+            vram = torch.cuda.memory_allocated()
+        else:
+            # TODO: We could add support for mps.current_allocated_memory() as well. Leaving out for now until we have
+            # time to test it properly.
+            vram = None
+
+        try:
+            malloc_info = LibcUtil().mallinfo2()
+        except OSError:
+            # This is expected in environments that do not have the 'libc.so.6' shared library.
+            malloc_info = None
+
+        return cls(process_ram, vram, malloc_info)
+
+
+def get_pretty_snapshot_diff(snapshot_1: MemorySnapshot, snapshot_2: MemorySnapshot) -> str:
+    """Get a pretty string describing the difference between two `MemorySnapshot`s."""
+
+    def get_msg_line(prefix: str, val1: int, val2: int):
+        diff = val2 - val1
+        return f"{prefix: <30} ({(diff/GB):+5.3f}): {(val1/GB):5.3f}GB -> {(val2/GB):5.3f}GB\n"
+
+    msg = ""
+
+    msg += get_msg_line("Process RAM", snapshot_1.process_ram, snapshot_2.process_ram)
+
+    if snapshot_1.malloc_info is not None and snapshot_2.malloc_info is not None:
+        msg += get_msg_line("libc mmap allocated", snapshot_1.malloc_info.hblkhd, snapshot_2.malloc_info.hblkhd)
+
+        msg += get_msg_line("libc arena used", snapshot_1.malloc_info.uordblks, snapshot_2.malloc_info.uordblks)
+
+        msg += get_msg_line("libc arena free", snapshot_1.malloc_info.fordblks, snapshot_2.malloc_info.fordblks)
+
+        libc_total_allocated_1 = snapshot_1.malloc_info.arena + snapshot_1.malloc_info.hblkhd
+        libc_total_allocated_2 = snapshot_2.malloc_info.arena + snapshot_2.malloc_info.hblkhd
+        msg += get_msg_line("libc total allocated", libc_total_allocated_1, libc_total_allocated_2)
+
+        libc_total_used_1 = snapshot_1.malloc_info.uordblks + snapshot_1.malloc_info.hblkhd
+        libc_total_used_2 = snapshot_2.malloc_info.uordblks + snapshot_2.malloc_info.hblkhd
+        msg += get_msg_line("libc total used", libc_total_used_1, libc_total_used_2)
+
+    if snapshot_1.vram is not None and snapshot_2.vram is not None:
+        msg += get_msg_line("VRAM", snapshot_1.vram, snapshot_2.vram)
+
+    return msg

invokeai/backend/model_management/model_cache.py

@@ -18,8 +18,10 @@ context. Use like this:
 
 import gc
 import hashlib
+import math
 import os
 import sys
+import time
 from contextlib import suppress
 from dataclasses import dataclass, field
 from pathlib import Path
@@ -28,6 +30,8 @@ from typing import Any, Dict, Optional, Type, Union, types
 import torch
 
 import invokeai.backend.util.logging as logger
+from invokeai.backend.model_management.memory_snapshot import MemorySnapshot, get_pretty_snapshot_diff
+from invokeai.backend.model_management.model_load_optimizations import skip_torch_weight_init
 
 from ..util.devices import choose_torch_device
 from .models import BaseModelType, ModelBase, ModelType, SubModelType
@@ -44,6 +48,8 @@ DEFAULT_MAX_VRAM_CACHE_SIZE = 2.75
 
 # actual size of a gig
 GIG = 1073741824
+# Size of a MB in bytes.
+MB = 2**20
 
 
 @dataclass
@@ -205,22 +211,44 @@ class ModelCache(object):
         cache_entry = self._cached_models.get(key, None)
         if cache_entry is None:
             self.logger.info(
-                f"Loading model {model_path}, type {base_model.value}:{model_type.value}{':'+submodel.value if submodel else ''}"
+                f"Loading model {model_path}, type"
+                f" {base_model.value}:{model_type.value}{':'+submodel.value if submodel else ''}"
             )
             if self.stats:
                 self.stats.misses += 1
 
-            # this will remove older cached models until
-            # there is sufficient room to load the requested model
-            self._make_cache_room(model_info.get_size(submodel))
+            self_reported_model_size_before_load = model_info.get_size(submodel)
+            # Remove old models from the cache to make room for the new model.
+            self._make_cache_room(self_reported_model_size_before_load)
 
-            # clean memory to make MemoryUsage() more accurate
-            gc.collect()
-            model = model_info.get_model(child_type=submodel, torch_dtype=self.precision)
-            if mem_used := model_info.get_size(submodel):
-                self.logger.debug(f"CPU RAM used for load: {(mem_used/GIG):.2f} GB")
+            # Load the model from disk and capture a memory snapshot before/after.
+            start_load_time = time.time()
+            snapshot_before = MemorySnapshot.capture()
+            with skip_torch_weight_init():
+                model = model_info.get_model(child_type=submodel, torch_dtype=self.precision)
+            snapshot_after = MemorySnapshot.capture()
+            end_load_time = time.time()
 
-            cache_entry = _CacheRecord(self, model, mem_used)
+            self_reported_model_size_after_load = model_info.get_size(submodel)
+
+            self.logger.debug(
+                f"Moved model '{key}' from disk to cpu in {(end_load_time-start_load_time):.2f}s.\n"
+                f"Self-reported size before/after load: {(self_reported_model_size_before_load/GIG):.3f}GB /"
+                f" {(self_reported_model_size_after_load/GIG):.3f}GB.\n"
+                f"{get_pretty_snapshot_diff(snapshot_before, snapshot_after)}"
+            )
+
+            # We only log a warning for over-reported (not under-reported) model sizes before load. There is a known
+            # issue where models report their fp32 size before load, and are then loaded as fp16. Once this issue is
+            # addressed, it would make sense to log a warning for both over-reported and under-reported model sizes.
+            if (self_reported_model_size_after_load - self_reported_model_size_before_load) > 10 * MB:
+                self.logger.warning(
+                    f"Model '{key}' mis-reported its size before load. Self-reported size before/after load:"
+                    f" {(self_reported_model_size_before_load/GIG):.2f}GB /"
+                    f" {(self_reported_model_size_after_load/GIG):.2f}GB."
+                )
+
+            cache_entry = _CacheRecord(self, model, self_reported_model_size_after_load)
             self._cached_models[key] = cache_entry
         else:
             if self.stats:
@@ -240,6 +268,45 @@ class ModelCache(object):
 
         return self.ModelLocker(self, key, cache_entry.model, gpu_load, cache_entry.size)
 
+    def _move_model_to_device(self, key: str, target_device: torch.device):
+        cache_entry = self._cached_models[key]
+
+        source_device = cache_entry.model.device
+        # Note: We compare device types only so that 'cuda' == 'cuda:0'. This would need to be revised to support
+        # multi-GPU.
+        if torch.device(source_device).type == torch.device(target_device).type:
+            return
+
+        start_model_to_time = time.time()
+        snapshot_before = MemorySnapshot.capture()
+        cache_entry.model.to(target_device)
+        snapshot_after = MemorySnapshot.capture()
+        end_model_to_time = time.time()
+        self.logger.debug(
+            f"Moved model '{key}' from {source_device} to"
+            f" {target_device} in {(end_model_to_time-start_model_to_time):.2f}s.\n"
+            f"Estimated model size: {(cache_entry.size/GIG):.3f} GB.\n"
+            f"{get_pretty_snapshot_diff(snapshot_before, snapshot_after)}"
+        )
+
+        if snapshot_before.vram is not None and snapshot_after.vram is not None:
+            vram_change = abs(snapshot_before.vram - snapshot_after.vram)
+
+            # If the estimated model size does not match the change in VRAM, log a warning.
+            if not math.isclose(
+                vram_change,
+                cache_entry.size,
+                rel_tol=0.1,
+                abs_tol=10 * MB,
+            ):
+                self.logger.warning(
+                    f"Moving model '{key}' from {source_device} to"
+                    f" {target_device} caused an unexpected change in VRAM usage. The model's"
+                    " estimated size may be incorrect. Estimated model size:"
+                    f" {(cache_entry.size/GIG):.3f} GB.\n"
+                    f"{get_pretty_snapshot_diff(snapshot_before, snapshot_after)}"
+                )
+
     class ModelLocker(object):
         def __init__(self, cache, key, model, gpu_load, size_needed):
             """
@@ -269,11 +336,7 @@ class ModelCache(object):
                     if self.cache.lazy_offloading:
                         self.cache._offload_unlocked_models(self.size_needed)
 
-                    if self.model.device != self.cache.execution_device:
-                        self.cache.logger.debug(f"Moving {self.key} into {self.cache.execution_device}")
-                        with VRAMUsage() as mem:
-                            self.model.to(self.cache.execution_device)  # move into GPU
-                        self.cache.logger.debug(f"GPU VRAM used for load: {(mem.vram_used/GIG):.2f} GB")
+                    self.cache._move_model_to_device(self.key, self.cache.execution_device)
 
                     self.cache.logger.debug(f"Locking {self.key} in {self.cache.execution_device}")
                     self.cache._print_cuda_stats()
@@ -286,7 +349,7 @@ class ModelCache(object):
             # in the event that the caller wants the model in RAM, we
             # move it into CPU if it is in GPU and not locked
             elif self.cache_entry.loaded and not self.cache_entry.locked:
-                self.model.to(self.cache.storage_device)
+                self.cache._move_model_to_device(self.key, self.cache.storage_device)
 
             return self.model
 
@@ -339,7 +402,8 @@ class ModelCache(object):
                 locked_models += 1
 
         self.logger.debug(
-            f"Current VRAM/RAM usage: {vram}/{ram}; cached_models/loaded_models/locked_models/ = {cached_models}/{loaded_models}/{locked_models}"
+            f"Current VRAM/RAM usage: {vram}/{ram}; cached_models/loaded_models/locked_models/ ="
+            f" {cached_models}/{loaded_models}/{locked_models}"
         )
 
     def _cache_size(self) -> int:
@@ -354,7 +418,8 @@ class ModelCache(object):
 
         if current_size + bytes_needed > maximum_size:
             self.logger.debug(
-                f"Max cache size exceeded: {(current_size/GIG):.2f}/{self.max_cache_size:.2f} GB, need an additional {(bytes_needed/GIG):.2f} GB"
+                f"Max cache size exceeded: {(current_size/GIG):.2f}/{self.max_cache_size:.2f} GB, need an additional"
+                f" {(bytes_needed/GIG):.2f} GB"
             )
 
         self.logger.debug(f"Before unloading: cached_models={len(self._cached_models)}")
@@ -387,7 +452,8 @@ class ModelCache(object):
 
             device = cache_entry.model.device if hasattr(cache_entry.model, "device") else None
             self.logger.debug(
-                f"Model: {model_key}, locks: {cache_entry._locks}, device: {device}, loaded: {cache_entry.loaded}, refs: {refs}"
+                f"Model: {model_key}, locks: {cache_entry._locks}, device: {device}, loaded: {cache_entry.loaded},"
+                f" refs: {refs}"
             )
 
             # 2 refs:
@@ -423,11 +489,9 @@ class ModelCache(object):
             if vram_in_use <= reserved:
                 break
             if not cache_entry.locked and cache_entry.loaded:
-                self.logger.debug(f"Offloading {model_key} from {self.execution_device} into {self.storage_device}")
-                with VRAMUsage() as mem:
-                    cache_entry.model.to(self.storage_device)
-                self.logger.debug(f"GPU VRAM freed: {(mem.vram_used/GIG):.2f} GB")
-                vram_in_use += mem.vram_used  # note vram_used is negative
+                self._move_model_to_device(model_key, self.storage_device)
+
+                vram_in_use = torch.cuda.memory_allocated()
                 self.logger.debug(f"{(vram_in_use/GIG):.2f}GB VRAM used for models; max allowed={(reserved/GIG):.2f}GB")
 
         gc.collect()
@@ -454,16 +518,3 @@ class ModelCache(object):
         with open(hashpath, "w") as f:
             f.write(hash)
         return hash
-
-
-class VRAMUsage(object):
-    def __init__(self):
-        self.vram = None
-        self.vram_used = 0
-
-    def __enter__(self):
-        self.vram = torch.cuda.memory_allocated()
-        return self
-
-    def __exit__(self, *args):
-        self.vram_used = torch.cuda.memory_allocated() - self.vram

invokeai/backend/model_management/model_load_optimizations.py

@@ -0,0 +1,30 @@
+from contextlib import contextmanager
+
+import torch
+
+
+def _no_op(*args, **kwargs):
+    pass
+
+
+@contextmanager
+def skip_torch_weight_init():
+    """A context manager that monkey-patches several of the common torch layers (torch.nn.Linear, torch.nn.Conv1d, etc.)
+    to skip weight initialization.
+
+    By default, `torch.nn.Linear` and `torch.nn.ConvNd` layers initialize their weights (according to a particular
+    distribution) when __init__ is called. This weight initialization step can take a significant amount of time, and is
+    completely unnecessary if the intent is to load checkpoint weights from disk for the layer. This context manager
+    monkey-patches common torch layers to skip the weight initialization step.
+    """
+    torch_modules = [torch.nn.Linear, torch.nn.modules.conv._ConvNd]
+    saved_functions = [m.reset_parameters for m in torch_modules]
+
+    try:
+        for torch_module in torch_modules:
+            torch_module.reset_parameters = _no_op
+
+        yield None
+    finally:
+        for torch_module, saved_function in zip(torch_modules, saved_functions):
+            torch_module.reset_parameters = saved_function

invokeai/backend/model_management/model_probe.py

@@ -57,6 +57,7 @@ class ModelProbe(object):
         "AutoencoderTiny": ModelType.Vae,
         "ControlNetModel": ModelType.ControlNet,
         "CLIPVisionModelWithProjection": ModelType.CLIPVision,
+        "T2IAdapter": ModelType.T2IAdapter,
     }
 
     @classmethod
@@ -408,6 +409,11 @@ class CLIPVisionCheckpointProbe(CheckpointProbeBase):
         raise NotImplementedError()
 
 
+class T2IAdapterCheckpointProbe(CheckpointProbeBase):
+    def get_base_type(self) -> BaseModelType:
+        raise NotImplementedError()
+
+
 ########################################################
 # classes for probing folders
 #######################################################
@@ -595,6 +601,26 @@ class CLIPVisionFolderProbe(FolderProbeBase):
         return BaseModelType.Any
 
 
+class T2IAdapterFolderProbe(FolderProbeBase):
+    def get_base_type(self) -> BaseModelType:
+        config_file = self.folder_path / "config.json"
+        if not config_file.exists():
+            raise InvalidModelException(f"Cannot determine base type for {self.folder_path}")
+        with open(config_file, "r") as file:
+            config = json.load(file)
+
+        adapter_type = config.get("adapter_type", None)
+        if adapter_type == "full_adapter_xl":
+            return BaseModelType.StableDiffusionXL
+        elif adapter_type == "full_adapter" or "light_adapter":
+            # I haven't seen any T2I adapter models for SD2, so assume that this is an SD1 adapter.
+            return BaseModelType.StableDiffusion1
+        else:
+            raise InvalidModelException(
+                f"Unable to determine base model for '{self.folder_path}' (adapter_type = {adapter_type})."
+            )
+
+
 ############## register probe classes ######
 ModelProbe.register_probe("diffusers", ModelType.Main, PipelineFolderProbe)
 ModelProbe.register_probe("diffusers", ModelType.Vae, VaeFolderProbe)
@@ -603,6 +629,7 @@ ModelProbe.register_probe("diffusers", ModelType.TextualInversion, TextualInvers
 ModelProbe.register_probe("diffusers", ModelType.ControlNet, ControlNetFolderProbe)
 ModelProbe.register_probe("diffusers", ModelType.IPAdapter, IPAdapterFolderProbe)
 ModelProbe.register_probe("diffusers", ModelType.CLIPVision, CLIPVisionFolderProbe)
+ModelProbe.register_probe("diffusers", ModelType.T2IAdapter, T2IAdapterFolderProbe)
 
 ModelProbe.register_probe("checkpoint", ModelType.Main, PipelineCheckpointProbe)
 ModelProbe.register_probe("checkpoint", ModelType.Vae, VaeCheckpointProbe)
@@ -611,5 +638,6 @@ ModelProbe.register_probe("checkpoint", ModelType.TextualInversion, TextualInver
 ModelProbe.register_probe("checkpoint", ModelType.ControlNet, ControlNetCheckpointProbe)
 ModelProbe.register_probe("checkpoint", ModelType.IPAdapter, IPAdapterCheckpointProbe)
 ModelProbe.register_probe("checkpoint", ModelType.CLIPVision, CLIPVisionCheckpointProbe)
+ModelProbe.register_probe("checkpoint", ModelType.T2IAdapter, T2IAdapterCheckpointProbe)
 
 ModelProbe.register_probe("onnx", ModelType.ONNX, ONNXFolderProbe)

invokeai/backend/model_management/models/__init__.py

@@ -25,6 +25,7 @@ from .lora import LoRAModel
 from .sdxl import StableDiffusionXLModel
 from .stable_diffusion import StableDiffusion1Model, StableDiffusion2Model
 from .stable_diffusion_onnx import ONNXStableDiffusion1Model, ONNXStableDiffusion2Model
+from .t2i_adapter import T2IAdapterModel
 from .textual_inversion import TextualInversionModel
 from .vae import VaeModel
 
@@ -38,6 +39,7 @@ MODEL_CLASSES = {
         ModelType.TextualInversion: TextualInversionModel,
         ModelType.IPAdapter: IPAdapterModel,
         ModelType.CLIPVision: CLIPVisionModel,
+        ModelType.T2IAdapter: T2IAdapterModel,
     },
     BaseModelType.StableDiffusion2: {
         ModelType.ONNX: ONNXStableDiffusion2Model,
@@ -48,6 +50,7 @@ MODEL_CLASSES = {
         ModelType.TextualInversion: TextualInversionModel,
         ModelType.IPAdapter: IPAdapterModel,
         ModelType.CLIPVision: CLIPVisionModel,
+        ModelType.T2IAdapter: T2IAdapterModel,
     },
     BaseModelType.StableDiffusionXL: {
         ModelType.Main: StableDiffusionXLModel,
@@ -59,6 +62,7 @@ MODEL_CLASSES = {
         ModelType.ONNX: ONNXStableDiffusion2Model,
         ModelType.IPAdapter: IPAdapterModel,
         ModelType.CLIPVision: CLIPVisionModel,
+        ModelType.T2IAdapter: T2IAdapterModel,
     },
     BaseModelType.StableDiffusionXLRefiner: {
         ModelType.Main: StableDiffusionXLModel,
@@ -70,6 +74,7 @@ MODEL_CLASSES = {
         ModelType.ONNX: ONNXStableDiffusion2Model,
         ModelType.IPAdapter: IPAdapterModel,
         ModelType.CLIPVision: CLIPVisionModel,
+        ModelType.T2IAdapter: T2IAdapterModel,
     },
     BaseModelType.Any: {
         ModelType.CLIPVision: CLIPVisionModel,
@@ -81,6 +86,7 @@ MODEL_CLASSES = {
         ModelType.ControlNet: ControlNetModel,
         ModelType.TextualInversion: TextualInversionModel,
         ModelType.IPAdapter: IPAdapterModel,
+        ModelType.T2IAdapter: T2IAdapterModel,
     },
     # BaseModelType.Kandinsky2_1: {
     #    ModelType.Main: Kandinsky2_1Model,

invokeai/backend/model_management/models/base.py

@@ -53,6 +53,7 @@ class ModelType(str, Enum):
     TextualInversion = "embedding"
     IPAdapter = "ip_adapter"
     CLIPVision = "clip_vision"
+    T2IAdapter = "t2i_adapter"
 
 
 class SubModelType(str, Enum):

invokeai/backend/model_management/models/ip_adapter.py

@@ -13,6 +13,7 @@ from invokeai.backend.model_management.models.base import (
     ModelConfigBase,
     ModelType,
     SubModelType,
+    calc_model_size_by_fs,
     classproperty,
 )
 
@@ -30,7 +31,7 @@ class IPAdapterModel(ModelBase):
         assert model_type == ModelType.IPAdapter
         super().__init__(model_path, base_model, model_type)
 
-        self.model_size = os.path.getsize(self.model_path)
+        self.model_size = calc_model_size_by_fs(self.model_path)
 
     @classmethod
     def detect_format(cls, path: str) -> str:
@@ -63,10 +64,13 @@ class IPAdapterModel(ModelBase):
         if child_type is not None:
             raise ValueError("There are no child models in an IP-Adapter model.")
 
-        return build_ip_adapter(
+        model = build_ip_adapter(
             ip_adapter_ckpt_path=os.path.join(self.model_path, "ip_adapter.bin"), device="cpu", dtype=torch_dtype
         )
 
+        self.model_size = model.calc_size()
+        return model
+
     @classmethod
     def convert_if_required(
         cls,

invokeai/backend/model_management/models/t2i_adapter.py

@@ -0,0 +1,102 @@
+import os
+from enum import Enum
+from typing import Literal, Optional
+
+import torch
+from diffusers import T2IAdapter
+
+from invokeai.backend.model_management.models.base import (
+    BaseModelType,
+    EmptyConfigLoader,
+    InvalidModelException,
+    ModelBase,
+    ModelConfigBase,
+    ModelNotFoundException,
+    ModelType,
+    SubModelType,
+    calc_model_size_by_data,
+    calc_model_size_by_fs,
+    classproperty,
+)
+
+
+class T2IAdapterModelFormat(str, Enum):
+    Diffusers = "diffusers"
+
+
+class T2IAdapterModel(ModelBase):
+    class DiffusersConfig(ModelConfigBase):
+        model_format: Literal[T2IAdapterModelFormat.Diffusers]
+
+    def __init__(self, model_path: str, base_model: BaseModelType, model_type: ModelType):
+        assert model_type == ModelType.T2IAdapter
+        super().__init__(model_path, base_model, model_type)
+
+        config = EmptyConfigLoader.load_config(self.model_path, config_name="config.json")
+
+        model_class_name = config.get("_class_name", None)
+        if model_class_name not in {"T2IAdapter"}:
+            raise InvalidModelException(f"Invalid T2I-Adapter model. Unknown _class_name: '{model_class_name}'.")
+
+        self.model_class = self._hf_definition_to_type(["diffusers", model_class_name])
+        self.model_size = calc_model_size_by_fs(self.model_path)
+
+    def get_size(self, child_type: Optional[SubModelType] = None):
+        if child_type is not None:
+            raise ValueError(f"T2I-Adapters do not have child models. Invalid child type: '{child_type}'.")
+        return self.model_size
+
+    def get_model(
+        self,
+        torch_dtype: Optional[torch.dtype],
+        child_type: Optional[SubModelType] = None,
+    ) -> T2IAdapter:
+        if child_type is not None:
+            raise ValueError(f"T2I-Adapters do not have child models. Invalid child type: '{child_type}'.")
+
+        model = None
+        for variant in ["fp16", None]:
+            try:
+                model = self.model_class.from_pretrained(
+                    self.model_path,
+                    torch_dtype=torch_dtype,
+                    variant=variant,
+                )
+                break
+            except Exception:
+                pass
+        if not model:
+            raise ModelNotFoundException()
+
+        # Calculate a more accurate size after loading the model into memory.
+        self.model_size = calc_model_size_by_data(model)
+        return model
+
+    @classproperty
+    def save_to_config(cls) -> bool:
+        return False
+
+    @classmethod
+    def detect_format(cls, path: str):
+        if not os.path.exists(path):
+            raise ModelNotFoundException(f"Model not found at '{path}'.")
+
+        if os.path.isdir(path):
+            if os.path.exists(os.path.join(path, "config.json")):
+                return T2IAdapterModelFormat.Diffusers
+
+        raise InvalidModelException(f"Unsupported T2I-Adapter format: '{path}'.")
+
+    @classmethod
+    def convert_if_required(
+        cls,
+        model_path: str,
+        output_path: str,
+        config: ModelConfigBase,
+        base_model: BaseModelType,
+    ) -> str:
+        format = cls.detect_format(model_path)
+        if format == T2IAdapterModelFormat.Diffusers:
+            return model_path
+        else:
+            raise ValueError(f"Unsupported format: '{format}'.")

invokeai/backend/stable_diffusion/diffusers_pipeline.py

@@ -24,6 +24,7 @@ from transformers import CLIPFeatureExtractor, CLIPTextModel, CLIPTokenizer
 
 from invokeai.app.services.config import InvokeAIAppConfig
 from invokeai.backend.ip_adapter.ip_adapter import IPAdapter
+from invokeai.backend.ip_adapter.unet_patcher import UNetPatcher
 from invokeai.backend.stable_diffusion.diffusion.conditioning_data import ConditioningData
 
 from ..util import auto_detect_slice_size, normalize_device
@@ -173,6 +174,16 @@ class IPAdapterData:
     end_step_percent: float = Field(default=1.0)
 
 
+@dataclass
+class T2IAdapterData:
+    """A structure containing the information required to apply conditioning from a single T2I-Adapter model."""
+
+    adapter_state: dict[torch.Tensor] = Field()
+    weight: Union[float, list[float]] = Field(default=1.0)
+    begin_step_percent: float = Field(default=0.0)
+    end_step_percent: float = Field(default=1.0)
+
+
 @dataclass
 class InvokeAIStableDiffusionPipelineOutput(StableDiffusionPipelineOutput):
     r"""
@@ -326,7 +337,8 @@ class StableDiffusionGeneratorPipeline(StableDiffusionPipeline):
         additional_guidance: List[Callable] = None,
         callback: Callable[[PipelineIntermediateState], None] = None,
         control_data: List[ControlNetData] = None,
-        ip_adapter_data: Optional[IPAdapterData] = None,
+        ip_adapter_data: Optional[list[IPAdapterData]] = None,
+        t2i_adapter_data: Optional[list[T2IAdapterData]] = None,
         mask: Optional[torch.Tensor] = None,
         masked_latents: Optional[torch.Tensor] = None,
         seed: Optional[int] = None,
@@ -379,6 +391,7 @@ class StableDiffusionGeneratorPipeline(StableDiffusionPipeline):
                 additional_guidance=additional_guidance,
                 control_data=control_data,
                 ip_adapter_data=ip_adapter_data,
+                t2i_adapter_data=t2i_adapter_data,
                 callback=callback,
             )
         finally:
@@ -398,7 +411,8 @@ class StableDiffusionGeneratorPipeline(StableDiffusionPipeline):
         *,
         additional_guidance: List[Callable] = None,
         control_data: List[ControlNetData] = None,
-        ip_adapter_data: Optional[IPAdapterData] = None,
+        ip_adapter_data: Optional[list[IPAdapterData]] = None,
+        t2i_adapter_data: Optional[list[T2IAdapterData]] = None,
         callback: Callable[[PipelineIntermediateState], None] = None,
     ):
         self._adjust_memory_efficient_attention(latents)
@@ -411,6 +425,7 @@ class StableDiffusionGeneratorPipeline(StableDiffusionPipeline):
         if timesteps.shape[0] == 0:
             return latents, attention_map_saver
 
+        ip_adapter_unet_patcher = None
         if conditioning_data.extra is not None and conditioning_data.extra.wants_cross_attention_control:
             attn_ctx = self.invokeai_diffuser.custom_attention_context(
                 self.invokeai_diffuser.model,
@@ -421,11 +436,8 @@ class StableDiffusionGeneratorPipeline(StableDiffusionPipeline):
         elif ip_adapter_data is not None:
             # TODO(ryand): Should we raise an exception if both custom attention and IP-Adapter attention are active?
             # As it is now, the IP-Adapter will silently be skipped.
-            weight = ip_adapter_data.weight[0] if isinstance(ip_adapter_data.weight, List) else ip_adapter_data.weight
-            attn_ctx = ip_adapter_data.ip_adapter_model.apply_ip_adapter_attention(
-                unet=self.invokeai_diffuser.model,
-                scale=weight,
-            )
+            ip_adapter_unet_patcher = UNetPatcher([ipa.ip_adapter_model for ipa in ip_adapter_data])
+            attn_ctx = ip_adapter_unet_patcher.apply_ip_adapter_attention(self.invokeai_diffuser.model)
             self.use_ip_adapter = True
         else:
             attn_ctx = nullcontext()
@@ -454,6 +466,8 @@ class StableDiffusionGeneratorPipeline(StableDiffusionPipeline):
                     additional_guidance=additional_guidance,
                     control_data=control_data,
                     ip_adapter_data=ip_adapter_data,
+                    t2i_adapter_data=t2i_adapter_data,
+                    ip_adapter_unet_patcher=ip_adapter_unet_patcher,
                 )
                 latents = step_output.prev_sample
 
@@ -499,7 +513,9 @@ class StableDiffusionGeneratorPipeline(StableDiffusionPipeline):
         total_step_count: int,
         additional_guidance: List[Callable] = None,
         control_data: List[ControlNetData] = None,
-        ip_adapter_data: Optional[IPAdapterData] = None,
+        ip_adapter_data: Optional[list[IPAdapterData]] = None,
+        t2i_adapter_data: Optional[list[T2IAdapterData]] = None,
+        ip_adapter_unet_patcher: Optional[UNetPatcher] = None,
     ):
         # invokeai_diffuser has batched timesteps, but diffusers schedulers expect a single value
         timestep = t[0]
@@ -512,26 +528,30 @@ class StableDiffusionGeneratorPipeline(StableDiffusionPipeline):
 
         # handle IP-Adapter
         if self.use_ip_adapter and ip_adapter_data is not None:  # somewhat redundant but logic is clearer
-            first_adapter_step = math.floor(ip_adapter_data.begin_step_percent * total_step_count)
-            last_adapter_step = math.ceil(ip_adapter_data.end_step_percent * total_step_count)
-            weight = (
-                ip_adapter_data.weight[step_index]
-                if isinstance(ip_adapter_data.weight, List)
-                else ip_adapter_data.weight
-            )
-            if step_index >= first_adapter_step and step_index <= last_adapter_step:
-                # only apply IP-Adapter if current step is within the IP-Adapter's begin/end step range
-                # ip_adapter_data.ip_adapter_model.set_scale(ip_adapter_data.weight)
-                ip_adapter_data.ip_adapter_model.set_scale(weight)
-            else:
-                # otherwise, set IP-Adapter scale to 0, so it has no effect
-                ip_adapter_data.ip_adapter_model.set_scale(0.0)
+            for i, single_ip_adapter_data in enumerate(ip_adapter_data):
+                first_adapter_step = math.floor(single_ip_adapter_data.begin_step_percent * total_step_count)
+                last_adapter_step = math.ceil(single_ip_adapter_data.end_step_percent * total_step_count)
+                weight = (
+                    single_ip_adapter_data.weight[step_index]
+                    if isinstance(single_ip_adapter_data.weight, List)
+                    else single_ip_adapter_data.weight
+                )
+                if step_index >= first_adapter_step and step_index <= last_adapter_step:
+                    # Only apply this IP-Adapter if the current step is within the IP-Adapter's begin/end step range.
+                    ip_adapter_unet_patcher.set_scale(i, weight)
+                else:
+                    # Otherwise, set the IP-Adapter's scale to 0, so it has no effect.
+                    ip_adapter_unet_patcher.set_scale(i, 0.0)
 
-        # handle ControlNet(s)
-        # default is no controlnet, so set controlnet processing output to None
-        controlnet_down_block_samples, controlnet_mid_block_sample = None, None
-        if control_data is not None:
-            controlnet_down_block_samples, controlnet_mid_block_sample = self.invokeai_diffuser.do_controlnet_step(
+        # Handle ControlNet(s) and T2I-Adapter(s)
+        down_block_additional_residuals = None
+        mid_block_additional_residual = None
+        if control_data is not None and t2i_adapter_data is not None:
+            # TODO(ryand): This is a limitation of the UNet2DConditionModel API, not a fundamental incompatibility
+            # between ControlNets and T2I-Adapters. We will try to fix this upstream in diffusers.
+            raise Exception("ControlNet(s) and T2I-Adapter(s) cannot be used simultaneously (yet).")
+        elif control_data is not None:
+            down_block_additional_residuals, mid_block_additional_residual = self.invokeai_diffuser.do_controlnet_step(
                 control_data=control_data,
                 sample=latent_model_input,
                 timestep=timestep,
@@ -539,6 +559,32 @@ class StableDiffusionGeneratorPipeline(StableDiffusionPipeline):
                 total_step_count=total_step_count,
                 conditioning_data=conditioning_data,
             )
+        elif t2i_adapter_data is not None:
+            accum_adapter_state = None
+            for single_t2i_adapter_data in t2i_adapter_data:
+                # Determine the T2I-Adapter weights for the current denoising step.
+                first_t2i_adapter_step = math.floor(single_t2i_adapter_data.begin_step_percent * total_step_count)
+                last_t2i_adapter_step = math.ceil(single_t2i_adapter_data.end_step_percent * total_step_count)
+                t2i_adapter_weight = (
+                    single_t2i_adapter_data.weight[step_index]
+                    if isinstance(single_t2i_adapter_data.weight, list)
+                    else single_t2i_adapter_data.weight
+                )
+                if step_index < first_t2i_adapter_step or step_index > last_t2i_adapter_step:
+                    # If the current step is outside of the T2I-Adapter's begin/end step range, then set its weight to 0
+                    # so it has no effect.
+                    t2i_adapter_weight = 0.0
+
+                # Apply the t2i_adapter_weight, and accumulate.
+                if accum_adapter_state is None:
+                    # Handle the first T2I-Adapter.
+                    accum_adapter_state = [val * t2i_adapter_weight for val in single_t2i_adapter_data.adapter_state]
+                else:
+                    # Add to the previous adapter states.
+                    for idx, value in enumerate(single_t2i_adapter_data.adapter_state):
+                        accum_adapter_state[idx] += value * t2i_adapter_weight
+
+            down_block_additional_residuals = accum_adapter_state
 
         uc_noise_pred, c_noise_pred = self.invokeai_diffuser.do_unet_step(
             sample=latent_model_input,
@@ -547,8 +593,8 @@ class StableDiffusionGeneratorPipeline(StableDiffusionPipeline):
             total_step_count=total_step_count,
             conditioning_data=conditioning_data,
             # extra:
-            down_block_additional_residuals=controlnet_down_block_samples,  # from controlnet(s)
-            mid_block_additional_residual=controlnet_mid_block_sample,  # from controlnet(s)
+            down_block_additional_residuals=down_block_additional_residuals,
+            mid_block_additional_residual=mid_block_additional_residual,
         )
 
         guidance_scale = conditioning_data.guidance_scale

invokeai/backend/stable_diffusion/diffusion/conditioning_data.py

@@ -81,7 +81,7 @@ class ConditioningData:
     """
     postprocessing_settings: Optional[PostprocessingSettings] = None
 
-    ip_adapter_conditioning: Optional[IPAdapterConditioningInfo] = None
+    ip_adapter_conditioning: Optional[list[IPAdapterConditioningInfo]] = None
 
     @property
     def dtype(self):

invokeai/backend/stable_diffusion/diffusion/shared_invokeai_diffusion.py

@@ -346,12 +346,10 @@ class InvokeAIDiffuserComponent:
         cross_attention_kwargs = None
         if conditioning_data.ip_adapter_conditioning is not None:
             cross_attention_kwargs = {
-                "ip_adapter_image_prompt_embeds": torch.cat(
-                    [
-                        conditioning_data.ip_adapter_conditioning.uncond_image_prompt_embeds,
-                        conditioning_data.ip_adapter_conditioning.cond_image_prompt_embeds,
-                    ]
-                )
+                "ip_adapter_image_prompt_embeds": [
+                    torch.cat([ipa_conditioning.uncond_image_prompt_embeds, ipa_conditioning.cond_image_prompt_embeds])
+                    for ipa_conditioning in conditioning_data.ip_adapter_conditioning
+                ]
             }
 
         added_cond_kwargs = None
@@ -418,7 +416,10 @@ class InvokeAIDiffuserComponent:
         cross_attention_kwargs = None
         if conditioning_data.ip_adapter_conditioning is not None:
             cross_attention_kwargs = {
-                "ip_adapter_image_prompt_embeds": conditioning_data.ip_adapter_conditioning.uncond_image_prompt_embeds
+                "ip_adapter_image_prompt_embeds": [
+                    ipa_conditioning.uncond_image_prompt_embeds
+                    for ipa_conditioning in conditioning_data.ip_adapter_conditioning
+                ]
             }
 
         added_cond_kwargs = None
@@ -444,7 +445,10 @@ class InvokeAIDiffuserComponent:
         cross_attention_kwargs = None
         if conditioning_data.ip_adapter_conditioning is not None:
             cross_attention_kwargs = {
-                "ip_adapter_image_prompt_embeds": conditioning_data.ip_adapter_conditioning.cond_image_prompt_embeds
+                "ip_adapter_image_prompt_embeds": [
+                    ipa_conditioning.cond_image_prompt_embeds
+                    for ipa_conditioning in conditioning_data.ip_adapter_conditioning
+                ]
             }
 
         added_cond_kwargs = None

invokeai/backend/util/test_utils.py

@@ -0,0 +1,67 @@
+import contextlib
+from pathlib import Path
+from typing import Optional, Union
+
+import pytest
+import torch
+
+from invokeai.app.services.config.invokeai_config import InvokeAIAppConfig
+from invokeai.backend.install.model_install_backend import ModelInstall
+from invokeai.backend.model_management.model_manager import ModelInfo
+from invokeai.backend.model_management.models.base import BaseModelType, ModelNotFoundException, ModelType, SubModelType
+
+
+@pytest.fixture(scope="session")
+def torch_device():
+    return "cuda" if torch.cuda.is_available() else "cpu"
+
+
+@pytest.fixture(scope="module")
+def model_installer():
+    """A global ModelInstall pytest fixture to be used by many tests."""
+    # HACK(ryand): InvokeAIAppConfig.get_config() returns a singleton config object. This can lead to weird interactions
+    # between tests that need to alter the config. For example, some tests change the 'root' directory in the config,
+    # which can cause `install_and_load_model(...)` to re-download the model unnecessarily. As a temporary workaround,
+    # we pass a kwarg to get_config, which causes the config to be re-loaded. To fix this properly, we should stop using
+    # a singleton.
+    return ModelInstall(InvokeAIAppConfig.get_config(log_level="info"))
+
+
+def install_and_load_model(
+    model_installer: ModelInstall,
+    model_path_id_or_url: Union[str, Path],
+    model_name: str,
+    base_model: BaseModelType,
+    model_type: ModelType,
+    submodel_type: Optional[SubModelType] = None,
+) -> ModelInfo:
+    """Install a model if it is not already installed, then get the ModelInfo for that model.
+
+    This is intended as a utility function for tests.
+
+    Args:
+        model_installer (ModelInstall): The model installer.
+        model_path_id_or_url (Union[str, Path]): The path, HF ID, URL, etc. where the model can be installed from if it
+            is not already installed.
+        model_name (str): The model name, forwarded to ModelManager.get_model(...).
+        base_model (BaseModelType): The base model, forwarded to ModelManager.get_model(...).
+        model_type (ModelType): The model type, forwarded to ModelManager.get_model(...).
+        submodel_type (Optional[SubModelType]): The submodel type, forwarded to ModelManager.get_model(...).
+
+    Returns:
+        ModelInfo
+    """
+    # If the requested model is already installed, return its ModelInfo.
+    with contextlib.suppress(ModelNotFoundException):
+        return model_installer.mgr.get_model(model_name, base_model, model_type, submodel_type)
+
+    # Install the requested model.
+    model_installer.heuristic_import(model_path_id_or_url)
+
+    try:
+        return model_installer.mgr.get_model(model_name, base_model, model_type, submodel_type)
+    except ModelNotFoundException as e:
+        raise Exception(
+            "Failed to get model info after installing it. There could be a mismatch between the requested model and"
+            f" the installation id ('{model_path_id_or_url}'). Error: {e}"
+        )

invokeai/configs/INITIAL_MODELS.yaml

@@ -60,6 +60,9 @@ sd-1/main/trinart_stable_diffusion_v2:
    description: An SD-1.5 model finetuned with ~40K assorted high resolution manga/anime-style images (2.13 GB)
    repo_id: naclbit/trinart_stable_diffusion_v2
    recommended: False
+sd-1/controlnet/qrcode_monster:
+   repo_id: monster-labs/control_v1p_sd15_qrcode_monster
+   subfolder: v2
 sd-1/controlnet/canny:
    repo_id: lllyasviel/control_v11p_sd15_canny
    recommended: True
@@ -93,6 +96,22 @@ sd-1/controlnet/tile:
    repo_id: lllyasviel/control_v11f1e_sd15_tile
 sd-1/controlnet/ip2p:
    repo_id: lllyasviel/control_v11e_sd15_ip2p
+sd-1/t2i_adapter/canny-sd15:
+   repo_id: TencentARC/t2iadapter_canny_sd15v2
+sd-1/t2i_adapter/sketch-sd15:
+   repo_id: TencentARC/t2iadapter_sketch_sd15v2
+sd-1/t2i_adapter/depth-sd15:
+   repo_id: TencentARC/t2iadapter_depth_sd15v2
+sd-1/t2i_adapter/zoedepth-sd15:
+   repo_id: TencentARC/t2iadapter_zoedepth_sd15v1
+sdxl/t2i_adapter/canny-sdxl:
+   repo_id: TencentARC/t2i-adapter-canny-sdxl-1.0
+sdxl/t2i_adapter/zoedepth-sdxl:
+   repo_id: TencentARC/t2i-adapter-depth-zoe-sdxl-1.0
+sdxl/t2i_adapter/lineart-sdxl:
+   repo_id: TencentARC/t2i-adapter-lineart-sdxl-1.0
+sdxl/t2i_adapter/sketch-sdxl:
+   repo_id: TencentARC/t2i-adapter-sketch-sdxl-1.0
 sd-1/embedding/EasyNegative:
    path: https://huggingface.co/embed/EasyNegative/resolve/main/EasyNegative.safetensors
    recommended: True

invokeai/frontend/install/model_install.py

@@ -98,15 +98,16 @@ class addModelsForm(CyclingForm, npyscreen.FormMultiPage):
         self.tabs = self.add_widget_intelligent(
             SingleSelectColumns,
             values=[
-                "STARTER MODELS",
-                "MAIN MODELS",
+                "STARTERS",
+                "MAINS",
                 "CONTROLNETS",
+                "T2I-ADAPTERS",
                 "IP-ADAPTERS",
-                "LORA/LYCORIS",
-                "TEXTUAL INVERSION",
+                "LORAS",
+                "TI EMBEDDINGS",
             ],
             value=[self.current_tab],
-            columns=6,
+            columns=7,
             max_height=2,
             relx=8,
             scroll_exit=True,
@@ -131,6 +132,12 @@ class addModelsForm(CyclingForm, npyscreen.FormMultiPage):
         )
         bottom_of_table = max(bottom_of_table, self.nextrely)
 
+        self.nextrely = top_of_table
+        self.t2i_models = self.add_model_widgets(
+            model_type=ModelType.T2IAdapter,
+            window_width=window_width,
+        )
+        bottom_of_table = max(bottom_of_table, self.nextrely)
         self.nextrely = top_of_table
         self.ipadapter_models = self.add_model_widgets(
             model_type=ModelType.IPAdapter,
@@ -351,6 +358,7 @@ class addModelsForm(CyclingForm, npyscreen.FormMultiPage):
             self.starter_pipelines,
             self.pipeline_models,
             self.controlnet_models,
+            self.t2i_models,
             self.ipadapter_models,
             self.lora_models,
             self.ti_models,
@@ -541,6 +549,7 @@ class addModelsForm(CyclingForm, npyscreen.FormMultiPage):
             self.starter_pipelines,
             self.pipeline_models,
             self.controlnet_models,
+            self.t2i_models,
             self.ipadapter_models,
             self.lora_models,
             self.ti_models,

invokeai/frontend/web/dist/assets/ThemeLocaleProvider-6771db0c.js

@@ -0,0 +1,280 @@
+import{w as s,h_ as T,v as l,a2 as I,h$ as R,ae as V,i0 as z,i1 as j,i2 as D,i3 as F,i4 as G,i5 as W,i6 as K,aG as H,i7 as U,i8 as Y}from"./index-90346e33.js";import{M as Z}from"./MantineProvider-486d4834.js";var P=String.raw,E=P`
+  :root,
+  :host {
+    --chakra-vh: 100vh;
+  }
+
+  @supports (height: -webkit-fill-available) {
+    :root,
+    :host {
+      --chakra-vh: -webkit-fill-available;
+    }
+  }
+
+  @supports (height: -moz-fill-available) {
+    :root,
+    :host {
+      --chakra-vh: -moz-fill-available;
+    }
+  }
+
+  @supports (height: 100dvh) {
+    :root,
+    :host {
+      --chakra-vh: 100dvh;
+    }
+  }
+`,B=()=>s.jsx(T,{styles:E}),J=({scope:e=""})=>s.jsx(T,{styles:P`
+      html {
+        line-height: 1.5;
+        -webkit-text-size-adjust: 100%;
+        font-family: system-ui, sans-serif;
+        -webkit-font-smoothing: antialiased;
+        text-rendering: optimizeLegibility;
+        -moz-osx-font-smoothing: grayscale;
+        touch-action: manipulation;
+      }
+
+      body {
+        position: relative;
+        min-height: 100%;
+        margin: 0;
+        font-feature-settings: "kern";
+      }
+
+      ${e} :where(*, *::before, *::after) {
+        border-width: 0;
+        border-style: solid;
+        box-sizing: border-box;
+        word-wrap: break-word;
+      }
+
+      main {
+        display: block;
+      }
+
+      ${e} hr {
+        border-top-width: 1px;
+        box-sizing: content-box;
+        height: 0;
+        overflow: visible;
+      }
+
+      ${e} :where(pre, code, kbd,samp) {
+        font-family: SFMono-Regular, Menlo, Monaco, Consolas, monospace;
+        font-size: 1em;
+      }
+
+      ${e} a {
+        background-color: transparent;
+        color: inherit;
+        text-decoration: inherit;
+      }
+
+      ${e} abbr[title] {
+        border-bottom: none;
+        text-decoration: underline;
+        -webkit-text-decoration: underline dotted;
+        text-decoration: underline dotted;
+      }
+
+      ${e} :where(b, strong) {
+        font-weight: bold;
+      }
+
+      ${e} small {
+        font-size: 80%;
+      }
+
+      ${e} :where(sub,sup) {
+        font-size: 75%;
+        line-height: 0;
+        position: relative;
+        vertical-align: baseline;
+      }
+
+      ${e} sub {
+        bottom: -0.25em;
+      }
+
+      ${e} sup {
+        top: -0.5em;
+      }
+
+      ${e} img {
+        border-style: none;
+      }
+
+      ${e} :where(button, input, optgroup, select, textarea) {
+        font-family: inherit;
+        font-size: 100%;
+        line-height: 1.15;
+        margin: 0;
+      }
+
+      ${e} :where(button, input) {
+        overflow: visible;
+      }
+
+      ${e} :where(button, select) {
+        text-transform: none;
+      }
+
+      ${e} :where(
+          button::-moz-focus-inner,
+          [type="button"]::-moz-focus-inner,
+          [type="reset"]::-moz-focus-inner,
+          [type="submit"]::-moz-focus-inner
+        ) {
+        border-style: none;
+        padding: 0;
+      }
+
+      ${e} fieldset {
+        padding: 0.35em 0.75em 0.625em;
+      }
+
+      ${e} legend {
+        box-sizing: border-box;
+        color: inherit;
+        display: table;
+        max-width: 100%;
+        padding: 0;
+        white-space: normal;
+      }
+
+      ${e} progress {
+        vertical-align: baseline;
+      }
+
+      ${e} textarea {
+        overflow: auto;
+      }
+
+      ${e} :where([type="checkbox"], [type="radio"]) {
+        box-sizing: border-box;
+        padding: 0;
+      }
+
+      ${e} input[type="number"]::-webkit-inner-spin-button,
+      ${e} input[type="number"]::-webkit-outer-spin-button {
+        -webkit-appearance: none !important;
+      }
+
+      ${e} input[type="number"] {
+        -moz-appearance: textfield;
+      }
+
+      ${e} input[type="search"] {
+        -webkit-appearance: textfield;
+        outline-offset: -2px;
+      }
+
+      ${e} input[type="search"]::-webkit-search-decoration {
+        -webkit-appearance: none !important;
+      }
+
+      ${e} ::-webkit-file-upload-button {
+        -webkit-appearance: button;
+        font: inherit;
+      }
+
+      ${e} details {
+        display: block;
+      }
+
+      ${e} summary {
+        display: list-item;
+      }
+
+      template {
+        display: none;
+      }
+
+      [hidden] {
+        display: none !important;
+      }
+
+      ${e} :where(
+          blockquote,
+          dl,
+          dd,
+          h1,
+          h2,
+          h3,
+          h4,
+          h5,
+          h6,
+          hr,
+          figure,
+          p,
+          pre
+        ) {
+        margin: 0;
+      }
+
+      ${e} button {
+        background: transparent;
+        padding: 0;
+      }
+
+      ${e} fieldset {
+        margin: 0;
+        padding: 0;
+      }
+
+      ${e} :where(ol, ul) {
+        margin: 0;
+        padding: 0;
+      }
+
+      ${e} textarea {
+        resize: vertical;
+      }
+
+      ${e} :where(button, [role="button"]) {
+        cursor: pointer;
+      }
+
+      ${e} button::-moz-focus-inner {
+        border: 0 !important;
+      }
+
+      ${e} table {
+        border-collapse: collapse;
+      }
+
+      ${e} :where(h1, h2, h3, h4, h5, h6) {
+        font-size: inherit;
+        font-weight: inherit;
+      }
+
+      ${e} :where(button, input, optgroup, select, textarea) {
+        padding: 0;
+        line-height: inherit;
+        color: inherit;
+      }
+
+      ${e} :where(img, svg, video, canvas, audio, iframe, embed, object) {
+        display: block;
+      }
+
+      ${e} :where(img, video) {
+        max-width: 100%;
+        height: auto;
+      }
+
+      [data-js-focus-visible]
+        :focus:not([data-focus-visible-added]):not(
+          [data-focus-visible-disabled]
+        ) {
+        outline: none;
+        box-shadow: none;
+      }
+
+      ${e} select::-ms-expand {
+        display: none;
+      }
+
+      ${E}
+    `}),g={light:"chakra-ui-light",dark:"chakra-ui-dark"};function Q(e={}){const{preventTransition:o=!0}=e,n={setDataset:r=>{const t=o?n.preventTransition():void 0;document.documentElement.dataset.theme=r,document.documentElement.style.colorScheme=r,t==null||t()},setClassName(r){document.body.classList.add(r?g.dark:g.light),document.body.classList.remove(r?g.light:g.dark)},query(){return window.matchMedia("(prefers-color-scheme: dark)")},getSystemTheme(r){var t;return((t=n.query().matches)!=null?t:r==="dark")?"dark":"light"},addListener(r){const t=n.query(),i=a=>{r(a.matches?"dark":"light")};return typeof t.addListener=="function"?t.addListener(i):t.addEventListener("change",i),()=>{typeof t.removeListener=="function"?t.removeListener(i):t.removeEventListener("change",i)}},preventTransition(){const r=document.createElement("style");return r.appendChild(document.createTextNode("*{-webkit-transition:none!important;-moz-transition:none!important;-o-transition:none!important;-ms-transition:none!important;transition:none!important}")),document.head.appendChild(r),()=>{window.getComputedStyle(document.body),requestAnimationFrame(()=>{requestAnimationFrame(()=>{document.head.removeChild(r)})})}}};return n}var X="chakra-ui-color-mode";function L(e){return{ssr:!1,type:"localStorage",get(o){if(!(globalThis!=null&&globalThis.document))return o;let n;try{n=localStorage.getItem(e)||o}catch{}return n||o},set(o){try{localStorage.setItem(e,o)}catch{}}}}var ee=L(X),M=()=>{};function S(e,o){return e.type==="cookie"&&e.ssr?e.get(o):o}function O(e){const{value:o,children:n,options:{useSystemColorMode:r,initialColorMode:t,disableTransitionOnChange:i}={},colorModeManager:a=ee}=e,d=t==="dark"?"dark":"light",[u,p]=l.useState(()=>S(a,d)),[y,b]=l.useState(()=>S(a)),{getSystemTheme:w,setClassName:k,setDataset:x,addListener:$}=l.useMemo(()=>Q({preventTransition:i}),[i]),v=t==="system"&&!u?y:u,c=l.useCallback(h=>{const f=h==="system"?w():h;p(f),k(f==="dark"),x(f),a.set(f)},[a,w,k,x]);I(()=>{t==="system"&&b(w())},[]),l.useEffect(()=>{const h=a.get();if(h){c(h);return}if(t==="system"){c("system");return}c(d)},[a,d,t,c]);const C=l.useCallback(()=>{c(v==="dark"?"light":"dark")},[v,c]);l.useEffect(()=>{if(r)return $(c)},[r,$,c]);const A=l.useMemo(()=>({colorMode:o??v,toggleColorMode:o?M:C,setColorMode:o?M:c,forced:o!==void 0}),[v,C,c,o]);return s.jsx(R.Provider,{value:A,children:n})}O.displayName="ColorModeProvider";var te=["borders","breakpoints","colors","components","config","direction","fonts","fontSizes","fontWeights","letterSpacings","lineHeights","radii","shadows","sizes","space","styles","transition","zIndices"];function re(e){return V(e)?te.every(o=>Object.prototype.hasOwnProperty.call(e,o)):!1}function m(e){return typeof e=="function"}function oe(...e){return o=>e.reduce((n,r)=>r(n),o)}var ne=e=>function(...n){let r=[...n],t=n[n.length-1];return re(t)&&r.length>1?r=r.slice(0,r.length-1):t=e,oe(...r.map(i=>a=>m(i)?i(a):ae(a,i)))(t)},ie=ne(j);function ae(...e){return z({},...e,_)}function _(e,o,n,r){if((m(e)||m(o))&&Object.prototype.hasOwnProperty.call(r,n))return(...t)=>{const i=m(e)?e(...t):e,a=m(o)?o(...t):o;return z({},i,a,_)}}var q=l.createContext({getDocument(){return document},getWindow(){return window}});q.displayName="EnvironmentContext";function N(e){const{children:o,environment:n,disabled:r}=e,t=l.useRef(null),i=l.useMemo(()=>n||{getDocument:()=>{var d,u;return(u=(d=t.current)==null?void 0:d.ownerDocument)!=null?u:document},getWindow:()=>{var d,u;return(u=(d=t.current)==null?void 0:d.ownerDocument.defaultView)!=null?u:window}},[n]),a=!r||!n;return s.jsxs(q.Provider,{value:i,children:[o,a&&s.jsx("span",{id:"__chakra_env",hidden:!0,ref:t})]})}N.displayName="EnvironmentProvider";var se=e=>{const{children:o,colorModeManager:n,portalZIndex:r,resetScope:t,resetCSS:i=!0,theme:a={},environment:d,cssVarsRoot:u,disableEnvironment:p,disableGlobalStyle:y}=e,b=s.jsx(N,{environment:d,disabled:p,children:o});return s.jsx(D,{theme:a,cssVarsRoot:u,children:s.jsxs(O,{colorModeManager:n,options:a.config,children:[i?s.jsx(J,{scope:t}):s.jsx(B,{}),!y&&s.jsx(F,{}),r?s.jsx(G,{zIndex:r,children:b}):b]})})},le=e=>function({children:n,theme:r=e,toastOptions:t,...i}){return s.jsxs(se,{theme:r,...i,children:[s.jsx(W,{value:t==null?void 0:t.defaultOptions,children:n}),s.jsx(K,{...t})]})},de=le(j);const ue=()=>l.useMemo(()=>({colorScheme:"dark",fontFamily:"'Inter Variable', sans-serif",components:{ScrollArea:{defaultProps:{scrollbarSize:10},styles:{scrollbar:{"&:hover":{backgroundColor:"var(--invokeai-colors-baseAlpha-300)"}},thumb:{backgroundColor:"var(--invokeai-colors-baseAlpha-300)"}}}}}),[]),ce=L("@@invokeai-color-mode");function he({children:e}){const{i18n:o}=H(),n=o.dir(),r=l.useMemo(()=>ie({...U,direction:n}),[n]);l.useEffect(()=>{document.body.dir=n},[n]);const t=ue();return s.jsx(Z,{theme:t,children:s.jsx(de,{theme:r,colorModeManager:ce,toastOptions:Y,children:e})})}const ve=l.memo(he);export{ve as default};

invokeai/frontend/web/dist/index.html

@@ -3,6 +3,9 @@
   <head>
     <meta charset="UTF-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate">
+    <meta http-equiv="Pragma" content="no-cache">
+    <meta http-equiv="Expires" content="0">
     <title>InvokeAI - A Stable Diffusion Toolkit</title>
     <link rel="shortcut icon" type="icon" href="./assets/favicon-0d253ced.ico" />
     <style>
@@ -12,7 +15,7 @@
         margin: 0;
       }
     </style>
-    <script type="module" crossorigin src="./assets/index-f6c3f475.js"></script>
+    <script type="module" crossorigin src="./assets/index-90346e33.js"></script>
   </head>
 
   <body dir="ltr">

invokeai/frontend/web/dist/locales/en.json

@@ -13,14 +13,15 @@
         "reset": "Reset",
         "rotateClockwise": "Rotate Clockwise",
         "rotateCounterClockwise": "Rotate Counter-Clockwise",
-        "showGallery": "Show Gallery",
+        "showGalleryPanel": "Show Gallery Panel",
         "showOptionsPanel": "Show Side Panel",
         "toggleAutoscroll": "Toggle autoscroll",
         "toggleLogViewer": "Toggle Log Viewer",
         "uploadImage": "Upload Image",
         "useThisParameter": "Use this parameter",
         "zoomIn": "Zoom In",
-        "zoomOut": "Zoom Out"
+        "zoomOut": "Zoom Out",
+        "loadMore": "Load More"
     },
     "boards": {
         "addBoard": "Add Board",
@@ -48,8 +49,10 @@
         "cancel": "Cancel",
         "close": "Close",
         "communityLabel": "Community",
-        "controlNet": "Controlnet",
+        "controlNet": "ControlNet",
+        "controlAdapter": "Control Adapter",
         "ipAdapter": "IP Adapter",
+        "t2iAdapter": "T2I Adapter",
         "darkMode": "Dark Mode",
         "discordLabel": "Discord",
         "dontAskMeAgain": "Don't ask me again",
@@ -57,6 +60,7 @@
         "githubLabel": "Github",
         "hotkeysLabel": "Hotkeys",
         "imagePrompt": "Image Prompt",
+        "imageFailedToLoad": "Unable to Load Image",
         "img2img": "Image To Image",
         "langArabic": "العربية",
         "langBrPortuguese": "Português do Brasil",
@@ -80,6 +84,7 @@
         "load": "Load",
         "loading": "Loading",
         "loadingInvokeAI": "Loading Invoke AI",
+        "learnMore": "Learn More",
         "modelManager": "Model Manager",
         "nodeEditor": "Node Editor",
         "nodes": "Workflow Editor",
@@ -110,6 +115,7 @@
         "statusModelChanged": "Model Changed",
         "statusModelConverted": "Model Converted",
         "statusPreparing": "Preparing",
+        "statusProcessing": "Processing",
         "statusProcessingCanceled": "Processing Canceled",
         "statusProcessingComplete": "Processing Complete",
         "statusRestoringFaces": "Restoring Faces",
@@ -126,6 +132,16 @@
         "upload": "Upload"
     },
     "controlnet": {
+        "controlAdapter": "Control Adapter",
+        "controlnet": "$t(controlnet.controlAdapter) #{{number}} ($t(common.controlNet))",
+        "ip_adapter": "$t(controlnet.controlAdapter) #{{number}} ($t(common.ipAdapter))",
+        "t2i_adapter": "$t(controlnet.controlAdapter) #{{number}} ($t(common.t2iAdapter))",
+        "addControlNet": "Add $t(common.controlNet)",
+        "addIPAdapter": "Add $t(common.ipAdapter)",
+        "addT2IAdapter": "Add $t(common.t2iAdapter)",
+        "controlNetEnabledT2IDisabled": "$t(common.controlNet) enabled, $t(common.t2iAdapter)s disabled",
+        "t2iEnabledControlNetDisabled": "$t(common.t2iAdapter) enabled, $t(common.controlNet)s disabled",
+        "controlNetT2IMutexDesc": "$t(common.controlNet) and $t(common.t2iAdapter) at same time is currently unsupported.",
         "amult": "a_mult",
         "autoConfigure": "Auto configure processor",
         "balanced": "Balanced",
@@ -133,6 +149,8 @@
         "bgth": "bg_th",
         "canny": "Canny",
         "cannyDescription": "Canny edge detection",
+        "colorMap": "Color",
+        "colorMapDescription": "Generates a color map from the image",
         "coarse": "Coarse",
         "contentShuffle": "Content Shuffle",
         "contentShuffleDescription": "Shuffles the content in an image",
@@ -156,6 +174,7 @@
         "hideAdvanced": "Hide Advanced",
         "highThreshold": "High Threshold",
         "imageResolution": "Image Resolution",
+        "colorMapTileSize": "Tile Size",
         "importImageFromCanvas": "Import Image From Canvas",
         "importMaskFromCanvas": "Import Mask From Canvas",
         "incompatibleBaseModel": "Incompatible base model:",
@@ -203,6 +222,81 @@
         "incompatibleModel": "Incompatible base model:",
         "noMatchingEmbedding": "No matching Embeddings"
     },
+    "queue": {
+        "queue": "Queue",
+        "queueFront": "Add to Front of Queue",
+        "queueBack": "Add to Queue",
+        "queueCountPrediction": "Add {{predicted}} to Queue",
+        "queueMaxExceeded": "Max of {{max_queue_size}} exceeded, would skip {{skip}}",
+        "queuedCount": "{{pending}} Pending",
+        "queueTotal": "{{total}} Total",
+        "queueEmpty": "Queue Empty",
+        "enqueueing": "Queueing Batch",
+        "resume": "Resume",
+        "resumeTooltip": "Resume Processor",
+        "resumeSucceeded": "Processor Resumed",
+        "resumeFailed": "Problem Resuming Processor",
+        "pause": "Pause",
+        "pauseTooltip": "Pause Processor",
+        "pauseSucceeded": "Processor Paused",
+        "pauseFailed": "Problem Pausing Processor",
+        "cancel": "Cancel",
+        "cancelTooltip": "Cancel Current Item",
+        "cancelSucceeded": "Item Canceled",
+        "cancelFailed": "Problem Canceling Item",
+        "prune": "Prune",
+        "pruneTooltip": "Prune {{item_count}} Completed Items",
+        "pruneSucceeded": "Pruned {{item_count}} Completed Items from Queue",
+        "pruneFailed": "Problem Pruning Queue",
+        "clear": "Clear",
+        "clearTooltip": "Cancel and Clear All Items",
+        "clearSucceeded": "Queue Cleared",
+        "clearFailed": "Problem Clearing Queue",
+        "cancelBatch": "Cancel Batch",
+        "cancelItem": "Cancel Item",
+        "cancelBatchSucceeded": "Batch Canceled",
+        "cancelBatchFailed": "Problem Canceling Batch",
+        "clearQueueAlertDialog": "Clearing the queue immediately cancels any processing items and clears the queue entirely.",
+        "clearQueueAlertDialog2": "Are you sure you want to clear the queue?",
+        "current": "Current",
+        "next": "Next",
+        "status": "Status",
+        "total": "Total",
+        "pending": "Pending",
+        "in_progress": "In Progress",
+        "completed": "Completed",
+        "failed": "Failed",
+        "canceled": "Canceled",
+        "completedIn": "Completed in",
+        "batch": "Batch",
+        "item": "Item",
+        "session": "Session",
+        "batchValues": "Batch Values",
+        "notReady": "Unable to Queue",
+        "batchQueued": "Batch Queued",
+        "batchQueuedDesc": "Added {{item_count}} sessions to {{direction}} of queue",
+        "front": "front",
+        "back": "back",
+        "batchFailedToQueue": "Failed to Queue Batch",
+        "graphQueued": "Graph queued",
+        "graphFailedToQueue": "Failed to queue graph"
+    },
+    "invocationCache": {
+        "invocationCache": "Invocation Cache",
+        "cacheSize": "Cache Size",
+        "maxCacheSize": "Max Cache Size",
+        "hits": "Cache Hits",
+        "misses": "Cache Misses",
+        "clear": "Clear",
+        "clearSucceeded": "Invocation Cache Cleared",
+        "clearFailed": "Problem Clearing Invocation Cache",
+        "enable": "Enable",
+        "enableSucceeded": "Invocation Cache Enabled",
+        "enableFailed": "Problem Enabling Invocation Cache",
+        "disable": "Disable",
+        "disableSucceeded": "Invocation Cache Disabled",
+        "disableFailed": "Problem Disabling Invocation Cache"
+    },
     "gallery": {
         "allImagesLoaded": "All Images Loaded",
         "assets": "Assets",
@@ -615,7 +709,7 @@
         "noLoRAsAvailable": "No LoRAs available",
         "noMatchingLoRAs": "No matching LoRAs",
         "noMatchingModels": "No matching Models",
-        "noModelsAvailable": "No Modelss available",
+        "noModelsAvailable": "No models available",
         "selectLoRA": "Select a LoRA",
         "selectModel": "Select a Model"
     },
@@ -624,6 +718,8 @@
         "addNodeToolTip": "Add Node (Shift+A, Space)",
         "animatedEdges": "Animated Edges",
         "animatedEdgesHelp": "Animate selected edges and edges connected to selected nodes",
+        "boardField": "Board",
+        "boardFieldDescription": "A gallery board",
         "boolean": "Booleans",
         "booleanCollection": "Boolean Collection",
         "booleanCollectionDescription": "A collection of booleans.",
@@ -633,6 +729,7 @@
         "cannotConnectInputToInput": "Cannot connect input to input",
         "cannotConnectOutputToOutput": "Cannot connect output to output",
         "cannotConnectToSelf": "Cannot connect to self",
+        "cannotDuplicateConnection": "Cannot create duplicate connections",
         "clipField": "Clip",
         "clipFieldDescription": "Tokenizer and text_encoder submodels.",
         "collection": "Collection",
@@ -641,7 +738,8 @@
         "collectionItemDescription": "TODO",
         "colorCodeEdges": "Color-Code Edges",
         "colorCodeEdgesHelp": "Color-code edges according to their connected fields",
-        "colorCollectionDescription": "A collection of colors.",
+        "colorCollection": "A collection of colors.",
+        "colorCollectionDescription": "TODO",
         "colorField": "Color",
         "colorFieldDescription": "A RGBA color.",
         "colorPolymorphic": "Color Polymorphic",
@@ -688,7 +786,8 @@
         "imageFieldDescription": "Images may be passed between nodes.",
         "imagePolymorphic": "Image Polymorphic",
         "imagePolymorphicDescription": "A collection of images.",
-        "inputFields": "Input Feilds",
+        "inputField": "Input Field",
+        "inputFields": "Input Fields",
         "inputMayOnlyHaveOneConnection": "Input may only have one connection",
         "inputNode": "Input Node",
         "integer": "Integer",
@@ -698,6 +797,14 @@
         "integerPolymorphic": "Integer Polymorphic",
         "integerPolymorphicDescription": "A collection of integers.",
         "invalidOutputSchema": "Invalid output schema",
+        "ipAdapter": "IP-Adapter",
+        "ipAdapterCollection": "IP-Adapters Collection",
+        "ipAdapterCollectionDescription": "A collection of IP-Adapters.",
+        "ipAdapterDescription": "An Image Prompt Adapter (IP-Adapter).",
+        "ipAdapterModel": "IP-Adapter Model",
+        "ipAdapterModelDescription": "IP-Adapter Model Field",
+        "ipAdapterPolymorphic": "IP-Adapter Polymorphic",
+        "ipAdapterPolymorphicDescription": "A collection of IP-Adapters.",
         "latentsCollection": "Latents Collection",
         "latentsCollectionDescription": "Latents may be passed between nodes.",
         "latentsField": "Latents",
@@ -706,6 +813,7 @@
         "latentsPolymorphicDescription": "Latents may be passed between nodes.",
         "loadingNodes": "Loading Nodes...",
         "loadWorkflow": "Load Workflow",
+        "noWorkflow": "No Workflow",
         "loRAModelField": "LoRA",
         "loRAModelFieldDescription": "TODO",
         "mainModelField": "Model",
@@ -727,14 +835,15 @@
         "noImageFoundState": "No initial image found in state",
         "noMatchingNodes": "No matching nodes",
         "noNodeSelected": "No node selected",
-        "noOpacity": "Node Opacity",
+        "nodeOpacity": "Node Opacity",
         "noOutputRecorded": "No outputs recorded",
         "noOutputSchemaName": "No output schema name found in ref object",
         "notes": "Notes",
         "notesDescription": "Add notes about your workflow",
         "oNNXModelField": "ONNX Model",
         "oNNXModelFieldDescription": "ONNX model field.",
-        "outputFields": "Output Feilds",
+        "outputField": "Output Field",
+        "outputFields": "Output Fields",
         "outputNode": "Output node",
         "outputSchemaNotFound": "Output schema not found",
         "pickOne": "Pick One",
@@ -783,6 +892,7 @@
         "unknownNode": "Unknown Node",
         "unknownTemplate": "Unknown Template",
         "unkownInvocation": "Unknown Invocation type",
+        "updateNode": "Update Node",
         "updateApp": "Update App",
         "vaeField": "Vae",
         "vaeFieldDescription": "Vae submodel.",
@@ -806,7 +916,7 @@
         "zoomOutNodes": "Zoom Out"
     },
     "parameters": {
-        "aspectRatio": "Ratio",
+        "aspectRatio": "Aspect Ratio",
         "boundingBoxHeader": "Bounding Box",
         "boundingBoxHeight": "Bounding Box Height",
         "boundingBoxWidth": "Bounding Box Width",
@@ -819,6 +929,7 @@
         },
         "cfgScale": "CFG Scale",
         "clipSkip": "CLIP Skip",
+        "clipSkipWithLayerCount": "CLIP Skip {{layerCount}}",
         "closeViewer": "Close Viewer",
         "codeformerFidelity": "Fidelity",
         "coherenceMode": "Mode",
@@ -853,10 +964,12 @@
             "missingFieldTemplate": "Missing field template",
             "missingInputForField": "{{nodeLabel}} -> {{fieldLabel}} missing input",
             "missingNodeTemplate": "Missing node template",
-            "noControlImageForControlNet": "ControlNet {{index}} has no control image",
+            "noControlImageForControlAdapter": "Control Adapter {{number}} has no control image",
             "noInitialImageSelected": "No initial image selected",
-            "noModelForControlNet": "ControlNet {{index}} has no model selected.",
+            "noModelForControlAdapter": "Control Adapter {{number}} has no model selected.",
+            "incompatibleBaseModelForControlAdapter": "Control Adapter {{number}} model is invalid with main model.",
             "noModelSelected": "No model selected",
+            "noPrompts": "No prompts generated",
             "noNodesInGraph": "No nodes in graph",
             "readyToInvoke": "Ready to Invoke",
             "systemBusy": "System busy",
@@ -875,7 +988,12 @@
         "perlinNoise": "Perlin Noise",
         "positivePromptPlaceholder": "Positive Prompt",
         "randomizeSeed": "Randomize Seed",
+        "manualSeed": "Manual Seed",
+        "randomSeed": "Random Seed",
         "restoreFaces": "Restore Faces",
+        "iterations": "Iterations",
+        "iterationsWithCount_one": "{{count}} Iteration",
+        "iterationsWithCount_other": "{{count}} Iterations",
         "scale": "Scale",
         "scaleBeforeProcessing": "Scale Before Processing",
         "scaledHeight": "Scaled H",
@@ -886,13 +1004,17 @@
         "seamlessTiling": "Seamless Tiling",
         "seamlessXAxis": "X Axis",
         "seamlessYAxis": "Y Axis",
+        "seamlessX": "Seamless X",
+        "seamlessY": "Seamless Y",
+        "seamlessX&Y": "Seamless X & Y",
         "seamLowThreshold": "Low",
         "seed": "Seed",
         "seedWeights": "Seed Weights",
+        "imageActions": "Image Actions",
         "sendTo": "Send to",
         "sendToImg2Img": "Send to Image to Image",
         "sendToUnifiedCanvas": "Send To Unified Canvas",
-        "showOptionsPanel": "Show Options Panel",
+        "showOptionsPanel": "Show Side Panel (O or T)",
         "showPreview": "Show Preview",
         "shuffle": "Shuffle Seed",
         "steps": "Steps",
@@ -901,24 +1023,39 @@
         "tileSize": "Tile Size",
         "toggleLoopback": "Toggle Loopback",
         "type": "Type",
-        "upscale": "Upscale",
+        "upscale": "Upscale (Shift + U)",
         "upscaleImage": "Upscale Image",
         "upscaling": "Upscaling",
         "useAll": "Use All",
         "useCpuNoise": "Use CPU Noise",
+        "cpuNoise": "CPU Noise",
+        "gpuNoise": "GPU Noise",
         "useInitImg": "Use Initial Image",
         "usePrompt": "Use Prompt",
         "useSeed": "Use Seed",
         "variationAmount": "Variation Amount",
         "variations": "Variations",
         "vSymmetryStep": "V Symmetry Step",
-        "width": "Width"
+        "width": "Width",
+        "isAllowedToUpscale": {
+            "useX2Model": "Image is too large to upscale with x4 model, use x2 model",
+            "tooLarge": "Image is too large to upscale, select smaller image"
+        }
     },
-    "prompt": {
+    "dynamicPrompts": {
         "combinatorial": "Combinatorial Generation",
         "dynamicPrompts": "Dynamic Prompts",
         "enableDynamicPrompts": "Enable Dynamic Prompts",
-        "maxPrompts": "Max Prompts"
+        "maxPrompts": "Max Prompts",
+        "promptsWithCount_one": "{{count}} Prompt",
+        "promptsWithCount_other": "{{count}} Prompts",
+        "seedBehaviour": {
+            "label": "Seed Behaviour",
+            "perIterationLabel": "Seed per Iteration",
+            "perIterationDesc": "Use a different seed for each iteration",
+            "perPromptLabel": "Seed per Image",
+            "perPromptDesc": "Use a different seed for each image"
+        }
     },
     "sdxl": {
         "cfgScale": "CFG Scale",
@@ -969,7 +1106,8 @@
     },
     "toast": {
         "addedToBoard": "Added to board",
-        "baseModelChangedCleared": "Base model changed, cleared",
+        "baseModelChangedCleared_one": "Base model changed, cleared or disabled {{number}} incompatible submodel",
+        "baseModelChangedCleared_many": "$t(toast.baseModelChangedCleared_one)s",
         "canceled": "Processing Canceled",
         "canvasCopiedClipboard": "Canvas Copied to Clipboard",
         "canvasDownloaded": "Canvas Downloaded",
@@ -989,7 +1127,6 @@
         "imageSavingFailed": "Image Saving Failed",
         "imageUploaded": "Image Uploaded",
         "imageUploadFailed": "Image Upload Failed",
-        "incompatibleSubmodel": "incompatible submodel",
         "initialImageNotSet": "Initial Image Not Set",
         "initialImageNotSetDesc": "Could not load initial image",
         "initialImageSet": "Initial Image Set",
@@ -1066,6 +1203,210 @@
             "variations": "Try a variation with a value between 0.1 and 1.0 to change the result for a given seed. Interesting variations of the seed are between 0.1 and 0.3."
         }
     },
+    "popovers": {
+        "clipSkip": {
+            "heading": "CLIP Skip",
+            "paragraphs": [
+                "Choose how many layers of the CLIP model to skip.",
+                "Some models work better with certain CLIP Skip settings.",
+                "A higher value typically results in a less detailed image."
+            ]
+        },
+        "paramNegativeConditioning": {
+            "heading": "Negative Prompt",
+            "paragraphs": [
+                "The generation process avoids the concepts in the negative prompt. Use this to exclude qualities or objects from the output.",
+                "Supports Compel syntax and embeddings."
+            ]
+        },
+        "paramPositiveConditioning": {
+            "heading": "Positive Prompt",
+            "paragraphs": [
+                "Guides the generation process. You may use any words or phrases.",
+                "Compel and Dynamic Prompts syntaxes and embeddings."
+            ]
+        },
+        "paramScheduler": {
+            "heading": "Scheduler",
+            "paragraphs": [
+                "Scheduler defines how to iteratively add noise to an image or how to update a sample based on a model's output."
+            ]
+        },
+        "compositingBlur": {
+            "heading": "Blur",
+            "paragraphs": ["The blur radius of the mask."]
+        },
+        "compositingBlurMethod": {
+            "heading": "Blur Method",
+            "paragraphs": ["The method of blur applied to the masked area."]
+        },
+        "compositingCoherencePass": {
+            "heading": "Coherence Pass",
+            "paragraphs": [
+                "A second round of denoising helps to composite the Inpainted/Outpainted image."
+            ]
+        },
+        "compositingCoherenceMode": {
+            "heading": "Mode",
+            "paragraphs": ["The mode of the Coherence Pass."]
+        },
+        "compositingCoherenceSteps": {
+            "heading": "Steps",
+            "paragraphs": [
+                "Number of denoising steps used in the Coherence Pass.",
+                "Same as the main Steps parameter."
+            ]
+        },
+        "compositingStrength": {
+            "heading": "Strength",
+            "paragraphs": [
+                "Denoising strength for the Coherence Pass.",
+                "Same as the Image to Image Denoising Strength parameter."
+            ]
+        },
+        "compositingMaskAdjustments": {
+            "heading": "Mask Adjustments",
+            "paragraphs": ["Adjust the mask."]
+        },
+        "controlNetBeginEnd": {
+            "heading": "Begin / End Step Percentage",
+            "paragraphs": [
+                "Which steps of the denoising process will have the ControlNet applied.",
+                "ControlNets applied at the beginning of the process guide composition, and ControlNets applied at the end guide details."
+            ]
+        },
+        "controlNetControlMode": {
+            "heading": "Control Mode",
+            "paragraphs": [
+                "Lends more weight to either the prompt or ControlNet."
+            ]
+        },
+        "controlNetResizeMode": {
+            "heading": "Resize Mode",
+            "paragraphs": [
+                "How the ControlNet image will be fit to the image output size."
+            ]
+        },
+        "controlNet": {
+            "heading": "ControlNet",
+            "paragraphs": [
+                "ControlNets provide guidance to the generation process, helping create images with controlled composition, structure, or style, depending on the model selected."
+            ]
+        },
+        "controlNetWeight": {
+            "heading": "Weight",
+            "paragraphs": [
+                "How strongly the ControlNet will impact the generated image."
+            ]
+        },
+        "dynamicPrompts": {
+            "heading": "Dynamic Prompts",
+            "paragraphs": [
+                "Dynamic Prompts parses a single prompt into many.",
+                "The basic syntax is \"a {red|green|blue} ball\". This will produce three prompts: \"a red ball\", \"a green ball\" and \"a blue ball\".",
+                "You can use the syntax as many times as you like in a single prompt, but be sure to keep the number of prompts generated in check with the Max Prompts setting."
+            ]
+        },
+        "dynamicPromptsMaxPrompts": {
+            "heading": "Max Prompts",
+            "paragraphs": [
+                "Limits the number of prompts that can be generated by Dynamic Prompts."
+            ]
+        },
+        "dynamicPromptsSeedBehaviour": {
+            "heading": "Seed Behaviour",
+            "paragraphs": [
+                "Controls how the seed is used when generating prompts.",
+                "Per Iteration will use a unique seed for each iteration. Use this to explore prompt variations on a single seed.",
+                "For example, if you have 5 prompts, each image will use the same seed.",
+                "Per Image will use a unique seed for each image. This provides more variation."
+            ]
+        },
+        "infillMethod": {
+            "heading": "Infill Method",
+            "paragraphs": ["Method to infill the selected area."]
+        },
+        "lora": {
+            "heading": "LoRA Weight",
+            "paragraphs": [
+                "Higher LoRA weight will lead to larger impacts on the final image."
+            ]
+        },
+        "noiseUseCPU": {
+            "heading": "Use CPU Noise",
+            "paragraphs": [
+                "Controls whether noise is generated on the CPU or GPU.",
+                "With CPU Noise enabled, a particular seed will produce the same image on any machine.",
+                "There is no performance impact to enabling CPU Noise."
+            ]
+        },
+        "paramCFGScale": {
+            "heading": "CFG Scale",
+            "paragraphs": [
+                "Controls how much your prompt influences the generation process."
+            ]
+        },
+        "paramDenoisingStrength": {
+            "heading": "Denoising Strength",
+            "paragraphs": [
+                "How much noise is added to the input image.",
+                "0 will result in an identical image, while 1 will result in a completely new image."
+            ]
+        },
+        "paramIterations": {
+            "heading": "Iterations",
+            "paragraphs": [
+                "The number of images to generate.",
+                "If Dynamic Prompts is enabled, each of the prompts will be generated this many times."
+            ]
+        },
+        "paramModel": {
+            "heading": "Model",
+            "paragraphs": [
+                "Model used for the denoising steps.",
+                "Different models are typically trained to specialize in producing particular aesthetic results and content."
+            ]
+        },
+        "paramRatio": {
+            "heading": "Aspect Ratio",
+            "paragraphs": [
+                "The aspect ratio of the dimensions of the image generated.",
+                "An image size (in number of pixels) equivalent to 512x512 is recommended for SD1.5 models and a size equivalent to 1024x1024 is recommended for SDXL models."
+            ]
+        },
+        "paramSeed": {
+            "heading": "Seed",
+            "paragraphs": [
+                "Controls the starting noise used for generation.",
+                "Disable “Random Seed” to produce identical results with the same generation settings."
+            ]
+        },
+        "paramSteps": {
+            "heading": "Steps",
+            "paragraphs": [
+                "Number of steps that will be performed in each generation.",
+                "Higher step counts will typically create better images but will require more generation time."
+            ]
+        },
+        "paramVAE": {
+            "heading": "VAE",
+            "paragraphs": [
+                "Model used for translating AI output into the final image."
+            ]
+        },
+        "paramVAEPrecision": {
+            "heading": "VAE Precision",
+            "paragraphs": [
+                "The precision used during VAE encoding and decoding. FP16/half precision is more efficient, at the expense of minor image variations."
+            ]
+        },
+        "scaleBeforeProcessing": {
+            "heading": "Scale Before Processing",
+            "paragraphs": [
+                "Scales the selected area to the size best suited for the model before the image generation process."
+            ]
+        }
+    },
     "ui": {
         "hideProgressImages": "Hide Progress Images",
         "lockRatio": "Lock Ratio",
@@ -1128,6 +1469,8 @@
         "showCanvasDebugInfo": "Show Additional Canvas Info",
         "showGrid": "Show Grid",
         "showHide": "Show/Hide",
+        "showResultsOn": "Show Results (On)",
+        "showResultsOff": "Show Results (Off)",
         "showIntermediates": "Show Intermediates",
         "snapToGrid": "Snap to Grid",
         "undo": "Undo"

invokeai/frontend/web/index.html

@@ -3,6 +3,9 @@
   <head>
     <meta charset="UTF-8" />
     <meta name="viewport" content="width=device-width, initial-scale=1.0" />
+    <meta http-equiv="Cache-Control" content="no-cache, no-store, must-revalidate">
+    <meta http-equiv="Pragma" content="no-cache">
+    <meta http-equiv="Expires" content="0">
     <title>InvokeAI - A Stable Diffusion Toolkit</title>
     <link rel="shortcut icon" type="icon" href="favicon.ico" />
     <style>

invokeai/frontend/web/package.json

@@ -157,6 +157,7 @@
     "prettier": "^3.0.2",
     "rollup-plugin-visualizer": "^5.9.2",
     "ts-toolbelt": "^9.6.0",
+    "typescript": "^5.2.2",
     "vite": "^4.4.9",
     "vite-plugin-css-injected-by-js": "^3.3.0",
     "vite-plugin-dts": "^3.5.2",

invokeai/frontend/web/public/locales/en.json

@@ -49,8 +49,10 @@
         "cancel": "Cancel",
         "close": "Close",
         "communityLabel": "Community",
-        "controlNet": "Controlnet",
+        "controlNet": "ControlNet",
+        "controlAdapter": "Control Adapter",
         "ipAdapter": "IP Adapter",
+        "t2iAdapter": "T2I Adapter",
         "darkMode": "Dark Mode",
         "discordLabel": "Discord",
         "dontAskMeAgain": "Don't ask me again",
@@ -58,6 +60,7 @@
         "githubLabel": "Github",
         "hotkeysLabel": "Hotkeys",
         "imagePrompt": "Image Prompt",
+        "imageFailedToLoad": "Unable to Load Image",
         "img2img": "Image To Image",
         "langArabic": "العربية",
         "langBrPortuguese": "Português do Brasil",
@@ -79,7 +82,7 @@
         "lightMode": "Light Mode",
         "linear": "Linear",
         "load": "Load",
-        "loading": "Loading $t({{noun}})...",
+        "loading": "Loading",
         "loadingInvokeAI": "Loading Invoke AI",
         "learnMore": "Learn More",
         "modelManager": "Model Manager",
@@ -129,6 +132,16 @@
         "upload": "Upload"
     },
     "controlnet": {
+        "controlAdapter": "Control Adapter",
+        "controlnet": "$t(controlnet.controlAdapter) #{{number}} ($t(common.controlNet))",
+        "ip_adapter": "$t(controlnet.controlAdapter) #{{number}} ($t(common.ipAdapter))",
+        "t2i_adapter": "$t(controlnet.controlAdapter) #{{number}} ($t(common.t2iAdapter))",
+        "addControlNet": "Add $t(common.controlNet)",
+        "addIPAdapter": "Add $t(common.ipAdapter)",
+        "addT2IAdapter": "Add $t(common.t2iAdapter)",
+        "controlNetEnabledT2IDisabled": "$t(common.controlNet) enabled, $t(common.t2iAdapter)s disabled",
+        "t2iEnabledControlNetDisabled": "$t(common.t2iAdapter) enabled, $t(common.controlNet)s disabled",
+        "controlNetT2IMutexDesc": "$t(common.controlNet) and $t(common.t2iAdapter) at same time is currently unsupported.",
         "amult": "a_mult",
         "autoConfigure": "Auto configure processor",
         "balanced": "Balanced",
@@ -696,7 +709,7 @@
         "noLoRAsAvailable": "No LoRAs available",
         "noMatchingLoRAs": "No matching LoRAs",
         "noMatchingModels": "No matching Models",
-        "noModelsAvailable": "No Modelss available",
+        "noModelsAvailable": "No models available",
         "selectLoRA": "Select a LoRA",
         "selectModel": "Select a Model"
     },
@@ -716,6 +729,7 @@
         "cannotConnectInputToInput": "Cannot connect input to input",
         "cannotConnectOutputToOutput": "Cannot connect output to output",
         "cannotConnectToSelf": "Cannot connect to self",
+        "cannotDuplicateConnection": "Cannot create duplicate connections",
         "clipField": "Clip",
         "clipFieldDescription": "Tokenizer and text_encoder submodels.",
         "collection": "Collection",
@@ -783,6 +797,14 @@
         "integerPolymorphic": "Integer Polymorphic",
         "integerPolymorphicDescription": "A collection of integers.",
         "invalidOutputSchema": "Invalid output schema",
+        "ipAdapter": "IP-Adapter",
+        "ipAdapterCollection": "IP-Adapters Collection",
+        "ipAdapterCollectionDescription": "A collection of IP-Adapters.",
+        "ipAdapterDescription": "An Image Prompt Adapter (IP-Adapter).",
+        "ipAdapterModel": "IP-Adapter Model",
+        "ipAdapterModelDescription": "IP-Adapter Model Field",
+        "ipAdapterPolymorphic": "IP-Adapter Polymorphic",
+        "ipAdapterPolymorphicDescription": "A collection of IP-Adapters.",
         "latentsCollection": "Latents Collection",
         "latentsCollectionDescription": "Latents may be passed between nodes.",
         "latentsField": "Latents",
@@ -942,9 +964,10 @@
             "missingFieldTemplate": "Missing field template",
             "missingInputForField": "{{nodeLabel}} -> {{fieldLabel}} missing input",
             "missingNodeTemplate": "Missing node template",
-            "noControlImageForControlNet": "ControlNet {{index}} has no control image",
+            "noControlImageForControlAdapter": "Control Adapter {{number}} has no control image",
             "noInitialImageSelected": "No initial image selected",
-            "noModelForControlNet": "ControlNet {{index}} has no model selected.",
+            "noModelForControlAdapter": "Control Adapter {{number}} has no model selected.",
+            "incompatibleBaseModelForControlAdapter": "Control Adapter {{number}} model is invalid with main model.",
             "noModelSelected": "No model selected",
             "noPrompts": "No prompts generated",
             "noNodesInGraph": "No nodes in graph",
@@ -1013,7 +1036,11 @@
         "variationAmount": "Variation Amount",
         "variations": "Variations",
         "vSymmetryStep": "V Symmetry Step",
-        "width": "Width"
+        "width": "Width",
+        "isAllowedToUpscale": {
+            "useX2Model": "Image is too large to upscale with x4 model, use x2 model",
+            "tooLarge": "Image is too large to upscale, select smaller image"
+        }
     },
     "dynamicPrompts": {
         "combinatorial": "Combinatorial Generation",
@@ -1079,7 +1106,8 @@
     },
     "toast": {
         "addedToBoard": "Added to board",
-        "baseModelChangedCleared": "Base model changed, cleared",
+        "baseModelChangedCleared_one": "Base model changed, cleared or disabled {{number}} incompatible submodel",
+        "baseModelChangedCleared_many": "$t(toast.baseModelChangedCleared_one)s",
         "canceled": "Processing Canceled",
         "canvasCopiedClipboard": "Canvas Copied to Clipboard",
         "canvasDownloaded": "Canvas Downloaded",
@@ -1099,7 +1127,6 @@
         "imageSavingFailed": "Image Saving Failed",
         "imageUploaded": "Image Uploaded",
         "imageUploadFailed": "Image Upload Failed",
-        "incompatibleSubmodel": "incompatible submodel",
         "initialImageNotSet": "Initial Image Not Set",
         "initialImageNotSetDesc": "Could not load initial image",
         "initialImageSet": "Initial Image Set",
@@ -1442,6 +1469,8 @@
         "showCanvasDebugInfo": "Show Additional Canvas Info",
         "showGrid": "Show Grid",
         "showHide": "Show/Hide",
+        "showResultsOn": "Show Results (On)",
+        "showResultsOff": "Show Results (Off)",
         "showIntermediates": "Show Intermediates",
         "snapToGrid": "Snap to Grid",
         "undo": "Undo"

invokeai/frontend/web/src/app/components/App.tsx

@@ -1,6 +1,8 @@
 import { Flex, Grid } from '@chakra-ui/react';
+import { useStore } from '@nanostores/react';
 import { useLogger } from 'app/logging/useLogger';
 import { appStarted } from 'app/store/middleware/listenerMiddleware/listeners/appStarted';
+import { $headerComponent } from 'app/store/nanostores/headerComponent';
 import { useAppDispatch, useAppSelector } from 'app/store/storeHooks';
 import { PartialAppConfig } from 'app/types/invokeai';
 import ImageUploader from 'common/components/ImageUploader';
@@ -14,12 +16,10 @@ import i18n from 'i18n';
 import { size } from 'lodash-es';
 import { memo, useCallback, useEffect } from 'react';
 import { ErrorBoundary } from 'react-error-boundary';
-import { usePreselectedImage } from '../../features/parameters/hooks/usePreselectedImage';
 import AppErrorBoundaryFallback from './AppErrorBoundaryFallback';
 import GlobalHotkeys from './GlobalHotkeys';
+import PreselectedImage from './PreselectedImage';
 import Toaster from './Toaster';
-import { useStore } from '@nanostores/react';
-import { $headerComponent } from 'app/store/nanostores/headerComponent';
 
 const DEFAULT_CONFIG = {};
 
@@ -36,8 +36,7 @@ const App = ({ config = DEFAULT_CONFIG, selectedImage }: Props) => {
 
   const logger = useLogger('system');
   const dispatch = useAppDispatch();
-  const { handleSendToCanvas, handleSendToImg2Img, handleUseAllMetadata } =
-    usePreselectedImage(selectedImage?.imageName);
+
   const handleReset = useCallback(() => {
     localStorage.clear();
     location.reload();
@@ -59,24 +58,6 @@ const App = ({ config = DEFAULT_CONFIG, selectedImage }: Props) => {
     dispatch(appStarted());
   }, [dispatch]);
 
-  useEffect(() => {
-    if (selectedImage && selectedImage.action === 'sendToCanvas') {
-      handleSendToCanvas();
-    }
-  }, [selectedImage, handleSendToCanvas]);
-
-  useEffect(() => {
-    if (selectedImage && selectedImage.action === 'sendToImg2Img') {
-      handleSendToImg2Img();
-    }
-  }, [selectedImage, handleSendToImg2Img]);
-
-  useEffect(() => {
-    if (selectedImage && selectedImage.action === 'useAllParameters') {
-      handleUseAllMetadata();
-    }
-  }, [selectedImage, handleUseAllMetadata]);
-
   const headerComponent = useStore($headerComponent);
 
   return (
@@ -112,6 +93,7 @@ const App = ({ config = DEFAULT_CONFIG, selectedImage }: Props) => {
       <ChangeBoardModal />
       <Toaster />
       <GlobalHotkeys />
+      <PreselectedImage selectedImage={selectedImage} />
     </ErrorBoundary>
   );
 };

invokeai/frontend/web/src/app/components/PreselectedImage.tsx

@@ -0,0 +1,16 @@
+import { usePreselectedImage } from 'features/parameters/hooks/usePreselectedImage';
+import { memo } from 'react';
+
+type Props = {
+  selectedImage?: {
+    imageName: string;
+    action: 'sendToImg2Img' | 'sendToCanvas' | 'useAllParameters';
+  };
+};
+
+const PreselectedImage = (props: Props) => {
+  usePreselectedImage(props.selectedImage);
+  return null;
+};
+
+export default memo(PreselectedImage);

invokeai/frontend/web/src/app/components/ThemeLocaleProvider.tsx

@@ -5,7 +5,7 @@ import {
 } from '@chakra-ui/react';
 import { ReactNode, memo, useEffect, useMemo } from 'react';
 import { useTranslation } from 'react-i18next';
-import { theme as invokeAITheme } from 'theme/theme';
+import { TOAST_OPTIONS, theme as invokeAITheme } from 'theme/theme';
 
 import '@fontsource-variable/inter';
 import { MantineProvider } from '@mantine/core';
@@ -39,7 +39,11 @@ function ThemeLocaleProvider({ children }: ThemeLocaleProviderProps) {
 
   return (
     <MantineProvider theme={mantineTheme}>
-      <ChakraProvider theme={theme} colorModeManager={manager}>
+      <ChakraProvider
+        theme={theme}
+        colorModeManager={manager}
+        toastOptions={TOAST_OPTIONS}
+      >
         {children}
       </ChakraProvider>
     </MantineProvider>

invokeai/frontend/web/src/app/store/enhancers/reduxRemember/serialize.ts

@@ -1,5 +1,5 @@
 import { canvasPersistDenylist } from 'features/canvas/store/canvasPersistDenylist';
-import { controlNetDenylist } from 'features/controlNet/store/controlNetDenylist';
+import { controlAdaptersPersistDenylist } from 'features/controlAdapters/store/controlAdaptersPersistDenylist';
 import { dynamicPromptsPersistDenylist } from 'features/dynamicPrompts/store/dynamicPromptsPersistDenylist';
 import { galleryPersistDenylist } from 'features/gallery/store/galleryPersistDenylist';
 import { nodesPersistDenylist } from 'features/nodes/store/nodesPersistDenylist';
@@ -20,7 +20,7 @@ const serializationDenylist: {
   postprocessing: postprocessingPersistDenylist,
   system: systemPersistDenylist,
   ui: uiPersistDenylist,
-  controlNet: controlNetDenylist,
+  controlNet: controlAdaptersPersistDenylist,
   dynamicPrompts: dynamicPromptsPersistDenylist,
 };
 

invokeai/frontend/web/src/app/store/enhancers/reduxRemember/unserialize.ts

@@ -1,5 +1,5 @@
 import { initialCanvasState } from 'features/canvas/store/canvasSlice';
-import { initialControlNetState } from 'features/controlNet/store/controlNetSlice';
+import { initialControlAdapterState } from 'features/controlAdapters/store/controlAdaptersSlice';
 import { initialDynamicPromptsState } from 'features/dynamicPrompts/store/dynamicPromptsSlice';
 import { initialGalleryState } from 'features/gallery/store/gallerySlice';
 import { initialNodesState } from 'features/nodes/store/nodesSlice';
@@ -25,7 +25,7 @@ const initialStates: {
   config: initialConfigState,
   ui: initialUIState,
   hotkeys: initialHotkeysState,
-  controlNet: initialControlNetState,
+  controlAdapters: initialControlAdapterState,
   dynamicPrompts: initialDynamicPromptsState,
   sdxl: initialSDXLState,
 };

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/index.ts

@@ -54,21 +54,6 @@ import { addModelSelectedListener } from './listeners/modelSelected';
 import { addModelsLoadedListener } from './listeners/modelsLoaded';
 import { addDynamicPromptsListener } from './listeners/promptChanged';
 import { addReceivedOpenAPISchemaListener } from './listeners/receivedOpenAPISchema';
-import {
-  addSessionCanceledFulfilledListener,
-  addSessionCanceledPendingListener,
-  addSessionCanceledRejectedListener,
-} from './listeners/sessionCanceled';
-import {
-  addSessionCreatedFulfilledListener,
-  addSessionCreatedPendingListener,
-  addSessionCreatedRejectedListener,
-} from './listeners/sessionCreated';
-import {
-  addSessionInvokedFulfilledListener,
-  addSessionInvokedPendingListener,
-  addSessionInvokedRejectedListener,
-} from './listeners/sessionInvoked';
 import { addSocketConnectedEventListener as addSocketConnectedListener } from './listeners/socketio/socketConnected';
 import { addSocketDisconnectedEventListener as addSocketDisconnectedListener } from './listeners/socketio/socketDisconnected';
 import { addGeneratorProgressEventListener as addGeneratorProgressListener } from './listeners/socketio/socketGeneratorProgress';
@@ -86,6 +71,8 @@ import { addStagingAreaImageSavedListener } from './listeners/stagingAreaImageSa
 import { addTabChangedListener } from './listeners/tabChanged';
 import { addUpscaleRequestedListener } from './listeners/upscaleRequested';
 import { addWorkflowLoadedListener } from './listeners/workflowLoaded';
+import { addBatchEnqueuedListener } from './listeners/batchEnqueued';
+import { addControlAdapterAddedOrEnabledListener } from './listeners/controlAdapterAddedOrEnabled';
 
 export const listenerMiddleware = createListenerMiddleware();
 
@@ -136,6 +123,7 @@ addEnqueueRequestedCanvasListener();
 addEnqueueRequestedNodes();
 addEnqueueRequestedLinear();
 addAnyEnqueuedListener();
+addBatchEnqueuedListener();
 
 // Canvas actions
 addCanvasSavedToGalleryListener();
@@ -175,21 +163,6 @@ addSessionRetrievalErrorEventListener();
 addInvocationRetrievalErrorEventListener();
 addSocketQueueItemStatusChangedEventListener();
 
-// Session Created
-addSessionCreatedPendingListener();
-addSessionCreatedFulfilledListener();
-addSessionCreatedRejectedListener();
-
-// Session Invoked
-addSessionInvokedPendingListener();
-addSessionInvokedFulfilledListener();
-addSessionInvokedRejectedListener();
-
-// Session Canceled
-addSessionCanceledPendingListener();
-addSessionCanceledFulfilledListener();
-addSessionCanceledRejectedListener();
-
 // ControlNet
 addControlNetImageProcessedListener();
 addControlNetAutoProcessListener();
@@ -227,3 +200,7 @@ addTabChangedListener();
 
 // Dynamic prompts
 addDynamicPromptsListener();
+
+// Display toast when controlnet or t2i adapter enabled
+// TODO: Remove when they can both be enabled at same time
+addControlAdapterAddedOrEnabledListener();

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/batchEnqueued.ts

@@ -0,0 +1,101 @@
+import { createStandaloneToast } from '@chakra-ui/react';
+import { logger } from 'app/logging/logger';
+import { parseify } from 'common/util/serialize';
+import { zPydanticValidationError } from 'features/system/store/zodSchemas';
+import { t } from 'i18next';
+import { get, truncate, upperFirst } from 'lodash-es';
+import { queueApi } from 'services/api/endpoints/queue';
+import { TOAST_OPTIONS, theme } from 'theme/theme';
+import { startAppListening } from '..';
+
+const { toast } = createStandaloneToast({
+  theme: theme,
+  defaultOptions: TOAST_OPTIONS.defaultOptions,
+});
+
+export const addBatchEnqueuedListener = () => {
+  // success
+  startAppListening({
+    matcher: queueApi.endpoints.enqueueBatch.matchFulfilled,
+    effect: async (action) => {
+      const response = action.payload;
+      const arg = action.meta.arg.originalArgs;
+      logger('queue').debug(
+        { enqueueResult: parseify(response) },
+        'Batch enqueued'
+      );
+
+      if (!toast.isActive('batch-queued')) {
+        toast({
+          id: 'batch-queued',
+          title: t('queue.batchQueued'),
+          description: t('queue.batchQueuedDesc', {
+            item_count: response.enqueued,
+            direction: arg.prepend ? t('queue.front') : t('queue.back'),
+          }),
+          duration: 1000,
+          status: 'success',
+        });
+      }
+    },
+  });
+
+  // error
+  startAppListening({
+    matcher: queueApi.endpoints.enqueueBatch.matchRejected,
+    effect: async (action) => {
+      const response = action.payload;
+      const arg = action.meta.arg.originalArgs;
+
+      if (!response) {
+        toast({
+          title: t('queue.batchFailedToQueue'),
+          status: 'error',
+          description: 'Unknown Error',
+        });
+        logger('queue').error(
+          { batchConfig: parseify(arg), error: parseify(response) },
+          t('queue.batchFailedToQueue')
+        );
+        return;
+      }
+
+      const result = zPydanticValidationError.safeParse(response);
+      if (result.success) {
+        result.data.data.detail.map((e) => {
+          toast({
+            id: 'batch-failed-to-queue',
+            title: truncate(upperFirst(e.msg), { length: 128 }),
+            status: 'error',
+            description: truncate(
+              `Path:
+              ${e.loc.join('.')}`,
+              { length: 128 }
+            ),
+          });
+        });
+      } else {
+        let detail = 'Unknown Error';
+        let duration = undefined;
+        if (response.status === 403 && 'body' in response) {
+          detail = get(response, 'body.detail', 'Unknown Error');
+        } else if (response.status === 403 && 'error' in response) {
+          detail = get(response, 'error.detail', 'Unknown Error');
+        } else if (response.status === 403 && 'data' in response) {
+          detail = get(response, 'data.detail', 'Unknown Error');
+          duration = 15000;
+        }
+        toast({
+          title: t('queue.batchFailedToQueue'),
+          status: 'error',
+          description: detail,
+          ...(duration ? { duration } : {}),
+        });
+      }
+      logger('queue').error(
+        { batchConfig: parseify(arg), error: parseify(response) },
+        t('queue.batchFailedToQueue')
+      );
+    },
+  });
+};

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/boardAndImagesDeleted.ts

@@ -1,8 +1,5 @@
 import { resetCanvas } from 'features/canvas/store/canvasSlice';
-import {
-  controlNetReset,
-  ipAdapterStateReset,
-} from 'features/controlNet/store/controlNetSlice';
+import { controlAdaptersReset } from 'features/controlAdapters/store/controlAdaptersSlice';
 import { getImageUsage } from 'features/deleteImageModal/store/selectors';
 import { nodeEditorReset } from 'features/nodes/store/nodesSlice';
 import { clearInitialImage } from 'features/parameters/store/generationSlice';
@@ -20,8 +17,7 @@ export const addDeleteBoardAndImagesFulfilledListener = () => {
       let wasInitialImageReset = false;
       let wasCanvasReset = false;
       let wasNodeEditorReset = false;
-      let wasControlNetReset = false;
-      let wasIPAdapterReset = false;
+      let wereControlAdaptersReset = false;
 
       const state = getState();
       deleted_images.forEach((image_name) => {
@@ -42,14 +38,9 @@ export const addDeleteBoardAndImagesFulfilledListener = () => {
           wasNodeEditorReset = true;
         }
 
-        if (imageUsage.isControlNetImage && !wasControlNetReset) {
-          dispatch(controlNetReset());
-          wasControlNetReset = true;
-        }
-
-        if (imageUsage.isIPAdapterImage && !wasIPAdapterReset) {
-          dispatch(ipAdapterStateReset());
-          wasIPAdapterReset = true;
+        if (imageUsage.isControlImage && !wereControlAdaptersReset) {
+          dispatch(controlAdaptersReset());
+          wereControlAdaptersReset = true;
         }
       });
     },

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/boardIdSelected.ts

@@ -25,7 +25,7 @@ export const addBoardIdSelectedListener = () => {
       const state = getState();
 
       const board_id = boardIdSelected.match(action)
-        ? action.payload
+        ? action.payload.boardId
         : state.gallery.selectedBoardId;
 
       const galleryView = galleryViewChanged.match(action)
@@ -55,7 +55,12 @@ export const addBoardIdSelectedListener = () => {
 
         if (boardImagesData) {
           const firstImage = imagesSelectors.selectAll(boardImagesData)[0];
-          dispatch(imageSelected(firstImage ?? null));
+          const selectedImage = imagesSelectors.selectById(
+            boardImagesData,
+            action.payload.selectedImageName
+          );
+
+          dispatch(imageSelected(selectedImage || firstImage || null));
         } else {
           // board has no images - deselect
           dispatch(imageSelected(null));

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/canvasImageToControlNet.ts

@@ -1,22 +1,23 @@
 import { logger } from 'app/logging/logger';
-import { canvasImageToControlNet } from 'features/canvas/store/actions';
 import { getBaseLayerBlob } from 'features/canvas/util/getBaseLayerBlob';
-import { controlNetImageChanged } from 'features/controlNet/store/controlNetSlice';
+import { controlAdapterImageChanged } from 'features/controlAdapters/store/controlAdaptersSlice';
 import { addToast } from 'features/system/store/systemSlice';
+import { t } from 'i18next';
 import { imagesApi } from 'services/api/endpoints/images';
 import { startAppListening } from '..';
-import { t } from 'i18next';
+import { canvasImageToControlAdapter } from 'features/canvas/store/actions';
 
 export const addCanvasImageToControlNetListener = () => {
   startAppListening({
-    actionCreator: canvasImageToControlNet,
+    actionCreator: canvasImageToControlAdapter,
     effect: async (action, { dispatch, getState }) => {
       const log = logger('canvas');
       const state = getState();
+      const { id } = action.payload;
 
-      let blob;
+      let blob: Blob;
       try {
-        blob = await getBaseLayerBlob(state);
+        blob = await getBaseLayerBlob(state, true);
       } catch (err) {
         log.error(String(err));
         dispatch(
@@ -36,10 +37,10 @@ export const addCanvasImageToControlNetListener = () => {
           file: new File([blob], 'savedCanvas.png', {
             type: 'image/png',
           }),
-          image_category: 'mask',
+          image_category: 'control',
           is_intermediate: false,
           board_id: autoAddBoardId === 'none' ? undefined : autoAddBoardId,
-          crop_visible: true,
+          crop_visible: false,
           postUploadAction: {
             type: 'TOAST',
             toastOptions: { title: t('toast.canvasSentControlnetAssets') },
@@ -50,8 +51,8 @@ export const addCanvasImageToControlNetListener = () => {
       const { image_name } = imageDTO;
 
       dispatch(
-        controlNetImageChanged({
-          controlNetId: action.payload.controlNet.controlNetId,
+        controlAdapterImageChanged({
+          id,
           controlImage: image_name,
         })
       );

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/canvasMaskToControlNet.ts

@@ -1,19 +1,19 @@
 import { logger } from 'app/logging/logger';
-import { canvasMaskToControlNet } from 'features/canvas/store/actions';
+import { canvasMaskToControlAdapter } from 'features/canvas/store/actions';
 import { getCanvasData } from 'features/canvas/util/getCanvasData';
-import { controlNetImageChanged } from 'features/controlNet/store/controlNetSlice';
+import { controlAdapterImageChanged } from 'features/controlAdapters/store/controlAdaptersSlice';
 import { addToast } from 'features/system/store/systemSlice';
+import { t } from 'i18next';
 import { imagesApi } from 'services/api/endpoints/images';
 import { startAppListening } from '..';
-import { t } from 'i18next';
 
 export const addCanvasMaskToControlNetListener = () => {
   startAppListening({
-    actionCreator: canvasMaskToControlNet,
+    actionCreator: canvasMaskToControlAdapter,
     effect: async (action, { dispatch, getState }) => {
       const log = logger('canvas');
       const state = getState();
-
+      const { id } = action.payload;
       const canvasBlobsAndImageData = await getCanvasData(
         state.canvas.layerState,
         state.canvas.boundingBoxCoordinates,
@@ -50,7 +50,7 @@ export const addCanvasMaskToControlNetListener = () => {
           image_category: 'mask',
           is_intermediate: false,
           board_id: autoAddBoardId === 'none' ? undefined : autoAddBoardId,
-          crop_visible: true,
+          crop_visible: false,
           postUploadAction: {
             type: 'TOAST',
             toastOptions: { title: t('toast.maskSentControlnetAssets') },
@@ -61,8 +61,8 @@ export const addCanvasMaskToControlNetListener = () => {
       const { image_name } = imageDTO;
 
       dispatch(
-        controlNetImageChanged({
-          controlNetId: action.payload.controlNet.controlNetId,
+        controlAdapterImageChanged({
+          id,
           controlImage: image_name,
         })
       );

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/controlAdapterAddedOrEnabled.ts

@@ -0,0 +1,87 @@
+import { isAnyOf } from '@reduxjs/toolkit';
+import {
+  controlAdapterAdded,
+  controlAdapterAddedFromImage,
+  controlAdapterIsEnabledChanged,
+  controlAdapterRecalled,
+  selectControlAdapterAll,
+  selectControlAdapterById,
+} from 'features/controlAdapters/store/controlAdaptersSlice';
+import { ControlAdapterType } from 'features/controlAdapters/store/types';
+import { addToast } from 'features/system/store/systemSlice';
+import i18n from 'i18n';
+import { startAppListening } from '..';
+
+const isAnyControlAdapterAddedOrEnabled = isAnyOf(
+  controlAdapterAdded,
+  controlAdapterAddedFromImage,
+  controlAdapterRecalled,
+  controlAdapterIsEnabledChanged
+);
+
+/**
+ * Until we can have both controlnet and t2i adapter enabled at once, they are mutually exclusive
+ * This displays a toast when one is enabled and the other is already enabled, or one is added
+ * with the other enabled
+ */
+export const addControlAdapterAddedOrEnabledListener = () => {
+  startAppListening({
+    matcher: isAnyControlAdapterAddedOrEnabled,
+    effect: async (action, { dispatch, getOriginalState }) => {
+      const controlAdapters = getOriginalState().controlAdapters;
+
+      const hasEnabledControlNets = selectControlAdapterAll(
+        controlAdapters
+      ).some((ca) => ca.isEnabled && ca.type === 'controlnet');
+
+      const hasEnabledT2IAdapters = selectControlAdapterAll(
+        controlAdapters
+      ).some((ca) => ca.isEnabled && ca.type === 't2i_adapter');
+
+      let caType: ControlAdapterType | null = null;
+
+      if (controlAdapterAdded.match(action)) {
+        caType = action.payload.type;
+      }
+
+      if (controlAdapterAddedFromImage.match(action)) {
+        caType = action.payload.type;
+      }
+
+      if (controlAdapterRecalled.match(action)) {
+        caType = action.payload.type;
+      }
+
+      if (controlAdapterIsEnabledChanged.match(action)) {
+        const _caType = selectControlAdapterById(
+          controlAdapters,
+          action.payload.id
+        )?.type;
+        if (!_caType) {
+          return;
+        }
+        caType = _caType;
+      }
+
+      if (
+        (caType === 'controlnet' && hasEnabledT2IAdapters) ||
+        (caType === 't2i_adapter' && hasEnabledControlNets)
+      ) {
+        const title =
+          caType === 'controlnet'
+            ? i18n.t('controlnet.controlNetEnabledT2IDisabled')
+            : i18n.t('controlnet.t2iEnabledControlNetDisabled');
+
+        const description = i18n.t('controlnet.controlNetT2IMutexDesc');
+
+        dispatch(
+          addToast({
+            title,
+            description,
+            status: 'warning',
+          })
+        );
+      }
+    },
+  });
+};

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/controlNetAutoProcess.ts

@@ -1,15 +1,24 @@
 import { AnyListenerPredicate } from '@reduxjs/toolkit';
 import { logger } from 'app/logging/logger';
 import { RootState } from 'app/store/store';
-import { controlNetImageProcessed } from 'features/controlNet/store/actions';
+import { controlAdapterImageProcessed } from 'features/controlAdapters/store/actions';
 import {
-  controlNetAutoConfigToggled,
-  controlNetImageChanged,
-  controlNetModelChanged,
-  controlNetProcessorParamsChanged,
-  controlNetProcessorTypeChanged,
-} from 'features/controlNet/store/controlNetSlice';
+  controlAdapterAutoConfigToggled,
+  controlAdapterImageChanged,
+  controlAdapterModelChanged,
+  controlAdapterProcessorParamsChanged,
+  controlAdapterProcessortTypeChanged,
+  selectControlAdapterById,
+} from 'features/controlAdapters/store/controlAdaptersSlice';
 import { startAppListening } from '..';
+import { isControlNetOrT2IAdapter } from 'features/controlAdapters/store/types';
+
+type AnyControlAdapterParamChangeAction =
+  | ReturnType<typeof controlAdapterProcessorParamsChanged>
+  | ReturnType<typeof controlAdapterModelChanged>
+  | ReturnType<typeof controlAdapterImageChanged>
+  | ReturnType<typeof controlAdapterProcessortTypeChanged>
+  | ReturnType<typeof controlAdapterAutoConfigToggled>;
 
 const predicate: AnyListenerPredicate<RootState> = (
   action,
@@ -17,35 +26,37 @@ const predicate: AnyListenerPredicate<RootState> = (
   prevState
 ) => {
   const isActionMatched =
-    controlNetProcessorParamsChanged.match(action) ||
-    controlNetModelChanged.match(action) ||
-    controlNetImageChanged.match(action) ||
-    controlNetProcessorTypeChanged.match(action) ||
-    controlNetAutoConfigToggled.match(action);
+    controlAdapterProcessorParamsChanged.match(action) ||
+    controlAdapterModelChanged.match(action) ||
+    controlAdapterImageChanged.match(action) ||
+    controlAdapterProcessortTypeChanged.match(action) ||
+    controlAdapterAutoConfigToggled.match(action);
 
   if (!isActionMatched) {
     return false;
   }
 
-  if (controlNetAutoConfigToggled.match(action)) {
+  const { id } = action.payload;
+  const prevCA = selectControlAdapterById(prevState.controlAdapters, id);
+  const ca = selectControlAdapterById(state.controlAdapters, id);
+  if (
+    !prevCA ||
+    !isControlNetOrT2IAdapter(prevCA) ||
+    !ca ||
+    !isControlNetOrT2IAdapter(ca)
+  ) {
+    return false;
+  }
+
+  if (controlAdapterAutoConfigToggled.match(action)) {
     // do not process if the user just disabled auto-config
-    if (
-      prevState.controlNet.controlNets[action.payload.controlNetId]
-        ?.shouldAutoConfig === true
-    ) {
+    if (prevCA.shouldAutoConfig === true) {
       return false;
     }
   }
 
-  const cn = state.controlNet.controlNets[action.payload.controlNetId];
-
-  if (!cn) {
-    // something is wrong, the controlNet should exist
-    return false;
-  }
-
-  const { controlImage, processorType, shouldAutoConfig } = cn;
-  if (controlNetModelChanged.match(action) && !shouldAutoConfig) {
+  const { controlImage, processorType, shouldAutoConfig } = ca;
+  if (controlAdapterModelChanged.match(action) && !shouldAutoConfig) {
     // do not process if the action is a model change but the processor settings are dirty
     return false;
   }
@@ -67,7 +78,7 @@ export const addControlNetAutoProcessListener = () => {
     predicate,
     effect: async (action, { dispatch, cancelActiveListeners, delay }) => {
       const log = logger('session');
-      const { controlNetId } = action.payload;
+      const { id } = (action as AnyControlAdapterParamChangeAction).payload;
 
       // Cancel any in-progress instances of this listener
       cancelActiveListeners();
@@ -75,7 +86,7 @@ export const addControlNetAutoProcessListener = () => {
       // Delay before starting actual work
       await delay(300);
 
-      dispatch(controlNetImageProcessed({ controlNetId }));
+      dispatch(controlAdapterImageProcessed({ id }));
     },
   });
 };

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/controlNetImageProcessed.ts

@@ -1,7 +1,11 @@
 import { logger } from 'app/logging/logger';
 import { parseify } from 'common/util/serialize';
-import { controlNetImageProcessed } from 'features/controlNet/store/actions';
-import { controlNetProcessedImageChanged } from 'features/controlNet/store/controlNetSlice';
+import {
+  pendingControlImagesCleared,
+  controlAdapterImageChanged,
+  selectControlAdapterById,
+  controlAdapterProcessedImageChanged,
+} from 'features/controlAdapters/store/controlAdaptersSlice';
 import { SAVE_IMAGE } from 'features/nodes/util/graphBuilders/constants';
 import { addToast } from 'features/system/store/systemSlice';
 import { t } from 'i18next';
@@ -11,28 +15,34 @@ import { isImageOutput } from 'services/api/guards';
 import { Graph, ImageDTO } from 'services/api/types';
 import { socketInvocationComplete } from 'services/events/actions';
 import { startAppListening } from '..';
+import { controlAdapterImageProcessed } from 'features/controlAdapters/store/actions';
+import { isControlNetOrT2IAdapter } from 'features/controlAdapters/store/types';
 
 export const addControlNetImageProcessedListener = () => {
   startAppListening({
-    actionCreator: controlNetImageProcessed,
+    actionCreator: controlAdapterImageProcessed,
     effect: async (action, { dispatch, getState, take }) => {
       const log = logger('session');
-      const { controlNetId } = action.payload;
-      const controlNet = getState().controlNet.controlNets[controlNetId];
+      const { id } = action.payload;
+      const ca = selectControlAdapterById(getState().controlAdapters, id);
 
-      if (!controlNet?.controlImage) {
+      if (!ca?.controlImage || !isControlNetOrT2IAdapter(ca)) {
         log.error('Unable to process ControlNet image');
         return;
       }
 
+      if (ca.processorType === 'none' || ca.processorNode.type === 'none') {
+        return;
+      }
+
       // ControlNet one-off procressing graph is just the processor node, no edges.
       // Also we need to grab the image.
       const graph: Graph = {
         nodes: {
-          [controlNet.processorNode.id]: {
-            ...controlNet.processorNode,
+          [ca.processorNode.id]: {
+            ...ca.processorNode,
             is_intermediate: true,
-            image: { image_name: controlNet.controlImage },
+            image: { image_name: ca.controlImage },
           },
           [SAVE_IMAGE]: {
             id: SAVE_IMAGE,
@@ -44,7 +54,7 @@ export const addControlNetImageProcessedListener = () => {
         edges: [
           {
             source: {
-              node_id: controlNet.processorNode.id,
+              node_id: ca.processorNode.id,
               field: 'image',
             },
             destination: {
@@ -99,14 +109,36 @@ export const addControlNetImageProcessedListener = () => {
 
           // Update the processed image in the store
           dispatch(
-            controlNetProcessedImageChanged({
-              controlNetId,
+            controlAdapterProcessedImageChanged({
+              id,
               processedControlImage: processedControlImage.image_name,
             })
           );
         }
-      } catch {
+      } catch (error) {
         log.error({ graph: parseify(graph) }, t('queue.graphFailedToQueue'));
+
+        // handle usage-related errors
+        if (error instanceof Object) {
+          if ('data' in error && 'status' in error) {
+            if (error.status === 403) {
+              // eslint-disable-next-line @typescript-eslint/no-explicit-any
+              const detail = (error.data as any)?.detail || 'Unknown Error';
+              dispatch(
+                addToast({
+                  title: t('queue.graphFailedToQueue'),
+                  status: 'error',
+                  description: detail,
+                  duration: 15000,
+                })
+              );
+              dispatch(pendingControlImagesCleared());
+              dispatch(controlAdapterImageChanged({ id, controlImage: null }));
+              return;
+            }
+          }
+        }
+
         dispatch(
           addToast({
             title: t('queue.graphFailedToQueue'),

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/enqueueRequestedCanvas.ts

@@ -12,8 +12,6 @@ import { getCanvasGenerationMode } from 'features/canvas/util/getCanvasGeneratio
 import { canvasGraphBuilt } from 'features/nodes/store/actions';
 import { buildCanvasGraph } from 'features/nodes/util/graphBuilders/buildCanvasGraph';
 import { prepareLinearUIBatch } from 'features/nodes/util/graphBuilders/buildLinearBatchConfig';
-import { addToast } from 'features/system/store/systemSlice';
-import { t } from 'i18next';
 import { imagesApi } from 'services/api/endpoints/images';
 import { queueApi } from 'services/api/endpoints/queue';
 import { ImageDTO } from 'services/api/types';
@@ -140,8 +138,6 @@ export const addEnqueueRequestedCanvasListener = () => {
         const enqueueResult = await req.unwrap();
         req.reset();
 
-        log.debug({ enqueueResult: parseify(enqueueResult) }, 'Batch enqueued');
-
         const batchId = enqueueResult.batch.batch_id as string; // we know the is a string, backend provides it
 
         // Prep the canvas staging area if it is not yet initialized
@@ -158,28 +154,8 @@ export const addEnqueueRequestedCanvasListener = () => {
 
         // Associate the session with the canvas session ID
         dispatch(canvasBatchIdAdded(batchId));
-
-        dispatch(
-          addToast({
-            title: t('queue.batchQueued'),
-            description: t('queue.batchQueuedDesc', {
-              item_count: enqueueResult.enqueued,
-              direction: prepend ? t('queue.front') : t('queue.back'),
-            }),
-            status: 'success',
-          })
-        );
       } catch {
-        log.error(
-          { batchConfig: parseify(batchConfig) },
-          t('queue.batchFailedToQueue')
-        );
-        dispatch(
-          addToast({
-            title: t('queue.batchFailedToQueue'),
-            status: 'error',
-          })
-        );
+        // no-op
       }
     },
   });

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/enqueueRequestedLinear.ts

@@ -1,13 +1,9 @@
-import { logger } from 'app/logging/logger';
 import { enqueueRequested } from 'app/store/actions';
-import { parseify } from 'common/util/serialize';
 import { prepareLinearUIBatch } from 'features/nodes/util/graphBuilders/buildLinearBatchConfig';
 import { buildLinearImageToImageGraph } from 'features/nodes/util/graphBuilders/buildLinearImageToImageGraph';
 import { buildLinearSDXLImageToImageGraph } from 'features/nodes/util/graphBuilders/buildLinearSDXLImageToImageGraph';
 import { buildLinearSDXLTextToImageGraph } from 'features/nodes/util/graphBuilders/buildLinearSDXLTextToImageGraph';
 import { buildLinearTextToImageGraph } from 'features/nodes/util/graphBuilders/buildLinearTextToImageGraph';
-import { addToast } from 'features/system/store/systemSlice';
-import { t } from 'i18next';
 import { queueApi } from 'services/api/endpoints/queue';
 import { startAppListening } from '..';
 
@@ -18,7 +14,6 @@ export const addEnqueueRequestedLinear = () => {
       (action.payload.tabName === 'txt2img' ||
         action.payload.tabName === 'img2img'),
     effect: async (action, { getState, dispatch }) => {
-      const log = logger('queue');
       const state = getState();
       const model = state.generation.model;
       const { prepend } = action.payload;
@@ -41,38 +36,12 @@ export const addEnqueueRequestedLinear = () => {
 
       const batchConfig = prepareLinearUIBatch(state, graph, prepend);
 
-      try {
-        const req = dispatch(
-          queueApi.endpoints.enqueueBatch.initiate(batchConfig, {
-            fixedCacheKey: 'enqueueBatch',
-          })
-        );
-        const enqueueResult = await req.unwrap();
-        req.reset();
-
-        log.debug({ enqueueResult: parseify(enqueueResult) }, 'Batch enqueued');
-        dispatch(
-          addToast({
-            title: t('queue.batchQueued'),
-            description: t('queue.batchQueuedDesc', {
-              item_count: enqueueResult.enqueued,
-              direction: prepend ? t('queue.front') : t('queue.back'),
-            }),
-            status: 'success',
-          })
-        );
-      } catch {
-        log.error(
-          { batchConfig: parseify(batchConfig) },
-          t('queue.batchFailedToQueue')
-        );
-        dispatch(
-          addToast({
-            title: t('queue.batchFailedToQueue'),
-            status: 'error',
-          })
-        );
-      }
+      const req = dispatch(
+        queueApi.endpoints.enqueueBatch.initiate(batchConfig, {
+          fixedCacheKey: 'enqueueBatch',
+        })
+      );
+      req.reset();
     },
   });
 };

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/enqueueRequestedNodes.ts

@@ -1,9 +1,5 @@
-import { logger } from 'app/logging/logger';
 import { enqueueRequested } from 'app/store/actions';
-import { parseify } from 'common/util/serialize';
 import { buildNodesGraph } from 'features/nodes/util/graphBuilders/buildNodesGraph';
-import { addToast } from 'features/system/store/systemSlice';
-import { t } from 'i18next';
 import { queueApi } from 'services/api/endpoints/queue';
 import { BatchConfig } from 'services/api/types';
 import { startAppListening } from '..';
@@ -13,9 +9,7 @@ export const addEnqueueRequestedNodes = () => {
     predicate: (action): action is ReturnType<typeof enqueueRequested> =>
       enqueueRequested.match(action) && action.payload.tabName === 'nodes',
     effect: async (action, { getState, dispatch }) => {
-      const log = logger('queue');
       const state = getState();
-      const { prepend } = action.payload;
       const graph = buildNodesGraph(state.nodes);
       const batchConfig: BatchConfig = {
         batch: {
@@ -25,38 +19,12 @@ export const addEnqueueRequestedNodes = () => {
         prepend: action.payload.prepend,
       };
 
-      try {
-        const req = dispatch(
-          queueApi.endpoints.enqueueBatch.initiate(batchConfig, {
-            fixedCacheKey: 'enqueueBatch',
-          })
-        );
-        const enqueueResult = await req.unwrap();
-        req.reset();
-
-        log.debug({ enqueueResult: parseify(enqueueResult) }, 'Batch enqueued');
-        dispatch(
-          addToast({
-            title: t('queue.batchQueued'),
-            description: t('queue.batchQueuedDesc', {
-              item_count: enqueueResult.enqueued,
-              direction: prepend ? t('queue.front') : t('queue.back'),
-            }),
-            status: 'success',
-          })
-        );
-      } catch {
-        log.error(
-          { batchConfig: parseify(batchConfig) },
-          'Failed to enqueue batch'
-        );
-        dispatch(
-          addToast({
-            title: t('queue.batchFailedToQueue'),
-            status: 'error',
-          })
-        );
-      }
+      const req = dispatch(
+        queueApi.endpoints.enqueueBatch.initiate(batchConfig, {
+          fixedCacheKey: 'enqueueBatch',
+        })
+      );
+      req.reset();
     },
   });
 };

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/imageDeleted.ts

@@ -1,10 +1,10 @@
 import { logger } from 'app/logging/logger';
 import { resetCanvas } from 'features/canvas/store/canvasSlice';
 import {
-  controlNetImageChanged,
-  controlNetProcessedImageChanged,
-  ipAdapterImageChanged,
-} from 'features/controlNet/store/controlNetSlice';
+  controlAdapterImageChanged,
+  controlAdapterProcessedImageChanged,
+  selectControlAdapterAll,
+} from 'features/controlAdapters/store/controlAdaptersSlice';
 import { imageDeletionConfirmed } from 'features/deleteImageModal/store/actions';
 import { isModalOpenChanged } from 'features/deleteImageModal/store/slice';
 import { selectListImagesBaseQueryArgs } from 'features/gallery/store/gallerySelectors';
@@ -17,6 +17,7 @@ import { api } from 'services/api';
 import { imagesApi } from 'services/api/endpoints/images';
 import { imagesAdapter } from 'services/api/util';
 import { startAppListening } from '..';
+import { isControlNetOrT2IAdapter } from 'features/controlAdapters/store/types';
 
 export const addRequestedSingleImageDeletionListener = () => {
   startAppListening({
@@ -90,35 +91,28 @@ export const addRequestedSingleImageDeletionListener = () => {
           dispatch(clearInitialImage());
         }
 
-        // reset controlNets that use the deleted images
-        forEach(getState().controlNet.controlNets, (controlNet) => {
+        // reset control adapters that use the deleted images
+        forEach(selectControlAdapterAll(getState().controlAdapters), (ca) => {
           if (
-            controlNet.controlImage === imageDTO.image_name ||
-            controlNet.processedControlImage === imageDTO.image_name
+            ca.controlImage === imageDTO.image_name ||
+            (isControlNetOrT2IAdapter(ca) &&
+              ca.processedControlImage === imageDTO.image_name)
           ) {
             dispatch(
-              controlNetImageChanged({
-                controlNetId: controlNet.controlNetId,
+              controlAdapterImageChanged({
+                id: ca.id,
                 controlImage: null,
               })
             );
             dispatch(
-              controlNetProcessedImageChanged({
-                controlNetId: controlNet.controlNetId,
+              controlAdapterProcessedImageChanged({
+                id: ca.id,
                 processedControlImage: null,
               })
             );
           }
         });
 
-        // Remove IP Adapter Set Image if image is deleted.
-        if (
-          getState().controlNet.ipAdapterInfo.adapterImage?.image_name ===
-          imageDTO.image_name
-        ) {
-          dispatch(ipAdapterImageChanged(null));
-        }
-
         // reset nodes that use the deleted images
         getState().nodes.nodes.forEach((node) => {
           if (!isInvocationNode(node)) {
@@ -215,35 +209,28 @@ export const addRequestedMultipleImageDeletionListener = () => {
             dispatch(clearInitialImage());
           }
 
-          // reset controlNets that use the deleted images
-          forEach(getState().controlNet.controlNets, (controlNet) => {
+          // reset control adapters that use the deleted images
+          forEach(selectControlAdapterAll(getState().controlAdapters), (ca) => {
             if (
-              controlNet.controlImage === imageDTO.image_name ||
-              controlNet.processedControlImage === imageDTO.image_name
+              ca.controlImage === imageDTO.image_name ||
+              (isControlNetOrT2IAdapter(ca) &&
+                ca.processedControlImage === imageDTO.image_name)
             ) {
               dispatch(
-                controlNetImageChanged({
-                  controlNetId: controlNet.controlNetId,
+                controlAdapterImageChanged({
+                  id: ca.id,
                   controlImage: null,
                 })
               );
               dispatch(
-                controlNetProcessedImageChanged({
-                  controlNetId: controlNet.controlNetId,
+                controlAdapterProcessedImageChanged({
+                  id: ca.id,
                   processedControlImage: null,
                 })
               );
             }
           });
 
-          // Remove IP Adapter Set Image if image is deleted.
-          if (
-            getState().controlNet.ipAdapterInfo.adapterImage?.image_name ===
-            imageDTO.image_name
-          ) {
-            dispatch(ipAdapterImageChanged(null));
-          }
-
           // reset nodes that use the deleted images
           getState().nodes.nodes.forEach((node) => {
             if (!isInvocationNode(node)) {

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/imageDropped.ts

@@ -3,11 +3,9 @@ import { logger } from 'app/logging/logger';
 import { parseify } from 'common/util/serialize';
 import { setInitialCanvasImage } from 'features/canvas/store/canvasSlice';
 import {
-  controlNetImageChanged,
-  controlNetIsEnabledChanged,
-  ipAdapterImageChanged,
-  isIPAdapterEnabledChanged,
-} from 'features/controlNet/store/controlNetSlice';
+  controlAdapterImageChanged,
+  controlAdapterIsEnabledChanged,
+} from 'features/controlAdapters/store/controlAdaptersSlice';
 import {
   TypesafeDraggableData,
   TypesafeDroppableData,
@@ -90,39 +88,26 @@ export const addImageDroppedListener = () => {
        * Image dropped on ControlNet
        */
       if (
-        overData.actionType === 'SET_CONTROLNET_IMAGE' &&
+        overData.actionType === 'SET_CONTROL_ADAPTER_IMAGE' &&
         activeData.payloadType === 'IMAGE_DTO' &&
         activeData.payload.imageDTO
       ) {
-        const { controlNetId } = overData.context;
+        const { id } = overData.context;
         dispatch(
-          controlNetImageChanged({
+          controlAdapterImageChanged({
+            id,
             controlImage: activeData.payload.imageDTO.image_name,
-            controlNetId,
           })
         );
         dispatch(
-          controlNetIsEnabledChanged({
-            controlNetId,
+          controlAdapterIsEnabledChanged({
+            id,
             isEnabled: true,
           })
         );
         return;
       }
 
-      /**
-       * Image dropped on IP Adapter image
-       */
-      if (
-        overData.actionType === 'SET_IP_ADAPTER_IMAGE' &&
-        activeData.payloadType === 'IMAGE_DTO' &&
-        activeData.payload.imageDTO
-      ) {
-        dispatch(ipAdapterImageChanged(activeData.payload.imageDTO));
-        dispatch(isIPAdapterEnabledChanged(true));
-        return;
-      }
-
       /**
        * Image dropped on Canvas
        */

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/imageToDeleteSelected.ts

@@ -18,8 +18,7 @@ export const addImageToDeleteSelectedListener = () => {
       const isImageInUse =
         imagesUsage.some((i) => i.isCanvasImage) ||
         imagesUsage.some((i) => i.isInitialImage) ||
-        imagesUsage.some((i) => i.isControlNetImage) ||
-        imagesUsage.some((i) => i.isIPAdapterImage) ||
+        imagesUsage.some((i) => i.isControlImage) ||
         imagesUsage.some((i) => i.isNodesImage);
 
       if (shouldConfirmOnDelete || isImageInUse) {

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/imageUploaded.ts

@@ -2,11 +2,9 @@ import { UseToastOptions } from '@chakra-ui/react';
 import { logger } from 'app/logging/logger';
 import { setInitialCanvasImage } from 'features/canvas/store/canvasSlice';
 import {
-  controlNetImageChanged,
-  controlNetIsEnabledChanged,
-  ipAdapterImageChanged,
-  isIPAdapterEnabledChanged,
-} from 'features/controlNet/store/controlNetSlice';
+  controlAdapterImageChanged,
+  controlAdapterIsEnabledChanged,
+} from 'features/controlAdapters/store/controlAdaptersSlice';
 import { fieldImageValueChanged } from 'features/nodes/store/nodesSlice';
 import { initialImageChanged } from 'features/parameters/store/generationSlice';
 import { addToast } from 'features/system/store/systemSlice';
@@ -16,11 +14,6 @@ import { boardsApi } from 'services/api/endpoints/boards';
 import { startAppListening } from '..';
 import { imagesApi } from '../../../../../services/api/endpoints/images';
 
-const DEFAULT_UPLOADED_TOAST: UseToastOptions = {
-  title: t('toast.imageUploaded'),
-  status: 'success',
-};
-
 export const addImageUploadedFulfilledListener = () => {
   startAppListening({
     matcher: imagesApi.endpoints.uploadImage.matchFulfilled,
@@ -43,6 +36,11 @@ export const addImageUploadedFulfilledListener = () => {
         return;
       }
 
+      const DEFAULT_UPLOADED_TOAST: UseToastOptions = {
+        title: t('toast.imageUploaded'),
+        status: 'success',
+      };
+
       // default action - just upload and alert user
       if (postUploadAction?.type === 'TOAST') {
         const { toastOptions } = postUploadAction;
@@ -87,17 +85,17 @@ export const addImageUploadedFulfilledListener = () => {
         return;
       }
 
-      if (postUploadAction?.type === 'SET_CONTROLNET_IMAGE') {
-        const { controlNetId } = postUploadAction;
+      if (postUploadAction?.type === 'SET_CONTROL_ADAPTER_IMAGE') {
+        const { id } = postUploadAction;
         dispatch(
-          controlNetIsEnabledChanged({
-            controlNetId,
+          controlAdapterIsEnabledChanged({
+            id,
             isEnabled: true,
           })
         );
         dispatch(
-          controlNetImageChanged({
-            controlNetId,
+          controlAdapterImageChanged({
+            id,
             controlImage: imageDTO.image_name,
           })
         );
@@ -110,18 +108,6 @@ export const addImageUploadedFulfilledListener = () => {
         return;
       }
 
-      if (postUploadAction?.type === 'SET_IP_ADAPTER_IMAGE') {
-        dispatch(ipAdapterImageChanged(imageDTO));
-        dispatch(isIPAdapterEnabledChanged(true));
-        dispatch(
-          addToast({
-            ...DEFAULT_UPLOADED_TOAST,
-            description: t('toast.setIPAdapterImage'),
-          })
-        );
-        return;
-      }
-
       if (postUploadAction?.type === 'SET_INITIAL_IMAGE') {
         dispatch(initialImageChanged(imageDTO));
         dispatch(

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/modelSelected.ts

@@ -1,9 +1,9 @@
 import { logger } from 'app/logging/logger';
 import { setBoundingBoxDimensions } from 'features/canvas/store/canvasSlice';
 import {
-  controlNetRemoved,
-  ipAdapterStateReset,
-} from 'features/controlNet/store/controlNetSlice';
+  controlAdapterIsEnabledChanged,
+  selectControlAdapterAll,
+} from 'features/controlAdapters/store/controlAdaptersSlice';
 import { loraRemoved } from 'features/lora/store/loraSlice';
 import { modelSelected } from 'features/parameters/store/actions';
 import {
@@ -15,9 +15,9 @@ import {
 import { zMainOrOnnxModel } from 'features/parameters/types/parameterSchemas';
 import { addToast } from 'features/system/store/systemSlice';
 import { makeToast } from 'features/system/util/makeToast';
+import { t } from 'i18next';
 import { forEach } from 'lodash-es';
 import { startAppListening } from '..';
-import { t } from 'i18next';
 
 export const addModelSelectedListener = () => {
   startAppListening({
@@ -60,33 +60,27 @@ export const addModelSelectedListener = () => {
         }
 
         // handle incompatible controlnets
-        const { controlNets } = state.controlNet;
-        forEach(controlNets, (controlNet, controlNetId) => {
-          if (controlNet.model?.base_model !== base_model) {
-            dispatch(controlNetRemoved({ controlNetId }));
+        selectControlAdapterAll(state.controlAdapters).forEach((ca) => {
+          if (ca.model?.base_model !== base_model) {
+            dispatch(
+              controlAdapterIsEnabledChanged({ id: ca.id, isEnabled: false })
+            );
             modelsCleared += 1;
           }
         });
 
-        // handle incompatible IP-Adapter
-        const { ipAdapterInfo } = state.controlNet;
-        if (
-          ipAdapterInfo.model &&
-          ipAdapterInfo.model.base_model !== base_model
-        ) {
-          dispatch(ipAdapterStateReset());
-          modelsCleared += 1;
-        }
-
         if (modelsCleared > 0) {
           dispatch(
             addToast(
               makeToast({
-                title: `${t(
-                  'toast.baseModelChangedCleared'
-                )} ${modelsCleared} ${t('toast.incompatibleSubmodel')}${
-                  modelsCleared === 1 ? '' : 's'
-                }`,
+                title: t(
+                  modelsCleared === 1
+                    ? 'toast.baseModelChangedCleared_one'
+                    : 'toast.baseModelChangedCleared_many',
+                  {
+                    number: modelsCleared,
+                  }
+                ),
                 status: 'warning',
               })
             )

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/modelsLoaded.ts

@@ -1,5 +1,10 @@
 import { logger } from 'app/logging/logger';
-import { controlNetRemoved } from 'features/controlNet/store/controlNetSlice';
+import {
+  controlAdapterModelCleared,
+  selectAllControlNets,
+  selectAllIPAdapters,
+  selectAllT2IAdapters,
+} from 'features/controlAdapters/store/controlAdaptersSlice';
 import { loraRemoved } from 'features/lora/store/loraSlice';
 import {
   modelChanged,
@@ -216,21 +221,71 @@ export const addModelsLoadedListener = () => {
         `ControlNet models loaded (${action.payload.ids.length})`
       );
 
-      const controlNets = getState().controlNet.controlNets;
-
-      forEach(controlNets, (controlNet, controlNetId) => {
-        const isControlNetAvailable = some(
+      selectAllControlNets(getState().controlAdapters).forEach((ca) => {
+        const isModelAvailable = some(
           action.payload.entities,
           (m) =>
-            m?.model_name === controlNet?.model?.model_name &&
-            m?.base_model === controlNet?.model?.base_model
+            m?.model_name === ca?.model?.model_name &&
+            m?.base_model === ca?.model?.base_model
         );
 
-        if (isControlNetAvailable) {
+        if (isModelAvailable) {
           return;
         }
 
-        dispatch(controlNetRemoved({ controlNetId }));
+        dispatch(controlAdapterModelCleared({ id: ca.id }));
+      });
+    },
+  });
+  startAppListening({
+    matcher: modelsApi.endpoints.getT2IAdapterModels.matchFulfilled,
+    effect: async (action, { getState, dispatch }) => {
+      // ControlNet models loaded - need to remove missing ControlNets from state
+      const log = logger('models');
+      log.info(
+        { models: action.payload.entities },
+        `ControlNet models loaded (${action.payload.ids.length})`
+      );
+
+      selectAllT2IAdapters(getState().controlAdapters).forEach((ca) => {
+        const isModelAvailable = some(
+          action.payload.entities,
+          (m) =>
+            m?.model_name === ca?.model?.model_name &&
+            m?.base_model === ca?.model?.base_model
+        );
+
+        if (isModelAvailable) {
+          return;
+        }
+
+        dispatch(controlAdapterModelCleared({ id: ca.id }));
+      });
+    },
+  });
+  startAppListening({
+    matcher: modelsApi.endpoints.getIPAdapterModels.matchFulfilled,
+    effect: async (action, { getState, dispatch }) => {
+      // ControlNet models loaded - need to remove missing ControlNets from state
+      const log = logger('models');
+      log.info(
+        { models: action.payload.entities },
+        `IP Adapter models loaded (${action.payload.ids.length})`
+      );
+
+      selectAllIPAdapters(getState().controlAdapters).forEach((ca) => {
+        const isModelAvailable = some(
+          action.payload.entities,
+          (m) =>
+            m?.model_name === ca?.model?.model_name &&
+            m?.base_model === ca?.model?.base_model
+        );
+
+        if (isModelAvailable) {
+          return;
+        }
+
+        dispatch(controlAdapterModelCleared({ id: ca.id }));
       });
     },
   });

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/sessionCanceled.ts

@@ -1,44 +0,0 @@
-import { logger } from 'app/logging/logger';
-import { serializeError } from 'serialize-error';
-import { sessionCanceled } from 'services/api/thunks/session';
-import { startAppListening } from '..';
-
-export const addSessionCanceledPendingListener = () => {
-  startAppListening({
-    actionCreator: sessionCanceled.pending,
-    effect: () => {
-      //
-    },
-  });
-};
-
-export const addSessionCanceledFulfilledListener = () => {
-  startAppListening({
-    actionCreator: sessionCanceled.fulfilled,
-    effect: (action) => {
-      const log = logger('session');
-      const { session_id } = action.meta.arg;
-      log.debug({ session_id }, `Session canceled (${session_id})`);
-    },
-  });
-};
-
-export const addSessionCanceledRejectedListener = () => {
-  startAppListening({
-    actionCreator: sessionCanceled.rejected,
-    effect: (action) => {
-      const log = logger('session');
-      const { session_id } = action.meta.arg;
-      if (action.payload) {
-        const { error } = action.payload;
-        log.error(
-          {
-            session_id,
-            error: serializeError(error),
-          },
-          `Problem canceling session`
-        );
-      }
-    },
-  });
-};

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/sessionCreated.ts

@@ -1,45 +0,0 @@
-import { logger } from 'app/logging/logger';
-import { parseify } from 'common/util/serialize';
-import { serializeError } from 'serialize-error';
-import { sessionCreated } from 'services/api/thunks/session';
-import { startAppListening } from '..';
-
-export const addSessionCreatedPendingListener = () => {
-  startAppListening({
-    actionCreator: sessionCreated.pending,
-    effect: () => {
-      //
-    },
-  });
-};
-
-export const addSessionCreatedFulfilledListener = () => {
-  startAppListening({
-    actionCreator: sessionCreated.fulfilled,
-    effect: (action) => {
-      const log = logger('session');
-      const session = action.payload;
-      log.debug(
-        { session: parseify(session) },
-        `Session created (${session.id})`
-      );
-    },
-  });
-};
-
-export const addSessionCreatedRejectedListener = () => {
-  startAppListening({
-    actionCreator: sessionCreated.rejected,
-    effect: (action) => {
-      const log = logger('session');
-      if (action.payload) {
-        const { error, status } = action.payload;
-        const graph = parseify(action.meta.arg);
-        log.error(
-          { graph, status, error: serializeError(error) },
-          `Problem creating session`
-        );
-      }
-    },
-  });
-};

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/sessionInvoked.ts

@@ -1,44 +0,0 @@
-import { logger } from 'app/logging/logger';
-import { serializeError } from 'serialize-error';
-import { sessionInvoked } from 'services/api/thunks/session';
-import { startAppListening } from '..';
-
-export const addSessionInvokedPendingListener = () => {
-  startAppListening({
-    actionCreator: sessionInvoked.pending,
-    effect: () => {
-      //
-    },
-  });
-};
-
-export const addSessionInvokedFulfilledListener = () => {
-  startAppListening({
-    actionCreator: sessionInvoked.fulfilled,
-    effect: (action) => {
-      const log = logger('session');
-      const { session_id } = action.meta.arg;
-      log.debug({ session_id }, `Session invoked (${session_id})`);
-    },
-  });
-};
-
-export const addSessionInvokedRejectedListener = () => {
-  startAppListening({
-    actionCreator: sessionInvoked.rejected,
-    effect: (action) => {
-      const log = logger('session');
-      const { session_id } = action.meta.arg;
-      if (action.payload) {
-        const { error } = action.payload;
-        log.error(
-          {
-            session_id,
-            error: serializeError(error),
-          },
-          `Problem invoking session`
-        );
-      }
-    },
-  });
-};

invokeai/frontend/web/src/app/store/middleware/listenerMiddleware/listeners/socketio/socketInvocationComplete.ts

@@ -8,6 +8,7 @@ import {
 } from 'features/gallery/store/gallerySlice';
 import { IMAGE_CATEGORIES } from 'features/gallery/store/types';
 import { CANVAS_OUTPUT } from 'features/nodes/util/graphBuilders/constants';
+import { boardsApi } from 'services/api/endpoints/boards';
 import { imagesApi } from 'services/api/endpoints/images';
 import { isImageOutput } from 'services/api/guards';
 import { imagesAdapter } from 'services/api/util';
@@ -70,10 +71,21 @@ export const addInvocationCompleteEventListener = () => {
             )
           );
 
+          // update the total images for the board
+          dispatch(
+            boardsApi.util.updateQueryData(
+              'getBoardImagesTotal',
+              imageDTO.board_id ?? 'none',
+              (draft) => {
+                // eslint-disable-next-line @typescript-eslint/no-unused-vars
+                draft.total += 1;
+              }
+            )
+          );
+
           dispatch(
             imagesApi.util.invalidateTags([
-              { type: 'BoardImagesTotal', id: imageDTO.board_id },
-              { type: 'BoardAssetsTotal', id: imageDTO.board_id },
+              { type: 'Board', id: imageDTO.board_id ?? 'none' },
             ])
           );
 
@@ -81,9 +93,32 @@ export const addInvocationCompleteEventListener = () => {
 
           // If auto-switch is enabled, select the new image
           if (shouldAutoSwitch) {
-            // if auto-add is enabled, switch the board as the image comes in
-            dispatch(galleryViewChanged('images'));
-            dispatch(boardIdSelected(imageDTO.board_id ?? 'none'));
+            // if auto-add is enabled, switch the gallery view and board if needed as the image comes in
+            if (gallery.galleryView !== 'images') {
+              dispatch(galleryViewChanged('images'));
+            }
+
+            if (
+              imageDTO.board_id &&
+              imageDTO.board_id !== gallery.selectedBoardId
+            ) {
+              dispatch(
+                boardIdSelected({
+                  boardId: imageDTO.board_id,
+                  selectedImageName: imageDTO.image_name,
+                })
+              );
+            }
+
+            if (!imageDTO.board_id && gallery.selectedBoardId !== 'none') {
+              dispatch(
+                boardIdSelected({
+                  boardId: 'none',
+                  selectedImageName: imageDTO.image_name,
+                })
+              );
+            }
+
             dispatch(imageSelected(imageDTO));
           }
         }