A very primitive working version of peft patching. It is very slow. LoRAs don't get unloaded yet, so can only be run once. And the results are *slightly* different than the old implementation. I suspect this is because the lora weight is not being applied to the UNet, but there could be other issues as well.

README.md

@@ -2,102 +2,21 @@
 
 ![project hero](https://github.com/invoke-ai/InvokeAI/assets/31807370/6e3728c7-e90e-4711-905c-3b55844ff5be)
 
-# Invoke - Professional Creative AI Tools for Visual Media
+# Invoke - Professional Creative AI Tools for Visual Media 
+##  To learn more about Invoke, or implement our Business solutions, visit [invoke.com](https://www.invoke.com/about)
+  
 
-#### To learn more about Invoke, or implement our Business solutions, visit [invoke.com]
 
-[![discord badge]][discord link] [![latest release badge]][latest release link] [![github stars badge]][github stars link] [![github forks badge]][github forks link] [![CI checks on main badge]][CI checks on main link] [![latest commit to main badge]][latest commit to main link] [![github open issues badge]][github open issues link] [![github open prs badge]][github open prs link] [![translation status badge]][translation status link]
+[![discord badge]][discord link]
 
-</div>
+[![latest release badge]][latest release link] [![github stars badge]][github stars link] [![github forks badge]][github forks link]
 
-Invoke is a leading creative engine built to empower professionals and enthusiasts alike. Generate and create stunning visual media using the latest AI-driven technologies. Invoke offers an industry leading web-based UI, and serves as the foundation for multiple commercial products.
+[![CI checks on main badge]][CI checks on main link] [![latest commit to main badge]][latest commit to main link]
 
-[Installation and Updates][installation docs] - [Documentation and Tutorials][docs home] - [Bug Reports][github issues] - [Contributing][contributing docs]
+[![github open issues badge]][github open issues link] [![github open prs badge]][github open prs link] [![translation status badge]][translation status link]
 
-<div align="center">
-
-![Highlighted Features - Canvas and Workflows](https://github.com/invoke-ai/InvokeAI/assets/31807370/708f7a82-084f-4860-bfbe-e2588c53548d)
-
-</div>
-
-## Quick Start
-
-1. Download and unzip the installer from the bottom of the [latest release][latest release link].
-2. Run the installer script.
-
-   - **Windows**: Double-click on the `install.bat` script.
-   - **macOS**: Open a Terminal window, drag the file `install.sh` from Finder into the Terminal, and press enter.
-   - **Linux**: Run `install.sh`.
-
-3. When prompted, enter a location for the install and select your GPU type.
-4. Once the install finishes, find the directory you selected during install. The default location is `C:\Users\Username\invokeai` for Windows or `~/invokeai` for Linux/macOS.
-5. Run the launcher script (`invoke.bat` for Windows, `invoke.sh` for macOS and Linux) the same way you ran the installer script in step 2.
-6. Select option 1 to start the application. Once it starts up, open your browser and go to <http://localhost:9090>.
-7. Open the model manager tab to install a starter model and then you'll be ready to generate.
-
-More detail, including hardware requirements and manual install instructions, are available in the [installation documentation][installation docs].
-
-## Troubleshooting, FAQ and Support
-
-Please review our [FAQ][faq] for solutions to common installation problems and other issues.
-
-For more help, please join our [Discord][discord link].
-
-## Features
-
-Full details on features can be found in [our documentation][features docs].
-
-### Web Server & UI
-
-Invoke runs a locally hosted web server & React UI with an industry-leading user experience.
-
-### Unified Canvas
-
-The Unified Canvas is a fully integrated canvas implementation with support for all core generation capabilities, in/out-painting, brush tools, and more. This creative tool unlocks the capability for artists to create with AI as a creative collaborator, and can be used to augment AI-generated imagery, sketches, photography, renders, and more.
-
-### Workflows & Nodes
-
-Invoke offers a fully featured workflow management solution, enabling users to combine the power of node-based workflows with the easy of a UI. This allows for customizable generation pipelines to be developed and shared by users looking to create specific workflows to support their production use-cases.
-
-### Board & Gallery Management
-
-Invoke features an organized gallery system for easily storing, accessing, and remixing your content in the Invoke workspace. Images can be dragged/dropped onto any Image-base UI element in the application, and rich metadata within the Image allows for easy recall of key prompts or settings used in your workflow.
-
-### Other features
-
-- Support for both ckpt and diffusers models
-- SD1.5, SD2.0, and SDXL support
-- Upscaling Tools
-- Embedding Manager & Support
-- Model Manager & Support
-- Workflow creation & management
-- Node-Based Architecture
-
-## Contributing
-
-Anyone who wishes to contribute to this project - whether documentation, features, bug fixes, code cleanup, testing, or code reviews - is very much encouraged to do so.
-
-Get started with contributing by reading our [contribution documentation][contributing docs], joining the [#dev-chat] or the GitHub discussion board.
-
-We hope you enjoy using Invoke as much as we enjoy creating it, and we hope you will elect to become part of our community.
-
-## Thanks
-
-Invoke is a combined effort of [passionate and talented people from across the world][contributors]. We thank them for their time, hard work and effort.
-
-Original portions of the software are Copyright © 2024 by respective contributors.
-
-[features docs]: https://invoke-ai.github.io/InvokeAI/features/
-[faq]: https://invoke-ai.github.io/InvokeAI/help/FAQ/
-[contributors]: https://invoke-ai.github.io/InvokeAI/other/CONTRIBUTORS/
-[invoke.com]: https://www.invoke.com/about
-[github issues]: https://github.com/invoke-ai/InvokeAI/issues
-[docs home]: https://invoke-ai.github.io/InvokeAI
-[installation docs]: https://invoke-ai.github.io/InvokeAI/installation/INSTALLATION/
-[#dev-chat]: https://discord.com/channels/1020123559063990373/1049495067846524939
-[contributing docs]: https://invoke-ai.github.io/InvokeAI/contributing/CONTRIBUTING/
 [CI checks on main badge]: https://flat.badgen.net/github/checks/invoke-ai/InvokeAI/main?label=CI%20status%20on%20main&cache=900&icon=github
-[CI checks on main link]: https://github.com/invoke-ai/InvokeAI/actions?query=branch%3Amain
+[CI checks on main link]:https://github.com/invoke-ai/InvokeAI/actions?query=branch%3Amain
 [discord badge]: https://flat.badgen.net/discord/members/ZmtBAhwWhy?icon=discord
 [discord link]: https://discord.gg/ZmtBAhwWhy
 [github forks badge]: https://flat.badgen.net/github/forks/invoke-ai/InvokeAI?icon=github
@@ -111,6 +30,402 @@ Original portions of the software are Copyright © 2024 by respective contributo
 [latest commit to main badge]: https://flat.badgen.net/github/last-commit/invoke-ai/InvokeAI/main?icon=github&color=yellow&label=last%20dev%20commit&cache=900
 [latest commit to main link]: https://github.com/invoke-ai/InvokeAI/commits/main
 [latest release badge]: https://flat.badgen.net/github/release/invoke-ai/InvokeAI/development?icon=github
-[latest release link]: https://github.com/invoke-ai/InvokeAI/releases/latest
+[latest release link]: https://github.com/invoke-ai/InvokeAI/releases
 [translation status badge]: https://hosted.weblate.org/widgets/invokeai/-/svg-badge.svg
 [translation status link]: https://hosted.weblate.org/engage/invokeai/
+
+</div>
+
+InvokeAI is a leading creative engine built to empower professionals
+and enthusiasts alike. Generate and create stunning visual media using
+the latest AI-driven technologies. InvokeAI offers an industry leading
+Web Interface, interactive Command Line Interface, and also serves as
+the foundation for multiple commercial products.
+
+**Quick links**: [[How to
+  Install](https://invoke-ai.github.io/InvokeAI/installation/INSTALLATION/)] [<a
+  href="https://discord.gg/ZmtBAhwWhy">Discord Server</a>] [<a
+  href="https://invoke-ai.github.io/InvokeAI/">Documentation and
+  Tutorials</a>]
+  [<a href="https://github.com/invoke-ai/InvokeAI/issues">Bug Reports</a>]
+  [<a
+  href="https://github.com/invoke-ai/InvokeAI/discussions">Discussion,
+  Ideas & Q&A</a>] 
+   [<a
+  href="https://invoke-ai.github.io/InvokeAI/contributing/CONTRIBUTING/">Contributing</a>] 
+
+<div align="center">
+
+
+![Highlighted Features - Canvas and Workflows](https://github.com/invoke-ai/InvokeAI/assets/31807370/708f7a82-084f-4860-bfbe-e2588c53548d)
+
+
+</div>
+
+## Table of Contents
+
+Table of Contents 📝
+
+**Getting Started**
+1. 🏁 [Quick Start](#quick-start) 
+3. 🖥️ [Hardware Requirements](#hardware-requirements) 
+
+**More About Invoke**
+1. 🌟 [Features](#features) 
+2. 📣 [Latest Changes](#latest-changes) 
+3. 🛠️ [Troubleshooting](#troubleshooting) 
+
+**Supporting the Project**
+1. 🤝 [Contributing](#contributing) 
+2. 👥 [Contributors](#contributors) 
+3. 💕 [Support](#support) 
+
+## Quick Start
+
+For full installation and upgrade instructions, please see:
+[InvokeAI Installation Overview](https://invoke-ai.github.io/InvokeAI/installation/INSTALLATION/)
+
+If upgrading from version 2.3, please read [Migrating a 2.3 root
+directory to 3.0](#migrating-to-3) first.
+
+### Automatic Installer (suggested for 1st time users)
+
+1. Go to the bottom of the [Latest Release Page](https://github.com/invoke-ai/InvokeAI/releases/latest)
+
+2. Download the .zip file for your OS (Windows/macOS/Linux).
+
+3. Unzip the file.
+
+4. **Windows:** double-click on the `install.bat` script. **macOS:** Open a Terminal window, drag the file `install.sh` from Finder
+into the Terminal, and press return. **Linux:** run `install.sh`.
+
+5. You'll be asked to confirm the location of the folder in which
+to install InvokeAI and its image generation model files. Pick a
+location with at least 15 GB of free memory. More if you plan on
+installing lots of models.
+
+6. Wait while the installer does its thing. After installing the software,
+the installer will launch a script that lets you configure InvokeAI and
+select a set of starting image generation models.
+
+7. Find the folder that InvokeAI was installed into (it is not the
+same as the unpacked zip file directory!) The default location of this
+folder (if you didn't change it in step 5) is `~/invokeai` on
+Linux/Mac systems, and `C:\Users\YourName\invokeai` on Windows. This directory will contain launcher scripts named `invoke.sh` and `invoke.bat`.
+
+8. On Windows systems, double-click on the `invoke.bat` file. On
+macOS, open a Terminal window, drag `invoke.sh` from the folder into
+the Terminal, and press return. On Linux, run `invoke.sh`
+
+9. Press 2 to open the "browser-based UI", press enter/return, wait a
+minute or two for Stable Diffusion to start up, then open your browser
+and go to http://localhost:9090.
+
+10. Type `banana sushi` in the box on the top left and click `Invoke`
+
+### Command-Line Installation (for developers and users familiar with Terminals)
+
+You must have Python 3.10 through 3.11 installed on your machine. Earlier or
+later versions are not supported.
+Node.js also needs to be installed along with `pnpm` (can be installed with
+the command `npm install -g pnpm` if needed)
+
+1. Open a command-line window on your machine. The PowerShell is recommended for Windows.
+2. Create a directory to install InvokeAI into. You'll need at least 15 GB of free space:
+
+    ```terminal
+    mkdir invokeai
+    ````
+
+3. Create a virtual environment named `.venv` inside this directory and activate it:
+
+    ```terminal
+    cd invokeai
+    python -m venv .venv --prompt InvokeAI
+    ```
+
+4. Activate the virtual environment (do it every time you run InvokeAI)
+
+    _For Linux/Mac users:_
+
+    ```sh
+    source .venv/bin/activate
+    ```
+
+    _For Windows users:_
+
+    ```ps
+    .venv\Scripts\activate
+    ```
+
+5. Install the InvokeAI module and its dependencies. Choose the command suited for your platform & GPU.
+
+    _For Windows/Linux with an NVIDIA GPU:_
+
+    ```terminal
+    pip install "InvokeAI[xformers]" --use-pep517 --extra-index-url https://download.pytorch.org/whl/cu121
+    ```
+
+    _For Linux with an AMD GPU:_
+
+    ```sh
+    pip install InvokeAI --use-pep517 --extra-index-url https://download.pytorch.org/whl/rocm5.6
+    ```
+
+    _For non-GPU systems:_
+    ```terminal
+    pip install InvokeAI --use-pep517 --extra-index-url https://download.pytorch.org/whl/cpu
+    ``` 
+
+    _For Macintoshes, either Intel or M1/M2/M3:_
+
+    ```sh
+    pip install InvokeAI --use-pep517
+    ```
+
+6. Configure InvokeAI and install a starting set of image generation models (you only need to do this once):
+
+    ```terminal
+    invokeai-configure --root .
+    ```
+	Don't miss the dot at the end!
+
+7. Launch the web server (do it every time you run InvokeAI):
+
+    ```terminal
+    invokeai-web
+    ```
+
+8. Point your browser to http://localhost:9090 to bring up the web interface.
+
+9. Type `banana sushi` in the box on the top left and click `Invoke`.
+
+Be sure to activate the virtual environment each time before re-launching InvokeAI,
+using `source .venv/bin/activate` or `.venv\Scripts\activate`.
+
+## Detailed Installation Instructions
+
+This fork is supported across Linux, Windows and Macintosh. Linux
+users can use either an Nvidia-based card (with CUDA support) or an
+AMD card (using the ROCm driver). For full installation and upgrade
+instructions, please see:
+[InvokeAI Installation Overview](https://invoke-ai.github.io/InvokeAI/installation/INSTALL_SOURCE/)
+
+<a name="migrating-to-3"></a>
+### Migrating a v2.3 InvokeAI root directory
+
+The InvokeAI root directory is where the InvokeAI startup file,
+installed models, and generated images are stored. It is ordinarily
+named `invokeai` and located in your home directory. The contents and
+layout of this directory has changed between versions 2.3 and 3.0 and
+cannot be used directly.
+
+We currently recommend that you use the installer to create a new root
+directory named differently from the 2.3 one, e.g. `invokeai-3` and
+then use a migration script to copy your 2.3 models into the new
+location. However, if you choose, you can upgrade this directory in
+place.  This section gives both recipes.
+
+#### Creating a new root directory and migrating old models
+
+This is the safer recipe because it leaves your old root directory in
+place to fall back on.
+
+1. Follow the instructions above to create and install InvokeAI in a
+directory that has a different name from the 2.3 invokeai directory.
+In this example, we will use "invokeai-3"
+
+2. When you are prompted to select models to install, select a minimal
+set of models, such as stable-diffusion-v1.5 only.
+
+3. After installation is complete launch `invokeai.sh` (Linux/Mac) or
+`invokeai.bat` and select option 8 "Open the developers console". This
+will take you to the command line.
+
+4. Issue the command `invokeai-migrate3 --from /path/to/v2.3-root --to
+/path/to/invokeai-3-root`. Provide the correct `--from` and `--to`
+paths for your v2.3 and v3.0 root directories respectively.
+
+This will copy and convert your old models from 2.3 format to 3.0
+format and create a new `models` directory in the 3.0 directory. The
+old models directory (which contains the models selected at install
+time) will be renamed `models.orig` and can be deleted once you have
+confirmed that the migration was successful.
+
+ If you wish, you can pass the 2.3 root directory to both `--from` and
+`--to` in order to update in place. Warning: this directory will no
+longer be usable with InvokeAI 2.3.
+
+#### Migrating in place
+
+For the adventurous, you may do an in-place upgrade from 2.3 to 3.0
+without touching the command line. ***This recipe does not work on
+Windows platforms due to a bug in the Windows version of the 2.3
+upgrade script.** See the next section for a Windows recipe.
+
+##### For Mac and Linux Users:
+
+1. Launch the InvokeAI launcher script in your current v2.3 root directory.
+
+2. Select option [9] "Update InvokeAI" to bring up the updater dialog.
+
+3. Select option [1] to upgrade to the latest release.
+
+4. Once the upgrade is finished you will be returned to the launcher
+menu. Select option [6] "Re-run the configure script to fix a broken
+install or to complete a major upgrade".
+
+This will run the configure script against the v2.3 directory and
+update it to the 3.0 format. The following files will be replaced:
+
+  - The invokeai.init file, replaced by invokeai.yaml
+  - The models directory
+  - The configs/models.yaml model index
+  
+The original versions of these files will be saved with the suffix
+".orig" appended to the end. Once you have confirmed that the upgrade
+worked, you can safely remove these files. Alternatively you can
+restore a working v2.3 directory by removing the new files and
+restoring the ".orig" files' original names.
+
+##### For Windows Users:
+
+Windows Users can upgrade with the
+
+1. Enter the 2.3 root directory you wish to upgrade
+2. Launch `invoke.sh` or `invoke.bat`
+3. Select the "Developer's console" option [8]
+4. Type the following commands
+
+```
+pip install "invokeai @ https://github.com/invoke-ai/InvokeAI/archive/refs/tags/v3.0.0" --use-pep517 --upgrade
+invokeai-configure --root .
+```
+(Replace `v3.0.0` with the current release number if this document is out of date).
+
+The first command will install and upgrade new software to run
+InvokeAI. The second will prepare the 2.3 directory for use with 3.0.
+You may now launch the WebUI in the usual way, by selecting option [1]
+from the launcher script
+
+#### Migrating Images
+
+The migration script will migrate your invokeai settings and models,
+including textual inversion models, LoRAs and merges that you may have
+installed previously. However it does **not** migrate the generated
+images stored in your 2.3-format outputs directory. To do this, you 
+need to run an additional step:
+
+1. From a working InvokeAI 3.0 root directory, start the launcher and
+enter menu option [8] to open the "developer's console".
+
+2. At the developer's console command line, type the command:
+
+```bash
+invokeai-import-images
+```
+
+3. This will lead you through the process of confirming the desired
+   source and destination for the imported images. The images will
+   appear in the gallery board of your choice, and contain the
+   original prompt, model name, and other parameters used to generate
+   the image.
+   
+(Many kudos to **techjedi** for contributing this script.)
+
+## Hardware Requirements
+
+InvokeAI is supported across Linux, Windows and macOS. Linux
+users can use either an Nvidia-based card (with CUDA support) or an
+AMD card (using the ROCm driver).
+
+### System
+
+You will need one of the following:
+
+- An NVIDIA-based graphics card with 4 GB or more VRAM memory. 6-8 GB
+  of VRAM is highly recommended for rendering using the Stable
+  Diffusion XL models
+- An Apple computer with an M1 chip.
+- An AMD-based graphics card with 4GB or more VRAM memory (Linux
+  only), 6-8 GB for XL rendering.
+
+We do not recommend the GTX 1650 or 1660 series video cards. They are
+unable to run in half-precision mode and do not have sufficient VRAM
+to render 512x512 images.
+
+**Memory** - At least 12 GB Main Memory RAM.
+
+**Disk** - At least 12 GB of free disk space for the machine learning model, Python, and all its dependencies.
+
+## Features
+
+Feature documentation can be reviewed by navigating to [the InvokeAI Documentation page](https://invoke-ai.github.io/InvokeAI/features/)
+
+### *Web Server & UI*
+
+InvokeAI offers a locally hosted Web Server & React Frontend, with an industry leading user experience. The Web-based UI allows for simple and intuitive workflows, and is responsive for use on mobile devices and tablets accessing the web server.
+
+### *Unified Canvas*
+
+The Unified Canvas is a fully integrated canvas implementation with support for all core generation capabilities, in/outpainting, brush tools, and more. This creative tool unlocks the capability for artists to create with AI as a creative collaborator, and can be used to augment AI-generated imagery, sketches, photography, renders, and more.
+
+### *Workflows & Nodes*
+
+InvokeAI offers a fully featured workflow management solution, enabling users to combine the power of nodes based workflows with the easy of a UI. This allows for customizable generation pipelines to be developed and shared by users looking to create specific workflows to support their production use-cases.
+
+### *Board & Gallery Management*
+
+Invoke AI provides an organized gallery system for easily storing, accessing, and remixing your content in the Invoke workspace. Images can be dragged/dropped onto any Image-base UI element in the application, and rich metadata within the Image allows for easy recall of key prompts or settings used in your workflow. 
+
+### Other features
+
+- *Support for both ckpt and diffusers models*
+- *SD 2.0, 2.1, XL support*
+- *Upscaling Tools*
+- *Embedding Manager & Support*
+- *Model Manager & Support*
+- *Workflow creation & management*
+- *Node-Based Architecture*
+
+
+### Latest Changes
+
+For our latest changes, view our [Release
+Notes](https://github.com/invoke-ai/InvokeAI/releases) and the
+[CHANGELOG](docs/CHANGELOG.md).
+
+### Troubleshooting / FAQ
+
+Please check out our **[FAQ](https://invoke-ai.github.io/InvokeAI/help/FAQ/)** to get solutions for common installation
+problems and other issues. For more help, please join our [Discord][discord link]
+
+## Contributing
+
+Anyone who wishes to contribute to this project, whether documentation, features, bug fixes, code
+cleanup, testing, or code reviews, is very much encouraged to do so.
+
+Get started with contributing by reading our [Contribution documentation](https://invoke-ai.github.io/InvokeAI/contributing/CONTRIBUTING/), joining the [#dev-chat](https://discord.com/channels/1020123559063990373/1049495067846524939) or the GitHub discussion board.
+
+If you are unfamiliar with how
+to contribute to GitHub projects, we have a new contributor checklist you can follow to get started contributing: 
+[New Contributor Checklist](https://invoke-ai.github.io/InvokeAI/contributing/contribution_guides/newContributorChecklist/).
+
+We hope you enjoy using our software as much as we enjoy creating it,
+and we hope that some of those of you who are reading this will elect
+to become part of our community.
+
+Welcome to InvokeAI!
+
+### Contributors
+
+This fork is a combined effort of various people from across the world.
+[Check out the list of all these amazing people](https://invoke-ai.github.io/InvokeAI/other/CONTRIBUTORS/). We thank them for
+their time, hard work and effort.
+
+### Support
+
+For support, please use this repository's GitHub Issues tracking service, or join the [Discord][discord link].
+
+Original portions of the software are Copyright (c) 2023 by respective contributors.
+

docs/contributing/frontend/WORKFLOWS.md

@@ -117,13 +117,13 @@ Stateless fields do not store their value in the node, so their field instances
 
 "Custom" fields will always be treated as stateless fields.
 
-##### Single and Collection Fields
+##### Collection and Scalar Fields
 
-Field types have a name and cardinality property which may identify it as a **SINGLE**, **COLLECTION** or **SINGLE_OR_COLLECTION** field.
+Field types have a name and two flags which may identify it as a **collection** or **collection or scalar** field.
 
-- If a field is annotated in python as a singular value or class, its field type is parsed as a **SINGLE** type (e.g. `int`, `ImageField`, `str`).
-- If a field is annotated in python as a list, its field type is parsed as a **COLLECTION** type (e.g. `list[int]`).
-- If it is annotated as a union of a type and list, the type will be parsed as a **SINGLE_OR_COLLECTION** type (e.g. `Union[int, list[int]]`). Fields may not be unions of different types (e.g. `Union[int, list[str]]` and `Union[int, str]` are not allowed).
+If a field is annotated in python as a list, its field type is parsed and flagged as a **collection** type (e.g. `list[int]`).
+
+If it is annotated as a union of a type and list, the type will be flagged as a **collection or scalar** type (e.g. `Union[int, list[int]]`). Fields may not be unions of different types (e.g. `Union[int, list[str]]` and `Union[int, str]` are not allowed).
 
 ## Implementation
 
@@ -173,7 +173,8 @@ Field types are represented as structured objects:
 ```ts
 type FieldType = {
   name: string;
-  cardinality: 'SINGLE' | 'COLLECTION' | 'SINGLE_OR_COLLECTION';
+  isCollection: boolean;
+  isCollectionOrScalar: boolean;
 };
 ```
 
@@ -185,7 +186,7 @@ There are 4 general cases for field type parsing.
 
 When a field is annotated as a primitive values (e.g. `int`, `str`, `float`), the field type parsing is fairly straightforward. The field is represented by a simple OpenAPI **schema object**, which has a `type` property.
 
-We create a field type name from this `type` string (e.g. `string` -> `StringField`). The cardinality is `"SINGLE"`.
+We create a field type name from this `type` string (e.g. `string` -> `StringField`).
 
 ##### Complex Types
 
@@ -199,13 +200,13 @@ We need to **dereference** the schema to pull these out. Dereferencing may requi
 
 When a field is annotated as a list of a single type, the schema object has an `items` property. They may be a schema object or reference object and must be parsed to determine the item type.
 
-We use the item type for field type name. The cardinality is `"COLLECTION"`.
+We use the item type for field type name, adding `isCollection: true` to the field type.
 
-##### Single or Collection Types
+##### Collection or Scalar Types
 
 When a field is annotated as a union of a type and list of that type, the schema object has an `anyOf` property, which holds a list of valid types for the union.
 
-After verifying that the union has two members (a type and list of the same type), we use the type for field type name, with cardinality `"SINGLE_OR_COLLECTION"`.
+After verifying that the union has two members (a type and list of the same type), we use the type for field type name, adding `isCollectionOrScalar: true` to the field type.
 
 ##### Optional Fields
 

docs/features/CONFIGURATION.md

@@ -51,11 +51,13 @@ The settings in this file will override the defaults. You only need
 to change this file if the default for a particular setting doesn't
 work for you.
 
-You'll find an example file next to `invokeai.yaml` that shows the default values.
-
 Some settings, like [Model Marketplace API Keys], require the YAML
 to be formatted correctly. Here is a [basic guide to YAML files].
 
+You can fix a broken `invokeai.yaml` by deleting it and running the
+configuration script again -- option [6] in the launcher, "Re-run the
+configure script".
+
 #### Custom Config File Location
 
 You can use any config file with the `--config` CLI arg. Pass in the path to the `invokeai.yaml` file you want to use.

docs/features/GALLERY.md

@@ -1,92 +0,0 @@
----
-title: InvokeAI Gallery Panel
----
-
-# :material-web: InvokeAI Gallery Panel
-
-## Quick guided walkthrough of the Gallery Panel's features
-
-The Gallery Panel is a fast way to review, find, and make use of images you've
-generated and loaded. The Gallery is divided into Boards. The Uncategorized board is always 
-present but you can create your own for better organization.
-
-![image](../assets/gallery/gallery.png)
-
-### Board Display and Settings
-
-At the very top of the Gallery Panel are the boards disclosure and settings buttons.
-
-![image](../assets/gallery/top_controls.png)
-
-The disclosure button shows the name of the currently selected board and allows you to show and hide the board thumbnails (shown in the image below).
-
-![image](../assets/gallery/board_thumbnails.png)
-
-The settings button opens a list of options.
-
-![image](../assets/gallery/board_settings.png)
-
-- ***Image Size*** this slider lets you control the size of the image previews (images of three different sizes).
-- ***Auto-Switch to New Images*** if you turn this on, whenever a new image is generated, it will automatically be loaded into the current image panel on the Text to Image tab and into the result panel on the [Image to Image](IMG2IMG.md) tab. This will happen invisibly if you are on any other tab when the image is generated.
-- ***Auto-Assign Board on Click*** whenever an image is generated or saved, it always gets put in a board. The board it gets put into is marked with AUTO (image of board marked). Turning on Auto-Assign Board on Click will make whichever board you last selected be the destination when you click Invoke. That means you can click Invoke, select a different board, and then click Invoke again and the two images will be put in two different boards. (bold)It's the board selected when Invoke is clicked that's used, not the board that's selected when the image is finished generating.(bold) Turning this off, enables the Auto-Add Board drop down which lets you set one specific board to always put generated images into. This also enables and disables the Auto-add to this Board menu item described below.
-- ***Always Show Image Size Badge*** this toggles whether to show image sizes for each image preview (show two images, one with sizes shown, one without)
-
-Below these two buttons, you'll see the Search Boards text entry area. You use this to search for specific boards by the name of the board.
-Next to it is the Add Board (+) button which lets you add new boards. Boards can be renamed by clicking on the name of the board under its thumbnail and typing in the new name.
-
-### Board Thumbnail Menu
-
-Each board has a context menu (ctrl+click / right-click).
-
-![image](../assets/gallery/thumbnail_menu.png)
-
-- ***Auto-add to this Board*** if you've disabled Auto-Assign Board on Click in the board settings, you can use this option to set this board to be where new images are put.
-- ***Download Board*** this will add all the images in the board into a zip file and provide a link to it in a notification (image of notification)
-- ***Delete Board*** this will delete the board
-> [!CAUTION]
-> This will delete all the images in the board and the board itself.
-
-### Board Contents
-
-Every board is organized by two tabs, Images and Assets.
-
-![image](../assets/gallery/board_tabs.png)
-
-Images are the Invoke-generated images that are placed into the board. Assets are images that you upload into Invoke to be used as an [Image Prompt](https://support.invoke.ai/support/solutions/articles/151000159340-using-the-image-prompt-adapter-ip-adapter-) or in the [Image to Image](IMG2IMG.md) tab.
-
-### Image Thumbnail Menu
-
-Every image generated by Invoke has its generation information stored as text inside the image file itself. This can be read directly by selecting the image and clicking on the Info button ![image](../assets/gallery/info_button.png) in any of the image result panels. 
-
-Each image also has a context menu (ctrl+click / right-click).
-
-![image](../assets/gallery/image_menu.png)
-
- The options are (items marked with an * will not work with images that lack generation information):
-- ***Open in New Tab*** this will open the image alone in a new browser tab, separate from the Invoke interface.
-- ***Download Image*** this will trigger your browser to download the image.
-- ***Load Workflow **** this will load any workflow settings into the Workflow tab and automatically open it.
-- ***Remix Image **** this will load all of the image's generation information, (bold)excluding its Seed, into the left hand control panel
-- ***Use Prompt **** this will load only the image's text prompts into the left-hand control panel
-- ***Use Seed **** this will load only the image's Seed into the left-hand control panel
-- ***Use All **** this will load all of the image's generation information into the left-hand control panel
-- ***Send to Image to Image*** this will put the image into the left-hand panel in the Image to Image tab ana automatically open it
-- ***Send to Unified Canvas*** This will (bold)replace whatever is already present(bold) in the Unified Canvas tab with the image and automatically open the tab
-- ***Change Board*** this will oipen a small window that will let you move the image to a different board. This is the same as dragging the image to that board's thumbnail.
-- ***Star Image*** this will add the image to the board's list of starred images that are always kept at the top of the gallery. This is the same as clicking on the star on the top right-hand side of the image that appears when you hover over the image with the mouse
-- ***Delete Image*** this will delete the image from the board
-> [!CAUTION] 
-> This will delete the image entirely from Invoke.
-
-## Summary
-
-This walkthrough only covers the Gallery interface and Boards. Actually generating images is handled by [Prompts](PROMPTS.md), the [Image to Image](IMG2IMG.md) tab, and the [Unified Canvas](UNIFIED_CANVAS.md).
-
-## Acknowledgements
-
-A huge shout-out to the core team working to make the Web GUI a reality,
-including [psychedelicious](https://github.com/psychedelicious),
-[Kyle0654](https://github.com/Kyle0654) and
-[blessedcoolant](https://github.com/blessedcoolant).
-[hipsterusername](https://github.com/hipsterusername) was the team's unofficial
-cheerleader and added tooltips/docs.

docs/features/PROMPTS.md

@@ -108,6 +108,40 @@ Can be used with .and():
 Each will give you different results - try them out and see what you prefer!
 
 
+
+### Cross-Attention Control ('prompt2prompt')
+
+Sometimes an image you generate is almost right, and you just want to change one
+detail without affecting the rest. You could use a photo editor and inpainting
+to overpaint the area, but that's a pain. Here's where `prompt2prompt` comes in
+handy.
+
+Generate an image with a given prompt, record the seed of the image, and then
+use the `prompt2prompt` syntax to substitute words in the original prompt for
+words in a new prompt. This works for `img2img` as well.
+
+For example, consider the prompt `a cat.swap(dog) playing with a ball in the forest`. Normally, because the words interact with each other when doing a stable diffusion image generation, these two prompts would generate different compositions:
+  - `a cat playing with a ball in the forest`
+  - `a dog playing with a ball in the forest`
+
+| `a cat playing with a ball in the forest` | `a dog playing with a ball in the forest` |
+| --- | --- |
+| img | img |
+
+
+      - For multiple word swaps, use parentheses: `a (fluffy cat).swap(barking dog) playing with a ball in the forest`.
+      - To swap a comma, use quotes: `a ("fluffy, grey cat").swap("big, barking dog") playing with a ball in the forest`.
+- Supports options `t_start` and `t_end` (each 0-1) loosely corresponding to (bloc97's)[(https://github.com/bloc97/CrossAttentionControl)] `prompt_edit_tokens_start/_end` but with the math swapped to make it easier to
+  intuitively understand. `t_start` and `t_end` are used to control on which steps cross-attention control should run. With the default values `t_start=0` and `t_end=1`, cross-attention control is active on every step of image generation. Other values can be used to turn cross-attention control off for part of the image generation process.
+    - For example, if doing a diffusion with 10 steps for the prompt is `a cat.swap(dog, t_start=0.3, t_end=1.0) playing with a ball in the forest`, the first 3 steps will be run as `a cat playing with a ball in the forest`, while the last 7 steps will run as `a dog playing with a ball in the forest`, but the pixels that represent `dog` will be locked to the pixels that would have represented `cat` if the `cat` prompt had been used instead.
+    - Conversely, for `a cat.swap(dog, t_start=0, t_end=0.7) playing with a ball in the forest`, the first 7 steps will run as `a dog playing with a ball in the forest` with the pixels that represent `dog` locked to the same pixels that would have represented `cat` if the `cat` prompt was being used instead. The final 3 steps will just run `a cat playing with a ball in the forest`.
+    > For img2img, the step sequence does not start at 0 but instead at `(1.0-strength)` - so if the img2img `strength` is `0.7`, `t_start` and `t_end` must both be greater than `0.3` (`1.0-0.7`) to have any effect.
+
+Prompt2prompt `.swap()` is not compatible with xformers, which will be temporarily disabled when doing a `.swap()` - so you should expect to use more VRAM and run slower that with xformers enabled.
+
+The `prompt2prompt` code is based off
+[bloc97's colab](https://github.com/bloc97/CrossAttentionControl).
+
 ### Escaping parentheses and speech marks 
 
 If the model you are using has parentheses () or speech marks "" as part of its

docs/features/TRAINING.md

@@ -4,6 +4,278 @@ title: Training
 
 # :material-file-document: Training
 
-Invoke Training has moved to its own repository, with a dedicated UI for accessing common scripts like Textual Inversion and LoRA training.
+# Textual Inversion Training
+## **Personalizing Text-to-Image Generation**
 
-You can find more by visiting the repo at https://github.com/invoke-ai/invoke-training
+You may personalize the generated images to provide your own styles or objects
+by training a new LDM checkpoint and introducing a new vocabulary to the fixed
+model as a (.pt) embeddings file. Alternatively, you may use or train
+HuggingFace Concepts embeddings files (.bin) from
+<https://huggingface.co/sd-concepts-library> and its associated
+notebooks.
+
+## **Hardware and Software Requirements**
+
+You will need a GPU to perform training in a reasonable length of
+time, and at least 12 GB of VRAM. We recommend using the [`xformers`
+library](../installation/070_INSTALL_XFORMERS.md) to accelerate the
+training process further. During training, about ~8 GB is temporarily
+needed in order to store intermediate models, checkpoints and logs.
+
+## **Preparing for Training**
+
+To train, prepare a folder that contains 3-5 images that illustrate
+the object or concept. It is good to provide a variety of examples or
+poses to avoid overtraining the system. Format these images as PNG
+(preferred) or JPG. You do not need to resize or crop the images in
+advance, but for more control you may wish to do so.
+
+Place the training images in a directory on the machine InvokeAI runs
+on. We recommend placing them in a subdirectory of the
+`text-inversion-training-data` folder located in the InvokeAI root
+directory, ordinarily `~/invokeai` (Linux/Mac), or
+`C:\Users\your_name\invokeai` (Windows). For example, to create an
+embedding for the "psychedelic" style, you'd place the training images
+into the directory
+`~invokeai/text-inversion-training-data/psychedelic`.
+
+## **Launching Training Using the Console Front End**
+
+InvokeAI 2.3 and higher comes with a text console-based training front
+end. From within the `invoke.sh`/`invoke.bat` Invoke launcher script,
+start training tool selecting choice (3):
+
+```sh
+1 "Generate images with a browser-based interface"
+2 "Explore InvokeAI nodes using a command-line interface"
+3 "Textual inversion training"
+4 "Merge models (diffusers type only)"
+5 "Download and install models"
+6 "Change InvokeAI startup options"
+7 "Re-run the configure script to fix a broken install or to complete a major upgrade"
+8 "Open the developer console"
+9 "Update InvokeAI"
+```
+
+Alternatively, you can select option (8) or from the command line, with the InvokeAI virtual environment active,
+you can then launch the front end with the command `invokeai-ti --gui`.
+
+This will launch a text-based front end that will look like this:
+
+<figure markdown>
+![ti-frontend](../assets/textual-inversion/ti-frontend.png)
+</figure>
+
+The interface is keyboard-based. Move from field to field using
+control-N (^N) to move to the next field and control-P (^P) to the
+previous one. <Tab> and <shift-TAB> work as well. Once a field is
+active, use the cursor keys. In a checkbox group, use the up and down
+cursor keys to move from choice to choice, and <space> to select a
+choice. In a scrollbar, use the left and right cursor keys to increase
+and decrease the value of the scroll. In textfields, type the desired
+values.
+
+The number of parameters may look intimidating, but in most cases the
+predefined defaults work fine. The red circled fields in the above
+illustration are the ones you will adjust most frequently.
+
+### Model Name
+
+This will list all the diffusers models that are currently
+installed. Select the one you wish to use as the basis for your
+embedding. Be aware that if you use a SD-1.X-based model for your
+training, you will only be able to use this embedding with other
+SD-1.X-based models. Similarly, if you train on SD-2.X, you will only
+be able to use the embeddings with models based on SD-2.X.
+
+### Trigger Term
+
+This is the prompt term you will use to trigger the embedding. Type a
+single word or phrase you wish to use as the trigger, example
+"psychedelic" (without angle brackets). Within InvokeAI, you will then
+be able to activate the trigger using the syntax `<psychedelic>`.
+
+### Initializer
+
+This is a single character that is used internally during the training
+process as a placeholder for the trigger term. It defaults to "*" and
+can usually be left alone.
+
+### Resume from last saved checkpoint
+
+As training proceeds, textual inversion will write a series of
+intermediate files that can be used to resume training from where it
+was left off in the case of an interruption. This checkbox will be
+automatically selected if you provide a previously used trigger term
+and at least one checkpoint file is found on disk.
+
+Note that as of 20 January 2023, resume does not seem to be working
+properly due to an issue with the upstream code.
+
+### Data Training Directory
+
+This is the location of the images to be used for training. When you
+select a trigger term like "my-trigger", the frontend will prepopulate
+this field with `~/invokeai/text-inversion-training-data/my-trigger`,
+but you can change the path to wherever you want.
+
+### Output Destination Directory
+
+This is the location of the logs, checkpoint files, and embedding
+files created during training. When you select a trigger term like
+"my-trigger", the frontend will prepopulate this field with
+`~/invokeai/text-inversion-output/my-trigger`, but you can change the
+path to wherever you want.
+
+### Image resolution
+
+The images in the training directory will be automatically scaled to
+the value you use here. For best results, you will want to use the
+same default resolution of the underlying model (512 pixels for
+SD-1.5, 768 for the larger version of SD-2.1).
+
+### Center crop images
+
+If this is selected, your images will be center cropped to make them
+square before resizing them to the desired resolution. Center cropping
+can indiscriminately cut off the top of subjects' heads for portrait
+aspect images, so if you have images like this, you may wish to use a
+photoeditor to manually crop them to a square aspect ratio.
+
+### Mixed precision
+
+Select the floating point precision for the embedding. "no" will
+result in a full 32-bit precision, "fp16" will provide 16-bit
+precision, and "bf16" will provide mixed precision (only available
+when XFormers is used).
+
+### Max training steps
+
+How many steps the training will take before the model converges. Most
+training sets will converge with 2000-3000 steps.
+
+### Batch size
+
+This adjusts how many training images are processed simultaneously in
+each step. Higher values will cause the training process to run more
+quickly, but use more memory. The default size will run with GPUs with
+as little as 12 GB.
+
+### Learning rate
+
+The rate at which the system adjusts its internal weights during
+training. Higher values risk overtraining (getting the same image each
+time), and lower values will take more steps to train a good
+model. The default of 0.0005 is conservative; you may wish to increase
+it to 0.005 to speed up training.
+
+### Scale learning rate by number of GPUs, steps and batch size
+
+If this is selected (the default) the system will adjust the provided
+learning rate to improve performance.
+
+### Use xformers acceleration
+
+This will activate XFormers memory-efficient attention. You need to
+have XFormers installed for this to have an effect.
+
+### Learning rate scheduler
+
+This adjusts how the learning rate changes over the course of
+training. The default "constant" means to use a constant learning rate
+for the entire training session. The other values scale the learning
+rate according to various formulas.
+
+Only "constant" is supported by the XFormers library.
+
+### Gradient accumulation steps
+
+This is a parameter that allows you to use bigger batch sizes than
+your GPU's VRAM would ordinarily accommodate, at the cost of some
+performance.
+
+### Warmup steps
+
+If "constant_with_warmup" is selected in the learning rate scheduler,
+then this provides the number of warmup steps. Warmup steps have a
+very low learning rate, and are one way of preventing early
+overtraining.
+
+## The training run
+
+Start the training run by advancing to the OK button (bottom right)
+and pressing <enter>. A series of progress messages will be displayed
+as the training process proceeds. This may take an hour or two,
+depending on settings and the speed of your system. Various log and
+checkpoint files will be written into the output directory (ordinarily
+`~/invokeai/text-inversion-output/my-model/`)
+
+At the end of successful training, the system will copy the file
+`learned_embeds.bin` into the InvokeAI root directory's `embeddings`
+directory, using a subdirectory named after the trigger token. For
+example, if the trigger token was `psychedelic`, then look for the
+embeddings file in
+`~/invokeai/embeddings/psychedelic/learned_embeds.bin`
+
+You may now launch InvokeAI and try out a prompt that uses the trigger
+term. For example `a plate of banana sushi in <psychedelic> style`.
+
+## **Training with the Command-Line Script**
+
+Training can also be done using a traditional command-line script. It
+can be launched from within the "developer's console", or from the
+command line after activating InvokeAI's virtual environment.
+
+It accepts a large number of arguments, which can be summarized by
+passing the `--help` argument:
+
+```sh
+invokeai-ti --help
+```
+
+Typical usage is shown here:
+```sh
+invokeai-ti \
+       --model=stable-diffusion-1.5 \
+       --resolution=512 \
+       --learnable_property=style \
+       --initializer_token='*' \
+       --placeholder_token='<psychedelic>' \
+       --train_data_dir=/home/lstein/invokeai/training-data/psychedelic \
+       --output_dir=/home/lstein/invokeai/text-inversion-training/psychedelic \
+       --scale_lr \
+       --train_batch_size=8 \
+       --gradient_accumulation_steps=4 \
+       --max_train_steps=3000 \
+       --learning_rate=0.0005 \
+       --resume_from_checkpoint=latest \
+       --lr_scheduler=constant \
+       --mixed_precision=fp16 \
+       --only_save_embeds
+```
+
+## Troubleshooting
+
+### `Cannot load embedding for <trigger>. It was trained on a model with token dimension 1024, but the current model has token dimension 768`
+
+Messages like this indicate you trained the embedding on a different base model than the currently selected one.
+
+For example, in the error above, the training was done on SD2.1 (768x768) but it was used on SD1.5 (512x512).
+
+## Reading
+
+For more information on textual inversion, please see the following
+resources:
+
+* The [textual inversion repository](https://github.com/rinongal/textual_inversion) and
+  associated paper for details and limitations.
+* [HuggingFace's textual inversion training
+  page](https://huggingface.co/docs/diffusers/training/text_inversion)
+* [HuggingFace example script
+  documentation](https://github.com/huggingface/diffusers/tree/main/examples/textual_inversion)
+  (Note that this script is similar to, but not identical, to
+  `textual_inversion`, but produces embed files that are completely compatible.
+
+---
+
+copyright (c) 2023, Lincoln Stein and the InvokeAI Development Team

docs/features/WEB.md

@@ -54,7 +54,7 @@ main sections:
    of buttons at the top lets you modify and manipulate the image in
    various ways.
 
-3. A **gallery** section on the right that contains a history of the images you
+3. A **gallery** section on the left that contains a history of the images you
    have generated. These images are read and written to the directory specified
    in the `INVOKEAIROOT/invokeai.yaml` initialization file, usually a directory
    named `outputs` in `INVOKEAIROOT`.

docs/help/FAQ.md

@@ -40,25 +40,6 @@ Follow the same steps to scan and import the missing models.
 - Check the `ram` setting in `invokeai.yaml`. This setting tells Invoke how much of your system RAM can be used to cache models. Having this too high or too low can slow things down. That said, it's generally safest to not set this at all and instead let Invoke manage it.
 - Check the `vram` setting in `invokeai.yaml`. This setting tells Invoke how much of your GPU VRAM can be used to cache models. Counter-intuitively, if this setting is too high, Invoke will need to do a lot of shuffling of models as it juggles the VRAM cache and the currently-loaded model. The default value of 0.25 is generally works well for GPUs without 16GB or more VRAM. Even on a 24GB card, the default works well.
 - Check that your generations are happening on your GPU (if you have one). InvokeAI will log what is being used for generation upon startup. If your GPU isn't used, re-install to ensure the correct versions of torch get installed.
-- If you are on Windows, you may have exceeded your GPU's VRAM capacity and are using slower [shared GPU memory](#shared-gpu-memory-windows). There's a guide to opt out of this behaviour in the linked FAQ entry.
-
-## Shared GPU Memory (Windows)
-
-!!! tip "Nvidia GPUs with driver 536.40"
-
-    This only applies to current Nvidia cards with driver 536.40 or later, released in June 2023.
-
-When the GPU doesn't have enough VRAM for a task, Windows is able to allocate some of its CPU RAM to the GPU. This is much slower than VRAM, but it does allow the system to generate when it otherwise might no have enough VRAM.
-
-When shared GPU memory is used, generation slows down dramatically - but at least it doesn't crash.
-
-If you'd like to opt out of this behavior and instead get an error when you exceed your GPU's VRAM, follow [this guide from Nvidia](https://nvidia.custhelp.com/app/answers/detail/a_id/5490).
-
-Here's how to get the python path required in the linked guide:
-
-- Run `invoke.bat`.
-- Select option 2 for developer console.
-- At least one python path will be printed. Copy the path that includes your invoke installation directory (typically the first).
 
 ## Installer cannot find python (Windows)
 

docs/installation/010_INSTALL_AUTOMATED.md

@@ -1,10 +1,8 @@
-# Automatic Install & Updates
+# Automatic Install
 
-**The same packaged installer file can be used for both new installs and updates.**
-Using the installer for updates will leave everything you've added since installation, and just update the core libraries used to run Invoke.
-Simply use the same path you installed to originally.
+The installer is used for both new installs and updates.
 
-Both release and pre-release versions can be installed using the installer. It also supports install through a wheel if needed.
+Both release and pre-release versions can be installed using it. It also supports install a wheel if needed.
 
 Be sure to review the [installation requirements] and ensure your system has everything it needs to install Invoke.
 
@@ -98,7 +96,7 @@ Updating is exactly the same as installing - download the latest installer, choo
 
 If you have installation issues, please review the [FAQ]. You can also [create an issue] or ask for help on [discord].
 
-[installation requirements]: INSTALL_REQUIREMENTS.md
+[installation requirements]: INSTALLATION.md#installation-requirements
 [FAQ]: ../help/FAQ.md
 [install some models]: 050_INSTALLING_MODELS.md
 [configuration docs]: ../features/CONFIGURATION.md

docs/installation/020_INSTALL_MANUAL.md

@@ -10,7 +10,7 @@ InvokeAI is distributed as a python package on PyPI, installable with `pip`. The
 
 ### Requirements
 
-Before you start, go through the [installation requirements](./INSTALL_REQUIREMENTS.md).
+Before you start, go through the [installation requirements].
 
 ### Installation Walkthrough
 
@@ -79,7 +79,7 @@ Before you start, go through the [installation requirements](./INSTALL_REQUIREME
 
 1. Install the InvokeAI Package. The base command is `pip install InvokeAI --use-pep517`, but you may need to change this depending on your system and the desired features.
 
-    - You may need to provide an [extra index URL](https://pip.pypa.io/en/stable/cli/pip_install/#cmdoption-extra-index-url). Select your platform configuration using [this tool on the PyTorch website](https://pytorch.org/get-started/locally/). Copy the `--extra-index-url` string from this and append it to your install command.
+    - You may need to provide an [extra index URL]. Select your platform configuration using [this tool on the PyTorch website]. Copy the `--extra-index-url` string from this and append it to your install command.
 
         !!! example "Install with an extra index URL"
 
@@ -116,4 +116,4 @@ Before you start, go through the [installation requirements](./INSTALL_REQUIREME
 
     !!! warning
 
-        If the virtual environment is _not_ inside the root directory, then you _must_ specify the path to the root directory with `--root \path\to\invokeai` or the `INVOKEAI_ROOT` environment variable.
+        If the virtual environment is _not_ inside the root directory, then you _must_ specify the path to the root directory with `--root_dir \path\to\invokeai` or the `INVOKEAI_ROOT` environment variable.

docs/installation/INSTALLATION.md

@@ -1,4 +1,4 @@
-# Installation and Updating Overview
+# Installation Overview
 
 Before installing, review the [installation requirements] to ensure your system is set up properly.
 
@@ -6,21 +6,14 @@ See the [FAQ] for frequently-encountered installation issues.
 
 If you need more help, join our [discord] or [create an issue].
 
-<h2>Automatic Install & Updates </h2>
+<h2>Automatic Install</h2>
 
 ✅ The automatic install is the best way to run InvokeAI. Check out the [installation guide] to get started.
 
-⬆️ The same installer is also the best way to update InvokeAI - Simply rerun it for the same folder you installed to.
-
-The installation process simply manages installation for the core libraries & application dependencies that run Invoke.
-Any models, images, or other assets in the Invoke root folder won't be affected by the installation process.
-
 <h2>Manual Install</h2>
 
 If you are familiar with python and want more control over the packages that are installed, you can [install InvokeAI manually via PyPI].
 
-Updates are managed by reinstalling the latest version through PyPi.
-
 <h2>Developer Install</h2>
 
 If you want to contribute to InvokeAI, consult the [developer install guide].

docs/installation/INSTALL_DEVELOPMENT.md

@@ -23,7 +23,6 @@ If you have an interest in how InvokeAI works, or you would like to add features
 
 1. [Fork and clone] the [InvokeAI repo].
 1. Follow the [manual installation] docs to create a new virtual environment for the development install.
-   - Create a new folder outside the repo root for the installation and create the venv inside that folder.
    - When installing the InvokeAI package, add `-e` to the command so you get an [editable install].
 1. Install the [frontend dev toolchain] and do a production build of the UI as described.
 1. You can now run the app as described in the [manual installation] docs.

docs/installation/INSTALL_REQUIREMENTS.md

@@ -37,13 +37,13 @@ Invoke runs best with a dedicated GPU, but will fall back to running on CPU, alb
     === "Nvidia"
 
         ```
-        Any GPU with at least 8GB VRAM.
+        Any GPU with at least 8GB VRAM. Linux only.
         ```
 
     === "AMD"
 
         ```
-        Any GPU with at least 16GB VRAM. Linux only.
+        Any GPU with at least 16GB VRAM.
         ```
 
     === "Mac"

invokeai/app/api/routers/app_info.py

@@ -12,7 +12,8 @@ from pydantic import BaseModel, Field
 
 from invokeai.app.invocations.upscale import ESRGAN_MODELS
 from invokeai.app.services.invocation_cache.invocation_cache_common import InvocationCacheStatus
-from invokeai.backend.image_util.infill_methods.patchmatch import PatchMatch
+from invokeai.backend.image_util.patchmatch import PatchMatch
+from invokeai.backend.image_util.safety_checker import SafetyChecker
 from invokeai.backend.util.logging import logging
 from invokeai.version import __version__
 
@@ -99,7 +100,7 @@ async def get_app_deps() -> AppDependencyVersions:
 
 @app_router.get("/config", operation_id="get_config", status_code=200, response_model=AppConfig)
 async def get_config() -> AppConfig:
-    infill_methods = ["tile", "lama", "cv2", "color"]  # TODO: add mosaic back
+    infill_methods = ["tile", "lama", "cv2"]
     if PatchMatch.patchmatch_available():
         infill_methods.append("patchmatch")
 
@@ -108,7 +109,9 @@ async def get_config() -> AppConfig:
         upscaling_models.append(str(Path(model).stem))
     upscaler = Upscaler(upscaling_method="esrgan", upscaling_models=upscaling_models)
 
-    nsfw_methods = ["nsfw_checker"]
+    nsfw_methods = []
+    if SafetyChecker.safety_checker_available():
+        nsfw_methods.append("nsfw_checker")
 
     watermarking_methods = ["invisible_watermark"]
 

invokeai/app/api/routers/images.py

@@ -6,12 +6,13 @@ from fastapi import BackgroundTasks, Body, HTTPException, Path, Query, Request,
 from fastapi.responses import FileResponse
 from fastapi.routing import APIRouter
 from PIL import Image
-from pydantic import BaseModel, Field, JsonValue
+from pydantic import BaseModel, Field, ValidationError
 
-from invokeai.app.invocations.fields import MetadataField
+from invokeai.app.invocations.fields import MetadataField, MetadataFieldValidator
 from invokeai.app.services.image_records.image_records_common import ImageCategory, ImageRecordChanges, ResourceOrigin
 from invokeai.app.services.images.images_common import ImageDTO, ImageUrlsDTO
 from invokeai.app.services.shared.pagination import OffsetPaginatedResults
+from invokeai.app.services.workflow_records.workflow_records_common import WorkflowWithoutID, WorkflowWithoutIDValidator
 
 from ..dependencies import ApiDependencies
 
@@ -41,17 +42,13 @@ async def upload_image(
     board_id: Optional[str] = Query(default=None, description="The board to add this image to, if any"),
     session_id: Optional[str] = Query(default=None, description="The session ID associated with this upload, if any"),
     crop_visible: Optional[bool] = Query(default=False, description="Whether to crop the image"),
-    metadata: Optional[JsonValue] = Body(
-        default=None, description="The metadata to associate with the image", embed=True
-    ),
 ) -> ImageDTO:
     """Uploads an image"""
     if not file.content_type or not file.content_type.startswith("image"):
         raise HTTPException(status_code=415, detail="Not an image")
 
-    _metadata = None
-    _workflow = None
-    _graph = None
+    metadata = None
+    workflow = None
 
     contents = await file.read()
     try:
@@ -65,28 +62,22 @@ async def upload_image(
 
     # TODO: retain non-invokeai metadata on upload?
     # attempt to parse metadata from image
-    metadata_raw = metadata if isinstance(metadata, str) else pil_image.info.get("invokeai_metadata", None)
-    if isinstance(metadata_raw, str):
-        _metadata = metadata_raw
-    else:
-        ApiDependencies.invoker.services.logger.warn("Failed to parse metadata for uploaded image")
-        pass
+    metadata_raw = pil_image.info.get("invokeai_metadata", None)
+    if metadata_raw:
+        try:
+            metadata = MetadataFieldValidator.validate_json(metadata_raw)
+        except ValidationError:
+            ApiDependencies.invoker.services.logger.warn("Failed to parse metadata for uploaded image")
+            pass
 
     # attempt to parse workflow from image
     workflow_raw = pil_image.info.get("invokeai_workflow", None)
-    if isinstance(workflow_raw, str):
-        _workflow = workflow_raw
-    else:
-        ApiDependencies.invoker.services.logger.warn("Failed to parse workflow for uploaded image")
-        pass
-
-    # attempt to extract graph from image
-    graph_raw = pil_image.info.get("invokeai_graph", None)
-    if isinstance(graph_raw, str):
-        _graph = graph_raw
-    else:
-        ApiDependencies.invoker.services.logger.warn("Failed to parse graph for uploaded image")
-        pass
+    if workflow_raw is not None:
+        try:
+            workflow = WorkflowWithoutIDValidator.validate_json(workflow_raw)
+        except ValidationError:
+            ApiDependencies.invoker.services.logger.warn("Failed to parse metadata for uploaded image")
+            pass
 
     try:
         image_dto = ApiDependencies.invoker.services.images.create(
@@ -95,9 +86,8 @@ async def upload_image(
             image_category=image_category,
             session_id=session_id,
             board_id=board_id,
-            metadata=_metadata,
-            workflow=_workflow,
-            graph=_graph,
+            metadata=metadata,
+            workflow=workflow,
             is_intermediate=is_intermediate,
         )
 
@@ -195,21 +185,14 @@ async def get_image_metadata(
         raise HTTPException(status_code=404)
 
 
-class WorkflowAndGraphResponse(BaseModel):
-    workflow: Optional[str] = Field(description="The workflow used to generate the image, as stringified JSON")
-    graph: Optional[str] = Field(description="The graph used to generate the image, as stringified JSON")
-
-
 @images_router.get(
-    "/i/{image_name}/workflow", operation_id="get_image_workflow", response_model=WorkflowAndGraphResponse
+    "/i/{image_name}/workflow", operation_id="get_image_workflow", response_model=Optional[WorkflowWithoutID]
 )
 async def get_image_workflow(
     image_name: str = Path(description="The name of image whose workflow to get"),
-) -> WorkflowAndGraphResponse:
+) -> Optional[WorkflowWithoutID]:
     try:
-        workflow = ApiDependencies.invoker.services.images.get_workflow(image_name)
-        graph = ApiDependencies.invoker.services.images.get_graph(image_name)
-        return WorkflowAndGraphResponse(workflow=workflow, graph=graph)
+        return ApiDependencies.invoker.services.images.get_workflow(image_name)
     except Exception:
         raise HTTPException(status_code=404)
 

invokeai/app/api/routers/model_manager.py

@@ -6,7 +6,7 @@ import pathlib
 import shutil
 import traceback
 from copy import deepcopy
-from typing import Any, Dict, List, Optional, Type
+from typing import Any, Dict, List, Optional
 
 from fastapi import Body, Path, Query, Response, UploadFile
 from fastapi.responses import FileResponse
@@ -16,7 +16,6 @@ from pydantic import AnyHttpUrl, BaseModel, ConfigDict, Field
 from starlette.exceptions import HTTPException
 from typing_extensions import Annotated
 
-from invokeai.app.services.model_images.model_images_common import ModelImageFileNotFoundException
 from invokeai.app.services.model_install import ModelInstallJob
 from invokeai.app.services.model_records import (
     DuplicateModelException,
@@ -53,13 +52,6 @@ class ModelsList(BaseModel):
     model_config = ConfigDict(use_enum_values=True)
 
 
-def add_cover_image_to_model_config(config: AnyModelConfig, dependencies: Type[ApiDependencies]) -> AnyModelConfig:
-    """Add a cover image URL to a model configuration."""
-    cover_image = dependencies.invoker.services.model_images.get_url(config.key)
-    config.cover_image = cover_image
-    return config
-
-
 ##############################################################################
 # These are example inputs and outputs that are used in places where Swagger
 # is unable to generate a correct example.
@@ -126,7 +118,8 @@ async def list_model_records(
             record_store.search_by_attr(model_type=model_type, model_name=model_name, model_format=model_format)
         )
     for model in found_models:
-        model = add_cover_image_to_model_config(model, ApiDependencies)
+        cover_image = ApiDependencies.invoker.services.model_images.get_url(model.key)
+        model.cover_image = cover_image
     return ModelsList(models=found_models)
 
 
@@ -167,9 +160,12 @@ async def get_model_record(
     key: str = Path(description="Key of the model record to fetch."),
 ) -> AnyModelConfig:
     """Get a model record"""
+    record_store = ApiDependencies.invoker.services.model_manager.store
     try:
-        config = ApiDependencies.invoker.services.model_manager.store.get_model(key)
-        return add_cover_image_to_model_config(config, ApiDependencies)
+        config: AnyModelConfig = record_store.get_model(key)
+        cover_image = ApiDependencies.invoker.services.model_images.get_url(key)
+        config.cover_image = cover_image
+        return config
     except UnknownModelException as e:
         raise HTTPException(status_code=404, detail=str(e))
 
@@ -298,15 +294,14 @@ async def update_model_record(
     installer = ApiDependencies.invoker.services.model_manager.install
     try:
         record_store.update_model(key, changes=changes)
-        config = installer.sync_model_path(key)
-        config = add_cover_image_to_model_config(config, ApiDependencies)
+        model_response: AnyModelConfig = installer.sync_model_path(key)
         logger.info(f"Updated model: {key}")
     except UnknownModelException as e:
         raise HTTPException(status_code=404, detail=str(e))
     except ValueError as e:
         logger.error(str(e))
         raise HTTPException(status_code=409, detail=str(e))
-    return config
+    return model_response
 
 
 @model_manager_router.get(
@@ -653,14 +648,6 @@ async def convert_model(
         logger.error(str(e))
         raise HTTPException(status_code=409, detail=str(e))
 
-    # Update the model image if the model had one
-    try:
-        model_image = ApiDependencies.invoker.services.model_images.get(key)
-        ApiDependencies.invoker.services.model_images.save(model_image, new_key)
-        ApiDependencies.invoker.services.model_images.delete(key)
-    except ModelImageFileNotFoundException:
-        pass
-
     # delete the original safetensors file
     installer.delete(key)
 
@@ -668,8 +655,7 @@ async def convert_model(
     shutil.rmtree(cache_path)
 
     # return the config record for the new diffusers directory
-    new_config = store.get_model(new_key)
-    new_config = add_cover_image_to_model_config(new_config, ApiDependencies)
+    new_config: AnyModelConfig = store.get_model(new_key)
     return new_config
 
 

invokeai/app/api_app.py

@@ -28,7 +28,7 @@ from invokeai.app.api.no_cache_staticfiles import NoCacheStaticFiles
 from invokeai.app.invocations.model import ModelIdentifierField
 from invokeai.app.services.config.config_default import get_config
 from invokeai.app.services.session_processor.session_processor_common import ProgressImage
-from invokeai.backend.util.devices import TorchDevice
+from invokeai.backend.util.devices import get_torch_device_name
 
 from ..backend.util.logging import InvokeAILogger
 from .api.dependencies import ApiDependencies
@@ -63,7 +63,7 @@ logger = InvokeAILogger.get_logger(config=app_config)
 mimetypes.add_type("application/javascript", ".js")
 mimetypes.add_type("text/css", ".css")
 
-torch_device_name = TorchDevice.get_torch_device_name()
+torch_device_name = get_torch_device_name()
 logger.info(f"Using torch device: {torch_device_name}")
 
 
@@ -164,12 +164,6 @@ def custom_openapi() -> dict[str, Any]:
     for schema_key, schema_json in additional_schemas[1]["$defs"].items():
         openapi_schema["components"]["schemas"][schema_key] = schema_json
 
-    openapi_schema["components"]["schemas"]["InvocationOutputMap"] = {
-        "type": "object",
-        "properties": {},
-        "required": [],
-    }
-
     # Add a reference to the output type to additionalProperties of the invoker schema
     for invoker in all_invocations:
         invoker_name = invoker.__name__  # type: ignore [attr-defined] # this is a valid attribute
@@ -178,8 +172,6 @@ def custom_openapi() -> dict[str, Any]:
         invoker_schema = openapi_schema["components"]["schemas"][f"{invoker_name}"]
         outputs_ref = {"$ref": f"#/components/schemas/{output_type_title}"}
         invoker_schema["output"] = outputs_ref
-        openapi_schema["components"]["schemas"]["InvocationOutputMap"]["properties"][invoker.get_type()] = outputs_ref
-        openapi_schema["components"]["schemas"]["InvocationOutputMap"]["required"].append(invoker.get_type())
         invoker_schema["class"] = "invocation"
 
     # This code no longer seems to be necessary?

invokeai/app/invocations/compel.py

@@ -5,26 +5,20 @@ from compel import Compel, ReturnedEmbeddingsType
 from compel.prompt_parser import Blend, Conjunction, CrossAttentionControlSubstitute, FlattenedPrompt, Fragment
 from transformers import CLIPTextModel, CLIPTextModelWithProjection, CLIPTokenizer
 
-from invokeai.app.invocations.fields import (
-    ConditioningField,
-    FieldDescriptions,
-    Input,
-    InputField,
-    OutputField,
-    TensorField,
-    UIComponent,
-)
+from invokeai.app.invocations.fields import FieldDescriptions, Input, InputField, OutputField, UIComponent
 from invokeai.app.invocations.primitives import ConditioningOutput
 from invokeai.app.services.shared.invocation_context import InvocationContext
 from invokeai.app.util.ti_utils import generate_ti_list
-from invokeai.backend.lora import LoRAModelRaw
 from invokeai.backend.model_patcher import ModelPatcher
+from invokeai.backend.peft.peft_model import PeftModel
+from invokeai.backend.peft.peft_model_patcher import PeftModelPatcher
 from invokeai.backend.stable_diffusion.diffusion.conditioning_data import (
     BasicConditioningInfo,
     ConditioningFieldData,
+    ExtraConditioningInfo,
     SDXLConditioningInfo,
 )
-from invokeai.backend.util.devices import TorchDevice
+from invokeai.backend.util.devices import torch_dtype
 
 from .baseinvocation import BaseInvocation, BaseInvocationOutput, invocation, invocation_output
 from .model import CLIPField
@@ -43,7 +37,7 @@ from .model import CLIPField
     title="Prompt",
     tags=["prompt", "compel"],
     category="conditioning",
-    version="1.2.0",
+    version="1.1.1",
 )
 class CompelInvocation(BaseInvocation):
     """Parse prompt using compel package to conditioning."""
@@ -58,9 +52,6 @@ class CompelInvocation(BaseInvocation):
         description=FieldDescriptions.clip,
         input=Input.Connection,
     )
-    mask: Optional[TensorField] = InputField(
-        default=None, description="A mask defining the region that this conditioning prompt applies to."
-    )
 
     @torch.no_grad()
     def invoke(self, context: InvocationContext) -> ConditioningOutput:
@@ -71,15 +62,12 @@ class CompelInvocation(BaseInvocation):
         text_encoder_model = text_encoder_info.model
         assert isinstance(text_encoder_model, CLIPTextModel)
 
-        def _lora_loader() -> Iterator[Tuple[LoRAModelRaw, float]]:
+        def _lora_loader() -> Iterator[Tuple[PeftModel, float]]:
             for lora in self.clip.loras:
                 lora_info = context.models.load(lora.lora)
-                assert isinstance(lora_info.model, LoRAModelRaw)
+                assert isinstance(lora_info.model, PeftModel)
                 yield (lora_info.model, lora.weight)
                 del lora_info
-            return
-
-        # loras = [(context.models.get(**lora.dict(exclude={"weight"})).context.model, lora.weight) for lora in self.clip.loras]
 
         ti_list = generate_ti_list(self.prompt, text_encoder_info.config.base, context)
 
@@ -90,7 +78,7 @@ class CompelInvocation(BaseInvocation):
             ),
             text_encoder_info as text_encoder,
             # Apply the LoRA after text_encoder has been moved to its target device for faster patching.
-            ModelPatcher.apply_lora_text_encoder(text_encoder, _lora_loader()),
+            PeftModelPatcher.apply_peft_model_to_text_encoder(text_encoder, _lora_loader(), "text_encoder"),
             # Apply CLIP Skip after LoRA to prevent LoRA application from failing on skipped layers.
             ModelPatcher.apply_clip_skip(text_encoder_model, self.clip.skipped_layers),
         ):
@@ -99,7 +87,7 @@ class CompelInvocation(BaseInvocation):
                 tokenizer=tokenizer,
                 text_encoder=text_encoder,
                 textual_inversion_manager=ti_manager,
-                dtype_for_device_getter=TorchDevice.choose_torch_dtype,
+                dtype_for_device_getter=torch_dtype,
                 truncate_long_prompts=False,
             )
 
@@ -108,19 +96,27 @@ class CompelInvocation(BaseInvocation):
             if context.config.get().log_tokenization:
                 log_tokenization_for_conjunction(conjunction, tokenizer)
 
-            c, _options = compel.build_conditioning_tensor_for_conjunction(conjunction)
+            c, options = compel.build_conditioning_tensor_for_conjunction(conjunction)
+
+            ec = ExtraConditioningInfo(
+                tokens_count_including_eos_bos=get_max_token_count(tokenizer, conjunction),
+                cross_attention_control_args=options.get("cross_attention_control", None),
+            )
 
         c = c.detach().to("cpu")
 
-        conditioning_data = ConditioningFieldData(conditionings=[BasicConditioningInfo(embeds=c)])
+        conditioning_data = ConditioningFieldData(
+            conditionings=[
+                BasicConditioningInfo(
+                    embeds=c,
+                    extra_conditioning=ec,
+                )
+            ]
+        )
 
         conditioning_name = context.conditioning.save(conditioning_data)
-        return ConditioningOutput(
-            conditioning=ConditioningField(
-                conditioning_name=conditioning_name,
-                mask=self.mask,
-            )
-        )
+
+        return ConditioningOutput.build(conditioning_name)
 
 
 class SDXLPromptInvocationBase:
@@ -134,7 +130,7 @@ class SDXLPromptInvocationBase:
         get_pooled: bool,
         lora_prefix: str,
         zero_on_empty: bool,
-    ) -> Tuple[torch.Tensor, Optional[torch.Tensor]]:
+    ) -> Tuple[torch.Tensor, Optional[torch.Tensor], Optional[ExtraConditioningInfo]]:
         tokenizer_info = context.models.load(clip_field.tokenizer)
         tokenizer_model = tokenizer_info.model
         assert isinstance(tokenizer_model, CLIPTokenizer)
@@ -161,18 +157,15 @@ class SDXLPromptInvocationBase:
                 )
             else:
                 c_pooled = None
-            return c, c_pooled
+            return c, c_pooled, None
 
-        def _lora_loader() -> Iterator[Tuple[LoRAModelRaw, float]]:
+        def _lora_loader() -> Iterator[Tuple[PeftModel, float]]:
             for lora in clip_field.loras:
                 lora_info = context.models.load(lora.lora)
                 lora_model = lora_info.model
-                assert isinstance(lora_model, LoRAModelRaw)
+                assert isinstance(lora_model, PeftModel)
                 yield (lora_model, lora.weight)
                 del lora_info
-            return
-
-        # loras = [(context.models.get(**lora.dict(exclude={"weight"})).context.model, lora.weight) for lora in self.clip.loras]
 
         ti_list = generate_ti_list(prompt, text_encoder_info.config.base, context)
 
@@ -183,7 +176,7 @@ class SDXLPromptInvocationBase:
             ),
             text_encoder_info as text_encoder,
             # Apply the LoRA after text_encoder has been moved to its target device for faster patching.
-            ModelPatcher.apply_lora(text_encoder, _lora_loader(), lora_prefix),
+            PeftModelPatcher.apply_peft_model_to_text_encoder(text_encoder, _lora_loader(), lora_prefix),
             # Apply CLIP Skip after LoRA to prevent LoRA application from failing on skipped layers.
             ModelPatcher.apply_clip_skip(text_encoder_model, clip_field.skipped_layers),
         ):
@@ -193,7 +186,7 @@ class SDXLPromptInvocationBase:
                 tokenizer=tokenizer,
                 text_encoder=text_encoder,
                 textual_inversion_manager=ti_manager,
-                dtype_for_device_getter=TorchDevice.choose_torch_dtype,
+                dtype_for_device_getter=torch_dtype,
                 truncate_long_prompts=False,  # TODO:
                 returned_embeddings_type=ReturnedEmbeddingsType.PENULTIMATE_HIDDEN_STATES_NON_NORMALIZED,  # TODO: clip skip
                 requires_pooled=get_pooled,
@@ -206,12 +199,17 @@ class SDXLPromptInvocationBase:
                 log_tokenization_for_conjunction(conjunction, tokenizer)
 
             # TODO: ask for optimizations? to not run text_encoder twice
-            c, _options = compel.build_conditioning_tensor_for_conjunction(conjunction)
+            c, options = compel.build_conditioning_tensor_for_conjunction(conjunction)
             if get_pooled:
                 c_pooled = compel.conditioning_provider.get_pooled_embeddings([prompt])
             else:
                 c_pooled = None
 
+            ec = ExtraConditioningInfo(
+                tokens_count_including_eos_bos=get_max_token_count(tokenizer, conjunction),
+                cross_attention_control_args=options.get("cross_attention_control", None),
+            )
+
         del tokenizer
         del text_encoder
         del tokenizer_info
@@ -221,7 +219,7 @@ class SDXLPromptInvocationBase:
         if c_pooled is not None:
             c_pooled = c_pooled.detach().to("cpu")
 
-        return c, c_pooled
+        return c, c_pooled, ec
 
 
 @invocation(
@@ -229,7 +227,7 @@ class SDXLPromptInvocationBase:
     title="SDXL Prompt",
     tags=["sdxl", "compel", "prompt"],
     category="conditioning",
-    version="1.2.0",
+    version="1.1.1",
 )
 class SDXLCompelPromptInvocation(BaseInvocation, SDXLPromptInvocationBase):
     """Parse prompt using compel package to conditioning."""
@@ -252,19 +250,20 @@ class SDXLCompelPromptInvocation(BaseInvocation, SDXLPromptInvocationBase):
     target_height: int = InputField(default=1024, description="")
     clip: CLIPField = InputField(description=FieldDescriptions.clip, input=Input.Connection, title="CLIP 1")
     clip2: CLIPField = InputField(description=FieldDescriptions.clip, input=Input.Connection, title="CLIP 2")
-    mask: Optional[TensorField] = InputField(
-        default=None, description="A mask defining the region that this conditioning prompt applies to."
-    )
 
     @torch.no_grad()
     def invoke(self, context: InvocationContext) -> ConditioningOutput:
-        c1, c1_pooled = self.run_clip_compel(context, self.clip, self.prompt, False, "lora_te1_", zero_on_empty=True)
+        c1, c1_pooled, ec1 = self.run_clip_compel(
+            context, self.clip, self.prompt, False, "text_encoder", zero_on_empty=True
+        )
         if self.style.strip() == "":
-            c2, c2_pooled = self.run_clip_compel(
-                context, self.clip2, self.prompt, True, "lora_te2_", zero_on_empty=True
+            c2, c2_pooled, ec2 = self.run_clip_compel(
+                context, self.clip2, self.prompt, True, "text_encoder_2", zero_on_empty=True
             )
         else:
-            c2, c2_pooled = self.run_clip_compel(context, self.clip2, self.style, True, "lora_te2_", zero_on_empty=True)
+            c2, c2_pooled, ec2 = self.run_clip_compel(
+                context, self.clip2, self.style, True, "text_encoder_2", zero_on_empty=True
+            )
 
         original_size = (self.original_height, self.original_width)
         crop_coords = (self.crop_top, self.crop_left)
@@ -303,19 +302,17 @@ class SDXLCompelPromptInvocation(BaseInvocation, SDXLPromptInvocationBase):
         conditioning_data = ConditioningFieldData(
             conditionings=[
                 SDXLConditioningInfo(
-                    embeds=torch.cat([c1, c2], dim=-1), pooled_embeds=c2_pooled, add_time_ids=add_time_ids
+                    embeds=torch.cat([c1, c2], dim=-1),
+                    pooled_embeds=c2_pooled,
+                    add_time_ids=add_time_ids,
+                    extra_conditioning=ec1,
                 )
             ]
         )
 
         conditioning_name = context.conditioning.save(conditioning_data)
 
-        return ConditioningOutput(
-            conditioning=ConditioningField(
-                conditioning_name=conditioning_name,
-                mask=self.mask,
-            )
-        )
+        return ConditioningOutput.build(conditioning_name)
 
 
 @invocation(
@@ -343,7 +340,7 @@ class SDXLRefinerCompelPromptInvocation(BaseInvocation, SDXLPromptInvocationBase
     @torch.no_grad()
     def invoke(self, context: InvocationContext) -> ConditioningOutput:
         # TODO: if there will appear lora for refiner - write proper prefix
-        c2, c2_pooled = self.run_clip_compel(context, self.clip2, self.style, True, "<NONE>", zero_on_empty=False)
+        c2, c2_pooled, ec2 = self.run_clip_compel(context, self.clip2, self.style, True, "<NONE>", zero_on_empty=False)
 
         original_size = (self.original_height, self.original_width)
         crop_coords = (self.crop_top, self.crop_left)
@@ -352,7 +349,14 @@ class SDXLRefinerCompelPromptInvocation(BaseInvocation, SDXLPromptInvocationBase
 
         assert c2_pooled is not None
         conditioning_data = ConditioningFieldData(
-            conditionings=[SDXLConditioningInfo(embeds=c2, pooled_embeds=c2_pooled, add_time_ids=add_time_ids)]
+            conditionings=[
+                SDXLConditioningInfo(
+                    embeds=c2,
+                    pooled_embeds=c2_pooled,
+                    add_time_ids=add_time_ids,
+                    extra_conditioning=ec2,  # or None
+                )
+            ]
         )
 
         conditioning_name = context.conditioning.save(conditioning_data)

invokeai/app/invocations/controlnet_image_processors.py

@@ -24,6 +24,7 @@ from pydantic import BaseModel, Field, field_validator, model_validator
 from invokeai.app.invocations.fields import (
     FieldDescriptions,
     ImageField,
+    Input,
     InputField,
     OutputField,
     UIType,
@@ -34,16 +35,22 @@ from invokeai.app.invocations.model import ModelIdentifierField
 from invokeai.app.invocations.primitives import ImageOutput
 from invokeai.app.invocations.util import validate_begin_end_step, validate_weights
 from invokeai.app.services.shared.invocation_context import InvocationContext
-from invokeai.app.util.controlnet_utils import CONTROLNET_MODE_VALUES, CONTROLNET_RESIZE_VALUES, heuristic_resize
 from invokeai.backend.image_util.canny import get_canny_edges
 from invokeai.backend.image_util.depth_anything import DepthAnythingDetector
 from invokeai.backend.image_util.dw_openpose import DWOpenposeDetector
 from invokeai.backend.image_util.hed import HEDProcessor
 from invokeai.backend.image_util.lineart import LineartProcessor
 from invokeai.backend.image_util.lineart_anime import LineartAnimeProcessor
-from invokeai.backend.image_util.util import np_to_pil, pil_to_np
 
-from .baseinvocation import BaseInvocation, BaseInvocationOutput, Classification, invocation, invocation_output
+from .baseinvocation import BaseInvocation, BaseInvocationOutput, invocation, invocation_output
+
+CONTROLNET_MODE_VALUES = Literal["balanced", "more_prompt", "more_control", "unbalanced"]
+CONTROLNET_RESIZE_VALUES = Literal[
+    "just_resize",
+    "crop_resize",
+    "fill_resize",
+    "just_resize_simple",
+]
 
 
 class ControlField(BaseModel):
@@ -79,13 +86,13 @@ class ControlOutput(BaseInvocationOutput):
     control: ControlField = OutputField(description=FieldDescriptions.control)
 
 
-@invocation("controlnet", title="ControlNet", tags=["controlnet"], category="controlnet", version="1.1.2")
+@invocation("controlnet", title="ControlNet", tags=["controlnet"], category="controlnet", version="1.1.1")
 class ControlNetInvocation(BaseInvocation):
     """Collects ControlNet info to pass to other nodes"""
 
     image: ImageField = InputField(description="The control image")
     control_model: ModelIdentifierField = InputField(
-        description=FieldDescriptions.controlnet_model, ui_type=UIType.ControlNetModel
+        description=FieldDescriptions.controlnet_model, input=Input.Direct, ui_type=UIType.ControlNetModel
     )
     control_weight: Union[float, List[float]] = InputField(
         default=1.0, ge=-1, le=2, description="The weight given to the ControlNet"
@@ -164,13 +171,13 @@ class ImageProcessorInvocation(BaseInvocation, WithMetadata, WithBoard):
     title="Canny Processor",
     tags=["controlnet", "canny"],
     category="controlnet",
-    version="1.3.3",
+    version="1.3.2",
 )
 class CannyImageProcessorInvocation(ImageProcessorInvocation):
     """Canny edge detection for ControlNet"""
 
-    detect_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.detect_res)
-    image_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.image_res)
+    detect_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.detect_res)
+    image_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.image_res)
     low_threshold: int = InputField(
         default=100, ge=0, le=255, description="The low threshold of the Canny pixel gradient (0-255)"
     )
@@ -198,13 +205,13 @@ class CannyImageProcessorInvocation(ImageProcessorInvocation):
     title="HED (softedge) Processor",
     tags=["controlnet", "hed", "softedge"],
     category="controlnet",
-    version="1.2.3",
+    version="1.2.2",
 )
 class HedImageProcessorInvocation(ImageProcessorInvocation):
     """Applies HED edge detection to image"""
 
-    detect_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.detect_res)
-    image_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.image_res)
+    detect_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.detect_res)
+    image_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.image_res)
     # safe not supported in controlnet_aux v0.0.3
     # safe: bool = InputField(default=False, description=FieldDescriptions.safe_mode)
     scribble: bool = InputField(default=False, description=FieldDescriptions.scribble_mode)
@@ -227,13 +234,13 @@ class HedImageProcessorInvocation(ImageProcessorInvocation):
     title="Lineart Processor",
     tags=["controlnet", "lineart"],
     category="controlnet",
-    version="1.2.3",
+    version="1.2.2",
 )
 class LineartImageProcessorInvocation(ImageProcessorInvocation):
     """Applies line art processing to image"""
 
-    detect_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.detect_res)
-    image_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.image_res)
+    detect_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.detect_res)
+    image_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.image_res)
     coarse: bool = InputField(default=False, description="Whether to use coarse mode")
 
     def run_processor(self, image: Image.Image) -> Image.Image:
@@ -249,13 +256,13 @@ class LineartImageProcessorInvocation(ImageProcessorInvocation):
     title="Lineart Anime Processor",
     tags=["controlnet", "lineart", "anime"],
     category="controlnet",
-    version="1.2.3",
+    version="1.2.2",
 )
 class LineartAnimeImageProcessorInvocation(ImageProcessorInvocation):
     """Applies line art anime processing to image"""
 
-    detect_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.detect_res)
-    image_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.image_res)
+    detect_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.detect_res)
+    image_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.image_res)
 
     def run_processor(self, image: Image.Image) -> Image.Image:
         processor = LineartAnimeProcessor()
@@ -272,15 +279,15 @@ class LineartAnimeImageProcessorInvocation(ImageProcessorInvocation):
     title="Midas Depth Processor",
     tags=["controlnet", "midas"],
     category="controlnet",
-    version="1.2.4",
+    version="1.2.3",
 )
 class MidasDepthImageProcessorInvocation(ImageProcessorInvocation):
     """Applies Midas depth processing to image"""
 
     a_mult: float = InputField(default=2.0, ge=0, description="Midas parameter `a_mult` (a = a_mult * PI)")
     bg_th: float = InputField(default=0.1, ge=0, description="Midas parameter `bg_th`")
-    detect_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.detect_res)
-    image_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.image_res)
+    detect_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.detect_res)
+    image_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.image_res)
     # depth_and_normal not supported in controlnet_aux v0.0.3
     # depth_and_normal: bool = InputField(default=False, description="whether to use depth and normal mode")
 
@@ -303,13 +310,13 @@ class MidasDepthImageProcessorInvocation(ImageProcessorInvocation):
     title="Normal BAE Processor",
     tags=["controlnet"],
     category="controlnet",
-    version="1.2.3",
+    version="1.2.2",
 )
 class NormalbaeImageProcessorInvocation(ImageProcessorInvocation):
     """Applies NormalBae processing to image"""
 
-    detect_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.detect_res)
-    image_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.image_res)
+    detect_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.detect_res)
+    image_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.image_res)
 
     def run_processor(self, image):
         normalbae_processor = NormalBaeDetector.from_pretrained("lllyasviel/Annotators")
@@ -320,13 +327,13 @@ class NormalbaeImageProcessorInvocation(ImageProcessorInvocation):
 
 
 @invocation(
-    "mlsd_image_processor", title="MLSD Processor", tags=["controlnet", "mlsd"], category="controlnet", version="1.2.3"
+    "mlsd_image_processor", title="MLSD Processor", tags=["controlnet", "mlsd"], category="controlnet", version="1.2.2"
 )
 class MlsdImageProcessorInvocation(ImageProcessorInvocation):
     """Applies MLSD processing to image"""
 
-    detect_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.detect_res)
-    image_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.image_res)
+    detect_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.detect_res)
+    image_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.image_res)
     thr_v: float = InputField(default=0.1, ge=0, description="MLSD parameter `thr_v`")
     thr_d: float = InputField(default=0.1, ge=0, description="MLSD parameter `thr_d`")
 
@@ -343,13 +350,13 @@ class MlsdImageProcessorInvocation(ImageProcessorInvocation):
 
 
 @invocation(
-    "pidi_image_processor", title="PIDI Processor", tags=["controlnet", "pidi"], category="controlnet", version="1.2.3"
+    "pidi_image_processor", title="PIDI Processor", tags=["controlnet", "pidi"], category="controlnet", version="1.2.2"
 )
 class PidiImageProcessorInvocation(ImageProcessorInvocation):
     """Applies PIDI processing to image"""
 
-    detect_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.detect_res)
-    image_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.image_res)
+    detect_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.detect_res)
+    image_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.image_res)
     safe: bool = InputField(default=False, description=FieldDescriptions.safe_mode)
     scribble: bool = InputField(default=False, description=FieldDescriptions.scribble_mode)
 
@@ -370,13 +377,13 @@ class PidiImageProcessorInvocation(ImageProcessorInvocation):
     title="Content Shuffle Processor",
     tags=["controlnet", "contentshuffle"],
     category="controlnet",
-    version="1.2.3",
+    version="1.2.2",
 )
 class ContentShuffleImageProcessorInvocation(ImageProcessorInvocation):
     """Applies content shuffle processing to image"""
 
-    detect_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.detect_res)
-    image_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.image_res)
+    detect_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.detect_res)
+    image_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.image_res)
     h: int = InputField(default=512, ge=0, description="Content shuffle `h` parameter")
     w: int = InputField(default=512, ge=0, description="Content shuffle `w` parameter")
     f: int = InputField(default=256, ge=0, description="Content shuffle `f` parameter")
@@ -400,7 +407,7 @@ class ContentShuffleImageProcessorInvocation(ImageProcessorInvocation):
     title="Zoe (Depth) Processor",
     tags=["controlnet", "zoe", "depth"],
     category="controlnet",
-    version="1.2.3",
+    version="1.2.2",
 )
 class ZoeDepthImageProcessorInvocation(ImageProcessorInvocation):
     """Applies Zoe depth processing to image"""
@@ -416,15 +423,15 @@ class ZoeDepthImageProcessorInvocation(ImageProcessorInvocation):
     title="Mediapipe Face Processor",
     tags=["controlnet", "mediapipe", "face"],
     category="controlnet",
-    version="1.2.4",
+    version="1.2.3",
 )
 class MediapipeFaceProcessorInvocation(ImageProcessorInvocation):
     """Applies mediapipe face processing to image"""
 
     max_faces: int = InputField(default=1, ge=1, description="Maximum number of faces to detect")
     min_confidence: float = InputField(default=0.5, ge=0, le=1, description="Minimum confidence for face detection")
-    detect_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.detect_res)
-    image_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.image_res)
+    detect_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.detect_res)
+    image_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.image_res)
 
     def run_processor(self, image):
         mediapipe_face_processor = MediapipeFaceDetector()
@@ -443,7 +450,7 @@ class MediapipeFaceProcessorInvocation(ImageProcessorInvocation):
     title="Leres (Depth) Processor",
     tags=["controlnet", "leres", "depth"],
     category="controlnet",
-    version="1.2.3",
+    version="1.2.2",
 )
 class LeresImageProcessorInvocation(ImageProcessorInvocation):
     """Applies leres processing to image"""
@@ -451,8 +458,8 @@ class LeresImageProcessorInvocation(ImageProcessorInvocation):
     thr_a: float = InputField(default=0, description="Leres parameter `thr_a`")
     thr_b: float = InputField(default=0, description="Leres parameter `thr_b`")
     boost: bool = InputField(default=False, description="Whether to use boost mode")
-    detect_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.detect_res)
-    image_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.image_res)
+    detect_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.detect_res)
+    image_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.image_res)
 
     def run_processor(self, image):
         leres_processor = LeresDetector.from_pretrained("lllyasviel/Annotators")
@@ -472,7 +479,7 @@ class LeresImageProcessorInvocation(ImageProcessorInvocation):
     title="Tile Resample Processor",
     tags=["controlnet", "tile"],
     category="controlnet",
-    version="1.2.3",
+    version="1.2.2",
 )
 class TileResamplerProcessorInvocation(ImageProcessorInvocation):
     """Tile resampler processor"""
@@ -512,13 +519,13 @@ class TileResamplerProcessorInvocation(ImageProcessorInvocation):
     title="Segment Anything Processor",
     tags=["controlnet", "segmentanything"],
     category="controlnet",
-    version="1.2.4",
+    version="1.2.3",
 )
 class SegmentAnythingProcessorInvocation(ImageProcessorInvocation):
     """Applies segment anything processing to image"""
 
-    detect_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.detect_res)
-    image_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.image_res)
+    detect_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.detect_res)
+    image_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.image_res)
 
     def run_processor(self, image):
         # segment_anything_processor = SamDetector.from_pretrained("ybelkada/segment-anything", subfolder="checkpoints")
@@ -559,12 +566,12 @@ class SamDetectorReproducibleColors(SamDetector):
     title="Color Map Processor",
     tags=["controlnet"],
     category="controlnet",
-    version="1.2.3",
+    version="1.2.2",
 )
 class ColorMapImageProcessorInvocation(ImageProcessorInvocation):
     """Generates a color map from the provided image"""
 
-    color_map_tile_size: int = InputField(default=64, ge=1, description=FieldDescriptions.tile_size)
+    color_map_tile_size: int = InputField(default=64, ge=0, description=FieldDescriptions.tile_size)
 
     def run_processor(self, image: Image.Image):
         np_image = np.array(image, dtype=np.uint8)
@@ -591,7 +598,7 @@ DEPTH_ANYTHING_MODEL_SIZES = Literal["large", "base", "small"]
     title="Depth Anything Processor",
     tags=["controlnet", "depth", "depth anything"],
     category="controlnet",
-    version="1.1.2",
+    version="1.1.1",
 )
 class DepthAnythingImageProcessorInvocation(ImageProcessorInvocation):
     """Generates a depth map based on the Depth Anything algorithm"""
@@ -599,7 +606,7 @@ class DepthAnythingImageProcessorInvocation(ImageProcessorInvocation):
     model_size: DEPTH_ANYTHING_MODEL_SIZES = InputField(
         default="small", description="The size of the depth model to use"
     )
-    resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.image_res)
+    resolution: int = InputField(default=512, ge=64, multiple_of=64, description=FieldDescriptions.image_res)
 
     def run_processor(self, image: Image.Image):
         depth_anything_detector = DepthAnythingDetector()
@@ -614,7 +621,7 @@ class DepthAnythingImageProcessorInvocation(ImageProcessorInvocation):
     title="DW Openpose Image Processor",
     tags=["controlnet", "dwpose", "openpose"],
     category="controlnet",
-    version="1.1.1",
+    version="1.1.0",
 )
 class DWOpenposeImageProcessorInvocation(ImageProcessorInvocation):
     """Generates an openpose pose from an image using DWPose"""
@@ -622,7 +629,7 @@ class DWOpenposeImageProcessorInvocation(ImageProcessorInvocation):
     draw_body: bool = InputField(default=True)
     draw_face: bool = InputField(default=False)
     draw_hands: bool = InputField(default=False)
-    image_resolution: int = InputField(default=512, ge=1, description=FieldDescriptions.image_res)
+    image_resolution: int = InputField(default=512, ge=0, description=FieldDescriptions.image_res)
 
     def run_processor(self, image: Image.Image):
         dw_openpose = DWOpenposeDetector()
@@ -634,27 +641,3 @@ class DWOpenposeImageProcessorInvocation(ImageProcessorInvocation):
             resolution=self.image_resolution,
         )
         return processed_image
-
-
-@invocation(
-    "heuristic_resize",
-    title="Heuristic Resize",
-    tags=["image, controlnet"],
-    category="image",
-    version="1.0.1",
-    classification=Classification.Prototype,
-)
-class HeuristicResizeInvocation(BaseInvocation):
-    """Resize an image using a heuristic method. Preserves edge maps."""
-
-    image: ImageField = InputField(description="The image to resize")
-    width: int = InputField(default=512, ge=1, description="The width to resize to (px)")
-    height: int = InputField(default=512, ge=1, description="The height to resize to (px)")
-
-    def invoke(self, context: InvocationContext) -> ImageOutput:
-        image = context.images.get_pil(self.image.image_name, "RGB")
-        np_img = pil_to_np(image)
-        np_resized = heuristic_resize(np_img, (self.width, self.height))
-        resized = np_to_pil(np_resized)
-        image_dto = context.images.save(image=resized)
-        return ImageOutput.build(image_dto)

invokeai/app/invocations/fields.py

@@ -203,12 +203,6 @@ class DenoiseMaskField(BaseModel):
     gradient: bool = Field(default=False, description="Used for gradient inpainting")
 
 
-class TensorField(BaseModel):
-    """A tensor primitive field."""
-
-    tensor_name: str = Field(description="The name of a tensor.")
-
-
 class LatentsField(BaseModel):
     """A latents tensor primitive field"""
 
@@ -232,11 +226,7 @@ class ConditioningField(BaseModel):
     """A conditioning tensor primitive value"""
 
     conditioning_name: str = Field(description="The name of conditioning tensor")
-    mask: Optional[TensorField] = Field(
-        default=None,
-        description="The mask associated with this conditioning tensor. Excluded regions should be set to False, "
-        "included regions should be set to True.",
-    )
+    # endregion
 
 
 class MetadataField(RootModel[dict[str, Any]]):

invokeai/app/invocations/image.py

@@ -1,5 +1,6 @@
 # Copyright (c) 2022 Kyle Schouviller (https://github.com/kyle0654)
 
+from pathlib import Path
 from typing import Literal, Optional
 
 import cv2
@@ -503,7 +504,7 @@ class ImageInverseLerpInvocation(BaseInvocation, WithMetadata, WithBoard):
     title="Blur NSFW Image",
     tags=["image", "nsfw"],
     category="image",
-    version="1.2.3",
+    version="1.2.2",
 )
 class ImageNSFWBlurInvocation(BaseInvocation, WithMetadata, WithBoard):
     """Add blur to NSFW-flagged images"""
@@ -515,12 +516,23 @@ class ImageNSFWBlurInvocation(BaseInvocation, WithMetadata, WithBoard):
 
         logger = context.logger
         logger.debug("Running NSFW checker")
-        image = SafetyChecker.blur_if_nsfw(image)
+        if SafetyChecker.has_nsfw_concept(image):
+            logger.info("A potentially NSFW image has been detected. Image will be blurred.")
+            blurry_image = image.filter(filter=ImageFilter.GaussianBlur(radius=32))
+            caution = self._get_caution_img()
+            blurry_image.paste(caution, (0, 0), caution)
+            image = blurry_image
 
         image_dto = context.images.save(image=image)
 
         return ImageOutput.build(image_dto)
 
+    def _get_caution_img(self) -> Image.Image:
+        import invokeai.app.assets.images as image_assets
+
+        caution = Image.open(Path(image_assets.__path__[0]) / "caution.png")
+        return caution.resize((caution.width // 2, caution.height // 2))
+
 
 @invocation(
     "img_watermark",

invokeai/app/invocations/infill.py

@@ -1,91 +1,154 @@
-from abc import abstractmethod
-from typing import Literal, get_args
+# Copyright (c) 2022 Kyle Schouviller (https://github.com/kyle0654) and the InvokeAI Team
 
-from PIL import Image
+import math
+from typing import Literal, Optional, get_args
+
+import numpy as np
+from PIL import Image, ImageOps
 
 from invokeai.app.invocations.fields import ColorField, ImageField
 from invokeai.app.invocations.primitives import ImageOutput
 from invokeai.app.services.shared.invocation_context import InvocationContext
+from invokeai.app.util.download_with_progress import download_with_progress_bar
 from invokeai.app.util.misc import SEED_MAX
-from invokeai.backend.image_util.infill_methods.cv2_inpaint import cv2_inpaint
-from invokeai.backend.image_util.infill_methods.lama import LaMA
-from invokeai.backend.image_util.infill_methods.mosaic import infill_mosaic
-from invokeai.backend.image_util.infill_methods.patchmatch import PatchMatch, infill_patchmatch
-from invokeai.backend.image_util.infill_methods.tile import infill_tile
-from invokeai.backend.util.logging import InvokeAILogger
+from invokeai.backend.image_util.cv2_inpaint import cv2_inpaint
+from invokeai.backend.image_util.lama import LaMA
+from invokeai.backend.image_util.patchmatch import PatchMatch
 
 from .baseinvocation import BaseInvocation, invocation
 from .fields import InputField, WithBoard, WithMetadata
 from .image import PIL_RESAMPLING_MAP, PIL_RESAMPLING_MODES
 
-logger = InvokeAILogger.get_logger()
 
-
-def get_infill_methods():
-    methods = Literal["tile", "color", "lama", "cv2"]  # TODO: add mosaic back
+def infill_methods() -> list[str]:
+    methods = ["tile", "solid", "lama", "cv2"]
     if PatchMatch.patchmatch_available():
-        methods = Literal["patchmatch", "tile", "color", "lama", "cv2"]  # TODO: add mosaic back
+        methods.insert(0, "patchmatch")
     return methods
 
 
-INFILL_METHODS = get_infill_methods()
+INFILL_METHODS = Literal[tuple(infill_methods())]
 DEFAULT_INFILL_METHOD = "patchmatch" if "patchmatch" in get_args(INFILL_METHODS) else "tile"
 
 
-class InfillImageProcessorInvocation(BaseInvocation, WithMetadata, WithBoard):
-    """Base class for invocations that preprocess images for Infilling"""
+def infill_lama(im: Image.Image) -> Image.Image:
+    lama = LaMA()
+    return lama(im)
 
-    image: ImageField = InputField(description="The image to process")
 
-    @abstractmethod
-    def infill(self, image: Image.Image) -> Image.Image:
-        """Infill the image with the specified method"""
-        pass
+def infill_patchmatch(im: Image.Image) -> Image.Image:
+    if im.mode != "RGBA":
+        return im
 
-    def load_image(self, context: InvocationContext) -> tuple[Image.Image, bool]:
-        """Process the image to have an alpha channel before being infilled"""
-        image = context.images.get_pil(self.image.image_name)
-        has_alpha = True if image.mode == "RGBA" else False
-        return image, has_alpha
+    # Skip patchmatch if patchmatch isn't available
+    if not PatchMatch.patchmatch_available():
+        return im
 
-    def invoke(self, context: InvocationContext) -> ImageOutput:
-        # Retrieve and process image to be infilled
-        input_image, has_alpha = self.load_image(context)
+    # Patchmatch (note, we may want to expose patch_size? Increasing it significantly impacts performance though)
+    im_patched_np = PatchMatch.inpaint(im.convert("RGB"), ImageOps.invert(im.split()[-1]), patch_size=3)
+    im_patched = Image.fromarray(im_patched_np, mode="RGB")
+    return im_patched
 
-        # If the input image has no alpha channel, return it
-        if has_alpha is False:
-            return ImageOutput.build(context.images.get_dto(self.image.image_name))
 
-        # Perform Infill action
-        infilled_image = self.infill(input_image)
+def infill_cv2(im: Image.Image) -> Image.Image:
+    return cv2_inpaint(im)
 
-        # Create ImageDTO for Infilled Image
-        infilled_image_dto = context.images.save(image=infilled_image)
 
-        # Return Infilled Image
-        return ImageOutput.build(infilled_image_dto)
+def get_tile_images(image: np.ndarray, width=8, height=8):
+    _nrows, _ncols, depth = image.shape
+    _strides = image.strides
+
+    nrows, _m = divmod(_nrows, height)
+    ncols, _n = divmod(_ncols, width)
+    if _m != 0 or _n != 0:
+        return None
+
+    return np.lib.stride_tricks.as_strided(
+        np.ravel(image),
+        shape=(nrows, ncols, height, width, depth),
+        strides=(height * _strides[0], width * _strides[1], *_strides),
+        writeable=False,
+    )
+
+
+def tile_fill_missing(im: Image.Image, tile_size: int = 16, seed: Optional[int] = None) -> Image.Image:
+    # Only fill if there's an alpha layer
+    if im.mode != "RGBA":
+        return im
+
+    a = np.asarray(im, dtype=np.uint8)
+
+    tile_size_tuple = (tile_size, tile_size)
+
+    # Get the image as tiles of a specified size
+    tiles = get_tile_images(a, *tile_size_tuple).copy()
+
+    # Get the mask as tiles
+    tiles_mask = tiles[:, :, :, :, 3]
+
+    # Find any mask tiles with any fully transparent pixels (we will be replacing these later)
+    tmask_shape = tiles_mask.shape
+    tiles_mask = tiles_mask.reshape(math.prod(tiles_mask.shape))
+    n, ny = (math.prod(tmask_shape[0:2])), math.prod(tmask_shape[2:])
+    tiles_mask = tiles_mask > 0
+    tiles_mask = tiles_mask.reshape((n, ny)).all(axis=1)
+
+    # Get RGB tiles in single array and filter by the mask
+    tshape = tiles.shape
+    tiles_all = tiles.reshape((math.prod(tiles.shape[0:2]), *tiles.shape[2:]))
+    filtered_tiles = tiles_all[tiles_mask]
+
+    if len(filtered_tiles) == 0:
+        return im
+
+    # Find all invalid tiles and replace with a random valid tile
+    replace_count = (tiles_mask == False).sum()  # noqa: E712
+    rng = np.random.default_rng(seed=seed)
+    tiles_all[np.logical_not(tiles_mask)] = filtered_tiles[rng.choice(filtered_tiles.shape[0], replace_count), :, :, :]
+
+    # Convert back to an image
+    tiles_all = tiles_all.reshape(tshape)
+    tiles_all = tiles_all.swapaxes(1, 2)
+    st = tiles_all.reshape(
+        (
+            math.prod(tiles_all.shape[0:2]),
+            math.prod(tiles_all.shape[2:4]),
+            tiles_all.shape[4],
+        )
+    )
+    si = Image.fromarray(st, mode="RGBA")
+
+    return si
 
 
 @invocation("infill_rgba", title="Solid Color Infill", tags=["image", "inpaint"], category="inpaint", version="1.2.2")
-class InfillColorInvocation(InfillImageProcessorInvocation):
+class InfillColorInvocation(BaseInvocation, WithMetadata, WithBoard):
     """Infills transparent areas of an image with a solid color"""
 
+    image: ImageField = InputField(description="The image to infill")
     color: ColorField = InputField(
         default=ColorField(r=127, g=127, b=127, a=255),
         description="The color to use to infill",
     )
 
-    def infill(self, image: Image.Image):
+    def invoke(self, context: InvocationContext) -> ImageOutput:
+        image = context.images.get_pil(self.image.image_name)
+
         solid_bg = Image.new("RGBA", image.size, self.color.tuple())
         infilled = Image.alpha_composite(solid_bg, image.convert("RGBA"))
+
         infilled.paste(image, (0, 0), image.split()[-1])
-        return infilled
+
+        image_dto = context.images.save(image=infilled)
+
+        return ImageOutput.build(image_dto)
 
 
 @invocation("infill_tile", title="Tile Infill", tags=["image", "inpaint"], category="inpaint", version="1.2.3")
-class InfillTileInvocation(InfillImageProcessorInvocation):
+class InfillTileInvocation(BaseInvocation, WithMetadata, WithBoard):
     """Infills transparent areas of an image with tiles of the image"""
 
+    image: ImageField = InputField(description="The image to infill")
     tile_size: int = InputField(default=32, ge=1, description="The tile size (px)")
     seed: int = InputField(
         default=0,
@@ -94,74 +157,92 @@ class InfillTileInvocation(InfillImageProcessorInvocation):
         description="The seed to use for tile generation (omit for random)",
     )
 
-    def infill(self, image: Image.Image):
-        output = infill_tile(image, seed=self.seed, tile_size=self.tile_size)
-        return output.infilled
+    def invoke(self, context: InvocationContext) -> ImageOutput:
+        image = context.images.get_pil(self.image.image_name)
+
+        infilled = tile_fill_missing(image.copy(), seed=self.seed, tile_size=self.tile_size)
+        infilled.paste(image, (0, 0), image.split()[-1])
+
+        image_dto = context.images.save(image=infilled)
+
+        return ImageOutput.build(image_dto)
 
 
 @invocation(
     "infill_patchmatch", title="PatchMatch Infill", tags=["image", "inpaint"], category="inpaint", version="1.2.2"
 )
-class InfillPatchMatchInvocation(InfillImageProcessorInvocation):
+class InfillPatchMatchInvocation(BaseInvocation, WithMetadata, WithBoard):
     """Infills transparent areas of an image using the PatchMatch algorithm"""
 
+    image: ImageField = InputField(description="The image to infill")
     downscale: float = InputField(default=2.0, gt=0, description="Run patchmatch on downscaled image to speedup infill")
     resample_mode: PIL_RESAMPLING_MODES = InputField(default="bicubic", description="The resampling mode")
 
-    def infill(self, image: Image.Image):
+    def invoke(self, context: InvocationContext) -> ImageOutput:
+        image = context.images.get_pil(self.image.image_name).convert("RGBA")
+
         resample_mode = PIL_RESAMPLING_MAP[self.resample_mode]
 
+        infill_image = image.copy()
         width = int(image.width / self.downscale)
         height = int(image.height / self.downscale)
-
-        infilled = image.resize(
+        infill_image = infill_image.resize(
             (width, height),
             resample=resample_mode,
         )
-        infilled = infill_patchmatch(image)
+
+        if PatchMatch.patchmatch_available():
+            infilled = infill_patchmatch(infill_image)
+        else:
+            raise ValueError("PatchMatch is not available on this system")
+
         infilled = infilled.resize(
             (image.width, image.height),
             resample=resample_mode,
         )
-        infilled.paste(image, (0, 0), mask=image.split()[-1])
 
-        return infilled
+        infilled.paste(image, (0, 0), mask=image.split()[-1])
+        # image.paste(infilled, (0, 0), mask=image.split()[-1])
+
+        image_dto = context.images.save(image=infilled)
+
+        return ImageOutput.build(image_dto)
 
 
 @invocation("infill_lama", title="LaMa Infill", tags=["image", "inpaint"], category="inpaint", version="1.2.2")
-class LaMaInfillInvocation(InfillImageProcessorInvocation):
+class LaMaInfillInvocation(BaseInvocation, WithMetadata, WithBoard):
     """Infills transparent areas of an image using the LaMa model"""
 
-    def infill(self, image: Image.Image):
-        lama = LaMA()
-        return lama(image)
+    image: ImageField = InputField(description="The image to infill")
+
+    def invoke(self, context: InvocationContext) -> ImageOutput:
+        image = context.images.get_pil(self.image.image_name)
+
+        # Downloads the LaMa model if it doesn't already exist
+        download_with_progress_bar(
+            name="LaMa Inpainting Model",
+            url="https://github.com/Sanster/models/releases/download/add_big_lama/big-lama.pt",
+            dest_path=context.config.get().models_path / "core/misc/lama/lama.pt",
+        )
+
+        infilled = infill_lama(image.copy())
+
+        image_dto = context.images.save(image=infilled)
+
+        return ImageOutput.build(image_dto)
 
 
 @invocation("infill_cv2", title="CV2 Infill", tags=["image", "inpaint"], category="inpaint", version="1.2.2")
-class CV2InfillInvocation(InfillImageProcessorInvocation):
+class CV2InfillInvocation(BaseInvocation, WithMetadata, WithBoard):
     """Infills transparent areas of an image using OpenCV Inpainting"""
 
-    def infill(self, image: Image.Image):
-        return cv2_inpaint(image)
-
-
-# @invocation(
-#     "infill_mosaic", title="Mosaic Infill", tags=["image", "inpaint", "outpaint"], category="inpaint", version="1.0.0"
-# )
-class MosaicInfillInvocation(InfillImageProcessorInvocation):
-    """Infills transparent areas of an image with a mosaic pattern drawing colors from the rest of the image"""
-
     image: ImageField = InputField(description="The image to infill")
-    tile_width: int = InputField(default=64, description="Width of the tile")
-    tile_height: int = InputField(default=64, description="Height of the tile")
-    min_color: ColorField = InputField(
-        default=ColorField(r=0, g=0, b=0, a=255),
-        description="The min threshold for color",
-    )
-    max_color: ColorField = InputField(
-        default=ColorField(r=255, g=255, b=255, a=255),
-        description="The max threshold for color",
-    )
 
-    def infill(self, image: Image.Image):
-        return infill_mosaic(image, (self.tile_width, self.tile_height), self.min_color.tuple(), self.max_color.tuple())
+    def invoke(self, context: InvocationContext) -> ImageOutput:
+        image = context.images.get_pil(self.image.image_name)
+
+        infilled = infill_cv2(image.copy())
+
+        image_dto = context.images.save(image=infilled)
+
+        return ImageOutput.build(image_dto)

invokeai/app/invocations/ip_adapter.py

@@ -1,11 +1,11 @@
 from builtins import float
-from typing import List, Literal, Optional, Union
+from typing import List, Literal, Union
 
 from pydantic import BaseModel, Field, field_validator, model_validator
 from typing_extensions import Self
 
 from invokeai.app.invocations.baseinvocation import BaseInvocation, BaseInvocationOutput, invocation, invocation_output
-from invokeai.app.invocations.fields import FieldDescriptions, InputField, OutputField, TensorField, UIType
+from invokeai.app.invocations.fields import FieldDescriptions, Input, InputField, OutputField, UIType
 from invokeai.app.invocations.model import ModelIdentifierField
 from invokeai.app.invocations.primitives import ImageField
 from invokeai.app.invocations.util import validate_begin_end_step, validate_weights
@@ -23,19 +23,13 @@ class IPAdapterField(BaseModel):
     image: Union[ImageField, List[ImageField]] = Field(description="The IP-Adapter image prompt(s).")
     ip_adapter_model: ModelIdentifierField = Field(description="The IP-Adapter model to use.")
     image_encoder_model: ModelIdentifierField = Field(description="The name of the CLIP image encoder model.")
-    weight: Union[float, List[float]] = Field(default=1, description="The weight given to the IP-Adapter.")
-    target_blocks: List[str] = Field(default=[], description="The IP Adapter blocks to apply")
+    weight: Union[float, List[float]] = Field(default=1, description="The weight given to the ControlNet")
     begin_step_percent: float = Field(
         default=0, ge=0, le=1, description="When the IP-Adapter is first applied (% of total steps)"
     )
     end_step_percent: float = Field(
         default=1, ge=0, le=1, description="When the IP-Adapter is last applied (% of total steps)"
     )
-    mask: Optional[TensorField] = Field(
-        default=None,
-        description="The bool mask associated with this IP-Adapter. Excluded regions should be set to False, included "
-        "regions should be set to True.",
-    )
 
     @field_validator("weight")
     @classmethod
@@ -58,7 +52,7 @@ class IPAdapterOutput(BaseInvocationOutput):
 CLIP_VISION_MODEL_MAP = {"ViT-H": "ip_adapter_sd_image_encoder", "ViT-G": "ip_adapter_sdxl_image_encoder"}
 
 
-@invocation("ip_adapter", title="IP-Adapter", tags=["ip_adapter", "control"], category="ip_adapter", version="1.4.1")
+@invocation("ip_adapter", title="IP-Adapter", tags=["ip_adapter", "control"], category="ip_adapter", version="1.2.2")
 class IPAdapterInvocation(BaseInvocation):
     """Collects IP-Adapter info to pass to other nodes."""
 
@@ -67,29 +61,24 @@ class IPAdapterInvocation(BaseInvocation):
     ip_adapter_model: ModelIdentifierField = InputField(
         description="The IP-Adapter model.",
         title="IP-Adapter Model",
+        input=Input.Direct,
         ui_order=-1,
         ui_type=UIType.IPAdapterModel,
     )
-    clip_vision_model: Literal["ViT-H", "ViT-G"] = InputField(
+    clip_vision_model: Literal["auto", "ViT-H", "ViT-G"] = InputField(
         description="CLIP Vision model to use. Overrides model settings. Mandatory for checkpoint models.",
-        default="ViT-H",
+        default="auto",
         ui_order=2,
     )
     weight: Union[float, List[float]] = InputField(
         default=1, description="The weight given to the IP-Adapter", title="Weight"
     )
-    method: Literal["full", "style", "composition"] = InputField(
-        default="full", description="The method to apply the IP-Adapter"
-    )
     begin_step_percent: float = InputField(
         default=0, ge=0, le=1, description="When the IP-Adapter is first applied (% of total steps)"
     )
     end_step_percent: float = InputField(
         default=1, ge=0, le=1, description="When the IP-Adapter is last applied (% of total steps)"
     )
-    mask: Optional[TensorField] = InputField(
-        default=None, description="A mask defining the region that this IP-Adapter applies to."
-    )
 
     @field_validator("weight")
     @classmethod
@@ -107,43 +96,27 @@ class IPAdapterInvocation(BaseInvocation):
         ip_adapter_info = context.models.get_config(self.ip_adapter_model.key)
         assert isinstance(ip_adapter_info, (IPAdapterInvokeAIConfig, IPAdapterCheckpointConfig))
 
-        if isinstance(ip_adapter_info, IPAdapterInvokeAIConfig):
-            image_encoder_model_id = ip_adapter_info.image_encoder_model_id
-            image_encoder_model_name = image_encoder_model_id.split("/")[-1].strip()
+        if self.clip_vision_model == "auto":
+            if isinstance(ip_adapter_info, IPAdapterInvokeAIConfig):
+                image_encoder_model_id = ip_adapter_info.image_encoder_model_id
+                image_encoder_model_name = image_encoder_model_id.split("/")[-1].strip()
+            else:
+                raise RuntimeError(
+                    "You need to set the appropriate CLIP Vision model for checkpoint IP Adapter models."
+                )
         else:
             image_encoder_model_name = CLIP_VISION_MODEL_MAP[self.clip_vision_model]
 
         image_encoder_model = self._get_image_encoder(context, image_encoder_model_name)
 
-        if self.method == "style":
-            if ip_adapter_info.base == "sd-1":
-                target_blocks = ["up_blocks.1"]
-            elif ip_adapter_info.base == "sdxl":
-                target_blocks = ["up_blocks.0.attentions.1"]
-            else:
-                raise ValueError(f"Unsupported IP-Adapter base type: '{ip_adapter_info.base}'.")
-        elif self.method == "composition":
-            if ip_adapter_info.base == "sd-1":
-                target_blocks = ["down_blocks.2", "mid_block"]
-            elif ip_adapter_info.base == "sdxl":
-                target_blocks = ["down_blocks.2.attentions.1"]
-            else:
-                raise ValueError(f"Unsupported IP-Adapter base type: '{ip_adapter_info.base}'.")
-        elif self.method == "full":
-            target_blocks = ["block"]
-        else:
-            raise ValueError(f"Unexpected IP-Adapter method: '{self.method}'.")
-
         return IPAdapterOutput(
             ip_adapter=IPAdapterField(
                 image=self.image,
                 ip_adapter_model=self.ip_adapter_model,
                 image_encoder_model=ModelIdentifierField.from_config(image_encoder_model),
                 weight=self.weight,
-                target_blocks=target_blocks,
                 begin_step_percent=self.begin_step_percent,
                 end_step_percent=self.end_step_percent,
-                mask=self.mask,
             ),
         )
 

invokeai/app/invocations/latent.py

@@ -1,16 +1,16 @@
 # Copyright (c) 2023 Kyle Schouviller (https://github.com/kyle0654)
-import inspect
+
 import math
 from contextlib import ExitStack
 from functools import singledispatchmethod
-from typing import Any, Dict, Iterator, List, Literal, Optional, Tuple, Union
+from typing import Any, Iterator, List, Literal, Optional, Tuple, Union
 
 import einops
 import numpy as np
 import numpy.typing as npt
 import torch
-import torchvision
 import torchvision.transforms as T
+from diffusers import AutoencoderKL, AutoencoderTiny
 from diffusers.configuration_utils import ConfigMixin
 from diffusers.image_processor import VaeImageProcessor
 from diffusers.models.adapter import T2IAdapter
@@ -20,12 +20,9 @@ from diffusers.models.attention_processor import (
     LoRAXFormersAttnProcessor,
     XFormersAttnProcessor,
 )
-from diffusers.models.autoencoders.autoencoder_kl import AutoencoderKL
-from diffusers.models.autoencoders.autoencoder_tiny import AutoencoderTiny
 from diffusers.models.unets.unet_2d_condition import UNet2DConditionModel
-from diffusers.schedulers.scheduling_dpmsolver_sde import DPMSolverSDEScheduler
-from diffusers.schedulers.scheduling_tcd import TCDScheduler
-from diffusers.schedulers.scheduling_utils import SchedulerMixin as Scheduler
+from diffusers.schedulers import DPMSolverSDEScheduler
+from diffusers.schedulers import SchedulerMixin as Scheduler
 from PIL import Image, ImageFilter
 from pydantic import field_validator
 from torchvision.transforms.functional import resize as tv_resize
@@ -51,36 +48,31 @@ from invokeai.app.invocations.t2i_adapter import T2IAdapterField
 from invokeai.app.services.shared.invocation_context import InvocationContext
 from invokeai.app.util.controlnet_utils import prepare_control_image
 from invokeai.backend.ip_adapter.ip_adapter import IPAdapter, IPAdapterPlus
-from invokeai.backend.lora import LoRAModelRaw
 from invokeai.backend.model_manager import BaseModelType, LoadedModel
-from invokeai.backend.model_manager.config import MainConfigBase, ModelVariantType
 from invokeai.backend.model_patcher import ModelPatcher
+from invokeai.backend.peft.peft_model import PeftModel
+from invokeai.backend.peft.peft_model_patcher import PeftModelPatcher
 from invokeai.backend.stable_diffusion import PipelineIntermediateState, set_seamless
-from invokeai.backend.stable_diffusion.diffusion.conditioning_data import (
-    BasicConditioningInfo,
-    IPAdapterConditioningInfo,
-    IPAdapterData,
-    Range,
-    SDXLConditioningInfo,
-    TextConditioningData,
-    TextConditioningRegions,
-)
-from invokeai.backend.util.mask import to_standard_float_mask
+from invokeai.backend.stable_diffusion.diffusion.conditioning_data import ConditioningData, IPAdapterConditioningInfo
 from invokeai.backend.util.silence_warnings import SilenceWarnings
 
 from ...backend.stable_diffusion.diffusers_pipeline import (
     ControlNetData,
+    IPAdapterData,
     StableDiffusionGeneratorPipeline,
     T2IAdapterData,
     image_resized_to_grid_as_tensor,
 )
 from ...backend.stable_diffusion.schedulers import SCHEDULER_MAP
-from ...backend.util.devices import TorchDevice
+from ...backend.util.devices import choose_precision, choose_torch_device
 from .baseinvocation import BaseInvocation, BaseInvocationOutput, invocation, invocation_output
 from .controlnet_image_processors import ControlField
 from .model import ModelIdentifierField, UNetField, VAEField
 
-DEFAULT_PRECISION = TorchDevice.choose_torch_dtype()
+if choose_torch_device() == torch.device("mps"):
+    from torch import mps
+
+DEFAULT_PRECISION = choose_precision(choose_torch_device())
 
 
 @invocation_output("scheduler_output")
@@ -188,7 +180,7 @@ class GradientMaskOutput(BaseInvocationOutput):
     title="Create Gradient Mask",
     tags=["mask", "denoise"],
     category="latents",
-    version="1.1.0",
+    version="1.0.0",
 )
 class CreateGradientMaskInvocation(BaseInvocation):
     """Creates mask for denoising model run."""
@@ -201,32 +193,6 @@ class CreateGradientMaskInvocation(BaseInvocation):
     minimum_denoise: float = InputField(
         default=0.0, ge=0, le=1, description="Minimum denoise level for the coherence region", ui_order=4
     )
-    image: Optional[ImageField] = InputField(
-        default=None,
-        description="OPTIONAL: Only connect for specialized Inpainting models, masked_latents will be generated from the image with the VAE",
-        title="[OPTIONAL] Image",
-        ui_order=6,
-    )
-    unet: Optional[UNetField] = InputField(
-        description="OPTIONAL: If the Unet is a specialized Inpainting model, masked_latents will be generated from the image with the VAE",
-        default=None,
-        input=Input.Connection,
-        title="[OPTIONAL] UNet",
-        ui_order=5,
-    )
-    vae: Optional[VAEField] = InputField(
-        default=None,
-        description="OPTIONAL: Only connect for specialized Inpainting models, masked_latents will be generated from the image with the VAE",
-        title="[OPTIONAL] VAE",
-        input=Input.Connection,
-        ui_order=7,
-    )
-    tiled: bool = InputField(default=False, description=FieldDescriptions.tiled, ui_order=8)
-    fp32: bool = InputField(
-        default=DEFAULT_PRECISION == "float32",
-        description=FieldDescriptions.fp32,
-        ui_order=9,
-    )
 
     @torch.no_grad()
     def invoke(self, context: InvocationContext) -> GradientMaskOutput:
@@ -262,27 +228,8 @@ class CreateGradientMaskInvocation(BaseInvocation):
         expanded_mask_image = Image.fromarray((expanded_mask.squeeze(0).numpy() * 255).astype(np.uint8), mode="L")
         expanded_image_dto = context.images.save(expanded_mask_image)
 
-        masked_latents_name = None
-        if self.unet is not None and self.vae is not None and self.image is not None:
-            # all three fields must be present at the same time
-            main_model_config = context.models.get_config(self.unet.unet.key)
-            assert isinstance(main_model_config, MainConfigBase)
-            if main_model_config.variant is ModelVariantType.Inpaint:
-                mask = blur_tensor
-                vae_info: LoadedModel = context.models.load(self.vae.vae)
-                image = context.images.get_pil(self.image.image_name)
-                image_tensor = image_resized_to_grid_as_tensor(image.convert("RGB"))
-                if image_tensor.dim() == 3:
-                    image_tensor = image_tensor.unsqueeze(0)
-                img_mask = tv_resize(mask, image_tensor.shape[-2:], T.InterpolationMode.BILINEAR, antialias=False)
-                masked_image = image_tensor * torch.where(img_mask < 0.5, 0.0, 1.0)
-                masked_latents = ImageToLatentsInvocation.vae_encode(
-                    vae_info, self.fp32, self.tiled, masked_image.clone()
-                )
-                masked_latents_name = context.tensors.save(tensor=masked_latents)
-
         return GradientMaskOutput(
-            denoise_mask=DenoiseMaskField(mask_name=mask_name, masked_latents_name=masked_latents_name, gradient=True),
+            denoise_mask=DenoiseMaskField(mask_name=mask_name, masked_latents_name=None, gradient=True),
             expanded_mask_area=ImageField(image_name=expanded_image_dto.image_name),
         )
 
@@ -329,10 +276,10 @@ def get_scheduler(
 class DenoiseLatentsInvocation(BaseInvocation):
     """Denoises noisy latents to decodable images"""
 
-    positive_conditioning: Union[ConditioningField, list[ConditioningField]] = InputField(
+    positive_conditioning: ConditioningField = InputField(
         description=FieldDescriptions.positive_cond, input=Input.Connection, ui_order=0
     )
-    negative_conditioning: Union[ConditioningField, list[ConditioningField]] = InputField(
+    negative_conditioning: ConditioningField = InputField(
         description=FieldDescriptions.negative_cond, input=Input.Connection, ui_order=1
     )
     noise: Optional[LatentsField] = InputField(
@@ -343,7 +290,7 @@ class DenoiseLatentsInvocation(BaseInvocation):
     )
     steps: int = InputField(default=10, gt=0, description=FieldDescriptions.steps)
     cfg_scale: Union[float, List[float]] = InputField(
-        default=7.5, description=FieldDescriptions.cfg_scale, title="CFG Scale"
+        default=7.5, ge=1, description=FieldDescriptions.cfg_scale, title="CFG Scale"
     )
     denoising_start: float = InputField(
         default=0.0,
@@ -410,175 +357,34 @@ class DenoiseLatentsInvocation(BaseInvocation):
                 raise ValueError("cfg_scale must be greater than 1")
         return v
 
-    def _get_text_embeddings_and_masks(
-        self,
-        cond_list: list[ConditioningField],
-        context: InvocationContext,
-        device: torch.device,
-        dtype: torch.dtype,
-    ) -> tuple[Union[list[BasicConditioningInfo], list[SDXLConditioningInfo]], list[Optional[torch.Tensor]]]:
-        """Get the text embeddings and masks from the input conditioning fields."""
-        text_embeddings: Union[list[BasicConditioningInfo], list[SDXLConditioningInfo]] = []
-        text_embeddings_masks: list[Optional[torch.Tensor]] = []
-        for cond in cond_list:
-            cond_data = context.conditioning.load(cond.conditioning_name)
-            text_embeddings.append(cond_data.conditionings[0].to(device=device, dtype=dtype))
-
-            mask = cond.mask
-            if mask is not None:
-                mask = context.tensors.load(mask.tensor_name)
-            text_embeddings_masks.append(mask)
-
-        return text_embeddings, text_embeddings_masks
-
-    def _preprocess_regional_prompt_mask(
-        self, mask: Optional[torch.Tensor], target_height: int, target_width: int, dtype: torch.dtype
-    ) -> torch.Tensor:
-        """Preprocess a regional prompt mask to match the target height and width.
-        If mask is None, returns a mask of all ones with the target height and width.
-        If mask is not None, resizes the mask to the target height and width using 'nearest' interpolation.
-
-        Returns:
-            torch.Tensor: The processed mask. shape: (1, 1, target_height, target_width).
-        """
-
-        if mask is None:
-            return torch.ones((1, 1, target_height, target_width), dtype=dtype)
-
-        mask = to_standard_float_mask(mask, out_dtype=dtype)
-
-        tf = torchvision.transforms.Resize(
-            (target_height, target_width), interpolation=torchvision.transforms.InterpolationMode.NEAREST
-        )
-
-        # Add a batch dimension to the mask, because torchvision expects shape (batch, channels, h, w).
-        mask = mask.unsqueeze(0)  # Shape: (1, h, w) -> (1, 1, h, w)
-        resized_mask = tf(mask)
-        return resized_mask
-
-    def _concat_regional_text_embeddings(
-        self,
-        text_conditionings: Union[list[BasicConditioningInfo], list[SDXLConditioningInfo]],
-        masks: Optional[list[Optional[torch.Tensor]]],
-        latent_height: int,
-        latent_width: int,
-        dtype: torch.dtype,
-    ) -> tuple[Union[BasicConditioningInfo, SDXLConditioningInfo], Optional[TextConditioningRegions]]:
-        """Concatenate regional text embeddings into a single embedding and track the region masks accordingly."""
-        if masks is None:
-            masks = [None] * len(text_conditionings)
-        assert len(text_conditionings) == len(masks)
-
-        is_sdxl = type(text_conditionings[0]) is SDXLConditioningInfo
-
-        all_masks_are_none = all(mask is None for mask in masks)
-
-        text_embedding = []
-        pooled_embedding = None
-        add_time_ids = None
-        cur_text_embedding_len = 0
-        processed_masks = []
-        embedding_ranges = []
-
-        for prompt_idx, text_embedding_info in enumerate(text_conditionings):
-            mask = masks[prompt_idx]
-
-            if is_sdxl:
-                # We choose a random SDXLConditioningInfo's pooled_embeds and add_time_ids here, with a preference for
-                # prompts without a mask. We prefer prompts without a mask, because they are more likely to contain
-                # global prompt information.  In an ideal case, there should be exactly one global prompt without a
-                # mask, but we don't enforce this.
-
-                # HACK(ryand): The fact that we have to choose a single pooled_embedding and add_time_ids here is a
-                # fundamental interface issue. The SDXL Compel nodes are not designed to be used in the way that we use
-                # them for regional prompting. Ideally, the DenoiseLatents invocation should accept a single
-                # pooled_embeds tensor and a list of standard text embeds with region masks. This change would be a
-                # pretty major breaking change to a popular node, so for now we use this hack.
-                if pooled_embedding is None or mask is None:
-                    pooled_embedding = text_embedding_info.pooled_embeds
-                if add_time_ids is None or mask is None:
-                    add_time_ids = text_embedding_info.add_time_ids
-
-            text_embedding.append(text_embedding_info.embeds)
-            if not all_masks_are_none:
-                embedding_ranges.append(
-                    Range(
-                        start=cur_text_embedding_len, end=cur_text_embedding_len + text_embedding_info.embeds.shape[1]
-                    )
-                )
-                processed_masks.append(
-                    self._preprocess_regional_prompt_mask(mask, latent_height, latent_width, dtype=dtype)
-                )
-
-            cur_text_embedding_len += text_embedding_info.embeds.shape[1]
-
-        text_embedding = torch.cat(text_embedding, dim=1)
-        assert len(text_embedding.shape) == 3  # batch_size, seq_len, token_len
-
-        regions = None
-        if not all_masks_are_none:
-            regions = TextConditioningRegions(
-                masks=torch.cat(processed_masks, dim=1),
-                ranges=embedding_ranges,
-            )
-
-        if is_sdxl:
-            return (
-                SDXLConditioningInfo(embeds=text_embedding, pooled_embeds=pooled_embedding, add_time_ids=add_time_ids),
-                regions,
-            )
-        return BasicConditioningInfo(embeds=text_embedding), regions
-
     def get_conditioning_data(
         self,
         context: InvocationContext,
+        scheduler: Scheduler,
         unet: UNet2DConditionModel,
-        latent_height: int,
-        latent_width: int,
-    ) -> TextConditioningData:
-        # Normalize self.positive_conditioning and self.negative_conditioning to lists.
-        cond_list = self.positive_conditioning
-        if not isinstance(cond_list, list):
-            cond_list = [cond_list]
-        uncond_list = self.negative_conditioning
-        if not isinstance(uncond_list, list):
-            uncond_list = [uncond_list]
+        seed: int,
+    ) -> ConditioningData:
+        positive_cond_data = context.conditioning.load(self.positive_conditioning.conditioning_name)
+        c = positive_cond_data.conditionings[0].to(device=unet.device, dtype=unet.dtype)
 
-        cond_text_embeddings, cond_text_embedding_masks = self._get_text_embeddings_and_masks(
-            cond_list, context, unet.device, unet.dtype
-        )
-        uncond_text_embeddings, uncond_text_embedding_masks = self._get_text_embeddings_and_masks(
-            uncond_list, context, unet.device, unet.dtype
-        )
+        negative_cond_data = context.conditioning.load(self.negative_conditioning.conditioning_name)
+        uc = negative_cond_data.conditionings[0].to(device=unet.device, dtype=unet.dtype)
 
-        cond_text_embedding, cond_regions = self._concat_regional_text_embeddings(
-            text_conditionings=cond_text_embeddings,
-            masks=cond_text_embedding_masks,
-            latent_height=latent_height,
-            latent_width=latent_width,
-            dtype=unet.dtype,
-        )
-        uncond_text_embedding, uncond_regions = self._concat_regional_text_embeddings(
-            text_conditionings=uncond_text_embeddings,
-            masks=uncond_text_embedding_masks,
-            latent_height=latent_height,
-            latent_width=latent_width,
-            dtype=unet.dtype,
-        )
-
-        if isinstance(self.cfg_scale, list):
-            assert (
-                len(self.cfg_scale) == self.steps
-            ), "cfg_scale (list) must have the same length as the number of steps"
-
-        conditioning_data = TextConditioningData(
-            uncond_text=uncond_text_embedding,
-            cond_text=cond_text_embedding,
-            uncond_regions=uncond_regions,
-            cond_regions=cond_regions,
+        conditioning_data = ConditioningData(
+            unconditioned_embeddings=uc,
+            text_embeddings=c,
             guidance_scale=self.cfg_scale,
             guidance_rescale_multiplier=self.cfg_rescale_multiplier,
         )
+
+        conditioning_data = conditioning_data.add_scheduler_args_if_applicable(  # FIXME
+            scheduler,
+            # for ddim scheduler
+            eta=0.0,  # ddim_eta
+            # for ancestral and sde schedulers
+            # flip all bits to have noise different from initial
+            generator=torch.Generator(device=unet.device).manual_seed(seed ^ 0xFFFFFFFF),
+        )
         return conditioning_data
 
     def create_pipeline(
@@ -586,6 +392,13 @@ class DenoiseLatentsInvocation(BaseInvocation):
         unet: UNet2DConditionModel,
         scheduler: Scheduler,
     ) -> StableDiffusionGeneratorPipeline:
+        # TODO:
+        # configure_model_padding(
+        #    unet,
+        #    self.seamless,
+        #    self.seamless_axes,
+        # )
+
         class FakeVae:
             class FakeVaeConfig:
                 def __init__(self) -> None:
@@ -676,10 +489,8 @@ class DenoiseLatentsInvocation(BaseInvocation):
         self,
         context: InvocationContext,
         ip_adapter: Optional[Union[IPAdapterField, list[IPAdapterField]]],
+        conditioning_data: ConditioningData,
         exit_stack: ExitStack,
-        latent_height: int,
-        latent_width: int,
-        dtype: torch.dtype,
     ) -> Optional[list[IPAdapterData]]:
         """If IP-Adapter is enabled, then this function loads the requisite models, and adds the image prompt embeddings
         to the `conditioning_data` (in-place).
@@ -695,6 +506,7 @@ class DenoiseLatentsInvocation(BaseInvocation):
             return None
 
         ip_adapter_data_list = []
+        conditioning_data.ip_adapter_conditioning = []
         for single_ip_adapter in ip_adapter:
             ip_adapter_model: Union[IPAdapter, IPAdapterPlus] = exit_stack.enter_context(
                 context.models.load(single_ip_adapter.ip_adapter_model)
@@ -717,20 +529,16 @@ class DenoiseLatentsInvocation(BaseInvocation):
                     single_ipa_images, image_encoder_model
                 )
 
-            mask = single_ip_adapter.mask
-            if mask is not None:
-                mask = context.tensors.load(mask.tensor_name)
-            mask = self._preprocess_regional_prompt_mask(mask, latent_height, latent_width, dtype=dtype)
+                conditioning_data.ip_adapter_conditioning.append(
+                    IPAdapterConditioningInfo(image_prompt_embeds, uncond_image_prompt_embeds)
+                )
 
             ip_adapter_data_list.append(
                 IPAdapterData(
                     ip_adapter_model=ip_adapter_model,
                     weight=single_ip_adapter.weight,
-                    target_blocks=single_ip_adapter.target_blocks,
                     begin_step_percent=single_ip_adapter.begin_step_percent,
                     end_step_percent=single_ip_adapter.end_step_percent,
-                    ip_adapter_conditioning=IPAdapterConditioningInfo(image_prompt_embeds, uncond_image_prompt_embeds),
-                    mask=mask,
                 )
             )
 
@@ -820,8 +628,7 @@ class DenoiseLatentsInvocation(BaseInvocation):
         steps: int,
         denoising_start: float,
         denoising_end: float,
-        seed: int,
-    ) -> Tuple[int, List[int], int, Dict[str, Any]]:
+    ) -> Tuple[int, List[int], int]:
         assert isinstance(scheduler, ConfigMixin)
         if scheduler.config.get("cpu_only", False):
             scheduler.set_timesteps(steps, device="cpu")
@@ -849,17 +656,7 @@ class DenoiseLatentsInvocation(BaseInvocation):
         timesteps = timesteps[t_start_idx : t_start_idx + t_end_idx]
         num_inference_steps = len(timesteps) // scheduler.order
 
-        scheduler_step_kwargs: Dict[str, Any] = {}
-        scheduler_step_signature = inspect.signature(scheduler.step)
-        if "generator" in scheduler_step_signature.parameters:
-            # At some point, someone decided that schedulers that accept a generator should use the original seed with
-            # all bits flipped. I don't know the original rationale for this, but now we must keep it like this for
-            # reproducibility.
-            scheduler_step_kwargs.update({"generator": torch.Generator(device=device).manual_seed(seed ^ 0xFFFFFFFF)})
-        if isinstance(scheduler, TCDScheduler):
-            scheduler_step_kwargs.update({"eta": 1.0})
-
-        return num_inference_steps, timesteps, init_timestep, scheduler_step_kwargs
+        return num_inference_steps, timesteps, init_timestep
 
     def prep_inpaint_mask(
         self, context: InvocationContext, latents: torch.Tensor
@@ -918,13 +715,12 @@ class DenoiseLatentsInvocation(BaseInvocation):
             def step_callback(state: PipelineIntermediateState) -> None:
                 context.util.sd_step_callback(state, unet_config.base)
 
-            def _lora_loader() -> Iterator[Tuple[LoRAModelRaw, float]]:
+            def _lora_loader() -> Iterator[Tuple[PeftModel, float]]:
                 for lora in self.unet.loras:
                     lora_info = context.models.load(lora.lora)
-                    assert isinstance(lora_info.model, LoRAModelRaw)
+                    assert isinstance(lora_info.model, PeftModel)
                     yield (lora_info.model, lora.weight)
                     del lora_info
-                return
 
             unet_info = context.models.load(self.unet.unet)
             assert isinstance(unet_info.model, UNet2DConditionModel)
@@ -934,7 +730,7 @@ class DenoiseLatentsInvocation(BaseInvocation):
                 set_seamless(unet_info.model, self.unet.seamless_axes),  # FIXME
                 unet_info as unet,
                 # Apply the LoRA after unet has been moved to its target device for faster patching.
-                ModelPatcher.apply_lora_unet(unet, _lora_loader()),
+                PeftModelPatcher.apply_peft_model_to_unet(unet, _lora_loader()),
             ):
                 assert isinstance(unet, UNet2DConditionModel)
                 latents = latents.to(device=unet.device, dtype=unet.dtype)
@@ -953,11 +749,7 @@ class DenoiseLatentsInvocation(BaseInvocation):
                 )
 
                 pipeline = self.create_pipeline(unet, scheduler)
-
-                _, _, latent_height, latent_width = latents.shape
-                conditioning_data = self.get_conditioning_data(
-                    context=context, unet=unet, latent_height=latent_height, latent_width=latent_width
-                )
+                conditioning_data = self.get_conditioning_data(context, scheduler, unet, seed)
 
                 controlnet_data = self.prep_control_data(
                     context=context,
@@ -971,19 +763,16 @@ class DenoiseLatentsInvocation(BaseInvocation):
                 ip_adapter_data = self.prep_ip_adapter_data(
                     context=context,
                     ip_adapter=self.ip_adapter,
+                    conditioning_data=conditioning_data,
                     exit_stack=exit_stack,
-                    latent_height=latent_height,
-                    latent_width=latent_width,
-                    dtype=unet.dtype,
                 )
 
-                num_inference_steps, timesteps, init_timestep, scheduler_step_kwargs = self.init_scheduler(
+                num_inference_steps, timesteps, init_timestep = self.init_scheduler(
                     scheduler,
                     device=unet.device,
                     steps=self.steps,
                     denoising_start=self.denoising_start,
                     denoising_end=self.denoising_end,
-                    seed=seed,
                 )
 
                 result_latents = pipeline.latents_from_embeddings(
@@ -996,7 +785,6 @@ class DenoiseLatentsInvocation(BaseInvocation):
                     masked_latents=masked_latents,
                     gradient_mask=gradient_mask,
                     num_inference_steps=num_inference_steps,
-                    scheduler_step_kwargs=scheduler_step_kwargs,
                     conditioning_data=conditioning_data,
                     control_data=controlnet_data,
                     ip_adapter_data=ip_adapter_data,
@@ -1006,10 +794,12 @@ class DenoiseLatentsInvocation(BaseInvocation):
 
             # https://discuss.huggingface.co/t/memory-usage-by-later-pipeline-stages/23699
             result_latents = result_latents.to("cpu")
-            TorchDevice.empty_cache()
+            torch.cuda.empty_cache()
+            if choose_torch_device() == torch.device("mps"):
+                mps.empty_cache()
 
             name = context.tensors.save(tensor=result_latents)
-        return LatentsOutput.build(latents_name=name, latents=result_latents, seed=None)
+        return LatentsOutput.build(latents_name=name, latents=result_latents, seed=seed)
 
 
 @invocation(
@@ -1073,7 +863,9 @@ class LatentsToImageInvocation(BaseInvocation, WithMetadata, WithBoard):
                 vae.disable_tiling()
 
             # clear memory as vae decode can request a lot
-            TorchDevice.empty_cache()
+            torch.cuda.empty_cache()
+            if choose_torch_device() == torch.device("mps"):
+                mps.empty_cache()
 
             with torch.inference_mode():
                 # copied from diffusers pipeline
@@ -1085,7 +877,9 @@ class LatentsToImageInvocation(BaseInvocation, WithMetadata, WithBoard):
 
                 image = VaeImageProcessor.numpy_to_pil(np_image)[0]
 
-        TorchDevice.empty_cache()
+        torch.cuda.empty_cache()
+        if choose_torch_device() == torch.device("mps"):
+            mps.empty_cache()
 
         image_dto = context.images.save(image=image)
 
@@ -1124,7 +918,9 @@ class ResizeLatentsInvocation(BaseInvocation):
 
     def invoke(self, context: InvocationContext) -> LatentsOutput:
         latents = context.tensors.load(self.latents.latents_name)
-        device = TorchDevice.choose_torch_device()
+
+        # TODO:
+        device = choose_torch_device()
 
         resized_latents = torch.nn.functional.interpolate(
             latents.to(device),
@@ -1135,8 +931,9 @@ class ResizeLatentsInvocation(BaseInvocation):
 
         # https://discuss.huggingface.co/t/memory-usage-by-later-pipeline-stages/23699
         resized_latents = resized_latents.to("cpu")
-
-        TorchDevice.empty_cache()
+        torch.cuda.empty_cache()
+        if device == torch.device("mps"):
+            mps.empty_cache()
 
         name = context.tensors.save(tensor=resized_latents)
         return LatentsOutput.build(latents_name=name, latents=resized_latents, seed=self.latents.seed)
@@ -1163,7 +960,8 @@ class ScaleLatentsInvocation(BaseInvocation):
     def invoke(self, context: InvocationContext) -> LatentsOutput:
         latents = context.tensors.load(self.latents.latents_name)
 
-        device = TorchDevice.choose_torch_device()
+        # TODO:
+        device = choose_torch_device()
 
         # resizing
         resized_latents = torch.nn.functional.interpolate(
@@ -1175,7 +973,9 @@ class ScaleLatentsInvocation(BaseInvocation):
 
         # https://discuss.huggingface.co/t/memory-usage-by-later-pipeline-stages/23699
         resized_latents = resized_latents.to("cpu")
-        TorchDevice.empty_cache()
+        torch.cuda.empty_cache()
+        if device == torch.device("mps"):
+            mps.empty_cache()
 
         name = context.tensors.save(tensor=resized_latents)
         return LatentsOutput.build(latents_name=name, latents=resized_latents, seed=self.latents.seed)
@@ -1307,7 +1107,8 @@ class BlendLatentsInvocation(BaseInvocation):
         if latents_a.shape != latents_b.shape:
             raise Exception("Latents to blend must be the same size.")
 
-        device = TorchDevice.choose_torch_device()
+        # TODO:
+        device = choose_torch_device()
 
         def slerp(
             t: Union[float, npt.NDArray[Any]],  # FIXME: maybe use np.float32 here?
@@ -1360,8 +1161,9 @@ class BlendLatentsInvocation(BaseInvocation):
 
         # https://discuss.huggingface.co/t/memory-usage-by-later-pipeline-stages/23699
         blended_latents = blended_latents.to("cpu")
-
-        TorchDevice.empty_cache()
+        torch.cuda.empty_cache()
+        if device == torch.device("mps"):
+            mps.empty_cache()
 
         name = context.tensors.save(tensor=blended_latents)
         return LatentsOutput.build(latents_name=name, latents=blended_latents)
@@ -1452,7 +1254,7 @@ class IdealSizeInvocation(BaseInvocation):
         return tuple((x - x % multiple_of) for x in args)
 
     def invoke(self, context: InvocationContext) -> IdealSizeOutput:
-        unet_config = context.models.get_config(self.unet.unet.key)
+        unet_config = context.models.get_config(**self.unet.unet.model_dump())
         aspect = self.width / self.height
         dimension: float = 512
         if unet_config.base == BaseModelType.StableDiffusion2:

invokeai/app/invocations/mask.py

@@ -1,120 +0,0 @@
-import numpy as np
-import torch
-
-from invokeai.app.invocations.baseinvocation import BaseInvocation, Classification, InvocationContext, invocation
-from invokeai.app.invocations.fields import ImageField, InputField, TensorField, WithMetadata
-from invokeai.app.invocations.primitives import MaskOutput
-
-
-@invocation(
-    "rectangle_mask",
-    title="Create Rectangle Mask",
-    tags=["conditioning"],
-    category="conditioning",
-    version="1.0.1",
-)
-class RectangleMaskInvocation(BaseInvocation, WithMetadata):
-    """Create a rectangular mask."""
-
-    width: int = InputField(description="The width of the entire mask.")
-    height: int = InputField(description="The height of the entire mask.")
-    x_left: int = InputField(description="The left x-coordinate of the rectangular masked region (inclusive).")
-    y_top: int = InputField(description="The top y-coordinate of the rectangular masked region (inclusive).")
-    rectangle_width: int = InputField(description="The width of the rectangular masked region.")
-    rectangle_height: int = InputField(description="The height of the rectangular masked region.")
-
-    def invoke(self, context: InvocationContext) -> MaskOutput:
-        mask = torch.zeros((1, self.height, self.width), dtype=torch.bool)
-        mask[:, self.y_top : self.y_top + self.rectangle_height, self.x_left : self.x_left + self.rectangle_width] = (
-            True
-        )
-
-        mask_tensor_name = context.tensors.save(mask)
-        return MaskOutput(
-            mask=TensorField(tensor_name=mask_tensor_name),
-            width=self.width,
-            height=self.height,
-        )
-
-
-@invocation(
-    "alpha_mask_to_tensor",
-    title="Alpha Mask to Tensor",
-    tags=["conditioning"],
-    category="conditioning",
-    version="1.0.0",
-    classification=Classification.Beta,
-)
-class AlphaMaskToTensorInvocation(BaseInvocation):
-    """Convert a mask image to a tensor. Opaque regions are 1 and transparent regions are 0."""
-
-    image: ImageField = InputField(description="The mask image to convert.")
-    invert: bool = InputField(default=False, description="Whether to invert the mask.")
-
-    def invoke(self, context: InvocationContext) -> MaskOutput:
-        image = context.images.get_pil(self.image.image_name)
-        mask = torch.zeros((1, image.height, image.width), dtype=torch.bool)
-        if self.invert:
-            mask[0] = torch.tensor(np.array(image)[:, :, 3] == 0, dtype=torch.bool)
-        else:
-            mask[0] = torch.tensor(np.array(image)[:, :, 3] > 0, dtype=torch.bool)
-
-        return MaskOutput(
-            mask=TensorField(tensor_name=context.tensors.save(mask)),
-            height=mask.shape[1],
-            width=mask.shape[2],
-        )
-
-
-@invocation(
-    "invert_tensor_mask",
-    title="Invert Tensor Mask",
-    tags=["conditioning"],
-    category="conditioning",
-    version="1.0.0",
-    classification=Classification.Beta,
-)
-class InvertTensorMaskInvocation(BaseInvocation):
-    """Inverts a tensor mask."""
-
-    mask: TensorField = InputField(description="The tensor mask to convert.")
-
-    def invoke(self, context: InvocationContext) -> MaskOutput:
-        mask = context.tensors.load(self.mask.tensor_name)
-        inverted = ~mask
-
-        return MaskOutput(
-            mask=TensorField(tensor_name=context.tensors.save(inverted)),
-            height=inverted.shape[1],
-            width=inverted.shape[2],
-        )
-
-
-@invocation(
-    "image_mask_to_tensor",
-    title="Image Mask to Tensor",
-    tags=["conditioning"],
-    category="conditioning",
-    version="1.0.0",
-)
-class ImageMaskToTensorInvocation(BaseInvocation, WithMetadata):
-    """Convert a mask image to a tensor. Converts the image to grayscale and uses thresholding at the specified value."""
-
-    image: ImageField = InputField(description="The mask image to convert.")
-    cutoff: int = InputField(ge=0, le=255, description="Cutoff (<)", default=128)
-    invert: bool = InputField(default=False, description="Whether to invert the mask.")
-
-    def invoke(self, context: InvocationContext) -> MaskOutput:
-        image = context.images.get_pil(self.image.image_name, mode="L")
-
-        mask = torch.zeros((1, image.height, image.width), dtype=torch.bool)
-        if self.invert:
-            mask[0] = torch.tensor(np.array(image)[:, :] >= self.cutoff, dtype=torch.bool)
-        else:
-            mask[0] = torch.tensor(np.array(image)[:, :] < self.cutoff, dtype=torch.bool)
-
-        return MaskOutput(
-            mask=TensorField(tensor_name=context.tensors.save(mask)),
-            height=mask.shape[1],
-            width=mask.shape[2],
-        )

invokeai/app/invocations/metadata.py

@@ -3,6 +3,7 @@ from typing import Any, Literal, Optional, Union
 from pydantic import BaseModel, ConfigDict, Field
 
 from invokeai.app.invocations.baseinvocation import BaseInvocation, BaseInvocationOutput, invocation, invocation_output
+from invokeai.app.invocations.controlnet_image_processors import CONTROLNET_MODE_VALUES, CONTROLNET_RESIZE_VALUES
 from invokeai.app.invocations.fields import (
     FieldDescriptions,
     ImageField,
@@ -13,7 +14,6 @@ from invokeai.app.invocations.fields import (
 )
 from invokeai.app.invocations.model import ModelIdentifierField
 from invokeai.app.services.shared.invocation_context import InvocationContext
-from invokeai.app.util.controlnet_utils import CONTROLNET_MODE_VALUES, CONTROLNET_RESIZE_VALUES
 
 from ...version import __version__
 
@@ -36,7 +36,6 @@ class IPAdapterMetadataField(BaseModel):
     image: ImageField = Field(description="The IP-Adapter image prompt.")
     ip_adapter_model: ModelIdentifierField = Field(description="The IP-Adapter model.")
     clip_vision_model: Literal["ViT-H", "ViT-G"] = Field(description="The CLIP Vision model")
-    method: Literal["full", "style", "composition"] = Field(description="Method to apply IP Weights with")
     weight: Union[float, list[float]] = Field(description="The weight given to the IP-Adapter")
     begin_step_percent: float = Field(description="When the IP-Adapter is first applied (% of total steps)")
     end_step_percent: float = Field(description="When the IP-Adapter is last applied (% of total steps)")

invokeai/app/invocations/model.py

@@ -11,7 +11,6 @@ from invokeai.backend.model_manager.config import AnyModelConfig, BaseModelType,
 from .baseinvocation import (
     BaseInvocation,
     BaseInvocationOutput,
-    Classification,
     invocation,
     invocation_output,
 )
@@ -94,46 +93,19 @@ class ModelLoaderOutput(UNetOutput, CLIPOutput, VAEOutput):
     pass
 
 
-@invocation_output("model_identifier_output")
-class ModelIdentifierOutput(BaseInvocationOutput):
-    """Model identifier output"""
-
-    model: ModelIdentifierField = OutputField(description="Model identifier", title="Model")
-
-
-@invocation(
-    "model_identifier",
-    title="Model identifier",
-    tags=["model"],
-    category="model",
-    version="1.0.0",
-    classification=Classification.Prototype,
-)
-class ModelIdentifierInvocation(BaseInvocation):
-    """Selects any model, outputting it its identifier. Be careful with this one! The identifier will be accepted as
-    input for any model, even if the model types don't match. If you connect this to a mismatched input, you'll get an
-    error."""
-
-    model: ModelIdentifierField = InputField(description="The model to select", title="Model")
-
-    def invoke(self, context: InvocationContext) -> ModelIdentifierOutput:
-        if not context.models.exists(self.model.key):
-            raise Exception(f"Unknown model {self.model.key}")
-
-        return ModelIdentifierOutput(model=self.model)
-
-
 @invocation(
     "main_model_loader",
     title="Main Model",
     tags=["model"],
     category="model",
-    version="1.0.3",
+    version="1.0.2",
 )
 class MainModelLoaderInvocation(BaseInvocation):
     """Loads a main model, outputting its submodels."""
 
-    model: ModelIdentifierField = InputField(description=FieldDescriptions.main_model, ui_type=UIType.MainModel)
+    model: ModelIdentifierField = InputField(
+        description=FieldDescriptions.main_model, input=Input.Direct, ui_type=UIType.MainModel
+    )
     # TODO: precision?
 
     def invoke(self, context: InvocationContext) -> ModelLoaderOutput:
@@ -162,12 +134,12 @@ class LoRALoaderOutput(BaseInvocationOutput):
     clip: Optional[CLIPField] = OutputField(default=None, description=FieldDescriptions.clip, title="CLIP")
 
 
-@invocation("lora_loader", title="LoRA", tags=["model"], category="model", version="1.0.3")
+@invocation("lora_loader", title="LoRA", tags=["model"], category="model", version="1.0.2")
 class LoRALoaderInvocation(BaseInvocation):
     """Apply selected lora to unet and text_encoder."""
 
     lora: ModelIdentifierField = InputField(
-        description=FieldDescriptions.lora_model, title="LoRA", ui_type=UIType.LoRAModel
+        description=FieldDescriptions.lora_model, input=Input.Direct, title="LoRA", ui_type=UIType.LoRAModel
     )
     weight: float = InputField(default=0.75, description=FieldDescriptions.lora_weight)
     unet: Optional[UNetField] = InputField(
@@ -218,75 +190,6 @@ class LoRALoaderInvocation(BaseInvocation):
         return output
 
 
-@invocation_output("lora_selector_output")
-class LoRASelectorOutput(BaseInvocationOutput):
-    """Model loader output"""
-
-    lora: LoRAField = OutputField(description="LoRA model and weight", title="LoRA")
-
-
-@invocation("lora_selector", title="LoRA Selector", tags=["model"], category="model", version="1.0.1")
-class LoRASelectorInvocation(BaseInvocation):
-    """Selects a LoRA model and weight."""
-
-    lora: ModelIdentifierField = InputField(
-        description=FieldDescriptions.lora_model, title="LoRA", ui_type=UIType.LoRAModel
-    )
-    weight: float = InputField(default=0.75, description=FieldDescriptions.lora_weight)
-
-    def invoke(self, context: InvocationContext) -> LoRASelectorOutput:
-        return LoRASelectorOutput(lora=LoRAField(lora=self.lora, weight=self.weight))
-
-
-@invocation("lora_collection_loader", title="LoRA Collection Loader", tags=["model"], category="model", version="1.0.0")
-class LoRACollectionLoader(BaseInvocation):
-    """Applies a collection of LoRAs to the provided UNet and CLIP models."""
-
-    loras: LoRAField | list[LoRAField] = InputField(
-        description="LoRA models and weights. May be a single LoRA or collection.", title="LoRAs"
-    )
-    unet: Optional[UNetField] = InputField(
-        default=None,
-        description=FieldDescriptions.unet,
-        input=Input.Connection,
-        title="UNet",
-    )
-    clip: Optional[CLIPField] = InputField(
-        default=None,
-        description=FieldDescriptions.clip,
-        input=Input.Connection,
-        title="CLIP",
-    )
-
-    def invoke(self, context: InvocationContext) -> LoRALoaderOutput:
-        output = LoRALoaderOutput()
-        loras = self.loras if isinstance(self.loras, list) else [self.loras]
-        added_loras: list[str] = []
-
-        for lora in loras:
-            if lora.lora.key in added_loras:
-                continue
-
-            if not context.models.exists(lora.lora.key):
-                raise Exception(f"Unknown lora: {lora.lora.key}!")
-
-            assert lora.lora.base in (BaseModelType.StableDiffusion1, BaseModelType.StableDiffusion2)
-
-            added_loras.append(lora.lora.key)
-
-            if self.unet is not None:
-                if output.unet is None:
-                    output.unet = self.unet.model_copy(deep=True)
-                output.unet.loras.append(lora)
-
-            if self.clip is not None:
-                if output.clip is None:
-                    output.clip = self.clip.model_copy(deep=True)
-                output.clip.loras.append(lora)
-
-        return output
-
-
 @invocation_output("sdxl_lora_loader_output")
 class SDXLLoRALoaderOutput(BaseInvocationOutput):
     """SDXL LoRA Loader Output"""
@@ -301,13 +204,13 @@ class SDXLLoRALoaderOutput(BaseInvocationOutput):
     title="SDXL LoRA",
     tags=["lora", "model"],
     category="model",
-    version="1.0.3",
+    version="1.0.2",
 )
 class SDXLLoRALoaderInvocation(BaseInvocation):
     """Apply selected lora to unet and text_encoder."""
 
     lora: ModelIdentifierField = InputField(
-        description=FieldDescriptions.lora_model, title="LoRA", ui_type=UIType.LoRAModel
+        description=FieldDescriptions.lora_model, input=Input.Direct, title="LoRA", ui_type=UIType.LoRAModel
     )
     weight: float = InputField(default=0.75, description=FieldDescriptions.lora_weight)
     unet: Optional[UNetField] = InputField(
@@ -376,78 +279,12 @@ class SDXLLoRALoaderInvocation(BaseInvocation):
         return output
 
 
-@invocation(
-    "sdxl_lora_collection_loader",
-    title="SDXL LoRA Collection Loader",
-    tags=["model"],
-    category="model",
-    version="1.0.0",
-)
-class SDXLLoRACollectionLoader(BaseInvocation):
-    """Applies a collection of SDXL LoRAs to the provided UNet and CLIP models."""
-
-    loras: LoRAField | list[LoRAField] = InputField(
-        description="LoRA models and weights. May be a single LoRA or collection.", title="LoRAs"
-    )
-    unet: Optional[UNetField] = InputField(
-        default=None,
-        description=FieldDescriptions.unet,
-        input=Input.Connection,
-        title="UNet",
-    )
-    clip: Optional[CLIPField] = InputField(
-        default=None,
-        description=FieldDescriptions.clip,
-        input=Input.Connection,
-        title="CLIP",
-    )
-    clip2: Optional[CLIPField] = InputField(
-        default=None,
-        description=FieldDescriptions.clip,
-        input=Input.Connection,
-        title="CLIP 2",
-    )
-
-    def invoke(self, context: InvocationContext) -> SDXLLoRALoaderOutput:
-        output = SDXLLoRALoaderOutput()
-        loras = self.loras if isinstance(self.loras, list) else [self.loras]
-        added_loras: list[str] = []
-
-        for lora in loras:
-            if lora.lora.key in added_loras:
-                continue
-
-            if not context.models.exists(lora.lora.key):
-                raise Exception(f"Unknown lora: {lora.lora.key}!")
-
-            assert lora.lora.base is BaseModelType.StableDiffusionXL
-
-            added_loras.append(lora.lora.key)
-
-            if self.unet is not None:
-                if output.unet is None:
-                    output.unet = self.unet.model_copy(deep=True)
-                output.unet.loras.append(lora)
-
-            if self.clip is not None:
-                if output.clip is None:
-                    output.clip = self.clip.model_copy(deep=True)
-                output.clip.loras.append(lora)
-
-            if self.clip2 is not None:
-                if output.clip2 is None:
-                    output.clip2 = self.clip2.model_copy(deep=True)
-                output.clip2.loras.append(lora)
-
-        return output
-
-
-@invocation("vae_loader", title="VAE", tags=["vae", "model"], category="model", version="1.0.3")
+@invocation("vae_loader", title="VAE", tags=["vae", "model"], category="model", version="1.0.2")
 class VAELoaderInvocation(BaseInvocation):
     """Loads a VAE model, outputting a VaeLoaderOutput"""
 
     vae_model: ModelIdentifierField = InputField(
-        description=FieldDescriptions.vae_model, title="VAE", ui_type=UIType.VAEModel
+        description=FieldDescriptions.vae_model, input=Input.Direct, title="VAE", ui_type=UIType.VAEModel
     )
 
     def invoke(self, context: InvocationContext) -> VAEOutput:

invokeai/app/invocations/noise.py

@@ -9,7 +9,7 @@ from invokeai.app.invocations.fields import FieldDescriptions, InputField, Laten
 from invokeai.app.services.shared.invocation_context import InvocationContext
 from invokeai.app.util.misc import SEED_MAX
 
-from ...backend.util.devices import TorchDevice
+from ...backend.util.devices import choose_torch_device, torch_dtype
 from .baseinvocation import (
     BaseInvocation,
     BaseInvocationOutput,
@@ -46,7 +46,7 @@ def get_noise(
             height // downsampling_factor,
             width // downsampling_factor,
         ],
-        dtype=TorchDevice.choose_torch_dtype(device=device),
+        dtype=torch_dtype(device),
         device=noise_device_type,
         generator=generator,
     ).to("cpu")
@@ -111,14 +111,14 @@ class NoiseInvocation(BaseInvocation):
 
     @field_validator("seed", mode="before")
     def modulo_seed(cls, v):
-        """Return the seed modulo (SEED_MAX + 1) to ensure it is within the valid range."""
+        """Returns the seed modulo (SEED_MAX + 1) to ensure it is within the valid range."""
         return v % (SEED_MAX + 1)
 
     def invoke(self, context: InvocationContext) -> NoiseOutput:
         noise = get_noise(
             width=self.width,
             height=self.height,
-            device=TorchDevice.choose_torch_device(),
+            device=choose_torch_device(),
             seed=self.seed,
             use_cpu=self.use_cpu,
         )

invokeai/app/invocations/primitives.py

@@ -15,7 +15,6 @@ from invokeai.app.invocations.fields import (
     InputField,
     LatentsField,
     OutputField,
-    TensorField,
     UIComponent,
 )
 from invokeai.app.services.images.images_common import ImageDTO
@@ -406,19 +405,9 @@ class ColorInvocation(BaseInvocation):
 
 # endregion
 
-
 # region Conditioning
 
 
-@invocation_output("mask_output")
-class MaskOutput(BaseInvocationOutput):
-    """A torch mask tensor."""
-
-    mask: TensorField = OutputField(description="The mask.")
-    width: int = OutputField(description="The width of the mask in pixels.")
-    height: int = OutputField(description="The height of the mask in pixels.")
-
-
 @invocation_output("conditioning_output")
 class ConditioningOutput(BaseInvocationOutput):
     """Base class for nodes that output a single conditioning tensor"""

invokeai/app/invocations/sdxl.py

@@ -1,4 +1,4 @@
-from invokeai.app.invocations.fields import FieldDescriptions, InputField, OutputField, UIType
+from invokeai.app.invocations.fields import FieldDescriptions, Input, InputField, OutputField, UIType
 from invokeai.app.services.shared.invocation_context import InvocationContext
 from invokeai.backend.model_manager import SubModelType
 
@@ -30,12 +30,12 @@ class SDXLRefinerModelLoaderOutput(BaseInvocationOutput):
     vae: VAEField = OutputField(description=FieldDescriptions.vae, title="VAE")
 
 
-@invocation("sdxl_model_loader", title="SDXL Main Model", tags=["model", "sdxl"], category="model", version="1.0.3")
+@invocation("sdxl_model_loader", title="SDXL Main Model", tags=["model", "sdxl"], category="model", version="1.0.2")
 class SDXLModelLoaderInvocation(BaseInvocation):
     """Loads an sdxl base model, outputting its submodels."""
 
     model: ModelIdentifierField = InputField(
-        description=FieldDescriptions.sdxl_main_model, ui_type=UIType.SDXLMainModel
+        description=FieldDescriptions.sdxl_main_model, input=Input.Direct, ui_type=UIType.SDXLMainModel
     )
     # TODO: precision?
 
@@ -67,13 +67,13 @@ class SDXLModelLoaderInvocation(BaseInvocation):
     title="SDXL Refiner Model",
     tags=["model", "sdxl", "refiner"],
     category="model",
-    version="1.0.3",
+    version="1.0.2",
 )
 class SDXLRefinerModelLoaderInvocation(BaseInvocation):
     """Loads an sdxl refiner model, outputting its submodels."""
 
     model: ModelIdentifierField = InputField(
-        description=FieldDescriptions.sdxl_refiner_model, ui_type=UIType.SDXLRefinerModel
+        description=FieldDescriptions.sdxl_refiner_model, input=Input.Direct, ui_type=UIType.SDXLRefinerModel
     )
     # TODO: precision?
 

invokeai/app/invocations/t2i_adapter.py

@@ -8,11 +8,11 @@ from invokeai.app.invocations.baseinvocation import (
     invocation,
     invocation_output,
 )
-from invokeai.app.invocations.fields import FieldDescriptions, ImageField, InputField, OutputField, UIType
+from invokeai.app.invocations.controlnet_image_processors import CONTROLNET_RESIZE_VALUES
+from invokeai.app.invocations.fields import FieldDescriptions, ImageField, Input, InputField, OutputField, UIType
 from invokeai.app.invocations.model import ModelIdentifierField
 from invokeai.app.invocations.util import validate_begin_end_step, validate_weights
 from invokeai.app.services.shared.invocation_context import InvocationContext
-from invokeai.app.util.controlnet_utils import CONTROLNET_RESIZE_VALUES
 
 
 class T2IAdapterField(BaseModel):
@@ -45,7 +45,7 @@ class T2IAdapterOutput(BaseInvocationOutput):
 
 
 @invocation(
-    "t2i_adapter", title="T2I-Adapter", tags=["t2i_adapter", "control"], category="t2i_adapter", version="1.0.3"
+    "t2i_adapter", title="T2I-Adapter", tags=["t2i_adapter", "control"], category="t2i_adapter", version="1.0.2"
 )
 class T2IAdapterInvocation(BaseInvocation):
     """Collects T2I-Adapter info to pass to other nodes."""
@@ -55,6 +55,7 @@ class T2IAdapterInvocation(BaseInvocation):
     t2i_adapter_model: ModelIdentifierField = InputField(
         description="The T2I-Adapter model.",
         title="T2I-Adapter Model",
+        input=Input.Direct,
         ui_order=-1,
         ui_type=UIType.T2IAdapterModel,
     )

invokeai/app/invocations/upscale.py

@@ -4,6 +4,7 @@ from typing import Literal
 
 import cv2
 import numpy as np
+import torch
 from PIL import Image
 from pydantic import ConfigDict
 
@@ -13,7 +14,7 @@ from invokeai.app.services.shared.invocation_context import InvocationContext
 from invokeai.app.util.download_with_progress import download_with_progress_bar
 from invokeai.backend.image_util.basicsr.rrdbnet_arch import RRDBNet
 from invokeai.backend.image_util.realesrgan.realesrgan import RealESRGAN
-from invokeai.backend.util.devices import TorchDevice
+from invokeai.backend.util.devices import choose_torch_device
 
 from .baseinvocation import BaseInvocation, invocation
 from .fields import InputField, WithBoard, WithMetadata
@@ -34,6 +35,9 @@ ESRGAN_MODEL_URLS: dict[str, str] = {
     "RealESRGAN_x2plus.pth": "https://github.com/xinntao/Real-ESRGAN/releases/download/v0.2.1/RealESRGAN_x2plus.pth",
 }
 
+if choose_torch_device() == torch.device("mps"):
+    from torch import mps
+
 
 @invocation("esrgan", title="Upscale (RealESRGAN)", tags=["esrgan", "upscale"], category="esrgan", version="1.3.2")
 class ESRGANInvocation(BaseInvocation, WithMetadata, WithBoard):
@@ -116,7 +120,9 @@ class ESRGANInvocation(BaseInvocation, WithMetadata, WithBoard):
         upscaled_image = upscaler.upscale(cv2_image)
         pil_image = Image.fromarray(cv2.cvtColor(upscaled_image, cv2.COLOR_BGR2RGB)).convert("RGBA")
 
-        TorchDevice.empty_cache()
+        torch.cuda.empty_cache()
+        if choose_torch_device() == torch.device("mps"):
+            mps.empty_cache()
 
         image_dto = context.images.save(image=pil_image)
 

invokeai/app/services/config/config_default.py

@@ -27,12 +27,12 @@ DEFAULT_RAM_CACHE = 10.0
 DEFAULT_VRAM_CACHE = 0.25
 DEFAULT_CONVERT_CACHE = 20.0
 DEVICE = Literal["auto", "cpu", "cuda", "cuda:1", "mps"]
-PRECISION = Literal["auto", "float16", "bfloat16", "float32"]
+PRECISION = Literal["auto", "float16", "bfloat16", "float32", "autocast"]
 ATTENTION_TYPE = Literal["auto", "normal", "xformers", "sliced", "torch-sdp"]
 ATTENTION_SLICE_SIZE = Literal["auto", "balanced", "max", 1, 2, 3, 4, 5, 6, 7, 8]
 LOG_FORMAT = Literal["plain", "color", "syslog", "legacy"]
 LOG_LEVEL = Literal["debug", "info", "warning", "error", "critical"]
-CONFIG_SCHEMA_VERSION = "4.0.1"
+CONFIG_SCHEMA_VERSION = "4.0.0"
 
 
 def get_default_ram_cache_size() -> float:
@@ -105,7 +105,7 @@ class InvokeAIAppConfig(BaseSettings):
         lazy_offload: Keep models in VRAM until their space is needed.
         log_memory_usage: If True, a memory snapshot will be captured before and after every model cache operation, and the result will be logged (at debug level). There is a time cost to capturing the memory snapshots, so it is recommended to only enable this feature if you are actively inspecting the model cache's behaviour.
         device: Preferred execution device. `auto` will choose the device depending on the hardware platform and the installed torch capabilities.<br>Valid values: `auto`, `cpu`, `cuda`, `cuda:1`, `mps`
-        precision: Floating point precision. `float16` will consume half the memory of `float32` but produce slightly lower-quality images. The `auto` setting will guess the proper precision based on your video card and operating system.<br>Valid values: `auto`, `float16`, `bfloat16`, `float32`
+        precision: Floating point precision. `float16` will consume half the memory of `float32` but produce slightly lower-quality images. The `auto` setting will guess the proper precision based on your video card and operating system.<br>Valid values: `auto`, `float16`, `bfloat16`, `float32`, `autocast`
         sequential_guidance: Whether to calculate guidance in serial instead of in parallel, lowering memory requirements.
         attention_type: Attention type.<br>Valid values: `auto`, `normal`, `xformers`, `sliced`, `torch-sdp`
         attention_slice_size: Slice size, valid when attention_type=="sliced".<br>Valid values: `auto`, `balanced`, `max`, `1`, `2`, `3`, `4`, `5`, `6`, `7`, `8`
@@ -370,9 +370,6 @@ def migrate_v3_config_dict(config_dict: dict[str, Any]) -> InvokeAIAppConfig:
             # `max_vram_cache_size` was renamed to `vram` some time in v3, but both names were used
             if k == "max_vram_cache_size" and "vram" not in category_dict:
                 parsed_config_dict["vram"] = v
-            # autocast was removed in v4.0.1
-            if k == "precision" and v == "autocast":
-                parsed_config_dict["precision"] = "auto"
             if k == "conf_path":
                 parsed_config_dict["legacy_models_yaml_path"] = v
             if k == "legacy_conf_dir":
@@ -395,28 +392,6 @@ def migrate_v3_config_dict(config_dict: dict[str, Any]) -> InvokeAIAppConfig:
     return config
 
 
-def migrate_v4_0_0_config_dict(config_dict: dict[str, Any]) -> InvokeAIAppConfig:
-    """Migrate v4.0.0 config dictionary to a current config object.
-
-    Args:
-        config_dict: A dictionary of settings from a v4.0.0 config file.
-
-    Returns:
-        An instance of `InvokeAIAppConfig` with the migrated settings.
-    """
-    parsed_config_dict: dict[str, Any] = {}
-    for k, v in config_dict.items():
-        # autocast was removed from precision in v4.0.1
-        if k == "precision" and v == "autocast":
-            parsed_config_dict["precision"] = "auto"
-        else:
-            parsed_config_dict[k] = v
-        if k == "schema_version":
-            parsed_config_dict[k] = CONFIG_SCHEMA_VERSION
-    config = DefaultInvokeAIAppConfig.model_validate(parsed_config_dict)
-    return config
-
-
 def load_and_migrate_config(config_path: Path) -> InvokeAIAppConfig:
     """Load and migrate a config file to the latest version.
 
@@ -443,21 +418,17 @@ def load_and_migrate_config(config_path: Path) -> InvokeAIAppConfig:
             raise RuntimeError(f"Failed to load and migrate v3 config file {config_path}: {e}") from e
         migrated_config.write_file(config_path)
         return migrated_config
-
-    if loaded_config_dict["schema_version"] == "4.0.0":
-        loaded_config_dict = migrate_v4_0_0_config_dict(loaded_config_dict)
-        loaded_config_dict.write_file(config_path)
-
-    # Attempt to load as a v4 config file
-    try:
-        # Meta is not included in the model fields, so we need to validate it separately
-        config = InvokeAIAppConfig.model_validate(loaded_config_dict)
-        assert (
-            config.schema_version == CONFIG_SCHEMA_VERSION
-        ), f"Invalid schema version, expected {CONFIG_SCHEMA_VERSION}: {config.schema_version}"
-        return config
-    except Exception as e:
-        raise RuntimeError(f"Failed to load config file {config_path}: {e}") from e
+    else:
+        # Attempt to load as a v4 config file
+        try:
+            # Meta is not included in the model fields, so we need to validate it separately
+            config = InvokeAIAppConfig.model_validate(loaded_config_dict)
+            assert (
+                config.schema_version == CONFIG_SCHEMA_VERSION
+            ), f"Invalid schema version, expected {CONFIG_SCHEMA_VERSION}: {config.schema_version}"
+            return config
+        except Exception as e:
+            raise RuntimeError(f"Failed to load config file {config_path}: {e}") from e
 
 
 @lru_cache(maxsize=1)

invokeai/app/services/download/download_default.py

@@ -318,8 +318,10 @@ class DownloadQueueService(DownloadQueueServiceBase):
         in_progress_path.rename(job.download_path)
 
     def _validate_filename(self, directory: str, filename: str) -> bool:
-        pc_name_max = get_pc_name_max(directory)
-        pc_path_max = get_pc_path_max(directory)
+        pc_name_max = os.pathconf(directory, "PC_NAME_MAX") if hasattr(os, "pathconf") else 260  # hardcoded for windows
+        pc_path_max = (
+            os.pathconf(directory, "PC_PATH_MAX") if hasattr(os, "pathconf") else 32767
+        )  # hardcoded for windows with long names enabled
         if "/" in filename:
             return False
         if filename.startswith(".."):
@@ -417,26 +419,6 @@ class DownloadQueueService(DownloadQueueServiceBase):
             self._logger.warning(excp)
 
 
-def get_pc_name_max(directory: str) -> int:
-    if hasattr(os, "pathconf"):
-        try:
-            return os.pathconf(directory, "PC_NAME_MAX")
-        except OSError:
-            # macOS w/ external drives raise OSError
-            pass
-    return 260  # hardcoded for windows
-
-
-def get_pc_path_max(directory: str) -> int:
-    if hasattr(os, "pathconf"):
-        try:
-            return os.pathconf(directory, "PC_PATH_MAX")
-        except OSError:
-            # some platforms may not have this value
-            pass
-    return 32767  # hardcoded for windows with long names enabled
-
-
 # Example on_progress event handler to display a TQDM status bar
 # Activate with:
 #   download_service.download(DownloadJob('http://foo.bar/baz', '/tmp', on_progress=TqdmProgress().update))

invokeai/app/services/events/events_base.py

@@ -122,8 +122,6 @@ class EventServiceBase:
         source_node_id: str,
         error_type: str,
         error: str,
-        user_id: str | None,
-        project_id: str | None,
     ) -> None:
         """Emitted when an invocation has completed"""
         self.__emit_queue_event(
@@ -137,8 +135,6 @@ class EventServiceBase:
                 "source_node_id": source_node_id,
                 "error_type": error_type,
                 "error": error,
-                "user_id": user_id,
-                "project_id": project_id,
             },
         )
 

invokeai/app/services/image_files/image_files_base.py

@@ -4,6 +4,9 @@ from typing import Optional
 
 from PIL.Image import Image as PILImageType
 
+from invokeai.app.invocations.fields import MetadataField
+from invokeai.app.services.workflow_records.workflow_records_common import WorkflowWithoutID
+
 
 class ImageFileStorageBase(ABC):
     """Low-level service responsible for storing and retrieving image files."""
@@ -30,9 +33,8 @@ class ImageFileStorageBase(ABC):
         self,
         image: PILImageType,
         image_name: str,
-        metadata: Optional[str] = None,
-        workflow: Optional[str] = None,
-        graph: Optional[str] = None,
+        metadata: Optional[MetadataField] = None,
+        workflow: Optional[WorkflowWithoutID] = None,
         thumbnail_size: int = 256,
     ) -> None:
         """Saves an image and a 256x256 WEBP thumbnail. Returns a tuple of the image name, thumbnail name, and created timestamp."""
@@ -44,11 +46,6 @@ class ImageFileStorageBase(ABC):
         pass
 
     @abstractmethod
-    def get_workflow(self, image_name: str) -> Optional[str]:
+    def get_workflow(self, image_name: str) -> Optional[WorkflowWithoutID]:
         """Gets the workflow of an image."""
         pass
-
-    @abstractmethod
-    def get_graph(self, image_name: str) -> Optional[str]:
-        """Gets the graph of an image."""
-        pass

invokeai/app/services/image_files/image_files_disk.py

@@ -7,7 +7,9 @@ from PIL import Image, PngImagePlugin
 from PIL.Image import Image as PILImageType
 from send2trash import send2trash
 
+from invokeai.app.invocations.fields import MetadataField
 from invokeai.app.services.invoker import Invoker
+from invokeai.app.services.workflow_records.workflow_records_common import WorkflowWithoutID
 from invokeai.app.util.thumbnails import get_thumbnail_name, make_thumbnail
 
 from .image_files_base import ImageFileStorageBase
@@ -54,9 +56,8 @@ class DiskImageFileStorage(ImageFileStorageBase):
         self,
         image: PILImageType,
         image_name: str,
-        metadata: Optional[str] = None,
-        workflow: Optional[str] = None,
-        graph: Optional[str] = None,
+        metadata: Optional[MetadataField] = None,
+        workflow: Optional[WorkflowWithoutID] = None,
         thumbnail_size: int = 256,
     ) -> None:
         try:
@@ -67,14 +68,13 @@ class DiskImageFileStorage(ImageFileStorageBase):
             info_dict = {}
 
             if metadata is not None:
-                info_dict["invokeai_metadata"] = metadata
-                pnginfo.add_text("invokeai_metadata", metadata)
+                metadata_json = metadata.model_dump_json()
+                info_dict["invokeai_metadata"] = metadata_json
+                pnginfo.add_text("invokeai_metadata", metadata_json)
             if workflow is not None:
-                info_dict["invokeai_workflow"] = workflow
-                pnginfo.add_text("invokeai_workflow", workflow)
-            if graph is not None:
-                info_dict["invokeai_graph"] = graph
-                pnginfo.add_text("invokeai_graph", graph)
+                workflow_json = workflow.model_dump_json()
+                info_dict["invokeai_workflow"] = workflow_json
+                pnginfo.add_text("invokeai_workflow", workflow_json)
 
             # When saving the image, the image object's info field is not populated. We need to set it
             image.info = info_dict
@@ -129,18 +129,11 @@ class DiskImageFileStorage(ImageFileStorageBase):
         path = path if isinstance(path, Path) else Path(path)
         return path.exists()
 
-    def get_workflow(self, image_name: str) -> str | None:
+    def get_workflow(self, image_name: str) -> WorkflowWithoutID | None:
         image = self.get(image_name)
         workflow = image.info.get("invokeai_workflow", None)
-        if isinstance(workflow, str):
-            return workflow
-        return None
-
-    def get_graph(self, image_name: str) -> str | None:
-        image = self.get(image_name)
-        graph = image.info.get("invokeai_graph", None)
-        if isinstance(graph, str):
-            return graph
+        if workflow is not None:
+            return WorkflowWithoutID.model_validate_json(workflow)
         return None
 
     def __validate_storage_folders(self) -> None:

invokeai/app/services/image_records/image_records_base.py

@@ -80,7 +80,7 @@ class ImageRecordStorageBase(ABC):
         starred: Optional[bool] = False,
         session_id: Optional[str] = None,
         node_id: Optional[str] = None,
-        metadata: Optional[str] = None,
+        metadata: Optional[MetadataField] = None,
     ) -> datetime:
         """Saves an image record."""
         pass

invokeai/app/services/image_records/image_records_sqlite.py

@@ -328,9 +328,10 @@ class SqliteImageRecordStorage(ImageRecordStorageBase):
         starred: Optional[bool] = False,
         session_id: Optional[str] = None,
         node_id: Optional[str] = None,
-        metadata: Optional[str] = None,
+        metadata: Optional[MetadataField] = None,
     ) -> datetime:
         try:
+            metadata_json = metadata.model_dump_json() if metadata is not None else None
             self._lock.acquire()
             self._cursor.execute(
                 """--sql
@@ -357,7 +358,7 @@ class SqliteImageRecordStorage(ImageRecordStorageBase):
                     height,
                     node_id,
                     session_id,
-                    metadata,
+                    metadata_json,
                     is_intermediate,
                     starred,
                     has_workflow,

invokeai/app/services/images/images_base.py

@@ -12,6 +12,7 @@ from invokeai.app.services.image_records.image_records_common import (
 )
 from invokeai.app.services.images.images_common import ImageDTO
 from invokeai.app.services.shared.pagination import OffsetPaginatedResults
+from invokeai.app.services.workflow_records.workflow_records_common import WorkflowWithoutID
 
 
 class ImageServiceABC(ABC):
@@ -50,9 +51,8 @@ class ImageServiceABC(ABC):
         session_id: Optional[str] = None,
         board_id: Optional[str] = None,
         is_intermediate: Optional[bool] = False,
-        metadata: Optional[str] = None,
-        workflow: Optional[str] = None,
-        graph: Optional[str] = None,
+        metadata: Optional[MetadataField] = None,
+        workflow: Optional[WorkflowWithoutID] = None,
     ) -> ImageDTO:
         """Creates an image, storing the file and its metadata."""
         pass
@@ -87,12 +87,7 @@ class ImageServiceABC(ABC):
         pass
 
     @abstractmethod
-    def get_workflow(self, image_name: str) -> Optional[str]:
-        """Gets an image's workflow."""
-        pass
-
-    @abstractmethod
-    def get_graph(self, image_name: str) -> Optional[str]:
+    def get_workflow(self, image_name: str) -> Optional[WorkflowWithoutID]:
         """Gets an image's workflow."""
         pass
 

invokeai/app/services/images/images_default.py

@@ -5,6 +5,7 @@ from PIL.Image import Image as PILImageType
 from invokeai.app.invocations.fields import MetadataField
 from invokeai.app.services.invoker import Invoker
 from invokeai.app.services.shared.pagination import OffsetPaginatedResults
+from invokeai.app.services.workflow_records.workflow_records_common import WorkflowWithoutID
 
 from ..image_files.image_files_common import (
     ImageFileDeleteException,
@@ -41,9 +42,8 @@ class ImageService(ImageServiceABC):
         session_id: Optional[str] = None,
         board_id: Optional[str] = None,
         is_intermediate: Optional[bool] = False,
-        metadata: Optional[str] = None,
-        workflow: Optional[str] = None,
-        graph: Optional[str] = None,
+        metadata: Optional[MetadataField] = None,
+        workflow: Optional[WorkflowWithoutID] = None,
     ) -> ImageDTO:
         if image_origin not in ResourceOrigin:
             raise InvalidOriginException
@@ -64,7 +64,7 @@ class ImageService(ImageServiceABC):
                 image_category=image_category,
                 width=width,
                 height=height,
-                has_workflow=workflow is not None or graph is not None,
+                has_workflow=workflow is not None,
                 # Meta fields
                 is_intermediate=is_intermediate,
                 # Nullable fields
@@ -75,7 +75,7 @@ class ImageService(ImageServiceABC):
             if board_id is not None:
                 self.__invoker.services.board_image_records.add_image_to_board(board_id=board_id, image_name=image_name)
             self.__invoker.services.image_files.save(
-                image_name=image_name, image=image, metadata=metadata, workflow=workflow, graph=graph
+                image_name=image_name, image=image, metadata=metadata, workflow=workflow
             )
             image_dto = self.get_dto(image_name)
 
@@ -157,7 +157,7 @@ class ImageService(ImageServiceABC):
             self.__invoker.services.logger.error("Problem getting image metadata")
             raise e
 
-    def get_workflow(self, image_name: str) -> Optional[str]:
+    def get_workflow(self, image_name: str) -> Optional[WorkflowWithoutID]:
         try:
             return self.__invoker.services.image_files.get_workflow(image_name)
         except ImageFileNotFoundException:
@@ -167,16 +167,6 @@ class ImageService(ImageServiceABC):
             self.__invoker.services.logger.error("Problem getting image workflow")
             raise
 
-    def get_graph(self, image_name: str) -> Optional[str]:
-        try:
-            return self.__invoker.services.image_files.get_graph(image_name)
-        except ImageFileNotFoundException:
-            self.__invoker.services.logger.error("Image file not found")
-            raise
-        except Exception:
-            self.__invoker.services.logger.error("Problem getting image graph")
-            raise
-
     def get_path(self, image_name: str, thumbnail: bool = False) -> str:
         try:
             return str(self.__invoker.services.image_files.get_path(image_name, thumbnail))

invokeai/app/services/model_install/model_install_default.py

@@ -3,6 +3,7 @@
 import locale
 import os
 import re
+import signal
 import threading
 import time
 from hashlib import sha256
@@ -12,7 +13,6 @@ from shutil import copyfile, copytree, move, rmtree
 from tempfile import mkdtemp
 from typing import Any, Dict, List, Optional, Union
 
-import torch
 import yaml
 from huggingface_hub import HfFolder
 from pydantic.networks import AnyHttpUrl
@@ -42,8 +42,7 @@ from invokeai.backend.model_manager.metadata.metadata_base import HuggingFaceMet
 from invokeai.backend.model_manager.probe import ModelProbe
 from invokeai.backend.model_manager.search import ModelSearch
 from invokeai.backend.util import InvokeAILogger
-from invokeai.backend.util.catch_sigint import catch_sigint
-from invokeai.backend.util.devices import TorchDevice
+from invokeai.backend.util.devices import choose_precision, choose_torch_device
 
 from .model_install_base import (
     MODEL_SOURCE_TO_TYPE_MAP,
@@ -112,6 +111,17 @@ class ModelInstallService(ModelInstallServiceBase):
     def start(self, invoker: Optional[Invoker] = None) -> None:
         """Start the installer thread."""
 
+        # Yes, this is weird. When the installer thread is running, the
+        # thread masks the ^C signal. When we receive a
+        # sigINT, we stop the thread, reset sigINT, and send a new
+        # sigINT to the parent process.
+        def sigint_handler(signum, frame):
+            self.stop()
+            signal.signal(signal.SIGINT, signal.SIG_DFL)
+            signal.raise_signal(signal.SIGINT)
+
+        signal.signal(signal.SIGINT, sigint_handler)
+
         with self._lock:
             if self._running:
                 raise Exception("Attempt to start the installer service twice")
@@ -121,8 +131,7 @@ class ModelInstallService(ModelInstallServiceBase):
             # In normal use, we do not want to scan the models directory - it should never have orphaned models.
             # We should only do the scan when the flag is set (which should only be set when testing).
             if self.app_config.scan_models_on_startup:
-                with catch_sigint():
-                    self._register_orphaned_models()
+                self._register_orphaned_models()
 
             # Check all models' paths and confirm they exist. A model could be missing if it was installed on a volume
             # that isn't currently mounted. In this case, we don't want to delete the model from the database, but we do
@@ -625,10 +634,11 @@ class ModelInstallService(ModelInstallServiceBase):
             self._next_job_id += 1
         return id
 
-    def _guess_variant(self) -> Optional[ModelRepoVariant]:
+    @staticmethod
+    def _guess_variant() -> Optional[ModelRepoVariant]:
         """Guess the best HuggingFace variant type to download."""
-        precision = TorchDevice.choose_torch_dtype()
-        return ModelRepoVariant.FP16 if precision == torch.float16 else None
+        precision = choose_precision(choose_torch_device())
+        return ModelRepoVariant.FP16 if precision == "float16" else None
 
     def _import_local_model(self, source: LocalModelSource, config: Optional[Dict[str, Any]]) -> ModelInstallJob:
         return ModelInstallJob(
@@ -744,8 +754,6 @@ class ModelInstallService(ModelInstallServiceBase):
             self._download_cache[download_job.source] = install_job  # matches a download job to an install job
             install_job.download_parts.add(download_job)
 
-        # only start the jobs once install_job.download_parts is fully populated
-        for download_job in install_job.download_parts:
             self._download_queue.submit_download_job(
                 download_job,
                 on_start=self._download_started_callback,
@@ -754,7 +762,6 @@ class ModelInstallService(ModelInstallServiceBase):
                 on_error=self._download_error_callback,
                 on_cancelled=self._download_cancelled_callback,
             )
-
         return install_job
 
     def _stat_size(self, path: Path) -> int:

invokeai/app/services/model_load/model_load_base.py

@@ -5,7 +5,8 @@ from abc import ABC, abstractmethod
 from typing import Optional
 
 from invokeai.app.services.shared.invocation_context import InvocationContextData
-from invokeai.backend.model_manager import AnyModel, AnyModelConfig, SubModelType
+from invokeai.backend.model_manager import AnyModelConfig, SubModelType
+from invokeai.backend.model_manager.any_model_type import AnyModel
 from invokeai.backend.model_manager.load import LoadedModel
 from invokeai.backend.model_manager.load.convert_cache import ModelConvertCacheBase
 from invokeai.backend.model_manager.load.model_cache.model_cache_base import ModelCacheBase

invokeai/app/services/model_load/model_load_default.py

@@ -6,7 +6,8 @@ from typing import Optional, Type
 from invokeai.app.services.config import InvokeAIAppConfig
 from invokeai.app.services.invoker import Invoker
 from invokeai.app.services.shared.invocation_context import InvocationContextData
-from invokeai.backend.model_manager import AnyModel, AnyModelConfig, SubModelType
+from invokeai.backend.model_manager import AnyModelConfig, SubModelType
+from invokeai.backend.model_manager.any_model_type import AnyModel
 from invokeai.backend.model_manager.load import (
     LoadedModel,
     ModelLoaderRegistry,

invokeai/app/services/model_manager/__init__.py

@@ -1,6 +1,6 @@
 """Initialization file for model manager service."""
 
-from invokeai.backend.model_manager import AnyModel, AnyModelConfig, BaseModelType, ModelType, SubModelType
+from invokeai.backend.model_manager import AnyModelConfig, BaseModelType, ModelType, SubModelType
 from invokeai.backend.model_manager.load import LoadedModel
 
 from .model_manager_default import ModelManagerService, ModelManagerServiceBase
@@ -8,7 +8,6 @@ from .model_manager_default import ModelManagerService, ModelManagerServiceBase
 __all__ = [
     "ModelManagerServiceBase",
     "ModelManagerService",
-    "AnyModel",
     "AnyModelConfig",
     "BaseModelType",
     "ModelType",

invokeai/app/services/model_manager/model_manager_default.py

@@ -1,14 +1,12 @@
 # Copyright (c) 2023 Lincoln D. Stein and the InvokeAI Team
 """Implementation of ModelManagerServiceBase."""
 
-from typing import Optional
-
 import torch
 from typing_extensions import Self
 
 from invokeai.app.services.invoker import Invoker
 from invokeai.backend.model_manager.load import ModelCache, ModelConvertCache, ModelLoaderRegistry
-from invokeai.backend.util.devices import TorchDevice
+from invokeai.backend.util.devices import choose_torch_device
 from invokeai.backend.util.logging import InvokeAILogger
 
 from ..config import InvokeAIAppConfig
@@ -69,7 +67,7 @@ class ModelManagerService(ModelManagerServiceBase):
         model_record_service: ModelRecordServiceBase,
         download_queue: DownloadQueueServiceBase,
         events: EventServiceBase,
-        execution_device: Optional[torch.device] = None,
+        execution_device: torch.device = choose_torch_device(),
     ) -> Self:
         """
         Construct the model manager service instance.
@@ -82,9 +80,8 @@ class ModelManagerService(ModelManagerServiceBase):
         ram_cache = ModelCache(
             max_cache_size=app_config.ram,
             max_vram_cache_size=app_config.vram,
-            lazy_offloading=app_config.lazy_offload,
             logger=logger,
-            execution_device=execution_device or TorchDevice.choose_torch_device(),
+            execution_device=execution_device,
         )
         convert_cache = ModelConvertCache(cache_path=app_config.convert_cache_path, max_size=app_config.convert_cache)
         loader = ModelLoadService(

invokeai/app/services/object_serializer/object_serializer_disk.py

@@ -1,6 +1,6 @@
-import shutil
 import tempfile
 import typing
+from dataclasses import dataclass
 from pathlib import Path
 from typing import TYPE_CHECKING, Optional, TypeVar
 
@@ -17,6 +17,12 @@ if TYPE_CHECKING:
 T = TypeVar("T")
 
 
+@dataclass
+class DeleteAllResult:
+    deleted_count: int
+    freed_space_bytes: float
+
+
 class ObjectSerializerDisk(ObjectSerializerBase[T]):
     """Disk-backed storage for arbitrary python objects. Serialization is handled by `torch.save` and `torch.load`.
 
@@ -29,12 +35,6 @@ class ObjectSerializerDisk(ObjectSerializerBase[T]):
         self._ephemeral = ephemeral
         self._base_output_dir = output_dir
         self._base_output_dir.mkdir(parents=True, exist_ok=True)
-
-        if self._ephemeral:
-            # Remove dangling tempdirs that might have been left over from an earlier unplanned shutdown.
-            for temp_dir in filter(Path.is_dir, self._base_output_dir.glob("tmp*")):
-                shutil.rmtree(temp_dir)
-
         # Must specify `ignore_cleanup_errors` to avoid fatal errors during cleanup on Windows
         self._tempdir = (
             tempfile.TemporaryDirectory(dir=self._base_output_dir, ignore_cleanup_errors=True) if ephemeral else None

invokeai/app/services/session_processor/session_processor_default.py

@@ -86,12 +86,6 @@ class DefaultSessionProcessor(SessionProcessorBase):
             self._poll_now()
         elif event_name == "batch_enqueued":
             self._poll_now()
-        elif event_name == "queue_item_status_changed" and event[1]["data"]["queue_item"]["status"] in [
-            "completed",
-            "failed",
-            "canceled",
-        ]:
-            self._poll_now()
 
     def resume(self) -> SessionProcessorStatus:
         if not self._resume_event.is_set():
@@ -237,8 +231,6 @@ class DefaultSessionProcessor(SessionProcessorBase):
                                 source_node_id=source_invocation_id,
                                 error_type=e.__class__.__name__,
                                 error=error,
-                                user_id=None,
-                                project_id=None,
                             )
                             pass
 

invokeai/app/services/shared/invocation_context.py

@@ -180,9 +180,9 @@ class ImagesInterface(InvocationContextInterface):
         # If `metadata` is provided directly, use that. Else, use the metadata provided by `WithMetadata`, falling back to None.
         metadata_ = None
         if metadata:
-            metadata_ = metadata.model_dump_json()
-        elif isinstance(self._data.invocation, WithMetadata) and self._data.invocation.metadata:
-            metadata_ = self._data.invocation.metadata.model_dump_json()
+            metadata_ = metadata
+        elif isinstance(self._data.invocation, WithMetadata):
+            metadata_ = self._data.invocation.metadata
 
         # If `board_id` is provided directly, use that. Else, use the board provided by `WithBoard`, falling back to None.
         board_id_ = None
@@ -191,14 +191,6 @@ class ImagesInterface(InvocationContextInterface):
         elif isinstance(self._data.invocation, WithBoard) and self._data.invocation.board:
             board_id_ = self._data.invocation.board.board_id
 
-        workflow_ = None
-        if self._data.queue_item.workflow:
-            workflow_ = self._data.queue_item.workflow.model_dump_json()
-
-        graph_ = None
-        if self._data.queue_item.session.graph:
-            graph_ = self._data.queue_item.session.graph.model_dump_json()
-
         return self._services.images.create(
             image=image,
             is_intermediate=self._data.invocation.is_intermediate,
@@ -206,8 +198,7 @@ class ImagesInterface(InvocationContextInterface):
             board_id=board_id_,
             metadata=metadata_,
             image_origin=ResourceOrigin.INTERNAL,
-            workflow=workflow_,
-            graph=graph_,
+            workflow=self._data.queue_item.workflow,
             session_id=self._data.queue_item.session_id,
             node_id=self._data.invocation.id,
         )
@@ -254,18 +245,6 @@ class ImagesInterface(InvocationContextInterface):
         """
         return self._services.images.get_dto(image_name)
 
-    def get_path(self, image_name: str, thumbnail: bool = False) -> Path:
-        """Gets the internal path to an image or thumbnail.
-
-        Args:
-            image_name: The name of the image to get the path of.
-            thumbnail: Get the path of the thumbnail instead of the full image
-
-        Returns:
-            The local path of the image or thumbnail.
-        """
-        return self._services.images.get_path(image_name, thumbnail)
-
 
 class TensorsInterface(InvocationContextInterface):
     def save(self, tensor: Tensor) -> str:

invokeai/app/util/controlnet_utils.py

@@ -1,21 +1,13 @@
-from typing import Any, Literal, Union
+from typing import Union
 
 import cv2
 import numpy as np
 import torch
+from controlnet_aux.util import HWC3
+from diffusers.utils import PIL_INTERPOLATION
 from einops import rearrange
 from PIL import Image
 
-from invokeai.backend.image_util.util import nms, normalize_image_channel_count
-
-CONTROLNET_RESIZE_VALUES = Literal[
-    "just_resize",
-    "crop_resize",
-    "fill_resize",
-    "just_resize_simple",
-]
-CONTROLNET_MODE_VALUES = Literal["balanced", "more_prompt", "more_control", "unbalanced"]
-
 ###################################################################
 # Copy of scripts/lvminthin.py from Mikubill/sd-webui-controlnet
 ###################################################################
@@ -76,6 +68,17 @@ def lvmin_thin(x, prunings=True):
     return y
 
 
+def nake_nms(x):
+    f1 = np.array([[0, 0, 0], [1, 1, 1], [0, 0, 0]], dtype=np.uint8)
+    f2 = np.array([[0, 1, 0], [0, 1, 0], [0, 1, 0]], dtype=np.uint8)
+    f3 = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1]], dtype=np.uint8)
+    f4 = np.array([[0, 0, 1], [0, 1, 0], [1, 0, 0]], dtype=np.uint8)
+    y = np.zeros_like(x)
+    for f in [f1, f2, f3, f4]:
+        np.putmask(y, cv2.dilate(x, kernel=f) == x, x)
+    return y
+
+
 ################################################################################
 # copied from Mikubill/sd-webui-controlnet external_code.py and modified for InvokeAI
 ################################################################################
@@ -131,122 +134,98 @@ def pixel_perfect_resolution(
     return int(np.round(estimation))
 
 
-def clone_contiguous(x: np.ndarray[Any, Any]) -> np.ndarray[Any, Any]:
-    """Get a memory-contiguous clone of the given numpy array, as a safety measure and to improve computation efficiency."""
-    return np.ascontiguousarray(x).copy()
-
-
-def np_img_to_torch(np_img: np.ndarray[Any, Any], device: torch.device) -> torch.Tensor:
-    """Convert a numpy image to a PyTorch tensor. The image is normalized to 0-1, rearranged to BCHW format and sent to
-    the specified device."""
-
-    torch_img = torch.from_numpy(np_img)
-    normalized = torch_img.float() / 255.0
-    bchw = rearrange(normalized, "h w c -> 1 c h w")
-    on_device = bchw.to(device)
-    return on_device.clone()
-
-
-def heuristic_resize(np_img: np.ndarray[Any, Any], size: tuple[int, int]) -> np.ndarray[Any, Any]:
-    """Resizes an image using a heuristic to choose the best resizing strategy.
-
-    - If the image appears to be an edge map, special handling will be applied to ensure the edges are not distorted.
-    - Single-pixel edge maps use NMS and thinning to keep the edges as single-pixel lines.
-    - Low-color-count images are resized with nearest-neighbor to preserve color information (for e.g. segmentation maps).
-    - The alpha channel is handled separately to ensure it is resized correctly.
-
-    Args:
-        np_img (np.ndarray): The input image.
-        size (tuple[int, int]): The target size for the image.
-
-    Returns:
-        np.ndarray: The resized image.
-
-    Adapted from https://github.com/Mikubill/sd-webui-controlnet.
-    """
-
-    # Return early if the image is already at the requested size
-    if np_img.shape[0] == size[1] and np_img.shape[1] == size[0]:
-        return np_img
-
-    # If the image has an alpha channel, separate it for special handling later.
-    inpaint_mask = None
-    if np_img.ndim == 3 and np_img.shape[2] == 4:
-        inpaint_mask = np_img[:, :, 3]
-        np_img = np_img[:, :, 0:3]
-
-    new_size_is_smaller = (size[0] * size[1]) < (np_img.shape[0] * np_img.shape[1])
-    new_size_is_bigger = (size[0] * size[1]) > (np_img.shape[0] * np_img.shape[1])
-    unique_color_count = np.unique(np_img.reshape(-1, np_img.shape[2]), axis=0).shape[0]
-    is_one_pixel_edge = False
-    is_binary = False
-
-    if unique_color_count == 2:
-        # If the image has only two colors, it is likely binary. Check if the image has one-pixel edges.
-        is_binary = np.min(np_img) < 16 and np.max(np_img) > 240
-        if is_binary:
-            eroded = cv2.erode(np_img, np.ones(shape=(3, 3), dtype=np.uint8), iterations=1)
-            dilated = cv2.dilate(eroded, np.ones(shape=(3, 3), dtype=np.uint8), iterations=1)
-            one_pixel_edge_count = np.where(dilated < np_img)[0].shape[0]
-            all_edge_count = np.where(np_img > 127)[0].shape[0]
-            is_one_pixel_edge = one_pixel_edge_count * 2 > all_edge_count
-
-    if 2 < unique_color_count < 200:
-        # With a low color count, we assume this is a map where exact colors are important. Near-neighbor preserves
-        # the colors as needed.
-        interpolation = cv2.INTER_NEAREST
-    elif new_size_is_smaller:
-        # This works best for downscaling
-        interpolation = cv2.INTER_AREA
-    else:
-        # Fall back for other cases
-        interpolation = cv2.INTER_CUBIC  # Must be CUBIC because we now use nms. NEVER CHANGE THIS
-
-    # This may be further transformed depending on the binary nature of the image.
-    resized = cv2.resize(np_img, size, interpolation=interpolation)
-
-    if inpaint_mask is not None:
-        # Resize the inpaint mask to match the resized image using the same interpolation method.
-        inpaint_mask = cv2.resize(inpaint_mask, size, interpolation=interpolation)
-
-    # If the image is binary, we will perform some additional processing to ensure the edges are preserved.
-    if is_binary:
-        resized = np.mean(resized.astype(np.float32), axis=2).clip(0, 255).astype(np.uint8)
-        if is_one_pixel_edge:
-            # Use NMS and thinning to keep the edges as single-pixel lines.
-            resized = nms(resized)
-            _, resized = cv2.threshold(resized, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
-            resized = lvmin_thin(resized, prunings=new_size_is_bigger)
-        else:
-            _, resized = cv2.threshold(resized, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
-        resized = np.stack([resized] * 3, axis=2)
-
-    # Restore the alpha channel if it was present.
-    if inpaint_mask is not None:
-        inpaint_mask = (inpaint_mask > 127).astype(np.float32) * 255.0
-        inpaint_mask = inpaint_mask[:, :, None].clip(0, 255).astype(np.uint8)
-        resized = np.concatenate([resized, inpaint_mask], axis=2)
-
-    return resized
-
-
 ###########################################################################
 # Copied from detectmap_proc method in scripts/detectmap_proc.py in Mikubill/sd-webui-controlnet
 #    modified for InvokeAI
 ###########################################################################
-def np_img_resize(
-    np_img: np.ndarray,
-    resize_mode: CONTROLNET_RESIZE_VALUES,
-    h: int,
-    w: int,
-    device: torch.device = torch.device("cpu"),
-) -> tuple[torch.Tensor, np.ndarray[Any, Any]]:
-    np_img = normalize_image_channel_count(np_img)
+# def detectmap_proc(detected_map, module, resize_mode, h, w):
+def np_img_resize(np_img: np.ndarray, resize_mode: str, h: int, w: int, device: torch.device = torch.device("cpu")):
+    # if 'inpaint' in module:
+    #     np_img = np_img.astype(np.float32)
+    # else:
+    #     np_img = HWC3(np_img)
+    np_img = HWC3(np_img)
 
+    def safe_numpy(x):
+        # A very safe method to make sure that Apple/Mac works
+        y = x
+
+        # below is very boring but do not change these. If you change these Apple or Mac may fail.
+        y = y.copy()
+        y = np.ascontiguousarray(y)
+        y = y.copy()
+        return y
+
+    def get_pytorch_control(x):
+        # A very safe method to make sure that Apple/Mac works
+        y = x
+
+        # below is very boring but do not change these. If you change these Apple or Mac may fail.
+        y = torch.from_numpy(y)
+        y = y.float() / 255.0
+        y = rearrange(y, "h w c -> 1 c h w")
+        y = y.clone()
+        # y = y.to(devices.get_device_for("controlnet"))
+        y = y.to(device)
+        y = y.clone()
+        return y
+
+    def high_quality_resize(x: np.ndarray, size):
+        # Written by lvmin
+        # Super high-quality control map up-scaling, considering binary, seg, and one-pixel edges
+        inpaint_mask = None
+        if x.ndim == 3 and x.shape[2] == 4:
+            inpaint_mask = x[:, :, 3]
+            x = x[:, :, 0:3]
+
+        new_size_is_smaller = (size[0] * size[1]) < (x.shape[0] * x.shape[1])
+        new_size_is_bigger = (size[0] * size[1]) > (x.shape[0] * x.shape[1])
+        unique_color_count = np.unique(x.reshape(-1, x.shape[2]), axis=0).shape[0]
+        is_one_pixel_edge = False
+        is_binary = False
+        if unique_color_count == 2:
+            is_binary = np.min(x) < 16 and np.max(x) > 240
+            if is_binary:
+                xc = x
+                xc = cv2.erode(xc, np.ones(shape=(3, 3), dtype=np.uint8), iterations=1)
+                xc = cv2.dilate(xc, np.ones(shape=(3, 3), dtype=np.uint8), iterations=1)
+                one_pixel_edge_count = np.where(xc < x)[0].shape[0]
+                all_edge_count = np.where(x > 127)[0].shape[0]
+                is_one_pixel_edge = one_pixel_edge_count * 2 > all_edge_count
+
+        if 2 < unique_color_count < 200:
+            interpolation = cv2.INTER_NEAREST
+        elif new_size_is_smaller:
+            interpolation = cv2.INTER_AREA
+        else:
+            interpolation = cv2.INTER_CUBIC  # Must be CUBIC because we now use nms. NEVER CHANGE THIS
+
+        y = cv2.resize(x, size, interpolation=interpolation)
+        if inpaint_mask is not None:
+            inpaint_mask = cv2.resize(inpaint_mask, size, interpolation=interpolation)
+
+        if is_binary:
+            y = np.mean(y.astype(np.float32), axis=2).clip(0, 255).astype(np.uint8)
+            if is_one_pixel_edge:
+                y = nake_nms(y)
+                _, y = cv2.threshold(y, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
+                y = lvmin_thin(y, prunings=new_size_is_bigger)
+            else:
+                _, y = cv2.threshold(y, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
+            y = np.stack([y] * 3, axis=2)
+
+        if inpaint_mask is not None:
+            inpaint_mask = (inpaint_mask > 127).astype(np.float32) * 255.0
+            inpaint_mask = inpaint_mask[:, :, None].clip(0, 255).astype(np.uint8)
+            y = np.concatenate([y, inpaint_mask], axis=2)
+
+        return y
+
+    # if resize_mode == external_code.ResizeMode.RESIZE:
     if resize_mode == "just_resize":  # RESIZE
-        np_img = heuristic_resize(np_img, (w, h))
-        np_img = clone_contiguous(np_img)
-        return np_img_to_torch(np_img, device), np_img
+        np_img = high_quality_resize(np_img, (w, h))
+        np_img = safe_numpy(np_img)
+        return get_pytorch_control(np_img), np_img
 
     old_h, old_w, _ = np_img.shape
     old_w = float(old_w)
@@ -257,6 +236,7 @@ def np_img_resize(
     def safeint(x: Union[int, float]) -> int:
         return int(np.round(x))
 
+    # if resize_mode == external_code.ResizeMode.OUTER_FIT:
     if resize_mode == "fill_resize":  # OUTER_FIT
         k = min(k0, k1)
         borders = np.concatenate([np_img[0, :, :], np_img[-1, :, :], np_img[:, 0, :], np_img[:, -1, :]], axis=0)
@@ -265,23 +245,23 @@ def np_img_resize(
             # Inpaint hijack
             high_quality_border_color[3] = 255
         high_quality_background = np.tile(high_quality_border_color[None, None], [h, w, 1])
-        np_img = heuristic_resize(np_img, (safeint(old_w * k), safeint(old_h * k)))
+        np_img = high_quality_resize(np_img, (safeint(old_w * k), safeint(old_h * k)))
         new_h, new_w, _ = np_img.shape
         pad_h = max(0, (h - new_h) // 2)
         pad_w = max(0, (w - new_w) // 2)
         high_quality_background[pad_h : pad_h + new_h, pad_w : pad_w + new_w] = np_img
         np_img = high_quality_background
-        np_img = clone_contiguous(np_img)
-        return np_img_to_torch(np_img, device), np_img
+        np_img = safe_numpy(np_img)
+        return get_pytorch_control(np_img), np_img
     else:  # resize_mode == "crop_resize"  (INNER_FIT)
         k = max(k0, k1)
-        np_img = heuristic_resize(np_img, (safeint(old_w * k), safeint(old_h * k)))
+        np_img = high_quality_resize(np_img, (safeint(old_w * k), safeint(old_h * k)))
         new_h, new_w, _ = np_img.shape
         pad_h = max(0, (new_h - h) // 2)
         pad_w = max(0, (new_w - w) // 2)
         np_img = np_img[pad_h : pad_h + h, pad_w : pad_w + w]
-        np_img = clone_contiguous(np_img)
-        return np_img_to_torch(np_img, device), np_img
+        np_img = safe_numpy(np_img)
+        return get_pytorch_control(np_img), np_img
 
 
 def prepare_control_image(
@@ -289,12 +269,12 @@ def prepare_control_image(
     width: int,
     height: int,
     num_channels: int = 3,
-    device: str = "cuda",
-    dtype: torch.dtype = torch.float16,
-    control_mode: CONTROLNET_MODE_VALUES = "balanced",
-    resize_mode: CONTROLNET_RESIZE_VALUES = "just_resize_simple",
-    do_classifier_free_guidance: bool = True,
-) -> torch.Tensor:
+    device="cuda",
+    dtype=torch.float16,
+    do_classifier_free_guidance=True,
+    control_mode="balanced",
+    resize_mode="just_resize_simple",
+):
     """Pre-process images for ControlNets or T2I-Adapters.
 
     Args:
@@ -312,15 +292,26 @@ def prepare_control_image(
         resize_mode (str, optional): Defaults to "just_resize_simple".
 
     Raises:
+        NotImplementedError: If resize_mode == "crop_resize_simple".
+        NotImplementedError: If resize_mode == "fill_resize_simple".
         ValueError: If `resize_mode` is not recognized.
         ValueError: If `num_channels` is out of range.
 
     Returns:
         torch.Tensor: The pre-processed input tensor.
     """
-    if resize_mode == "just_resize_simple":
+    if (
+        resize_mode == "just_resize_simple"
+        or resize_mode == "crop_resize_simple"
+        or resize_mode == "fill_resize_simple"
+    ):
         image = image.convert("RGB")
-        image = image.resize((width, height), resample=Image.LANCZOS)
+        if resize_mode == "just_resize_simple":
+            image = image.resize((width, height), resample=PIL_INTERPOLATION["lanczos"])
+        elif resize_mode == "crop_resize_simple":
+            raise NotImplementedError(f"prepare_control_image is not implemented for resize_mode='{resize_mode}'.")
+        elif resize_mode == "fill_resize_simple":
+            raise NotImplementedError(f"prepare_control_image is not implemented for resize_mode='{resize_mode}'.")
         nimage = np.array(image)
         nimage = nimage[None, :]
         nimage = np.concatenate([nimage], axis=0)
@@ -337,7 +328,8 @@ def prepare_control_image(
             resize_mode=resize_mode,
             h=height,
             w=width,
-            device=torch.device(device),
+            # device=torch.device('cpu')
+            device=device,
         )
     else:
         raise ValueError(f"Unsupported resize_mode: '{resize_mode}'.")

invokeai/backend/image_util/__init__.py

@@ -2,6 +2,7 @@
 Initialization file for invokeai.backend.image_util methods.
 """
 
-from .infill_methods.patchmatch import PatchMatch  # noqa: F401
+from .patchmatch import PatchMatch  # noqa: F401
 from .pngwriter import PngWriter, PromptFormatter, retrieve_metadata, write_metadata  # noqa: F401
+from .seamless import configure_model_padding  # noqa: F401
 from .util import InitImageResizer, make_grid  # noqa: F401

invokeai/backend/image_util/depth_anything/__init__.py

@@ -13,7 +13,7 @@ from invokeai.app.services.config.config_default import get_config
 from invokeai.app.util.download_with_progress import download_with_progress_bar
 from invokeai.backend.image_util.depth_anything.model.dpt import DPT_DINOv2
 from invokeai.backend.image_util.depth_anything.utilities.util import NormalizeImage, PrepareForNet, Resize
-from invokeai.backend.util.devices import TorchDevice
+from invokeai.backend.util.devices import choose_torch_device
 from invokeai.backend.util.logging import InvokeAILogger
 
 config = get_config()
@@ -56,7 +56,7 @@ class DepthAnythingDetector:
     def __init__(self) -> None:
         self.model = None
         self.model_size: Union[Literal["large", "base", "small"], None] = None
-        self.device = TorchDevice.choose_torch_device()
+        self.device = choose_torch_device()
 
     def load_model(self, model_size: Literal["large", "base", "small"] = "small"):
         DEPTH_ANYTHING_MODEL_PATH = config.models_path / DEPTH_ANYTHING_MODELS[model_size]["local"]
@@ -81,7 +81,7 @@ class DepthAnythingDetector:
             self.model.load_state_dict(torch.load(DEPTH_ANYTHING_MODEL_PATH.as_posix(), map_location="cpu"))
             self.model.eval()
 
-        self.model.to(self.device)
+        self.model.to(choose_torch_device())
         return self.model
 
     def __call__(self, image: Image.Image, resolution: int = 512) -> Image.Image:
@@ -94,7 +94,7 @@ class DepthAnythingDetector:
 
         image_height, image_width = np_image.shape[:2]
         np_image = transform({"image": np_image})["image"]
-        tensor_image = torch.from_numpy(np_image).unsqueeze(0).to(self.device)
+        tensor_image = torch.from_numpy(np_image).unsqueeze(0).to(choose_torch_device())
 
         with torch.no_grad():
             depth = self.model(tensor_image)

invokeai/backend/image_util/dw_openpose/wholebody.py

@@ -7,7 +7,7 @@ import onnxruntime as ort
 
 from invokeai.app.services.config.config_default import get_config
 from invokeai.app.util.download_with_progress import download_with_progress_bar
-from invokeai.backend.util.devices import TorchDevice
+from invokeai.backend.util.devices import choose_torch_device
 
 from .onnxdet import inference_detector
 from .onnxpose import inference_pose
@@ -28,9 +28,9 @@ config = get_config()
 
 class Wholebody:
     def __init__(self):
-        device = TorchDevice.choose_torch_device()
+        device = choose_torch_device()
 
-        providers = ["CUDAExecutionProvider"] if device.type == "cuda" else ["CPUExecutionProvider"]
+        providers = ["CUDAExecutionProvider"] if device == "cuda" else ["CPUExecutionProvider"]
 
         DET_MODEL_PATH = config.models_path / DWPOSE_MODELS["yolox_l.onnx"]["local"]
         download_with_progress_bar("yolox_l.onnx", DWPOSE_MODELS["yolox_l.onnx"]["url"], DET_MODEL_PATH)

invokeai/backend/image_util/hed.py

@@ -8,7 +8,7 @@ from huggingface_hub import hf_hub_download
 from PIL import Image
 
 from invokeai.backend.image_util.util import (
-    nms,
+    non_maximum_suppression,
     normalize_image_channel_count,
     np_to_pil,
     pil_to_np,
@@ -134,7 +134,7 @@ class HEDProcessor:
         detected_map = cv2.resize(detected_map, (width, height), interpolation=cv2.INTER_LINEAR)
 
         if scribble:
-            detected_map = nms(detected_map, 127, 3.0)
+            detected_map = non_maximum_suppression(detected_map, 127, 3.0)
             detected_map = cv2.GaussianBlur(detected_map, (0, 0), 3.0)
             detected_map[detected_map > 4] = 255
             detected_map[detected_map < 255] = 0

invokeai/backend/image_util/infill_methods/mosaic.py

@@ -1,60 +0,0 @@
-from typing import Tuple
-
-import numpy as np
-from PIL import Image
-
-
-def infill_mosaic(
-    image: Image.Image,
-    tile_shape: Tuple[int, int] = (64, 64),
-    min_color: Tuple[int, int, int, int] = (0, 0, 0, 0),
-    max_color: Tuple[int, int, int, int] = (255, 255, 255, 0),
-) -> Image.Image:
-    """
-    image:PIL - A PIL Image
-    tile_shape: Tuple[int,int] - Tile width & Tile Height
-    min_color: Tuple[int,int,int] - RGB values for the lowest color to clip to (0-255)
-    max_color: Tuple[int,int,int] - RGB values for the highest color to clip to (0-255)
-    """
-
-    np_image = np.array(image)  # Convert image to np array
-    alpha = np_image[:, :, 3]  # Get the mask from the alpha channel of the image
-    non_transparent_pixels = np_image[alpha != 0, :3]  # List of non-transparent pixels
-
-    # Create color tiles to paste in the empty areas of the image
-    tile_width, tile_height = tile_shape
-
-    # Clip the range of colors in the image to a particular spectrum only
-    r_min, g_min, b_min, _ = min_color
-    r_max, g_max, b_max, _ = max_color
-    non_transparent_pixels[:, 0] = np.clip(non_transparent_pixels[:, 0], r_min, r_max)
-    non_transparent_pixels[:, 1] = np.clip(non_transparent_pixels[:, 1], g_min, g_max)
-    non_transparent_pixels[:, 2] = np.clip(non_transparent_pixels[:, 2], b_min, b_max)
-
-    tiles = []
-    for _ in range(256):
-        color = non_transparent_pixels[np.random.randint(len(non_transparent_pixels))]
-        tile = np.zeros((tile_height, tile_width, 3), dtype=np.uint8)
-        tile[:, :] = color
-        tiles.append(tile)
-
-    # Fill the transparent area with tiles
-    filled_image = np.zeros((image.height, image.width, 3), dtype=np.uint8)
-
-    for x in range(image.width):
-        for y in range(image.height):
-            tile = tiles[np.random.randint(len(tiles))]
-            try:
-                filled_image[
-                    y - (y % tile_height) : y - (y % tile_height) + tile_height,
-                    x - (x % tile_width) : x - (x % tile_width) + tile_width,
-                ] = tile
-            except ValueError:
-                # Need to handle edge cases - literally
-                pass
-
-    filled_image = Image.fromarray(filled_image)  # Convert the filled tiles image to PIL
-    image = Image.composite(
-        image, filled_image, image.split()[-1]
-    )  # Composite the original image on top of the filled tiles
-    return image

invokeai/backend/image_util/infill_methods/patchmatch.py

@@ -1,67 +0,0 @@
-"""
-This module defines a singleton object, "patchmatch" that
-wraps the actual patchmatch object. It respects the global
-"try_patchmatch" attribute, so that patchmatch loading can
-be suppressed or deferred
-"""
-
-import numpy as np
-from PIL import Image
-
-import invokeai.backend.util.logging as logger
-from invokeai.app.services.config.config_default import get_config
-
-
-class PatchMatch:
-    """
-    Thin class wrapper around the patchmatch function.
-    """
-
-    patch_match = None
-    tried_load: bool = False
-
-    def __init__(self):
-        super().__init__()
-
-    @classmethod
-    def _load_patch_match(cls):
-        if cls.tried_load:
-            return
-        if get_config().patchmatch:
-            from patchmatch import patch_match as pm
-
-            if pm.patchmatch_available:
-                logger.info("Patchmatch initialized")
-                cls.patch_match = pm
-            else:
-                logger.info("Patchmatch not loaded (nonfatal)")
-        else:
-            logger.info("Patchmatch loading disabled")
-        cls.tried_load = True
-
-    @classmethod
-    def patchmatch_available(cls) -> bool:
-        cls._load_patch_match()
-        if not cls.patch_match:
-            return False
-        return cls.patch_match.patchmatch_available
-
-    @classmethod
-    def inpaint(cls, image: Image.Image) -> Image.Image:
-        if cls.patch_match is None or not cls.patchmatch_available():
-            return image
-
-        np_image = np.array(image)
-        mask = 255 - np_image[:, :, 3]
-        infilled = cls.patch_match.inpaint(np_image[:, :, :3], mask, patch_size=3)
-        return Image.fromarray(infilled, mode="RGB")
-
-
-def infill_patchmatch(image: Image.Image) -> Image.Image:
-    IS_PATCHMATCH_AVAILABLE = PatchMatch.patchmatch_available()
-
-    if not IS_PATCHMATCH_AVAILABLE:
-        logger.warning("PatchMatch is not available on this system")
-        return image
-
-    return PatchMatch.inpaint(image)

invokeai/backend/image_util/infill_methods/tile.ipynb

@@ -1,95 +0,0 @@
-{
- "cells": [
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "\"\"\"Smoke test for the tile infill\"\"\"\n",
-    "\n",
-    "from pathlib import Path\n",
-    "from typing import Optional\n",
-    "from PIL import Image\n",
-    "from invokeai.backend.image_util.infill_methods.tile import infill_tile\n",
-    "\n",
-    "images: list[tuple[str, Image.Image]] = []\n",
-    "\n",
-    "for i in sorted(Path(\"./test_images/\").glob(\"*.webp\")):\n",
-    "    images.append((i.name, Image.open(i)))\n",
-    "    images.append((i.name, Image.open(i).transpose(Image.FLIP_LEFT_RIGHT)))\n",
-    "    images.append((i.name, Image.open(i).transpose(Image.FLIP_TOP_BOTTOM)))\n",
-    "    images.append((i.name, Image.open(i).resize((512, 512))))\n",
-    "    images.append((i.name, Image.open(i).resize((1234, 461))))\n",
-    "\n",
-    "outputs: list[tuple[str, Image.Image, Image.Image, Optional[Image.Image]]] = []\n",
-    "\n",
-    "for name, image in images:\n",
-    "    try:\n",
-    "        output = infill_tile(image, seed=0, tile_size=32)\n",
-    "        outputs.append((name, image, output.infilled, output.tile_image))\n",
-    "    except ValueError as e:\n",
-    "        print(f\"Skipping image {name}: {e}\")"
-   ]
-  },
-  {
-   "cell_type": "code",
-   "execution_count": null,
-   "metadata": {},
-   "outputs": [],
-   "source": [
-    "# Display the images in jupyter notebook\n",
-    "import matplotlib.pyplot as plt\n",
-    "from PIL import ImageOps\n",
-    "\n",
-    "fig, axes = plt.subplots(len(outputs), 3, figsize=(10, 3 * len(outputs)))\n",
-    "plt.subplots_adjust(hspace=0)\n",
-    "\n",
-    "for i, (name, original, infilled, tile_image) in enumerate(outputs):\n",
-    "    # Add a border to each image, helps to see the edges\n",
-    "    size = original.size\n",
-    "    original = ImageOps.expand(original, border=5, fill=\"red\")\n",
-    "    filled = ImageOps.expand(infilled, border=5, fill=\"red\")\n",
-    "    if tile_image:\n",
-    "        tile_image = ImageOps.expand(tile_image, border=5, fill=\"red\")\n",
-    "\n",
-    "    axes[i, 0].imshow(original)\n",
-    "    axes[i, 0].axis(\"off\")\n",
-    "    axes[i, 0].set_title(f\"Original ({name} - {size})\")\n",
-    "\n",
-    "    if tile_image:\n",
-    "        axes[i, 1].imshow(tile_image)\n",
-    "        axes[i, 1].axis(\"off\")\n",
-    "        axes[i, 1].set_title(\"Tile Image\")\n",
-    "    else:\n",
-    "        axes[i, 1].axis(\"off\")\n",
-    "        axes[i, 1].set_title(\"NO TILES GENERATED (NO TRANSPARENCY)\")\n",
-    "\n",
-    "    axes[i, 2].imshow(filled)\n",
-    "    axes[i, 2].axis(\"off\")\n",
-    "    axes[i, 2].set_title(\"Filled\")"
-   ]
-  }
- ],
- "metadata": {
-  "kernelspec": {
-   "display_name": ".invokeai",
-   "language": "python",
-   "name": "python3"
-  },
-  "language_info": {
-   "codemirror_mode": {
-    "name": "ipython",
-    "version": 3
-   },
-   "file_extension": ".py",
-   "mimetype": "text/x-python",
-   "name": "python",
-   "nbconvert_exporter": "python",
-   "pygments_lexer": "ipython3",
-   "version": "3.10.12"
-  }
- },
- "nbformat": 4,
- "nbformat_minor": 2
-}

invokeai/backend/image_util/infill_methods/tile.py

@@ -1,122 +0,0 @@
-from dataclasses import dataclass
-from typing import Optional
-
-import numpy as np
-from PIL import Image
-
-
-def create_tile_pool(img_array: np.ndarray, tile_size: tuple[int, int]) -> list[np.ndarray]:
-    """
-    Create a pool of tiles from non-transparent areas of the image by systematically walking through the image.
-
-    Args:
-        img_array: numpy array of the image.
-        tile_size: tuple (tile_width, tile_height) specifying the size of each tile.
-
-    Returns:
-        A list of numpy arrays, each representing a tile.
-    """
-    tiles: list[np.ndarray] = []
-    rows, cols = img_array.shape[:2]
-    tile_width, tile_height = tile_size
-
-    for y in range(0, rows - tile_height + 1, tile_height):
-        for x in range(0, cols - tile_width + 1, tile_width):
-            tile = img_array[y : y + tile_height, x : x + tile_width]
-            # Check if the image has an alpha channel and the tile is completely opaque
-            if img_array.shape[2] == 4 and np.all(tile[:, :, 3] == 255):
-                tiles.append(tile)
-            elif img_array.shape[2] == 3:  # If no alpha channel, append the tile
-                tiles.append(tile)
-
-    if not tiles:
-        raise ValueError(
-            "Not enough opaque pixels to generate any tiles. Use a smaller tile size or a different image."
-        )
-
-    return tiles
-
-
-def create_filled_image(
-    img_array: np.ndarray, tile_pool: list[np.ndarray], tile_size: tuple[int, int], seed: int
-) -> np.ndarray:
-    """
-    Create an image of the same dimensions as the original, filled entirely with tiles from the pool.
-
-    Args:
-        img_array: numpy array of the original image.
-        tile_pool: A list of numpy arrays, each representing a tile.
-        tile_size: tuple (tile_width, tile_height) specifying the size of each tile.
-
-    Returns:
-        A numpy array representing the filled image.
-    """
-
-    rows, cols, _ = img_array.shape
-    tile_width, tile_height = tile_size
-
-    # Prep an empty RGB image
-    filled_img_array = np.zeros((rows, cols, 3), dtype=img_array.dtype)
-
-    # Make the random tile selection reproducible
-    rng = np.random.default_rng(seed)
-
-    for y in range(0, rows, tile_height):
-        for x in range(0, cols, tile_width):
-            # Pick a random tile from the pool
-            tile = tile_pool[rng.integers(len(tile_pool))]
-
-            # Calculate the space available (may be less than tile size near the edges)
-            space_y = min(tile_height, rows - y)
-            space_x = min(tile_width, cols - x)
-
-            # Crop the tile if necessary to fit into the available space
-            cropped_tile = tile[:space_y, :space_x, :3]
-
-            # Fill the available space with the (possibly cropped) tile
-            filled_img_array[y : y + space_y, x : x + space_x, :3] = cropped_tile
-
-    return filled_img_array
-
-
-@dataclass
-class InfillTileOutput:
-    infilled: Image.Image
-    tile_image: Optional[Image.Image] = None
-
-
-def infill_tile(image_to_infill: Image.Image, seed: int, tile_size: int) -> InfillTileOutput:
-    """Infills an image with random tiles from the image itself.
-
-    If the image is not an RGBA image, it is returned untouched.
-
-    Args:
-        image: The image to infill.
-        tile_size: The size of the tiles to use for infilling.
-
-    Raises:
-        ValueError: If there are not enough opaque pixels to generate any tiles.
-    """
-
-    if image_to_infill.mode != "RGBA":
-        return InfillTileOutput(infilled=image_to_infill)
-
-    # Internally, we want a tuple of (tile_width, tile_height). In the future, the tile size can be any rectangle.
-    _tile_size = (tile_size, tile_size)
-    np_image = np.array(image_to_infill, dtype=np.uint8)
-
-    # Create the pool of tiles that we will use to infill
-    tile_pool = create_tile_pool(np_image, _tile_size)
-
-    # Create an image from the tiles, same size as the original
-    tile_np_image = create_filled_image(np_image, tile_pool, _tile_size, seed)
-
-    # Paste the OG image over the tile image, effectively infilling the area
-    tile_image = Image.fromarray(tile_np_image, "RGB")
-    infilled = tile_image.copy()
-    infilled.paste(image_to_infill, (0, 0), image_to_infill.split()[-1])
-
-    # I think we want this to be "RGBA"?
-    infilled.convert("RGBA")
-
-    return InfillTileOutput(infilled=infilled, tile_image=tile_image)

invokeai/backend/image_util/lama.py

@@ -7,8 +7,7 @@ from PIL import Image
 
 import invokeai.backend.util.logging as logger
 from invokeai.app.services.config.config_default import get_config
-from invokeai.app.util.download_with_progress import download_with_progress_bar
-from invokeai.backend.util.devices import TorchDevice
+from invokeai.backend.util.devices import choose_torch_device
 
 
 def norm_img(np_img):
@@ -29,16 +28,8 @@ def load_jit_model(url_or_path, device):
 
 class LaMA:
     def __call__(self, input_image: Image.Image, *args: Any, **kwds: Any) -> Any:
-        device = TorchDevice.choose_torch_device()
+        device = choose_torch_device()
         model_location = get_config().models_path / "core/misc/lama/lama.pt"
-
-        if not model_location.exists():
-            download_with_progress_bar(
-                name="LaMa Inpainting Model",
-                url="https://github.com/Sanster/models/releases/download/add_big_lama/big-lama.pt",
-                dest_path=model_location,
-            )
-
         model = load_jit_model(model_location, device)
 
         image = np.asarray(input_image.convert("RGB"))

invokeai/backend/image_util/patchmatch.py

@@ -0,0 +1,49 @@
+"""
+This module defines a singleton object, "patchmatch" that
+wraps the actual patchmatch object. It respects the global
+"try_patchmatch" attribute, so that patchmatch loading can
+be suppressed or deferred
+"""
+
+import numpy as np
+
+import invokeai.backend.util.logging as logger
+from invokeai.app.services.config.config_default import get_config
+
+
+class PatchMatch:
+    """
+    Thin class wrapper around the patchmatch function.
+    """
+
+    patch_match = None
+    tried_load: bool = False
+
+    def __init__(self):
+        super().__init__()
+
+    @classmethod
+    def _load_patch_match(self):
+        if self.tried_load:
+            return
+        if get_config().patchmatch:
+            from patchmatch import patch_match as pm
+
+            if pm.patchmatch_available:
+                logger.info("Patchmatch initialized")
+            else:
+                logger.info("Patchmatch not loaded (nonfatal)")
+            self.patch_match = pm
+        else:
+            logger.info("Patchmatch loading disabled")
+        self.tried_load = True
+
+    @classmethod
+    def patchmatch_available(self) -> bool:
+        self._load_patch_match()
+        return self.patch_match and self.patch_match.patchmatch_available
+
+    @classmethod
+    def inpaint(self, *args, **kwargs) -> np.ndarray:
+        if self.patchmatch_available():
+            return self.patch_match.inpaint(*args, **kwargs)

invokeai/backend/image_util/realesrgan/realesrgan.py

@@ -11,7 +11,7 @@ from cv2.typing import MatLike
 from tqdm import tqdm
 
 from invokeai.backend.image_util.basicsr.rrdbnet_arch import RRDBNet
-from invokeai.backend.util.devices import TorchDevice
+from invokeai.backend.util.devices import choose_torch_device
 
 """
 Adapted from https://github.com/xinntao/Real-ESRGAN/blob/master/realesrgan/utils.py
@@ -65,7 +65,7 @@ class RealESRGAN:
         self.pre_pad = pre_pad
         self.mod_scale: Optional[int] = None
         self.half = half
-        self.device = TorchDevice.choose_torch_device()
+        self.device = choose_torch_device()
 
         loadnet = torch.load(model_path, map_location=torch.device("cpu"))
 

invokeai/backend/image_util/safety_checker.py

@@ -8,15 +8,14 @@ from pathlib import Path
 
 import numpy as np
 from diffusers.pipelines.stable_diffusion.safety_checker import StableDiffusionSafetyChecker
-from PIL import Image, ImageFilter
+from PIL import Image
 from transformers import AutoFeatureExtractor
 
 import invokeai.backend.util.logging as logger
 from invokeai.app.services.config.config_default import get_config
-from invokeai.backend.util.devices import TorchDevice
+from invokeai.backend.util.devices import choose_torch_device
 from invokeai.backend.util.silence_warnings import SilenceWarnings
 
-repo_id = "CompVis/stable-diffusion-safety-checker"
 CHECKER_PATH = "core/convert/stable-diffusion-safety-checker"
 
 
@@ -25,34 +24,34 @@ class SafetyChecker:
     Wrapper around SafetyChecker model.
     """
 
-    feature_extractor = None
     safety_checker = None
+    feature_extractor = None
+    tried_load: bool = False
 
     @classmethod
     def _load_safety_checker(cls):
-        if cls.safety_checker is not None and cls.feature_extractor is not None:
+        if cls.tried_load:
             return
 
         try:
-            model_path = get_config().models_path / CHECKER_PATH
-            if model_path.exists():
-                cls.feature_extractor = AutoFeatureExtractor.from_pretrained(model_path)
-                cls.safety_checker = StableDiffusionSafetyChecker.from_pretrained(model_path)
-            else:
-                model_path.mkdir(parents=True, exist_ok=True)
-                cls.feature_extractor = AutoFeatureExtractor.from_pretrained(repo_id)
-                cls.feature_extractor.save_pretrained(model_path, safe_serialization=True)
-                cls.safety_checker = StableDiffusionSafetyChecker.from_pretrained(repo_id)
-                cls.safety_checker.save_pretrained(model_path, safe_serialization=True)
+            cls.safety_checker = StableDiffusionSafetyChecker.from_pretrained(get_config().models_path / CHECKER_PATH)
+            cls.feature_extractor = AutoFeatureExtractor.from_pretrained(get_config().models_path / CHECKER_PATH)
         except Exception as e:
             logger.warning(f"Could not load NSFW checker: {str(e)}")
+        cls.tried_load = True
+
+    @classmethod
+    def safety_checker_available(cls) -> bool:
+        return Path(get_config().models_path, CHECKER_PATH).exists()
 
     @classmethod
     def has_nsfw_concept(cls, image: Image.Image) -> bool:
+        if not cls.safety_checker_available() and cls.tried_load:
+            return False
         cls._load_safety_checker()
         if cls.safety_checker is None or cls.feature_extractor is None:
             return False
-        device = TorchDevice.choose_torch_device()
+        device = choose_torch_device()
         features = cls.feature_extractor([image], return_tensors="pt")
         features.to(device)
         cls.safety_checker.to(device)
@@ -61,24 +60,3 @@ class SafetyChecker:
         with SilenceWarnings():
             checked_image, has_nsfw_concept = cls.safety_checker(images=x_image, clip_input=features.pixel_values)
         return has_nsfw_concept[0]
-
-    @classmethod
-    def blur_if_nsfw(cls, image: Image.Image) -> Image.Image:
-        if cls.has_nsfw_concept(image):
-            logger.warning("A potentially NSFW image has been detected. Image will be blurred.")
-            blurry_image = image.filter(filter=ImageFilter.GaussianBlur(radius=32))
-            caution = cls._get_caution_img()
-            # Center the caution image on the blurred image
-            x = (blurry_image.width - caution.width) // 2
-            y = (blurry_image.height - caution.height) // 2
-            blurry_image.paste(caution, (x, y), caution)
-            image = blurry_image
-
-        return image
-
-    @classmethod
-    def _get_caution_img(cls) -> Image.Image:
-        import invokeai.app.assets.images as image_assets
-
-        caution = Image.open(Path(image_assets.__path__[0]) / "caution.png")
-        return caution.resize((caution.width // 2, caution.height // 2))

invokeai/backend/image_util/seamless.py

@@ -0,0 +1,52 @@
+import torch.nn as nn
+
+
+def _conv_forward_asymmetric(self, input, weight, bias):
+    """
+    Patch for Conv2d._conv_forward that supports asymmetric padding
+    """
+    working = nn.functional.pad(input, self.asymmetric_padding["x"], mode=self.asymmetric_padding_mode["x"])
+    working = nn.functional.pad(working, self.asymmetric_padding["y"], mode=self.asymmetric_padding_mode["y"])
+    return nn.functional.conv2d(
+        working,
+        weight,
+        bias,
+        self.stride,
+        nn.modules.utils._pair(0),
+        self.dilation,
+        self.groups,
+    )
+
+
+def configure_model_padding(model, seamless, seamless_axes):
+    """
+    Modifies the 2D convolution layers to use a circular padding mode based on
+    the `seamless` and `seamless_axes` options.
+    """
+    # TODO: get an explicit interface for this in diffusers: https://github.com/huggingface/diffusers/issues/556
+    for m in model.modules():
+        if isinstance(m, (nn.Conv2d, nn.ConvTranspose2d)):
+            if seamless:
+                m.asymmetric_padding_mode = {}
+                m.asymmetric_padding = {}
+                m.asymmetric_padding_mode["x"] = "circular" if ("x" in seamless_axes) else "constant"
+                m.asymmetric_padding["x"] = (
+                    m._reversed_padding_repeated_twice[0],
+                    m._reversed_padding_repeated_twice[1],
+                    0,
+                    0,
+                )
+                m.asymmetric_padding_mode["y"] = "circular" if ("y" in seamless_axes) else "constant"
+                m.asymmetric_padding["y"] = (
+                    0,
+                    0,
+                    m._reversed_padding_repeated_twice[2],
+                    m._reversed_padding_repeated_twice[3],
+                )
+                m._conv_forward = _conv_forward_asymmetric.__get__(m, nn.Conv2d)
+            else:
+                m._conv_forward = nn.Conv2d._conv_forward.__get__(m, nn.Conv2d)
+                if hasattr(m, "asymmetric_padding_mode"):
+                    del m.asymmetric_padding_mode
+                if hasattr(m, "asymmetric_padding"):
+                    del m.asymmetric_padding

invokeai/backend/image_util/util.py

@@ -1,5 +1,4 @@
 from math import ceil, floor, sqrt
-from typing import Optional
 
 import cv2
 import numpy as np
@@ -144,21 +143,20 @@ def resize_image_to_resolution(input_image: np.ndarray, resolution: int) -> np.n
     h = float(input_image.shape[0])
     w = float(input_image.shape[1])
     scaling_factor = float(resolution) / min(h, w)
-    h = int(h * scaling_factor)
-    w = int(w * scaling_factor)
+    h *= scaling_factor
+    w *= scaling_factor
+    h = int(np.round(h / 64.0)) * 64
+    w = int(np.round(w / 64.0)) * 64
     if scaling_factor > 1:
         return cv2.resize(input_image, (w, h), interpolation=cv2.INTER_LANCZOS4)
     else:
         return cv2.resize(input_image, (w, h), interpolation=cv2.INTER_AREA)
 
 
-def nms(np_img: np.ndarray, threshold: Optional[int] = None, sigma: Optional[float] = None) -> np.ndarray:
+def non_maximum_suppression(image: np.ndarray, threshold: int, sigma: float):
     """
     Apply non-maximum suppression to an image.
 
-    If both threshold and sigma are provided, the image will blurred before the suppression and thresholded afterwards,
-    resulting in a binary output image.
-
     This function is adapted from https://github.com/lllyasviel/ControlNet.
 
     Args:
@@ -168,36 +166,23 @@ def nms(np_img: np.ndarray, threshold: Optional[int] = None, sigma: Optional[flo
 
     Returns:
         The image after non-maximum suppression.
-
-    Raises:
-        ValueError: If only one of threshold and sigma provided.
     """
 
-    # Raise a value error if only one of threshold and sigma is provided
-    if (threshold is None) != (sigma is None):
-        raise ValueError("Both threshold and sigma must be provided if one is provided.")
-
-    if sigma is not None and threshold is not None:
-        # Blurring the image can help to thin out features
-        np_img = cv2.GaussianBlur(np_img.astype(np.float32), (0, 0), sigma)
+    image = cv2.GaussianBlur(image.astype(np.float32), (0, 0), sigma)
 
     filter_1 = np.array([[0, 0, 0], [1, 1, 1], [0, 0, 0]], dtype=np.uint8)
     filter_2 = np.array([[0, 1, 0], [0, 1, 0], [0, 1, 0]], dtype=np.uint8)
     filter_3 = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1]], dtype=np.uint8)
     filter_4 = np.array([[0, 0, 1], [0, 1, 0], [1, 0, 0]], dtype=np.uint8)
 
-    nms_img = np.zeros_like(np_img)
+    y = np.zeros_like(image)
 
     for f in [filter_1, filter_2, filter_3, filter_4]:
-        np.putmask(nms_img, cv2.dilate(np_img, kernel=f) == np_img, np_img)
+        np.putmask(y, cv2.dilate(image, kernel=f) == image, image)
 
-    if sigma is not None and threshold is not None:
-        # We blurred - now threshold to get a binary image
-        thresholded = np.zeros_like(nms_img, dtype=np.uint8)
-        thresholded[nms_img > threshold] = 255
-        return thresholded
-
-    return nms_img
+    z = np.zeros_like(y, dtype=np.uint8)
+    z[y > threshold] = 255
+    return z
 
 
 def safe_step(x: np.ndarray, step: int = 2) -> np.ndarray:

invokeai/backend/ip_adapter/attention_processor.py

@@ -0,0 +1,182 @@
+# copied from https://github.com/tencent-ailab/IP-Adapter (Apache License 2.0)
+#   and modified as needed
+
+# tencent-ailab comment:
+# modified from https://github.com/huggingface/diffusers/blob/main/src/diffusers/models/attention_processor.py
+import torch
+import torch.nn as nn
+import torch.nn.functional as F
+from diffusers.models.attention_processor import AttnProcessor2_0 as DiffusersAttnProcessor2_0
+
+from invokeai.backend.ip_adapter.ip_attention_weights import IPAttentionProcessorWeights
+
+
+# Create a version of AttnProcessor2_0 that is a sub-class of nn.Module. This is required for IP-Adapter state_dict
+# loading.
+class AttnProcessor2_0(DiffusersAttnProcessor2_0, nn.Module):
+    def __init__(self):
+        DiffusersAttnProcessor2_0.__init__(self)
+        nn.Module.__init__(self)
+
+    def __call__(
+        self,
+        attn,
+        hidden_states,
+        encoder_hidden_states=None,
+        attention_mask=None,
+        temb=None,
+        ip_adapter_image_prompt_embeds=None,
+    ):
+        """Re-definition of DiffusersAttnProcessor2_0.__call__(...) that accepts and ignores the
+        ip_adapter_image_prompt_embeds parameter.
+        """
+        return DiffusersAttnProcessor2_0.__call__(
+            self, attn, hidden_states, encoder_hidden_states, attention_mask, temb
+        )
+
+
+class IPAttnProcessor2_0(torch.nn.Module):
+    r"""
+    Attention processor for IP-Adapater for PyTorch 2.0.
+    Args:
+        hidden_size (`int`):
+            The hidden size of the attention layer.
+        cross_attention_dim (`int`):
+            The number of channels in the `encoder_hidden_states`.
+        scale (`float`, defaults to 1.0):
+            the weight scale of image prompt.
+    """
+
+    def __init__(self, weights: list[IPAttentionProcessorWeights], scales: list[float]):
+        super().__init__()
+
+        if not hasattr(F, "scaled_dot_product_attention"):
+            raise ImportError("AttnProcessor2_0 requires PyTorch 2.0, to use it, please upgrade PyTorch to 2.0.")
+
+        assert len(weights) == len(scales)
+
+        self._weights = weights
+        self._scales = scales
+
+    def __call__(
+        self,
+        attn,
+        hidden_states,
+        encoder_hidden_states=None,
+        attention_mask=None,
+        temb=None,
+        ip_adapter_image_prompt_embeds=None,
+    ):
+        """Apply IP-Adapter attention.
+
+        Args:
+            ip_adapter_image_prompt_embeds (torch.Tensor): The image prompt embeddings.
+                Shape: (batch_size, num_ip_images, seq_len, ip_embedding_len).
+        """
+        residual = hidden_states
+
+        if attn.spatial_norm is not None:
+            hidden_states = attn.spatial_norm(hidden_states, temb)
+
+        input_ndim = hidden_states.ndim
+
+        if input_ndim == 4:
+            batch_size, channel, height, width = hidden_states.shape
+            hidden_states = hidden_states.view(batch_size, channel, height * width).transpose(1, 2)
+
+        batch_size, sequence_length, _ = (
+            hidden_states.shape if encoder_hidden_states is None else encoder_hidden_states.shape
+        )
+
+        if attention_mask is not None:
+            attention_mask = attn.prepare_attention_mask(attention_mask, sequence_length, batch_size)
+            # scaled_dot_product_attention expects attention_mask shape to be
+            # (batch, heads, source_length, target_length)
+            attention_mask = attention_mask.view(batch_size, attn.heads, -1, attention_mask.shape[-1])
+
+        if attn.group_norm is not None:
+            hidden_states = attn.group_norm(hidden_states.transpose(1, 2)).transpose(1, 2)
+
+        query = attn.to_q(hidden_states)
+
+        if encoder_hidden_states is None:
+            encoder_hidden_states = hidden_states
+        elif attn.norm_cross:
+            encoder_hidden_states = attn.norm_encoder_hidden_states(encoder_hidden_states)
+
+        key = attn.to_k(encoder_hidden_states)
+        value = attn.to_v(encoder_hidden_states)
+
+        inner_dim = key.shape[-1]
+        head_dim = inner_dim // attn.heads
+
+        query = query.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
+
+        key = key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
+        value = value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
+
+        # the output of sdp = (batch, num_heads, seq_len, head_dim)
+        # TODO: add support for attn.scale when we move to Torch 2.1
+        hidden_states = F.scaled_dot_product_attention(
+            query, key, value, attn_mask=attention_mask, dropout_p=0.0, is_causal=False
+        )
+
+        hidden_states = hidden_states.transpose(1, 2).reshape(batch_size, -1, attn.heads * head_dim)
+        hidden_states = hidden_states.to(query.dtype)
+
+        if encoder_hidden_states is not None:
+            # If encoder_hidden_states is not None, then we are doing cross-attention, not self-attention. In this case,
+            # we will apply IP-Adapter conditioning. We validate the inputs for IP-Adapter conditioning here.
+            assert ip_adapter_image_prompt_embeds is not None
+            assert len(ip_adapter_image_prompt_embeds) == len(self._weights)
+
+            for ipa_embed, ipa_weights, scale in zip(
+                ip_adapter_image_prompt_embeds, self._weights, self._scales, strict=True
+            ):
+                # The batch dimensions should match.
+                assert ipa_embed.shape[0] == encoder_hidden_states.shape[0]
+                # The token_len dimensions should match.
+                assert ipa_embed.shape[-1] == encoder_hidden_states.shape[-1]
+
+                ip_hidden_states = ipa_embed
+
+                # Expected ip_hidden_state shape: (batch_size, num_ip_images, ip_seq_len, ip_image_embedding)
+
+                ip_key = ipa_weights.to_k_ip(ip_hidden_states)
+                ip_value = ipa_weights.to_v_ip(ip_hidden_states)
+
+                # Expected ip_key and ip_value shape: (batch_size, num_ip_images, ip_seq_len, head_dim * num_heads)
+
+                ip_key = ip_key.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
+                ip_value = ip_value.view(batch_size, -1, attn.heads, head_dim).transpose(1, 2)
+
+                # Expected ip_key and ip_value shape: (batch_size, num_heads, num_ip_images * ip_seq_len, head_dim)
+
+                # TODO: add support for attn.scale when we move to Torch 2.1
+                ip_hidden_states = F.scaled_dot_product_attention(
+                    query, ip_key, ip_value, attn_mask=None, dropout_p=0.0, is_causal=False
+                )
+
+                # Expected ip_hidden_states shape: (batch_size, num_heads, query_seq_len, head_dim)
+
+                ip_hidden_states = ip_hidden_states.transpose(1, 2).reshape(batch_size, -1, attn.heads * head_dim)
+                ip_hidden_states = ip_hidden_states.to(query.dtype)
+
+                # Expected ip_hidden_states shape: (batch_size, query_seq_len, num_heads * head_dim)
+
+                hidden_states = hidden_states + scale * ip_hidden_states
+
+        # linear proj
+        hidden_states = attn.to_out[0](hidden_states)
+        # dropout
+        hidden_states = attn.to_out[1](hidden_states)
+
+        if input_ndim == 4:
+            hidden_states = hidden_states.transpose(-1, -2).reshape(batch_size, channel, height, width)
+
+        if attn.residual_connection:
+            hidden_states = hidden_states + residual
+
+        hidden_states = hidden_states / attn.rescale_output_factor
+
+        return hidden_states

invokeai/backend/ip_adapter/ip_adapter.py

@@ -12,7 +12,6 @@ from transformers import CLIPImageProcessor, CLIPVisionModelWithProjection
 
 from invokeai.backend.ip_adapter.ip_attention_weights import IPAttentionWeights
 
-from ..raw_model import RawModel
 from .resampler import Resampler
 
 
@@ -102,7 +101,7 @@ class MLPProjModel(torch.nn.Module):
         return clip_extra_context_tokens
 
 
-class IPAdapter(RawModel):
+class IPAdapter(torch.nn.Module):
     """IP-Adapter: https://arxiv.org/pdf/2308.06721.pdf"""
 
     def __init__(
@@ -112,6 +111,7 @@ class IPAdapter(RawModel):
         dtype: torch.dtype = torch.float16,
         num_tokens: int = 4,
     ):
+        super().__init__()
         self.device = device
         self.dtype = dtype
 

invokeai/backend/ip_adapter/unet_patcher.py

@@ -0,0 +1,53 @@
+from contextlib import contextmanager
+
+from diffusers.models import UNet2DConditionModel
+
+from invokeai.backend.ip_adapter.attention_processor import AttnProcessor2_0, IPAttnProcessor2_0
+from invokeai.backend.ip_adapter.ip_adapter import IPAdapter
+
+
+class UNetPatcher:
+    """A class that contains multiple IP-Adapters and can apply them to a UNet."""
+
+    def __init__(self, ip_adapters: list[IPAdapter]):
+        self._ip_adapters = ip_adapters
+        self._scales = [1.0] * len(self._ip_adapters)
+
+    def set_scale(self, idx: int, value: float):
+        self._scales[idx] = value
+
+    def _prepare_attention_processors(self, unet: UNet2DConditionModel):
+        """Prepare a dict of attention processors that can be injected into a unet, and load the IP-Adapter attention
+        weights into them.
+
+        Note that the `unet` param is only used to determine attention block dimensions and naming.
+        """
+        # Construct a dict of attention processors based on the UNet's architecture.
+        attn_procs = {}
+        for idx, name in enumerate(unet.attn_processors.keys()):
+            if name.endswith("attn1.processor"):
+                attn_procs[name] = AttnProcessor2_0()
+            else:
+                # Collect the weights from each IP Adapter for the idx'th attention processor.
+                attn_procs[name] = IPAttnProcessor2_0(
+                    [ip_adapter.attn_weights.get_attention_processor_weights(idx) for ip_adapter in self._ip_adapters],
+                    self._scales,
+                )
+        return attn_procs
+
+    @contextmanager
+    def apply_ip_adapter_attention(self, unet: UNet2DConditionModel):
+        """A context manager that patches `unet` with IP-Adapter attention processors."""
+
+        attn_procs = self._prepare_attention_processors(unet)
+
+        orig_attn_processors = unet.attn_processors
+
+        try:
+            # Note to future devs: set_attn_processor(...) does something slightly unexpected - it pops elements from the
+            # passed dict. So, if you wanted to keep the dict for future use, you'd have to make a moderately-shallow copy
+            # of it. E.g. `attn_procs_copy = {k: v for k, v in attn_procs.items()}`.
+            unet.set_attn_processor(attn_procs)
+            yield None
+        finally:
+            unet.set_attn_processor(orig_attn_processors)

invokeai/backend/lora.py

@@ -11,8 +11,6 @@ from typing_extensions import Self
 
 from invokeai.backend.model_manager import BaseModelType
 
-from .raw_model import RawModel
-
 
 class LoRALayerBase:
     # rank: Optional[int]
@@ -368,15 +366,13 @@ class IA3Layer(LoRALayerBase):
 AnyLoRALayer = Union[LoRALayer, LoHALayer, LoKRLayer, FullLayer, IA3Layer]
 
 
-class LoRAModelRaw(RawModel):  # (torch.nn.Module):
-    _name: str
-    layers: Dict[str, AnyLoRALayer]
-
+class LoRAModelRaw(torch.nn.Module):
     def __init__(
         self,
         name: str,
         layers: Dict[str, AnyLoRALayer],
     ):
+        super().__init__()
         self._name = name
         self.layers = layers
 

invokeai/backend/model_manager/__init__.py

@@ -1,7 +1,6 @@
 """Re-export frequently-used symbols from the Model Manager backend."""
 
 from .config import (
-    AnyModel,
     AnyModelConfig,
     BaseModelType,
     InvalidModelConfigException,
@@ -18,7 +17,6 @@ from .probe import ModelProbe
 from .search import ModelSearch
 
 __all__ = [
-    "AnyModel",
     "AnyModelConfig",
     "BaseModelType",
     "ModelRepoVariant",

invokeai/backend/model_manager/any_model_type.py

@@ -0,0 +1,12 @@
+from typing import Union
+
+import torch
+from diffusers.models.modeling_utils import ModelMixin
+
+from invokeai.backend.ip_adapter.ip_adapter import IPAdapter
+from invokeai.backend.onnx.onnx_runtime import IAIOnnxRuntimeModel
+from invokeai.backend.peft.peft_model import PeftModel
+from invokeai.backend.textual_inversion import TextualInversionModelRaw
+
+# ModelMixin is the base class for all diffusers and transformers models
+AnyModel = Union[ModelMixin, torch.nn.Module, IPAdapter, PeftModel, TextualInversionModelRaw, IAIOnnxRuntimeModel]

invokeai/backend/model_manager/config.py

@@ -24,20 +24,12 @@ import time
 from enum import Enum
 from typing import Literal, Optional, Type, TypeAlias, Union
 
-import torch
-from diffusers.models.modeling_utils import ModelMixin
 from pydantic import BaseModel, ConfigDict, Discriminator, Field, Tag, TypeAdapter
 from typing_extensions import Annotated, Any, Dict
 
 from invokeai.app.invocations.constants import SCHEDULER_NAME_VALUES
 from invokeai.app.util.misc import uuid_string
 
-from ..raw_model import RawModel
-
-# ModelMixin is the base class for all diffusers and transformers models
-# RawModel is the InvokeAI wrapper class for ip_adapters, loras, textual_inversion and onnx runtime
-AnyModel = Union[ModelMixin, RawModel, torch.nn.Module]
-
 
 class InvalidModelConfigException(Exception):
     """Exception for when config parser doesn't recognized this combination of model type and format."""
@@ -301,12 +293,12 @@ class MainConfigBase(ModelConfigBase):
     default_settings: Optional[MainModelDefaultSettings] = Field(
         description="Default settings for this model", default=None
     )
-    variant: ModelVariantType = ModelVariantType.Normal
 
 
 class MainCheckpointConfig(CheckpointConfigBase, MainConfigBase):
     """Model config for main checkpoint models."""
 
+    variant: ModelVariantType = ModelVariantType.Normal
     prediction_type: SchedulerPredictionType = SchedulerPredictionType.Epsilon
     upcast_attention: bool = False
 

invokeai/backend/model_manager/convert_ckpt_to_diffusers.py

@@ -15,7 +15,7 @@ from diffusers.pipelines.stable_diffusion.convert_from_ckpt import (
 )
 from omegaconf import DictConfig
 
-from . import AnyModel
+from invokeai.backend.model_manager.any_model_type import AnyModel
 
 
 def convert_ldm_vae_to_diffusers(

invokeai/backend/model_manager/load/load_base.py

@@ -10,8 +10,8 @@ from pathlib import Path
 from typing import Any, Optional
 
 from invokeai.app.services.config import InvokeAIAppConfig
+from invokeai.backend.model_manager.any_model_type import AnyModel
 from invokeai.backend.model_manager.config import (
-    AnyModel,
     AnyModelConfig,
     SubModelType,
 )

invokeai/backend/model_manager/load/load_default.py

@@ -7,18 +7,18 @@ from typing import Optional
 
 from invokeai.app.services.config import InvokeAIAppConfig
 from invokeai.backend.model_manager import (
-    AnyModel,
     AnyModelConfig,
     InvalidModelConfigException,
     SubModelType,
 )
+from invokeai.backend.model_manager.any_model_type import AnyModel
 from invokeai.backend.model_manager.config import DiffusersConfigBase, ModelType
 from invokeai.backend.model_manager.load.convert_cache import ModelConvertCacheBase
 from invokeai.backend.model_manager.load.load_base import LoadedModel, ModelLoaderBase
 from invokeai.backend.model_manager.load.model_cache.model_cache_base import ModelCacheBase, ModelLockerBase
 from invokeai.backend.model_manager.load.model_util import calc_model_size_by_data, calc_model_size_by_fs
 from invokeai.backend.model_manager.load.optimizations import skip_torch_weight_init
-from invokeai.backend.util.devices import TorchDevice
+from invokeai.backend.util.devices import choose_torch_device, torch_dtype
 
 
 # TO DO: The loader is not thread safe!
@@ -37,7 +37,7 @@ class ModelLoader(ModelLoaderBase):
         self._logger = logger
         self._ram_cache = ram_cache
         self._convert_cache = convert_cache
-        self._torch_dtype = TorchDevice.choose_torch_dtype()
+        self._torch_dtype = torch_dtype(choose_torch_device(), app_config)
 
     def load_model(self, model_config: AnyModelConfig, submodel_type: Optional[SubModelType] = None) -> LoadedModel:
         """

invokeai/backend/model_manager/load/model_cache/model_cache_base.py

@@ -14,7 +14,8 @@ from typing import Dict, Generic, Optional, TypeVar
 
 import torch
 
-from invokeai.backend.model_manager.config import AnyModel, SubModelType
+from invokeai.backend.model_manager.any_model_type import AnyModel
+from invokeai.backend.model_manager.config import SubModelType
 
 
 class ModelLockerBase(ABC):
@@ -117,7 +118,7 @@ class ModelCacheBase(ABC, Generic[T]):
 
     @property
     @abstractmethod
-    def stats(self) -> Optional[CacheStats]:
+    def stats(self) -> CacheStats:
         """Return collected CacheStats object."""
         pass
 

invokeai/backend/model_manager/load/model_cache/model_cache_default.py

@@ -28,14 +28,18 @@ from typing import Dict, List, Optional
 
 import torch
 
-from invokeai.backend.model_manager import AnyModel, SubModelType
+from invokeai.backend.model_manager import SubModelType
+from invokeai.backend.model_manager.any_model_type import AnyModel
 from invokeai.backend.model_manager.load.memory_snapshot import MemorySnapshot, get_pretty_snapshot_diff
-from invokeai.backend.util.devices import TorchDevice
+from invokeai.backend.util.devices import choose_torch_device
 from invokeai.backend.util.logging import InvokeAILogger
 
 from .model_cache_base import CacheRecord, CacheStats, ModelCacheBase, ModelLockerBase
 from .model_locker import ModelLocker
 
+if choose_torch_device() == torch.device("mps"):
+    from torch import mps
+
 # Maximum size of the cache, in gigs
 # Default is roughly enough to hold three fp16 diffusers models in RAM simultaneously
 DEFAULT_MAX_CACHE_SIZE = 6.0
@@ -241,7 +245,9 @@ class ModelCache(ModelCacheBase[AnyModel]):
                     f"Removing {cache_entry.key} from VRAM to free {(cache_entry.size/GIG):.2f}GB; vram free = {(torch.cuda.memory_allocated()/GIG):.2f}GB"
                 )
 
-        TorchDevice.empty_cache()
+        torch.cuda.empty_cache()
+        if choose_torch_device() == torch.device("mps"):
+            mps.empty_cache()
 
     def move_model_to_device(self, cache_entry: CacheRecord[AnyModel], target_device: torch.device) -> None:
         """Move model into the indicated device.
@@ -264,14 +270,12 @@ class ModelCache(ModelCacheBase[AnyModel]):
         if torch.device(source_device).type == torch.device(target_device).type:
             return
 
+        # may raise an exception here if insufficient GPU VRAM
+        self._check_free_vram(target_device, cache_entry.size)
+
         start_model_to_time = time.time()
         snapshot_before = self._capture_memory_snapshot()
-        try:
-            cache_entry.model.to(target_device)
-        except Exception as e:  # blow away cache entry
-            self._delete_cache_entry(cache_entry)
-            raise e
-
+        cache_entry.model.to(target_device)
         snapshot_after = self._capture_memory_snapshot()
         end_model_to_time = time.time()
         self.logger.debug(
@@ -326,11 +330,11 @@ class ModelCache(ModelCacheBase[AnyModel]):
                     f" {in_ram_models}/{in_vram_models}({locked_in_vram_models})"
                 )
 
-    def make_room(self, size: int) -> None:
+    def make_room(self, model_size: int) -> None:
         """Make enough room in the cache to accommodate a new model of indicated size."""
         # calculate how much memory this model will require
         # multiplier = 2 if self.precision==torch.float32 else 1
-        bytes_needed = size
+        bytes_needed = model_size
         maximum_size = self.max_cache_size * GIG  # stored in GB, convert to bytes
         current_size = self.cache_size()
 
@@ -385,11 +389,12 @@ class ModelCache(ModelCacheBase[AnyModel]):
             # 1 from onnx runtime object
             if not cache_entry.locked and refs <= (3 if "onnx" in model_key else 2):
                 self.logger.debug(
-                    f"Removing {model_key} from RAM cache to free at least {(size/GIG):.2f} GB (-{(cache_entry.size/GIG):.2f} GB)"
+                    f"Removing {model_key} from RAM cache to free at least {(model_size/GIG):.2f} GB (-{(cache_entry.size/GIG):.2f} GB)"
                 )
                 current_size -= cache_entry.size
                 models_cleared += 1
-                self._delete_cache_entry(cache_entry)
+                del self._cache_stack[pos]
+                del self._cached_models[model_key]
                 del cache_entry
 
             else:
@@ -411,9 +416,22 @@ class ModelCache(ModelCacheBase[AnyModel]):
                 self.stats.cleared = models_cleared
             gc.collect()
 
-        TorchDevice.empty_cache()
+        torch.cuda.empty_cache()
+        if choose_torch_device() == torch.device("mps"):
+            mps.empty_cache()
+
         self.logger.debug(f"After making room: cached_models={len(self._cached_models)}")
 
-    def _delete_cache_entry(self, cache_entry: CacheRecord[AnyModel]) -> None:
-        self._cache_stack.remove(cache_entry.key)
-        del self._cached_models[cache_entry.key]
+    def _check_free_vram(self, target_device: torch.device, needed_size: int) -> None:
+        if target_device.type != "cuda":
+            return
+        vram_device = (  # mem_get_info() needs an indexed device
+            target_device if target_device.index is not None else torch.device(str(target_device), index=0)
+        )
+        free_mem, _ = torch.cuda.mem_get_info(torch.device(vram_device))
+        if needed_size > free_mem:
+            needed_gb = round(needed_size / GIG, 2)
+            free_gb = round(free_mem / GIG, 2)
+            raise torch.cuda.OutOfMemoryError(
+                f"Insufficient VRAM to load model, requested {needed_gb}GB but only had {free_gb}GB free"
+            )

invokeai/backend/model_manager/load/model_cache/model_locker.py

@@ -4,7 +4,7 @@ Base class and implementation of a class that moves models in and out of VRAM.
 
 import torch
 
-from invokeai.backend.model_manager import AnyModel
+from invokeai.backend.model_manager.any_model_type import AnyModel
 
 from .model_cache_base import CacheRecord, ModelCacheBase, ModelLockerBase
 
@@ -34,6 +34,7 @@ class ModelLocker(ModelLockerBase):
 
         # NOTE that the model has to have the to() method in order for this code to move it into GPU!
         self._cache_entry.lock()
+
         try:
             if self._cache.lazy_offloading:
                 self._cache.offload_unlocked_models(self._cache_entry.size)
@@ -50,7 +51,6 @@ class ModelLocker(ModelLockerBase):
         except Exception:
             self._cache_entry.unlock()
             raise
-
         return self.model
 
     def unlock(self) -> None:

invokeai/backend/model_manager/load/model_loaders/controlnet.py

@@ -5,12 +5,12 @@ from pathlib import Path
 from typing import Optional
 
 from invokeai.backend.model_manager import (
-    AnyModel,
     AnyModelConfig,
     BaseModelType,
     ModelFormat,
     ModelType,
 )
+from invokeai.backend.model_manager.any_model_type import AnyModel
 from invokeai.backend.model_manager.config import CheckpointConfigBase
 from invokeai.backend.model_manager.convert_ckpt_to_diffusers import convert_controlnet_to_diffusers
 

invokeai/backend/model_manager/load/model_loaders/generic_diffusers.py

@@ -9,7 +9,6 @@ from diffusers.configuration_utils import ConfigMixin
 from diffusers.models.modeling_utils import ModelMixin
 
 from invokeai.backend.model_manager import (
-    AnyModel,
     AnyModelConfig,
     BaseModelType,
     InvalidModelConfigException,
@@ -17,6 +16,7 @@ from invokeai.backend.model_manager import (
     ModelType,
     SubModelType,
 )
+from invokeai.backend.model_manager.any_model_type import AnyModel
 from invokeai.backend.model_manager.config import DiffusersConfigBase
 
 from .. import ModelLoader, ModelLoaderRegistry

invokeai/backend/model_manager/load/model_loaders/ip_adapter.py

@@ -7,9 +7,9 @@ from typing import Optional
 import torch
 
 from invokeai.backend.ip_adapter.ip_adapter import build_ip_adapter
-from invokeai.backend.model_manager import AnyModel, AnyModelConfig, BaseModelType, ModelFormat, ModelType, SubModelType
+from invokeai.backend.model_manager import AnyModelConfig, BaseModelType, ModelFormat, ModelType, SubModelType
+from invokeai.backend.model_manager.any_model_type import AnyModel
 from invokeai.backend.model_manager.load import ModelLoader, ModelLoaderRegistry
-from invokeai.backend.raw_model import RawModel
 
 
 @ModelLoaderRegistry.register(base=BaseModelType.Any, type=ModelType.IPAdapter, format=ModelFormat.InvokeAI)
@@ -25,7 +25,7 @@ class IPAdapterInvokeAILoader(ModelLoader):
         if submodel_type is not None:
             raise ValueError("There are no submodels in an IP-Adapter model.")
         model_path = Path(config.path)
-        model: RawModel = build_ip_adapter(
+        model = build_ip_adapter(
             ip_adapter_ckpt_path=model_path,
             device=torch.device("cpu"),
             dtype=self._torch_dtype,

invokeai/backend/model_manager/load/model_loaders/lora.py

@@ -6,17 +6,17 @@ from pathlib import Path
 from typing import Optional
 
 from invokeai.app.services.config import InvokeAIAppConfig
-from invokeai.backend.lora import LoRAModelRaw
 from invokeai.backend.model_manager import (
-    AnyModel,
     AnyModelConfig,
     BaseModelType,
     ModelFormat,
     ModelType,
     SubModelType,
 )
+from invokeai.backend.model_manager.any_model_type import AnyModel
 from invokeai.backend.model_manager.load.convert_cache import ModelConvertCacheBase
 from invokeai.backend.model_manager.load.model_cache.model_cache_base import ModelCacheBase
+from invokeai.backend.peft.peft_model import PeftModel
 
 from .. import ModelLoader, ModelLoaderRegistry
 
@@ -47,7 +47,7 @@ class LoRALoader(ModelLoader):
             raise ValueError("There are no submodels in a LoRA model.")
         model_path = Path(config.path)
         assert self._model_base is not None
-        model = LoRAModelRaw.from_checkpoint(
+        model = PeftModel.from_checkpoint(
             file_path=model_path,
             dtype=self._torch_dtype,
             base_model=self._model_base,