Remove unused check_internet() function.

- Use the our adaptation of the HWC3 function with better types
- Extraction some of the util functions, name them better, add comments
- Improve type annotations
- Remove unreachable codepaths

README.md

@@ -2,21 +2,102 @@
 
 ![project hero](https://github.com/invoke-ai/InvokeAI/assets/31807370/6e3728c7-e90e-4711-905c-3b55844ff5be)
 
-# Invoke - Professional Creative AI Tools for Visual Media 
-##  To learn more about Invoke, or implement our Business solutions, visit [invoke.com](https://www.invoke.com/about)
-  
+# Invoke - Professional Creative AI Tools for Visual Media
 
+#### To learn more about Invoke, or implement our Business solutions, visit [invoke.com]
 
-[![discord badge]][discord link]
+[![discord badge]][discord link] [![latest release badge]][latest release link] [![github stars badge]][github stars link] [![github forks badge]][github forks link] [![CI checks on main badge]][CI checks on main link] [![latest commit to main badge]][latest commit to main link] [![github open issues badge]][github open issues link] [![github open prs badge]][github open prs link] [![translation status badge]][translation status link]
 
-[![latest release badge]][latest release link] [![github stars badge]][github stars link] [![github forks badge]][github forks link]
+</div>
 
-[![CI checks on main badge]][CI checks on main link] [![latest commit to main badge]][latest commit to main link]
+Invoke is a leading creative engine built to empower professionals and enthusiasts alike. Generate and create stunning visual media using the latest AI-driven technologies. Invoke offers an industry leading web-based UI, and serves as the foundation for multiple commercial products.
 
-[![github open issues badge]][github open issues link] [![github open prs badge]][github open prs link] [![translation status badge]][translation status link]
+[Installation][installation docs] - [Documentation and Tutorials][docs home] - [Bug Reports][github issues] - [Contributing][contributing docs]
 
+<div align="center">
+
+![Highlighted Features - Canvas and Workflows](https://github.com/invoke-ai/InvokeAI/assets/31807370/708f7a82-084f-4860-bfbe-e2588c53548d)
+
+</div>
+
+## Quick Start
+
+1. Download and unzip the installer from the bottom of the [latest release][latest release link].
+2. Run the installer script.
+
+   - **Windows**: Double-click on the `install.bat` script.
+   - **macOS**: Open a Terminal window, drag the file `install.sh` from Finder into the Terminal, and press enter.
+   - **Linux**: Run `install.sh`.
+
+3. When prompted, enter a location for the install and select your GPU type.
+4. Once the install finishes, find the directory you selected during install. The default location is `C:\Users\Username\invokeai` for Windows or `~/invokeai` for Linux/macOS.
+5. Run the launcher script (`invoke.bat` for Windows, `invoke.sh` for macOS and Linux) the same way you ran the installer script in step 2.
+6. Select option 1 to start the application. Once it starts up, open your browser and go to <http://localhost:9090>.
+7. Open the model manager tab to install a starter model and then you'll be ready to generate.
+
+More detail, including hardware requirements and manual install instructions, are available in the [installation documentation][installation docs].
+
+## Troubleshooting, FAQ and Support
+
+Please review our [FAQ][faq] for solutions to common installation problems and other issues.
+
+For more help, please join our [Discord][discord link].
+
+## Features
+
+Full details on features can be found in [our documentation][features docs].
+
+### Web Server & UI
+
+Invoke runs a locally hosted web server & React UI with an industry-leading user experience.
+
+### Unified Canvas
+
+The Unified Canvas is a fully integrated canvas implementation with support for all core generation capabilities, in/out-painting, brush tools, and more. This creative tool unlocks the capability for artists to create with AI as a creative collaborator, and can be used to augment AI-generated imagery, sketches, photography, renders, and more.
+
+### Workflows & Nodes
+
+Invoke offers a fully featured workflow management solution, enabling users to combine the power of node-based workflows with the easy of a UI. This allows for customizable generation pipelines to be developed and shared by users looking to create specific workflows to support their production use-cases.
+
+### Board & Gallery Management
+
+Invoke features an organized gallery system for easily storing, accessing, and remixing your content in the Invoke workspace. Images can be dragged/dropped onto any Image-base UI element in the application, and rich metadata within the Image allows for easy recall of key prompts or settings used in your workflow.
+
+### Other features
+
+- Support for both ckpt and diffusers models
+- SD1.5, SD2.0, and SDXL support
+- Upscaling Tools
+- Embedding Manager & Support
+- Model Manager & Support
+- Workflow creation & management
+- Node-Based Architecture
+
+## Contributing
+
+Anyone who wishes to contribute to this project - whether documentation, features, bug fixes, code cleanup, testing, or code reviews - is very much encouraged to do so.
+
+Get started with contributing by reading our [contribution documentation][contributing docs], joining the [#dev-chat] or the GitHub discussion board.
+
+We hope you enjoy using Invoke as much as we enjoy creating it, and we hope you will elect to become part of our community.
+
+## Thanks
+
+Invoke is a combined effort of [passionate and talented people from across the world][contributors]. We thank them for their time, hard work and effort.
+
+Original portions of the software are Copyright © 2024 by respective contributors.
+
+[features docs]: https://invoke-ai.github.io/InvokeAI/features/
+[faq]: https://invoke-ai.github.io/InvokeAI/help/FAQ/
+[contributors]: https://invoke-ai.github.io/InvokeAI/other/CONTRIBUTORS/
+[invoke.com]: https://www.invoke.com/about
+[github issues]: https://github.com/invoke-ai/InvokeAI/issues
+[docs home]: https://invoke-ai.github.io/InvokeAI
+[installation docs]: https://invoke-ai.github.io/InvokeAI/installation/INSTALLATION/
+[#dev-chat]: https://discord.com/channels/1020123559063990373/1049495067846524939
+[contributing docs]: https://invoke-ai.github.io/InvokeAI/contributing/CONTRIBUTING/
 [CI checks on main badge]: https://flat.badgen.net/github/checks/invoke-ai/InvokeAI/main?label=CI%20status%20on%20main&cache=900&icon=github
-[CI checks on main link]:https://github.com/invoke-ai/InvokeAI/actions?query=branch%3Amain
+[CI checks on main link]: https://github.com/invoke-ai/InvokeAI/actions?query=branch%3Amain
 [discord badge]: https://flat.badgen.net/discord/members/ZmtBAhwWhy?icon=discord
 [discord link]: https://discord.gg/ZmtBAhwWhy
 [github forks badge]: https://flat.badgen.net/github/forks/invoke-ai/InvokeAI?icon=github
@@ -30,402 +111,6 @@
 [latest commit to main badge]: https://flat.badgen.net/github/last-commit/invoke-ai/InvokeAI/main?icon=github&color=yellow&label=last%20dev%20commit&cache=900
 [latest commit to main link]: https://github.com/invoke-ai/InvokeAI/commits/main
 [latest release badge]: https://flat.badgen.net/github/release/invoke-ai/InvokeAI/development?icon=github
-[latest release link]: https://github.com/invoke-ai/InvokeAI/releases
+[latest release link]: https://github.com/invoke-ai/InvokeAI/releases/latest
 [translation status badge]: https://hosted.weblate.org/widgets/invokeai/-/svg-badge.svg
 [translation status link]: https://hosted.weblate.org/engage/invokeai/
-
-</div>
-
-InvokeAI is a leading creative engine built to empower professionals
-and enthusiasts alike. Generate and create stunning visual media using
-the latest AI-driven technologies. InvokeAI offers an industry leading
-Web Interface, interactive Command Line Interface, and also serves as
-the foundation for multiple commercial products.
-
-**Quick links**: [[How to
-  Install](https://invoke-ai.github.io/InvokeAI/installation/INSTALLATION/)] [<a
-  href="https://discord.gg/ZmtBAhwWhy">Discord Server</a>] [<a
-  href="https://invoke-ai.github.io/InvokeAI/">Documentation and
-  Tutorials</a>]
-  [<a href="https://github.com/invoke-ai/InvokeAI/issues">Bug Reports</a>]
-  [<a
-  href="https://github.com/invoke-ai/InvokeAI/discussions">Discussion,
-  Ideas & Q&A</a>] 
-   [<a
-  href="https://invoke-ai.github.io/InvokeAI/contributing/CONTRIBUTING/">Contributing</a>] 
-
-<div align="center">
-
-
-![Highlighted Features - Canvas and Workflows](https://github.com/invoke-ai/InvokeAI/assets/31807370/708f7a82-084f-4860-bfbe-e2588c53548d)
-
-
-</div>
-
-## Table of Contents
-
-Table of Contents 📝
-
-**Getting Started**
-1. 🏁 [Quick Start](#quick-start) 
-3. 🖥️ [Hardware Requirements](#hardware-requirements) 
-
-**More About Invoke**
-1. 🌟 [Features](#features) 
-2. 📣 [Latest Changes](#latest-changes) 
-3. 🛠️ [Troubleshooting](#troubleshooting) 
-
-**Supporting the Project**
-1. 🤝 [Contributing](#contributing) 
-2. 👥 [Contributors](#contributors) 
-3. 💕 [Support](#support) 
-
-## Quick Start
-
-For full installation and upgrade instructions, please see:
-[InvokeAI Installation Overview](https://invoke-ai.github.io/InvokeAI/installation/INSTALLATION/)
-
-If upgrading from version 2.3, please read [Migrating a 2.3 root
-directory to 3.0](#migrating-to-3) first.
-
-### Automatic Installer (suggested for 1st time users)
-
-1. Go to the bottom of the [Latest Release Page](https://github.com/invoke-ai/InvokeAI/releases/latest)
-
-2. Download the .zip file for your OS (Windows/macOS/Linux).
-
-3. Unzip the file.
-
-4. **Windows:** double-click on the `install.bat` script. **macOS:** Open a Terminal window, drag the file `install.sh` from Finder
-into the Terminal, and press return. **Linux:** run `install.sh`.
-
-5. You'll be asked to confirm the location of the folder in which
-to install InvokeAI and its image generation model files. Pick a
-location with at least 15 GB of free memory. More if you plan on
-installing lots of models.
-
-6. Wait while the installer does its thing. After installing the software,
-the installer will launch a script that lets you configure InvokeAI and
-select a set of starting image generation models.
-
-7. Find the folder that InvokeAI was installed into (it is not the
-same as the unpacked zip file directory!) The default location of this
-folder (if you didn't change it in step 5) is `~/invokeai` on
-Linux/Mac systems, and `C:\Users\YourName\invokeai` on Windows. This directory will contain launcher scripts named `invoke.sh` and `invoke.bat`.
-
-8. On Windows systems, double-click on the `invoke.bat` file. On
-macOS, open a Terminal window, drag `invoke.sh` from the folder into
-the Terminal, and press return. On Linux, run `invoke.sh`
-
-9. Press 2 to open the "browser-based UI", press enter/return, wait a
-minute or two for Stable Diffusion to start up, then open your browser
-and go to http://localhost:9090.
-
-10. Type `banana sushi` in the box on the top left and click `Invoke`
-
-### Command-Line Installation (for developers and users familiar with Terminals)
-
-You must have Python 3.10 through 3.11 installed on your machine. Earlier or
-later versions are not supported.
-Node.js also needs to be installed along with `pnpm` (can be installed with
-the command `npm install -g pnpm` if needed)
-
-1. Open a command-line window on your machine. The PowerShell is recommended for Windows.
-2. Create a directory to install InvokeAI into. You'll need at least 15 GB of free space:
-
-    ```terminal
-    mkdir invokeai
-    ````
-
-3. Create a virtual environment named `.venv` inside this directory and activate it:
-
-    ```terminal
-    cd invokeai
-    python -m venv .venv --prompt InvokeAI
-    ```
-
-4. Activate the virtual environment (do it every time you run InvokeAI)
-
-    _For Linux/Mac users:_
-
-    ```sh
-    source .venv/bin/activate
-    ```
-
-    _For Windows users:_
-
-    ```ps
-    .venv\Scripts\activate
-    ```
-
-5. Install the InvokeAI module and its dependencies. Choose the command suited for your platform & GPU.
-
-    _For Windows/Linux with an NVIDIA GPU:_
-
-    ```terminal
-    pip install "InvokeAI[xformers]" --use-pep517 --extra-index-url https://download.pytorch.org/whl/cu121
-    ```
-
-    _For Linux with an AMD GPU:_
-
-    ```sh
-    pip install InvokeAI --use-pep517 --extra-index-url https://download.pytorch.org/whl/rocm5.6
-    ```
-
-    _For non-GPU systems:_
-    ```terminal
-    pip install InvokeAI --use-pep517 --extra-index-url https://download.pytorch.org/whl/cpu
-    ``` 
-
-    _For Macintoshes, either Intel or M1/M2/M3:_
-
-    ```sh
-    pip install InvokeAI --use-pep517
-    ```
-
-6. Configure InvokeAI and install a starting set of image generation models (you only need to do this once):
-
-    ```terminal
-    invokeai-configure --root .
-    ```
-	Don't miss the dot at the end!
-
-7. Launch the web server (do it every time you run InvokeAI):
-
-    ```terminal
-    invokeai-web
-    ```
-
-8. Point your browser to http://localhost:9090 to bring up the web interface.
-
-9. Type `banana sushi` in the box on the top left and click `Invoke`.
-
-Be sure to activate the virtual environment each time before re-launching InvokeAI,
-using `source .venv/bin/activate` or `.venv\Scripts\activate`.
-
-## Detailed Installation Instructions
-
-This fork is supported across Linux, Windows and Macintosh. Linux
-users can use either an Nvidia-based card (with CUDA support) or an
-AMD card (using the ROCm driver). For full installation and upgrade
-instructions, please see:
-[InvokeAI Installation Overview](https://invoke-ai.github.io/InvokeAI/installation/INSTALL_SOURCE/)
-
-<a name="migrating-to-3"></a>
-### Migrating a v2.3 InvokeAI root directory
-
-The InvokeAI root directory is where the InvokeAI startup file,
-installed models, and generated images are stored. It is ordinarily
-named `invokeai` and located in your home directory. The contents and
-layout of this directory has changed between versions 2.3 and 3.0 and
-cannot be used directly.
-
-We currently recommend that you use the installer to create a new root
-directory named differently from the 2.3 one, e.g. `invokeai-3` and
-then use a migration script to copy your 2.3 models into the new
-location. However, if you choose, you can upgrade this directory in
-place.  This section gives both recipes.
-
-#### Creating a new root directory and migrating old models
-
-This is the safer recipe because it leaves your old root directory in
-place to fall back on.
-
-1. Follow the instructions above to create and install InvokeAI in a
-directory that has a different name from the 2.3 invokeai directory.
-In this example, we will use "invokeai-3"
-
-2. When you are prompted to select models to install, select a minimal
-set of models, such as stable-diffusion-v1.5 only.
-
-3. After installation is complete launch `invokeai.sh` (Linux/Mac) or
-`invokeai.bat` and select option 8 "Open the developers console". This
-will take you to the command line.
-
-4. Issue the command `invokeai-migrate3 --from /path/to/v2.3-root --to
-/path/to/invokeai-3-root`. Provide the correct `--from` and `--to`
-paths for your v2.3 and v3.0 root directories respectively.
-
-This will copy and convert your old models from 2.3 format to 3.0
-format and create a new `models` directory in the 3.0 directory. The
-old models directory (which contains the models selected at install
-time) will be renamed `models.orig` and can be deleted once you have
-confirmed that the migration was successful.
-
- If you wish, you can pass the 2.3 root directory to both `--from` and
-`--to` in order to update in place. Warning: this directory will no
-longer be usable with InvokeAI 2.3.
-
-#### Migrating in place
-
-For the adventurous, you may do an in-place upgrade from 2.3 to 3.0
-without touching the command line. ***This recipe does not work on
-Windows platforms due to a bug in the Windows version of the 2.3
-upgrade script.** See the next section for a Windows recipe.
-
-##### For Mac and Linux Users:
-
-1. Launch the InvokeAI launcher script in your current v2.3 root directory.
-
-2. Select option [9] "Update InvokeAI" to bring up the updater dialog.
-
-3. Select option [1] to upgrade to the latest release.
-
-4. Once the upgrade is finished you will be returned to the launcher
-menu. Select option [6] "Re-run the configure script to fix a broken
-install or to complete a major upgrade".
-
-This will run the configure script against the v2.3 directory and
-update it to the 3.0 format. The following files will be replaced:
-
-  - The invokeai.init file, replaced by invokeai.yaml
-  - The models directory
-  - The configs/models.yaml model index
-  
-The original versions of these files will be saved with the suffix
-".orig" appended to the end. Once you have confirmed that the upgrade
-worked, you can safely remove these files. Alternatively you can
-restore a working v2.3 directory by removing the new files and
-restoring the ".orig" files' original names.
-
-##### For Windows Users:
-
-Windows Users can upgrade with the
-
-1. Enter the 2.3 root directory you wish to upgrade
-2. Launch `invoke.sh` or `invoke.bat`
-3. Select the "Developer's console" option [8]
-4. Type the following commands
-
-```
-pip install "invokeai @ https://github.com/invoke-ai/InvokeAI/archive/refs/tags/v3.0.0" --use-pep517 --upgrade
-invokeai-configure --root .
-```
-(Replace `v3.0.0` with the current release number if this document is out of date).
-
-The first command will install and upgrade new software to run
-InvokeAI. The second will prepare the 2.3 directory for use with 3.0.
-You may now launch the WebUI in the usual way, by selecting option [1]
-from the launcher script
-
-#### Migrating Images
-
-The migration script will migrate your invokeai settings and models,
-including textual inversion models, LoRAs and merges that you may have
-installed previously. However it does **not** migrate the generated
-images stored in your 2.3-format outputs directory. To do this, you 
-need to run an additional step:
-
-1. From a working InvokeAI 3.0 root directory, start the launcher and
-enter menu option [8] to open the "developer's console".
-
-2. At the developer's console command line, type the command:
-
-```bash
-invokeai-import-images
-```
-
-3. This will lead you through the process of confirming the desired
-   source and destination for the imported images. The images will
-   appear in the gallery board of your choice, and contain the
-   original prompt, model name, and other parameters used to generate
-   the image.
-   
-(Many kudos to **techjedi** for contributing this script.)
-
-## Hardware Requirements
-
-InvokeAI is supported across Linux, Windows and macOS. Linux
-users can use either an Nvidia-based card (with CUDA support) or an
-AMD card (using the ROCm driver).
-
-### System
-
-You will need one of the following:
-
-- An NVIDIA-based graphics card with 4 GB or more VRAM memory. 6-8 GB
-  of VRAM is highly recommended for rendering using the Stable
-  Diffusion XL models
-- An Apple computer with an M1 chip.
-- An AMD-based graphics card with 4GB or more VRAM memory (Linux
-  only), 6-8 GB for XL rendering.
-
-We do not recommend the GTX 1650 or 1660 series video cards. They are
-unable to run in half-precision mode and do not have sufficient VRAM
-to render 512x512 images.
-
-**Memory** - At least 12 GB Main Memory RAM.
-
-**Disk** - At least 12 GB of free disk space for the machine learning model, Python, and all its dependencies.
-
-## Features
-
-Feature documentation can be reviewed by navigating to [the InvokeAI Documentation page](https://invoke-ai.github.io/InvokeAI/features/)
-
-### *Web Server & UI*
-
-InvokeAI offers a locally hosted Web Server & React Frontend, with an industry leading user experience. The Web-based UI allows for simple and intuitive workflows, and is responsive for use on mobile devices and tablets accessing the web server.
-
-### *Unified Canvas*
-
-The Unified Canvas is a fully integrated canvas implementation with support for all core generation capabilities, in/outpainting, brush tools, and more. This creative tool unlocks the capability for artists to create with AI as a creative collaborator, and can be used to augment AI-generated imagery, sketches, photography, renders, and more.
-
-### *Workflows & Nodes*
-
-InvokeAI offers a fully featured workflow management solution, enabling users to combine the power of nodes based workflows with the easy of a UI. This allows for customizable generation pipelines to be developed and shared by users looking to create specific workflows to support their production use-cases.
-
-### *Board & Gallery Management*
-
-Invoke AI provides an organized gallery system for easily storing, accessing, and remixing your content in the Invoke workspace. Images can be dragged/dropped onto any Image-base UI element in the application, and rich metadata within the Image allows for easy recall of key prompts or settings used in your workflow. 
-
-### Other features
-
-- *Support for both ckpt and diffusers models*
-- *SD 2.0, 2.1, XL support*
-- *Upscaling Tools*
-- *Embedding Manager & Support*
-- *Model Manager & Support*
-- *Workflow creation & management*
-- *Node-Based Architecture*
-
-
-### Latest Changes
-
-For our latest changes, view our [Release
-Notes](https://github.com/invoke-ai/InvokeAI/releases) and the
-[CHANGELOG](docs/CHANGELOG.md).
-
-### Troubleshooting / FAQ
-
-Please check out our **[FAQ](https://invoke-ai.github.io/InvokeAI/help/FAQ/)** to get solutions for common installation
-problems and other issues. For more help, please join our [Discord][discord link]
-
-## Contributing
-
-Anyone who wishes to contribute to this project, whether documentation, features, bug fixes, code
-cleanup, testing, or code reviews, is very much encouraged to do so.
-
-Get started with contributing by reading our [Contribution documentation](https://invoke-ai.github.io/InvokeAI/contributing/CONTRIBUTING/), joining the [#dev-chat](https://discord.com/channels/1020123559063990373/1049495067846524939) or the GitHub discussion board.
-
-If you are unfamiliar with how
-to contribute to GitHub projects, we have a new contributor checklist you can follow to get started contributing: 
-[New Contributor Checklist](https://invoke-ai.github.io/InvokeAI/contributing/contribution_guides/newContributorChecklist/).
-
-We hope you enjoy using our software as much as we enjoy creating it,
-and we hope that some of those of you who are reading this will elect
-to become part of our community.
-
-Welcome to InvokeAI!
-
-### Contributors
-
-This fork is a combined effort of various people from across the world.
-[Check out the list of all these amazing people](https://invoke-ai.github.io/InvokeAI/other/CONTRIBUTORS/). We thank them for
-their time, hard work and effort.
-
-### Support
-
-For support, please use this repository's GitHub Issues tracking service, or join the [Discord][discord link].
-
-Original portions of the software are Copyright (c) 2023 by respective contributors.
-

invokeai/app/api/dependencies.py

@@ -35,23 +35,6 @@ from ..services.urls.urls_default import LocalUrlService
 from ..services.workflow_records.workflow_records_sqlite import SqliteWorkflowRecordsStorage
 from .events import FastAPIEventService
 
-
-# TODO: is there a better way to achieve this?
-def check_internet() -> bool:
-    """
-    Return true if the internet is reachable.
-    It does this by pinging huggingface.co.
-    """
-    import urllib.request
-
-    host = "http://huggingface.co"
-    try:
-        urllib.request.urlopen(host, timeout=1)
-        return True
-    except Exception:
-        return False
-
-
 logger = InvokeAILogger.get_logger()
 
 

invokeai/app/api_app.py

@@ -2,6 +2,7 @@ import asyncio
 import logging
 import mimetypes
 import socket
+import time
 from contextlib import asynccontextmanager
 from inspect import signature
 from pathlib import Path
@@ -20,13 +21,10 @@ from fastapi_events.middleware import EventHandlerASGIMiddleware
 from pydantic.json_schema import models_json_schema
 from torch.backends.mps import is_available as is_mps_available
 
-# for PyCharm:
-# noinspection PyUnresolvedReferences
-import invokeai.backend.util.hotfixes  # noqa: F401 (monkeypatching on import)
 import invokeai.frontend.web as web_dir
 from invokeai.app.api.no_cache_staticfiles import NoCacheStaticFiles
 from invokeai.app.invocations.model import ModelIdentifierField
-from invokeai.app.services.config.config_default import get_config
+from invokeai.app.services.config.config_default import InvokeAIAppConfig, get_config
 from invokeai.app.services.session_processor.session_processor_common import ProgressImage
 from invokeai.backend.util.devices import TorchDevice
 
@@ -44,188 +42,189 @@ from .api.routers import (
     workflows,
 )
 from .api.sockets import SocketIO
-from .invocations.baseinvocation import (
-    BaseInvocation,
-    UIConfigBase,
-)
+from .invocations.baseinvocation import BaseInvocation, UIConfigBase
 from .invocations.fields import InputFieldJSONSchemaExtra, OutputFieldJSONSchemaExtra
 
-app_config = get_config()
+
+# TODO(ryand): Search for imports from api_app.py in the rest of the codebase and make sure I didn't break any of them.
+def build_app(app_config: InvokeAIAppConfig, logger: logging.Logger) -> FastAPI:
+    @asynccontextmanager
+    async def lifespan(app: FastAPI):
+        # Add startup event to load dependencies
+        ApiDependencies.initialize(config=app_config, event_handler_id=event_handler_id, logger=logger)
+        yield
+        # Shut down threads
+        ApiDependencies.shutdown()
+
+    app = FastAPI(
+        title="Invoke - Community Edition",
+        docs_url=None,
+        redoc_url=None,
+        separate_input_output_schemas=False,
+        lifespan=lifespan,
+    )
+
+    # Add event handler
+    event_handler_id: int = id(app)
+    app.add_middleware(
+        EventHandlerASGIMiddleware,
+        handlers=[local_handler],  # TODO: consider doing this in services to support different configurations
+        middleware_id=event_handler_id,
+    )
+
+    socket_io = SocketIO(app)
+
+    app.add_middleware(
+        CORSMiddleware,
+        allow_origins=app_config.allow_origins,
+        allow_credentials=app_config.allow_credentials,
+        allow_methods=app_config.allow_methods,
+        allow_headers=app_config.allow_headers,
+    )
+
+    app.add_middleware(GZipMiddleware, minimum_size=1000)
+
+    # Include all routers
+    app.include_router(utilities.utilities_router, prefix="/api")
+    app.include_router(model_manager.model_manager_router, prefix="/api")
+    app.include_router(download_queue.download_queue_router, prefix="/api")
+    app.include_router(images.images_router, prefix="/api")
+    app.include_router(boards.boards_router, prefix="/api")
+    app.include_router(board_images.board_images_router, prefix="/api")
+    app.include_router(app_info.app_router, prefix="/api")
+    app.include_router(session_queue.session_queue_router, prefix="/api")
+    app.include_router(workflows.workflows_router, prefix="/api")
+
+    add_custom_openapi(app)
+
+    @app.get("/docs", include_in_schema=False)
+    def overridden_swagger() -> HTMLResponse:
+        return get_swagger_ui_html(
+            openapi_url=app.openapi_url,  # type: ignore [arg-type] # this is always a string
+            title=f"{app.title} - Swagger UI",
+            swagger_favicon_url="static/docs/invoke-favicon-docs.svg",
+        )
+
+    @app.get("/redoc", include_in_schema=False)
+    def overridden_redoc() -> HTMLResponse:
+        return get_redoc_html(
+            openapi_url=app.openapi_url,  # type: ignore [arg-type] # this is always a string
+            title=f"{app.title} - Redoc",
+            redoc_favicon_url="static/docs/invoke-favicon-docs.svg",
+        )
+
+    web_root_path = Path(list(web_dir.__path__)[0])
+
+    try:
+        app.mount("/", NoCacheStaticFiles(directory=Path(web_root_path, "dist"), html=True), name="ui")
+    except RuntimeError:
+        logger.warn(f"No UI found at {web_root_path}/dist, skipping UI mount")
+
+    app.mount(
+        "/static", NoCacheStaticFiles(directory=Path(web_root_path, "static/")), name="static"
+    )  # docs favicon is in here
+
+    return app
 
 
-if is_mps_available():
-    import invokeai.backend.util.mps_fixes  # noqa: F401 (monkeypatching on import)
+def apply_monkeypatches() -> None:
+    # TODO(ryand): Don't monkeypatch on import!
+    if is_mps_available():
+        import invokeai.backend.util.mps_fixes  # noqa: F401 (monkeypatching on import)
+    import invokeai.backend.util.hotfixes  # noqa: F401 (monkeypatching on import)
 
 
-logger = InvokeAILogger.get_logger(config=app_config)
-# fix for windows mimetypes registry entries being borked
-# see https://github.com/invoke-ai/InvokeAI/discussions/3684#discussioncomment-6391352
-mimetypes.add_type("application/javascript", ".js")
-mimetypes.add_type("text/css", ".css")
-
-torch_device_name = TorchDevice.get_torch_device_name()
-logger.info(f"Using torch device: {torch_device_name}")
+def fix_mimetypes():
+    # fix for windows mimetypes registry entries being borked
+    # see https://github.com/invoke-ai/InvokeAI/discussions/3684#discussioncomment-6391352
+    mimetypes.add_type("application/javascript", ".js")
+    mimetypes.add_type("text/css", ".css")
 
 
-@asynccontextmanager
-async def lifespan(app: FastAPI):
-    # Add startup event to load dependencies
-    ApiDependencies.initialize(config=app_config, event_handler_id=event_handler_id, logger=logger)
-    yield
-    # Shut down threads
-    ApiDependencies.shutdown()
+def add_custom_openapi(app: FastAPI) -> None:
+    """Add a custom .openapi() method to the FastAPI app.
 
+    This is done based on the guidance here:
+    https://fastapi.tiangolo.com/how-to/extending-openapi/#normal-fastapi
+    """
 
-# Create the app
-# TODO: create this all in a method so configuration/etc. can be passed in?
-app = FastAPI(
-    title="Invoke - Community Edition",
-    docs_url=None,
-    redoc_url=None,
-    separate_input_output_schemas=False,
-    lifespan=lifespan,
-)
+    # Build a custom OpenAPI to include all outputs
+    # TODO: can outputs be included on metadata of invocation schemas somehow?
+    def custom_openapi() -> dict[str, Any]:
+        if app.openapi_schema:
+            return app.openapi_schema
+        openapi_schema = get_openapi(
+            title=app.title,
+            description="An API for invoking AI image operations",
+            version="1.0.0",
+            routes=app.routes,
+            separate_input_output_schemas=False,  # https://fastapi.tiangolo.com/how-to/separate-openapi-schemas/
+        )
 
-# Add event handler
-event_handler_id: int = id(app)
-app.add_middleware(
-    EventHandlerASGIMiddleware,
-    handlers=[local_handler],  # TODO: consider doing this in services to support different configurations
-    middleware_id=event_handler_id,
-)
+        # Add all outputs
+        all_invocations = BaseInvocation.get_invocations()
+        output_types = set()
+        output_type_titles = {}
+        for invoker in all_invocations:
+            output_type = signature(invoker.invoke).return_annotation
+            output_types.add(output_type)
 
-socket_io = SocketIO(app)
+        output_schemas = models_json_schema(
+            models=[(o, "serialization") for o in output_types], ref_template="#/components/schemas/{model}"
+        )
+        for schema_key, output_schema in output_schemas[1]["$defs"].items():
+            # TODO: note that we assume the schema_key here is the TYPE.__name__
+            # This could break in some cases, figure out a better way to do it
+            output_type_titles[schema_key] = output_schema["title"]
+            openapi_schema["components"]["schemas"][schema_key] = output_schema
+            openapi_schema["components"]["schemas"][schema_key]["class"] = "output"
 
-app.add_middleware(
-    CORSMiddleware,
-    allow_origins=app_config.allow_origins,
-    allow_credentials=app_config.allow_credentials,
-    allow_methods=app_config.allow_methods,
-    allow_headers=app_config.allow_headers,
-)
+        # Some models don't end up in the schemas as standalone definitions
+        additional_schemas = models_json_schema(
+            [
+                (UIConfigBase, "serialization"),
+                (InputFieldJSONSchemaExtra, "serialization"),
+                (OutputFieldJSONSchemaExtra, "serialization"),
+                (ModelIdentifierField, "serialization"),
+                (ProgressImage, "serialization"),
+            ],
+            ref_template="#/components/schemas/{model}",
+        )
+        for schema_key, schema_json in additional_schemas[1]["$defs"].items():
+            openapi_schema["components"]["schemas"][schema_key] = schema_json
 
-app.add_middleware(GZipMiddleware, minimum_size=1000)
+        # Add a reference to the output type to additionalProperties of the invoker schema
+        for invoker in all_invocations:
+            invoker_name = invoker.__name__  # type: ignore [attr-defined] # this is a valid attribute
+            output_type = signature(obj=invoker.invoke).return_annotation
+            output_type_title = output_type_titles[output_type.__name__]
+            invoker_schema = openapi_schema["components"]["schemas"][f"{invoker_name}"]
+            outputs_ref = {"$ref": f"#/components/schemas/{output_type_title}"}
+            invoker_schema["output"] = outputs_ref
+            invoker_schema["class"] = "invocation"
 
+        # This code no longer seems to be necessary?
+        # Leave it here just in case
+        #
+        # from invokeai.backend.model_manager import get_model_config_formats
+        # formats = get_model_config_formats()
+        # for model_config_name, enum_set in formats.items():
 
-# Include all routers
-app.include_router(utilities.utilities_router, prefix="/api")
-app.include_router(model_manager.model_manager_router, prefix="/api")
-app.include_router(download_queue.download_queue_router, prefix="/api")
-app.include_router(images.images_router, prefix="/api")
-app.include_router(boards.boards_router, prefix="/api")
-app.include_router(board_images.board_images_router, prefix="/api")
-app.include_router(app_info.app_router, prefix="/api")
-app.include_router(session_queue.session_queue_router, prefix="/api")
-app.include_router(workflows.workflows_router, prefix="/api")
+        #     if model_config_name in openapi_schema["components"]["schemas"]:
+        #         # print(f"Config with name {name} already defined")
+        #         continue
 
+        #     openapi_schema["components"]["schemas"][model_config_name] = {
+        #         "title": model_config_name,
+        #         "description": "An enumeration.",
+        #         "type": "string",
+        #         "enum": [v.value for v in enum_set],
+        #     }
 
-# Build a custom OpenAPI to include all outputs
-# TODO: can outputs be included on metadata of invocation schemas somehow?
-def custom_openapi() -> dict[str, Any]:
-    if app.openapi_schema:
+        app.openapi_schema = openapi_schema
         return app.openapi_schema
-    openapi_schema = get_openapi(
-        title=app.title,
-        description="An API for invoking AI image operations",
-        version="1.0.0",
-        routes=app.routes,
-        separate_input_output_schemas=False,  # https://fastapi.tiangolo.com/how-to/separate-openapi-schemas/
-    )
 
-    # Add all outputs
-    all_invocations = BaseInvocation.get_invocations()
-    output_types = set()
-    output_type_titles = {}
-    for invoker in all_invocations:
-        output_type = signature(invoker.invoke).return_annotation
-        output_types.add(output_type)
-
-    output_schemas = models_json_schema(
-        models=[(o, "serialization") for o in output_types], ref_template="#/components/schemas/{model}"
-    )
-    for schema_key, output_schema in output_schemas[1]["$defs"].items():
-        # TODO: note that we assume the schema_key here is the TYPE.__name__
-        # This could break in some cases, figure out a better way to do it
-        output_type_titles[schema_key] = output_schema["title"]
-        openapi_schema["components"]["schemas"][schema_key] = output_schema
-        openapi_schema["components"]["schemas"][schema_key]["class"] = "output"
-
-    # Some models don't end up in the schemas as standalone definitions
-    additional_schemas = models_json_schema(
-        [
-            (UIConfigBase, "serialization"),
-            (InputFieldJSONSchemaExtra, "serialization"),
-            (OutputFieldJSONSchemaExtra, "serialization"),
-            (ModelIdentifierField, "serialization"),
-            (ProgressImage, "serialization"),
-        ],
-        ref_template="#/components/schemas/{model}",
-    )
-    for schema_key, schema_json in additional_schemas[1]["$defs"].items():
-        openapi_schema["components"]["schemas"][schema_key] = schema_json
-
-    # Add a reference to the output type to additionalProperties of the invoker schema
-    for invoker in all_invocations:
-        invoker_name = invoker.__name__  # type: ignore [attr-defined] # this is a valid attribute
-        output_type = signature(obj=invoker.invoke).return_annotation
-        output_type_title = output_type_titles[output_type.__name__]
-        invoker_schema = openapi_schema["components"]["schemas"][f"{invoker_name}"]
-        outputs_ref = {"$ref": f"#/components/schemas/{output_type_title}"}
-        invoker_schema["output"] = outputs_ref
-        invoker_schema["class"] = "invocation"
-
-    # This code no longer seems to be necessary?
-    # Leave it here just in case
-    #
-    # from invokeai.backend.model_manager import get_model_config_formats
-    # formats = get_model_config_formats()
-    # for model_config_name, enum_set in formats.items():
-
-    #     if model_config_name in openapi_schema["components"]["schemas"]:
-    #         # print(f"Config with name {name} already defined")
-    #         continue
-
-    #     openapi_schema["components"]["schemas"][model_config_name] = {
-    #         "title": model_config_name,
-    #         "description": "An enumeration.",
-    #         "type": "string",
-    #         "enum": [v.value for v in enum_set],
-    #     }
-
-    app.openapi_schema = openapi_schema
-    return app.openapi_schema
-
-
-app.openapi = custom_openapi  # type: ignore [method-assign] # this is a valid assignment
-
-
-@app.get("/docs", include_in_schema=False)
-def overridden_swagger() -> HTMLResponse:
-    return get_swagger_ui_html(
-        openapi_url=app.openapi_url,  # type: ignore [arg-type] # this is always a string
-        title=f"{app.title} - Swagger UI",
-        swagger_favicon_url="static/docs/invoke-favicon-docs.svg",
-    )
-
-
-@app.get("/redoc", include_in_schema=False)
-def overridden_redoc() -> HTMLResponse:
-    return get_redoc_html(
-        openapi_url=app.openapi_url,  # type: ignore [arg-type] # this is always a string
-        title=f"{app.title} - Redoc",
-        redoc_favicon_url="static/docs/invoke-favicon-docs.svg",
-    )
-
-
-web_root_path = Path(list(web_dir.__path__)[0])
-
-try:
-    app.mount("/", NoCacheStaticFiles(directory=Path(web_root_path, "dist"), html=True), name="ui")
-except RuntimeError:
-    logger.warn(f"No UI found at {web_root_path}/dist, skipping UI mount")
-app.mount(
-    "/static", NoCacheStaticFiles(directory=Path(web_root_path, "static/")), name="static"
-)  # docs favicon is in here
+    app.openapi = custom_openapi  # type: ignore [method-assign] # this is a valid assignment
 
 
 def check_cudnn(logger: logging.Logger) -> None:
@@ -244,27 +243,39 @@ def check_cudnn(logger: logging.Logger) -> None:
             )
 
 
+def find_port(port: int) -> int:
+    """Find a port not in use starting at given port"""
+    # Taken from https://waylonwalker.com/python-find-available-port/, thanks Waylon!
+    # https://github.com/WaylonWalker
+    with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
+        if s.connect_ex(("localhost", port)) == 0:
+            return find_port(port=port + 1)
+        else:
+            return port
+
+
+def init_dev_reload(logger: logging.Logger) -> None:
+    try:
+        import jurigged
+    except ImportError as e:
+        logger.error(
+            'Can\'t start `--dev_reload` because jurigged is not found; `pip install -e ".[dev]"` to include development dependencies.',
+            exc_info=e,
+        )
+    else:
+        jurigged.watch(logger=InvokeAILogger.get_logger(name="jurigged").info)
+
+
 def invoke_api() -> None:
-    def find_port(port: int) -> int:
-        """Find a port not in use starting at given port"""
-        # Taken from https://waylonwalker.com/python-find-available-port/, thanks Waylon!
-        # https://github.com/WaylonWalker
-        with socket.socket(socket.AF_INET, socket.SOCK_STREAM) as s:
-            if s.connect_ex(("localhost", port)) == 0:
-                return find_port(port=port + 1)
-            else:
-                return port
+    start = time.time()
+    app_config = get_config()
+    logger = InvokeAILogger.get_logger(config=app_config)
+
+    apply_monkeypatches()
+    fix_mimetypes()
 
     if app_config.dev_reload:
-        try:
-            import jurigged
-        except ImportError as e:
-            logger.error(
-                'Can\'t start `--dev_reload` because jurigged is not found; `pip install -e ".[dev]"` to include development dependencies.',
-                exc_info=e,
-            )
-        else:
-            jurigged.watch(logger=InvokeAILogger.get_logger(name="jurigged").info)
+        init_dev_reload(logger)
 
     port = find_port(app_config.port)
     if port != app_config.port:
@@ -272,6 +283,11 @@ def invoke_api() -> None:
 
     check_cudnn(logger)
 
+    torch_device_name = TorchDevice.get_torch_device_name()
+    logger.info(f"Using torch device: {torch_device_name}")
+
+    app = build_app(app_config, logger)
+
     # Start our own event loop for eventing usage
     loop = asyncio.new_event_loop()
     config = uvicorn.Config(
@@ -291,6 +307,7 @@ def invoke_api() -> None:
         log.handlers.clear()
         for ch in logger.handlers:
             log.addHandler(ch)
+    logger.info(f"API started in {time.time() - start:.2f} seconds")
 
     loop.run_until_complete(server.serve())
 

invokeai/app/invocations/controlnet_image_processors.py

@@ -35,6 +35,7 @@ from invokeai.app.invocations.model import ModelIdentifierField
 from invokeai.app.invocations.primitives import ImageOutput
 from invokeai.app.invocations.util import validate_begin_end_step, validate_weights
 from invokeai.app.services.shared.invocation_context import InvocationContext
+from invokeai.app.util.controlnet_utils import CONTROLNET_MODE_VALUES, CONTROLNET_RESIZE_VALUES
 from invokeai.backend.image_util.canny import get_canny_edges
 from invokeai.backend.image_util.depth_anything import DepthAnythingDetector
 from invokeai.backend.image_util.dw_openpose import DWOpenposeDetector
@@ -44,14 +45,6 @@ from invokeai.backend.image_util.lineart_anime import LineartAnimeProcessor
 
 from .baseinvocation import BaseInvocation, BaseInvocationOutput, invocation, invocation_output
 
-CONTROLNET_MODE_VALUES = Literal["balanced", "more_prompt", "more_control", "unbalanced"]
-CONTROLNET_RESIZE_VALUES = Literal[
-    "just_resize",
-    "crop_resize",
-    "fill_resize",
-    "just_resize_simple",
-]
-
 
 class ControlField(BaseModel):
     image: ImageField = Field(description="The control image")

invokeai/app/invocations/latent.py

@@ -51,6 +51,7 @@ from invokeai.app.util.controlnet_utils import prepare_control_image
 from invokeai.backend.ip_adapter.ip_adapter import IPAdapter, IPAdapterPlus
 from invokeai.backend.lora import LoRAModelRaw
 from invokeai.backend.model_manager import BaseModelType, LoadedModel
+from invokeai.backend.model_manager.config import MainConfigBase, ModelVariantType
 from invokeai.backend.model_patcher import ModelPatcher
 from invokeai.backend.stable_diffusion import PipelineIntermediateState, set_seamless
 from invokeai.backend.stable_diffusion.diffusion.conditioning_data import (
@@ -185,7 +186,7 @@ class GradientMaskOutput(BaseInvocationOutput):
     title="Create Gradient Mask",
     tags=["mask", "denoise"],
     category="latents",
-    version="1.0.0",
+    version="1.1.0",
 )
 class CreateGradientMaskInvocation(BaseInvocation):
     """Creates mask for denoising model run."""
@@ -198,6 +199,32 @@ class CreateGradientMaskInvocation(BaseInvocation):
     minimum_denoise: float = InputField(
         default=0.0, ge=0, le=1, description="Minimum denoise level for the coherence region", ui_order=4
     )
+    image: Optional[ImageField] = InputField(
+        default=None,
+        description="OPTIONAL: Only connect for specialized Inpainting models, masked_latents will be generated from the image with the VAE",
+        title="[OPTIONAL] Image",
+        ui_order=6,
+    )
+    unet: Optional[UNetField] = InputField(
+        description="OPTIONAL: If the Unet is a specialized Inpainting model, masked_latents will be generated from the image with the VAE",
+        default=None,
+        input=Input.Connection,
+        title="[OPTIONAL] UNet",
+        ui_order=5,
+    )
+    vae: Optional[VAEField] = InputField(
+        default=None,
+        description="OPTIONAL: Only connect for specialized Inpainting models, masked_latents will be generated from the image with the VAE",
+        title="[OPTIONAL] VAE",
+        input=Input.Connection,
+        ui_order=7,
+    )
+    tiled: bool = InputField(default=False, description=FieldDescriptions.tiled, ui_order=8)
+    fp32: bool = InputField(
+        default=DEFAULT_PRECISION == "float32",
+        description=FieldDescriptions.fp32,
+        ui_order=9,
+    )
 
     @torch.no_grad()
     def invoke(self, context: InvocationContext) -> GradientMaskOutput:
@@ -233,8 +260,27 @@ class CreateGradientMaskInvocation(BaseInvocation):
         expanded_mask_image = Image.fromarray((expanded_mask.squeeze(0).numpy() * 255).astype(np.uint8), mode="L")
         expanded_image_dto = context.images.save(expanded_mask_image)
 
+        masked_latents_name = None
+        if self.unet is not None and self.vae is not None and self.image is not None:
+            # all three fields must be present at the same time
+            main_model_config = context.models.get_config(self.unet.unet.key)
+            assert isinstance(main_model_config, MainConfigBase)
+            if main_model_config.variant is ModelVariantType.Inpaint:
+                mask = blur_tensor
+                vae_info: LoadedModel = context.models.load(self.vae.vae)
+                image = context.images.get_pil(self.image.image_name)
+                image_tensor = image_resized_to_grid_as_tensor(image.convert("RGB"))
+                if image_tensor.dim() == 3:
+                    image_tensor = image_tensor.unsqueeze(0)
+                img_mask = tv_resize(mask, image_tensor.shape[-2:], T.InterpolationMode.BILINEAR, antialias=False)
+                masked_image = image_tensor * torch.where(img_mask < 0.5, 0.0, 1.0)
+                masked_latents = ImageToLatentsInvocation.vae_encode(
+                    vae_info, self.fp32, self.tiled, masked_image.clone()
+                )
+                masked_latents_name = context.tensors.save(tensor=masked_latents)
+
         return GradientMaskOutput(
-            denoise_mask=DenoiseMaskField(mask_name=mask_name, masked_latents_name=None, gradient=True),
+            denoise_mask=DenoiseMaskField(mask_name=mask_name, masked_latents_name=masked_latents_name, gradient=True),
             expanded_mask_area=ImageField(image_name=expanded_image_dto.image_name),
         )
 

invokeai/app/invocations/metadata.py

@@ -3,7 +3,6 @@ from typing import Any, Literal, Optional, Union
 from pydantic import BaseModel, ConfigDict, Field
 
 from invokeai.app.invocations.baseinvocation import BaseInvocation, BaseInvocationOutput, invocation, invocation_output
-from invokeai.app.invocations.controlnet_image_processors import CONTROLNET_MODE_VALUES, CONTROLNET_RESIZE_VALUES
 from invokeai.app.invocations.fields import (
     FieldDescriptions,
     ImageField,
@@ -14,6 +13,7 @@ from invokeai.app.invocations.fields import (
 )
 from invokeai.app.invocations.model import ModelIdentifierField
 from invokeai.app.services.shared.invocation_context import InvocationContext
+from invokeai.app.util.controlnet_utils import CONTROLNET_MODE_VALUES, CONTROLNET_RESIZE_VALUES
 
 from ...version import __version__
 

invokeai/app/invocations/t2i_adapter.py

@@ -8,11 +8,11 @@ from invokeai.app.invocations.baseinvocation import (
     invocation,
     invocation_output,
 )
-from invokeai.app.invocations.controlnet_image_processors import CONTROLNET_RESIZE_VALUES
 from invokeai.app.invocations.fields import FieldDescriptions, ImageField, Input, InputField, OutputField, UIType
 from invokeai.app.invocations.model import ModelIdentifierField
 from invokeai.app.invocations.util import validate_begin_end_step, validate_weights
 from invokeai.app.services.shared.invocation_context import InvocationContext
+from invokeai.app.util.controlnet_utils import CONTROLNET_RESIZE_VALUES
 
 
 class T2IAdapterField(BaseModel):

invokeai/app/services/object_serializer/object_serializer_disk.py

@@ -1,6 +1,6 @@
+import shutil
 import tempfile
 import typing
-from dataclasses import dataclass
 from pathlib import Path
 from typing import TYPE_CHECKING, Optional, TypeVar
 
@@ -17,12 +17,6 @@ if TYPE_CHECKING:
 T = TypeVar("T")
 
 
-@dataclass
-class DeleteAllResult:
-    deleted_count: int
-    freed_space_bytes: float
-
-
 class ObjectSerializerDisk(ObjectSerializerBase[T]):
     """Disk-backed storage for arbitrary python objects. Serialization is handled by `torch.save` and `torch.load`.
 
@@ -35,6 +29,12 @@ class ObjectSerializerDisk(ObjectSerializerBase[T]):
         self._ephemeral = ephemeral
         self._base_output_dir = output_dir
         self._base_output_dir.mkdir(parents=True, exist_ok=True)
+
+        if self._ephemeral:
+            # Remove dangling tempdirs that might have been left over from an earlier unplanned shutdown.
+            for temp_dir in filter(Path.is_dir, self._base_output_dir.glob("tmp*")):
+                shutil.rmtree(temp_dir)
+
         # Must specify `ignore_cleanup_errors` to avoid fatal errors during cleanup on Windows
         self._tempdir = (
             tempfile.TemporaryDirectory(dir=self._base_output_dir, ignore_cleanup_errors=True) if ephemeral else None

invokeai/app/util/controlnet_utils.py

@@ -1,13 +1,21 @@
-from typing import Union
+from typing import Any, Literal, Union
 
 import cv2
 import numpy as np
 import torch
-from controlnet_aux.util import HWC3
-from diffusers.utils import PIL_INTERPOLATION
 from einops import rearrange
 from PIL import Image
 
+from invokeai.backend.image_util.util import nms, normalize_image_channel_count
+
+CONTROLNET_RESIZE_VALUES = Literal[
+    "just_resize",
+    "crop_resize",
+    "fill_resize",
+    "just_resize_simple",
+]
+CONTROLNET_MODE_VALUES = Literal["balanced", "more_prompt", "more_control", "unbalanced"]
+
 ###################################################################
 # Copy of scripts/lvminthin.py from Mikubill/sd-webui-controlnet
 ###################################################################
@@ -68,17 +76,6 @@ def lvmin_thin(x, prunings=True):
     return y
 
 
-def nake_nms(x):
-    f1 = np.array([[0, 0, 0], [1, 1, 1], [0, 0, 0]], dtype=np.uint8)
-    f2 = np.array([[0, 1, 0], [0, 1, 0], [0, 1, 0]], dtype=np.uint8)
-    f3 = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1]], dtype=np.uint8)
-    f4 = np.array([[0, 0, 1], [0, 1, 0], [1, 0, 0]], dtype=np.uint8)
-    y = np.zeros_like(x)
-    for f in [f1, f2, f3, f4]:
-        np.putmask(y, cv2.dilate(x, kernel=f) == x, x)
-    return y
-
-
 ################################################################################
 # copied from Mikubill/sd-webui-controlnet external_code.py and modified for InvokeAI
 ################################################################################
@@ -134,98 +131,122 @@ def pixel_perfect_resolution(
     return int(np.round(estimation))
 
 
+def clone_contiguous(x: np.ndarray[Any, Any]) -> np.ndarray[Any, Any]:
+    """Get a memory-contiguous clone of the given numpy array, as a safety measure and to improve computation efficiency."""
+    return np.ascontiguousarray(x).copy()
+
+
+def np_img_to_torch(np_img: np.ndarray[Any, Any], device: torch.device) -> torch.Tensor:
+    """Convert a numpy image to a PyTorch tensor. The image is normalized to 0-1, rearranged to BCHW format and sent to
+    the specified device."""
+
+    torch_img = torch.from_numpy(np_img)
+    normalized = torch_img.float() / 255.0
+    bchw = rearrange(normalized, "h w c -> 1 c h w")
+    on_device = bchw.to(device)
+    return on_device.clone()
+
+
+def heuristic_resize(np_img: np.ndarray[Any, Any], size: tuple[int, int]) -> np.ndarray[Any, Any]:
+    """Resizes an image using a heuristic to choose the best resizing strategy.
+
+    - If the image appears to be an edge map, special handling will be applied to ensure the edges are not distorted.
+    - Single-pixel edge maps use NMS and thinning to keep the edges as single-pixel lines.
+    - Low-color-count images are resized with nearest-neighbor to preserve color information (for e.g. segmentation maps).
+    - The alpha channel is handled separately to ensure it is resized correctly.
+
+    Args:
+        np_img (np.ndarray): The input image.
+        size (tuple[int, int]): The target size for the image.
+
+    Returns:
+        np.ndarray: The resized image.
+
+    Adapted from https://github.com/Mikubill/sd-webui-controlnet.
+    """
+
+    # Return early if the image is already at the requested size
+    if np_img.shape[0] == size[1] and np_img.shape[1] == size[0]:
+        return np_img
+
+    # If the image has an alpha channel, separate it for special handling later.
+    inpaint_mask = None
+    if np_img.ndim == 3 and np_img.shape[2] == 4:
+        inpaint_mask = np_img[:, :, 3]
+        np_img = np_img[:, :, 0:3]
+
+    new_size_is_smaller = (size[0] * size[1]) < (np_img.shape[0] * np_img.shape[1])
+    new_size_is_bigger = (size[0] * size[1]) > (np_img.shape[0] * np_img.shape[1])
+    unique_color_count = np.unique(np_img.reshape(-1, np_img.shape[2]), axis=0).shape[0]
+    is_one_pixel_edge = False
+    is_binary = False
+
+    if unique_color_count == 2:
+        # If the image has only two colors, it is likely binary. Check if the image has one-pixel edges.
+        is_binary = np.min(np_img) < 16 and np.max(np_img) > 240
+        if is_binary:
+            eroded = cv2.erode(np_img, np.ones(shape=(3, 3), dtype=np.uint8), iterations=1)
+            dilated = cv2.dilate(eroded, np.ones(shape=(3, 3), dtype=np.uint8), iterations=1)
+            one_pixel_edge_count = np.where(dilated < np_img)[0].shape[0]
+            all_edge_count = np.where(np_img > 127)[0].shape[0]
+            is_one_pixel_edge = one_pixel_edge_count * 2 > all_edge_count
+
+    if 2 < unique_color_count < 200:
+        # With a low color count, we assume this is a map where exact colors are important. Near-neighbor preserves
+        # the colors as needed.
+        interpolation = cv2.INTER_NEAREST
+    elif new_size_is_smaller:
+        # This works best for downscaling
+        interpolation = cv2.INTER_AREA
+    else:
+        # Fall back for other cases
+        interpolation = cv2.INTER_CUBIC  # Must be CUBIC because we now use nms. NEVER CHANGE THIS
+
+    # This may be further transformed depending on the binary nature of the image.
+    resized = cv2.resize(np_img, size, interpolation=interpolation)
+
+    if inpaint_mask is not None:
+        # Resize the inpaint mask to match the resized image using the same interpolation method.
+        inpaint_mask = cv2.resize(inpaint_mask, size, interpolation=interpolation)
+
+    # If the image is binary, we will perform some additional processing to ensure the edges are preserved.
+    if is_binary:
+        resized = np.mean(resized.astype(np.float32), axis=2).clip(0, 255).astype(np.uint8)
+        if is_one_pixel_edge:
+            # Use NMS and thinning to keep the edges as single-pixel lines.
+            resized = nms(resized)
+            _, resized = cv2.threshold(resized, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
+            resized = lvmin_thin(resized, prunings=new_size_is_bigger)
+        else:
+            _, resized = cv2.threshold(resized, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
+        resized = np.stack([resized] * 3, axis=2)
+
+    # Restore the alpha channel if it was present.
+    if inpaint_mask is not None:
+        inpaint_mask = (inpaint_mask > 127).astype(np.float32) * 255.0
+        inpaint_mask = inpaint_mask[:, :, None].clip(0, 255).astype(np.uint8)
+        resized = np.concatenate([resized, inpaint_mask], axis=2)
+
+    return resized
+
+
 ###########################################################################
 # Copied from detectmap_proc method in scripts/detectmap_proc.py in Mikubill/sd-webui-controlnet
 #    modified for InvokeAI
 ###########################################################################
-# def detectmap_proc(detected_map, module, resize_mode, h, w):
-def np_img_resize(np_img: np.ndarray, resize_mode: str, h: int, w: int, device: torch.device = torch.device("cpu")):
-    # if 'inpaint' in module:
-    #     np_img = np_img.astype(np.float32)
-    # else:
-    #     np_img = HWC3(np_img)
-    np_img = HWC3(np_img)
+def np_img_resize(
+    np_img: np.ndarray,
+    resize_mode: CONTROLNET_RESIZE_VALUES,
+    h: int,
+    w: int,
+    device: torch.device = torch.device("cpu"),
+) -> tuple[torch.Tensor, np.ndarray[Any, Any]]:
+    np_img = normalize_image_channel_count(np_img)
 
-    def safe_numpy(x):
-        # A very safe method to make sure that Apple/Mac works
-        y = x
-
-        # below is very boring but do not change these. If you change these Apple or Mac may fail.
-        y = y.copy()
-        y = np.ascontiguousarray(y)
-        y = y.copy()
-        return y
-
-    def get_pytorch_control(x):
-        # A very safe method to make sure that Apple/Mac works
-        y = x
-
-        # below is very boring but do not change these. If you change these Apple or Mac may fail.
-        y = torch.from_numpy(y)
-        y = y.float() / 255.0
-        y = rearrange(y, "h w c -> 1 c h w")
-        y = y.clone()
-        # y = y.to(devices.get_device_for("controlnet"))
-        y = y.to(device)
-        y = y.clone()
-        return y
-
-    def high_quality_resize(x: np.ndarray, size):
-        # Written by lvmin
-        # Super high-quality control map up-scaling, considering binary, seg, and one-pixel edges
-        inpaint_mask = None
-        if x.ndim == 3 and x.shape[2] == 4:
-            inpaint_mask = x[:, :, 3]
-            x = x[:, :, 0:3]
-
-        new_size_is_smaller = (size[0] * size[1]) < (x.shape[0] * x.shape[1])
-        new_size_is_bigger = (size[0] * size[1]) > (x.shape[0] * x.shape[1])
-        unique_color_count = np.unique(x.reshape(-1, x.shape[2]), axis=0).shape[0]
-        is_one_pixel_edge = False
-        is_binary = False
-        if unique_color_count == 2:
-            is_binary = np.min(x) < 16 and np.max(x) > 240
-            if is_binary:
-                xc = x
-                xc = cv2.erode(xc, np.ones(shape=(3, 3), dtype=np.uint8), iterations=1)
-                xc = cv2.dilate(xc, np.ones(shape=(3, 3), dtype=np.uint8), iterations=1)
-                one_pixel_edge_count = np.where(xc < x)[0].shape[0]
-                all_edge_count = np.where(x > 127)[0].shape[0]
-                is_one_pixel_edge = one_pixel_edge_count * 2 > all_edge_count
-
-        if 2 < unique_color_count < 200:
-            interpolation = cv2.INTER_NEAREST
-        elif new_size_is_smaller:
-            interpolation = cv2.INTER_AREA
-        else:
-            interpolation = cv2.INTER_CUBIC  # Must be CUBIC because we now use nms. NEVER CHANGE THIS
-
-        y = cv2.resize(x, size, interpolation=interpolation)
-        if inpaint_mask is not None:
-            inpaint_mask = cv2.resize(inpaint_mask, size, interpolation=interpolation)
-
-        if is_binary:
-            y = np.mean(y.astype(np.float32), axis=2).clip(0, 255).astype(np.uint8)
-            if is_one_pixel_edge:
-                y = nake_nms(y)
-                _, y = cv2.threshold(y, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
-                y = lvmin_thin(y, prunings=new_size_is_bigger)
-            else:
-                _, y = cv2.threshold(y, 0, 255, cv2.THRESH_BINARY + cv2.THRESH_OTSU)
-            y = np.stack([y] * 3, axis=2)
-
-        if inpaint_mask is not None:
-            inpaint_mask = (inpaint_mask > 127).astype(np.float32) * 255.0
-            inpaint_mask = inpaint_mask[:, :, None].clip(0, 255).astype(np.uint8)
-            y = np.concatenate([y, inpaint_mask], axis=2)
-
-        return y
-
-    # if resize_mode == external_code.ResizeMode.RESIZE:
     if resize_mode == "just_resize":  # RESIZE
-        np_img = high_quality_resize(np_img, (w, h))
-        np_img = safe_numpy(np_img)
-        return get_pytorch_control(np_img), np_img
+        np_img = heuristic_resize(np_img, (w, h))
+        np_img = clone_contiguous(np_img)
+        return np_img_to_torch(np_img, device), np_img
 
     old_h, old_w, _ = np_img.shape
     old_w = float(old_w)
@@ -236,7 +257,6 @@ def np_img_resize(np_img: np.ndarray, resize_mode: str, h: int, w: int, device:
     def safeint(x: Union[int, float]) -> int:
         return int(np.round(x))
 
-    # if resize_mode == external_code.ResizeMode.OUTER_FIT:
     if resize_mode == "fill_resize":  # OUTER_FIT
         k = min(k0, k1)
         borders = np.concatenate([np_img[0, :, :], np_img[-1, :, :], np_img[:, 0, :], np_img[:, -1, :]], axis=0)
@@ -245,23 +265,23 @@ def np_img_resize(np_img: np.ndarray, resize_mode: str, h: int, w: int, device:
             # Inpaint hijack
             high_quality_border_color[3] = 255
         high_quality_background = np.tile(high_quality_border_color[None, None], [h, w, 1])
-        np_img = high_quality_resize(np_img, (safeint(old_w * k), safeint(old_h * k)))
+        np_img = heuristic_resize(np_img, (safeint(old_w * k), safeint(old_h * k)))
         new_h, new_w, _ = np_img.shape
         pad_h = max(0, (h - new_h) // 2)
         pad_w = max(0, (w - new_w) // 2)
         high_quality_background[pad_h : pad_h + new_h, pad_w : pad_w + new_w] = np_img
         np_img = high_quality_background
-        np_img = safe_numpy(np_img)
-        return get_pytorch_control(np_img), np_img
+        np_img = clone_contiguous(np_img)
+        return np_img_to_torch(np_img, device), np_img
     else:  # resize_mode == "crop_resize"  (INNER_FIT)
         k = max(k0, k1)
-        np_img = high_quality_resize(np_img, (safeint(old_w * k), safeint(old_h * k)))
+        np_img = heuristic_resize(np_img, (safeint(old_w * k), safeint(old_h * k)))
         new_h, new_w, _ = np_img.shape
         pad_h = max(0, (new_h - h) // 2)
         pad_w = max(0, (new_w - w) // 2)
         np_img = np_img[pad_h : pad_h + h, pad_w : pad_w + w]
-        np_img = safe_numpy(np_img)
-        return get_pytorch_control(np_img), np_img
+        np_img = clone_contiguous(np_img)
+        return np_img_to_torch(np_img, device), np_img
 
 
 def prepare_control_image(
@@ -269,12 +289,12 @@ def prepare_control_image(
     width: int,
     height: int,
     num_channels: int = 3,
-    device="cuda",
-    dtype=torch.float16,
-    do_classifier_free_guidance=True,
-    control_mode="balanced",
-    resize_mode="just_resize_simple",
-):
+    device: str = "cuda",
+    dtype: torch.dtype = torch.float16,
+    control_mode: CONTROLNET_MODE_VALUES = "balanced",
+    resize_mode: CONTROLNET_RESIZE_VALUES = "just_resize_simple",
+    do_classifier_free_guidance: bool = True,
+) -> torch.Tensor:
     """Pre-process images for ControlNets or T2I-Adapters.
 
     Args:
@@ -292,26 +312,15 @@ def prepare_control_image(
         resize_mode (str, optional): Defaults to "just_resize_simple".
 
     Raises:
-        NotImplementedError: If resize_mode == "crop_resize_simple".
-        NotImplementedError: If resize_mode == "fill_resize_simple".
         ValueError: If `resize_mode` is not recognized.
         ValueError: If `num_channels` is out of range.
 
     Returns:
         torch.Tensor: The pre-processed input tensor.
     """
-    if (
-        resize_mode == "just_resize_simple"
-        or resize_mode == "crop_resize_simple"
-        or resize_mode == "fill_resize_simple"
-    ):
+    if resize_mode == "just_resize_simple":
         image = image.convert("RGB")
-        if resize_mode == "just_resize_simple":
-            image = image.resize((width, height), resample=PIL_INTERPOLATION["lanczos"])
-        elif resize_mode == "crop_resize_simple":
-            raise NotImplementedError(f"prepare_control_image is not implemented for resize_mode='{resize_mode}'.")
-        elif resize_mode == "fill_resize_simple":
-            raise NotImplementedError(f"prepare_control_image is not implemented for resize_mode='{resize_mode}'.")
+        image = image.resize((width, height), resample=Image.LANCZOS)
         nimage = np.array(image)
         nimage = nimage[None, :]
         nimage = np.concatenate([nimage], axis=0)
@@ -328,8 +337,7 @@ def prepare_control_image(
             resize_mode=resize_mode,
             h=height,
             w=width,
-            # device=torch.device('cpu')
-            device=device,
+            device=torch.device(device),
         )
     else:
         raise ValueError(f"Unsupported resize_mode: '{resize_mode}'.")

invokeai/backend/image_util/hed.py

@@ -8,7 +8,7 @@ from huggingface_hub import hf_hub_download
 from PIL import Image
 
 from invokeai.backend.image_util.util import (
-    non_maximum_suppression,
+    nms,
     normalize_image_channel_count,
     np_to_pil,
     pil_to_np,
@@ -134,7 +134,7 @@ class HEDProcessor:
         detected_map = cv2.resize(detected_map, (width, height), interpolation=cv2.INTER_LINEAR)
 
         if scribble:
-            detected_map = non_maximum_suppression(detected_map, 127, 3.0)
+            detected_map = nms(detected_map, 127, 3.0)
             detected_map = cv2.GaussianBlur(detected_map, (0, 0), 3.0)
             detected_map[detected_map > 4] = 255
             detected_map[detected_map < 255] = 0

invokeai/backend/image_util/util.py

@@ -1,4 +1,5 @@
 from math import ceil, floor, sqrt
+from typing import Optional
 
 import cv2
 import numpy as np
@@ -153,10 +154,13 @@ def resize_image_to_resolution(input_image: np.ndarray, resolution: int) -> np.n
         return cv2.resize(input_image, (w, h), interpolation=cv2.INTER_AREA)
 
 
-def non_maximum_suppression(image: np.ndarray, threshold: int, sigma: float):
+def nms(np_img: np.ndarray, threshold: Optional[int] = None, sigma: Optional[float] = None) -> np.ndarray:
     """
     Apply non-maximum suppression to an image.
 
+    If both threshold and sigma are provided, the image will blurred before the suppression and thresholded afterwards,
+    resulting in a binary output image.
+
     This function is adapted from https://github.com/lllyasviel/ControlNet.
 
     Args:
@@ -166,23 +170,36 @@ def non_maximum_suppression(image: np.ndarray, threshold: int, sigma: float):
 
     Returns:
         The image after non-maximum suppression.
+
+    Raises:
+        ValueError: If only one of threshold and sigma provided.
     """
 
-    image = cv2.GaussianBlur(image.astype(np.float32), (0, 0), sigma)
+    # Raise a value error if only one of threshold and sigma is provided
+    if (threshold is None) != (sigma is None):
+        raise ValueError("Both threshold and sigma must be provided if one is provided.")
+
+    if sigma is not None and threshold is not None:
+        # Blurring the image can help to thin out features
+        np_img = cv2.GaussianBlur(np_img.astype(np.float32), (0, 0), sigma)
 
     filter_1 = np.array([[0, 0, 0], [1, 1, 1], [0, 0, 0]], dtype=np.uint8)
     filter_2 = np.array([[0, 1, 0], [0, 1, 0], [0, 1, 0]], dtype=np.uint8)
     filter_3 = np.array([[1, 0, 0], [0, 1, 0], [0, 0, 1]], dtype=np.uint8)
     filter_4 = np.array([[0, 0, 1], [0, 1, 0], [1, 0, 0]], dtype=np.uint8)
 
-    y = np.zeros_like(image)
+    nms_img = np.zeros_like(np_img)
 
     for f in [filter_1, filter_2, filter_3, filter_4]:
-        np.putmask(y, cv2.dilate(image, kernel=f) == image, image)
+        np.putmask(nms_img, cv2.dilate(np_img, kernel=f) == np_img, np_img)
 
-    z = np.zeros_like(y, dtype=np.uint8)
-    z[y > threshold] = 255
-    return z
+    if sigma is not None and threshold is not None:
+        # We blurred - now threshold to get a binary image
+        thresholded = np.zeros_like(nms_img, dtype=np.uint8)
+        thresholded[nms_img > threshold] = 255
+        return thresholded
+
+    return nms_img
 
 
 def safe_step(x: np.ndarray, step: int = 2) -> np.ndarray:

invokeai/backend/model_manager/config.py

@@ -301,12 +301,12 @@ class MainConfigBase(ModelConfigBase):
     default_settings: Optional[MainModelDefaultSettings] = Field(
         description="Default settings for this model", default=None
     )
+    variant: ModelVariantType = ModelVariantType.Normal
 
 
 class MainCheckpointConfig(CheckpointConfigBase, MainConfigBase):
     """Model config for main checkpoint models."""
 
-    variant: ModelVariantType = ModelVariantType.Normal
     prediction_type: SchedulerPredictionType = SchedulerPredictionType.Epsilon
     upcast_attention: bool = False
 

invokeai/backend/model_manager/starter_models.py

@@ -155,7 +155,7 @@ STARTER_MODELS: list[StarterModel] = [
     StarterModel(
         name="IP Adapter",
         base=BaseModelType.StableDiffusion1,
-        source="InvokeAI/ip_adapter_sd15",
+        source="https://huggingface.co/InvokeAI/ip_adapter_sd15/resolve/main/ip-adapter_sd15.safetensors",
         description="IP-Adapter for SD 1.5 models",
         type=ModelType.IPAdapter,
         dependencies=[ip_adapter_sd_image_encoder],
@@ -163,7 +163,7 @@ STARTER_MODELS: list[StarterModel] = [
     StarterModel(
         name="IP Adapter Plus",
         base=BaseModelType.StableDiffusion1,
-        source="InvokeAI/ip_adapter_plus_sd15",
+        source="https://huggingface.co/InvokeAI/ip_adapter_plus_sd15/resolve/main/ip-adapter-plus_sd15.safetensors",
         description="Refined IP-Adapter for SD 1.5 models",
         type=ModelType.IPAdapter,
         dependencies=[ip_adapter_sd_image_encoder],
@@ -171,7 +171,7 @@ STARTER_MODELS: list[StarterModel] = [
     StarterModel(
         name="IP Adapter Plus Face",
         base=BaseModelType.StableDiffusion1,
-        source="InvokeAI/ip_adapter_plus_face_sd15",
+        source="https://huggingface.co/InvokeAI/ip_adapter_plus_face_sd15/resolve/main/ip-adapter-plus-face_sd15.safetensors",
         description="Refined IP-Adapter for SD 1.5 models, adapted for faces",
         type=ModelType.IPAdapter,
         dependencies=[ip_adapter_sd_image_encoder],
@@ -179,7 +179,7 @@ STARTER_MODELS: list[StarterModel] = [
     StarterModel(
         name="IP Adapter SDXL",
         base=BaseModelType.StableDiffusionXL,
-        source="InvokeAI/ip_adapter_sdxl",
+        source="https://huggingface.co/InvokeAI/ip_adapter_sdxl_vit_h/resolve/main/ip-adapter_sdxl_vit-h.safetensors",
         description="IP-Adapter for SDXL models",
         type=ModelType.IPAdapter,
         dependencies=[ip_adapter_sdxl_image_encoder],

invokeai/frontend/web/package.json

@@ -51,6 +51,7 @@
     }
   },
   "dependencies": {
+    "@chakra-ui/react-use-size": "^2.1.0",
     "@dagrejs/dagre": "^1.1.1",
     "@dagrejs/graphlib": "^2.2.1",
     "@dnd-kit/core": "^6.1.0",

invokeai/frontend/web/pnpm-lock.yaml

@@ -8,6 +8,9 @@ dependencies:
   '@chakra-ui/react':
     specifier: ^2.8.2
     version: 2.8.2(@emotion/react@11.11.3)(@emotion/styled@11.11.0)(@types/react@18.2.59)(framer-motion@11.0.6)(react-dom@18.2.0)(react@18.2.0)
+  '@chakra-ui/react-use-size':
+    specifier: ^2.1.0
+    version: 2.1.0(react@18.2.0)
   '@dagrejs/dagre':
     specifier: ^1.1.1
     version: 1.1.1

invokeai/frontend/web/src/features/modelManagerV2/subpanels/ModelPanel/ControlNetOrT2IAdapterDefaultSettings/ControlNetOrT2IAdapterDefaultSettings.tsx

@@ -1,4 +1,4 @@
-import { Button, Flex, Heading, Text } from '@invoke-ai/ui-library';
+import { Button, Flex, Heading, SimpleGrid, Text } from '@invoke-ai/ui-library';
 import { useAppDispatch, useAppSelector } from 'app/store/storeHooks';
 import { useControlNetOrT2IAdapterDefaultSettings } from 'features/modelManagerV2/hooks/useControlNetOrT2IAdapterDefaultSettings';
 import { DefaultPreprocessor } from 'features/modelManagerV2/subpanels/ModelPanel/ControlNetOrT2IAdapterDefaultSettings/DefaultPreprocessor';
@@ -92,13 +92,9 @@ export const ControlNetOrT2IAdapterDefaultSettings = () => {
         </Button>
       </Flex>
 
-      <Flex flexDir="column" gap={8}>
-        <Flex gap={8}>
-          <Flex gap={4} w="full">
-            <DefaultPreprocessor control={control} name="preprocessor" />
-          </Flex>
-        </Flex>
-      </Flex>
+      <SimpleGrid columns={2} gap={8}>
+        <DefaultPreprocessor control={control} name="preprocessor" />
+      </SimpleGrid>
     </>
   );
 };

invokeai/frontend/web/src/features/modelManagerV2/subpanels/ModelPanel/MainModelDefaultSettings/MainModelDefaultSettings.tsx

@@ -1,4 +1,4 @@
-import { Button, Flex, Heading, Text } from '@invoke-ai/ui-library';
+import { Button, Flex, Heading, SimpleGrid, Text } from '@invoke-ai/ui-library';
 import { useAppDispatch, useAppSelector } from 'app/store/storeHooks';
 import { useMainModelDefaultSettings } from 'features/modelManagerV2/hooks/useMainModelDefaultSettings';
 import { DefaultHeight } from 'features/modelManagerV2/subpanels/ModelPanel/MainModelDefaultSettings/DefaultHeight';
@@ -122,40 +122,16 @@ export const MainModelDefaultSettings = () => {
         </Button>
       </Flex>
 
-      <Flex flexDir="column" gap={8}>
-        <Flex gap={8}>
-          <Flex gap={4} w="full">
-            <DefaultVae control={control} name="vae" />
-          </Flex>
-          <Flex gap={4} w="full">
-            <DefaultVaePrecision control={control} name="vaePrecision" />
-          </Flex>
-        </Flex>
-        <Flex gap={8}>
-          <Flex gap={4} w="full">
-            <DefaultScheduler control={control} name="scheduler" />
-          </Flex>
-          <Flex gap={4} w="full">
-            <DefaultSteps control={control} name="steps" />
-          </Flex>
-        </Flex>
-        <Flex gap={8}>
-          <Flex gap={4} w="full">
-            <DefaultCfgScale control={control} name="cfgScale" />
-          </Flex>
-          <Flex gap={4} w="full">
-            <DefaultCfgRescaleMultiplier control={control} name="cfgRescaleMultiplier" />
-          </Flex>
-        </Flex>
-        <Flex gap={8}>
-          <Flex gap={4} w="full">
-            <DefaultWidth control={control} optimalDimension={optimalDimension} />
-          </Flex>
-          <Flex gap={4} w="full">
-            <DefaultHeight control={control} optimalDimension={optimalDimension} />
-          </Flex>
-        </Flex>
-      </Flex>
+      <SimpleGrid columns={2} gap={8}>
+        <DefaultVae control={control} name="vae" />
+        <DefaultVaePrecision control={control} name="vaePrecision" />
+        <DefaultScheduler control={control} name="scheduler" />
+        <DefaultSteps control={control} name="steps" />
+        <DefaultCfgScale control={control} name="cfgScale" />
+        <DefaultCfgRescaleMultiplier control={control} name="cfgRescaleMultiplier" />
+        <DefaultWidth control={control} optimalDimension={optimalDimension} />
+        <DefaultHeight control={control} optimalDimension={optimalDimension} />
+      </SimpleGrid>
     </>
   );
 };

invokeai/frontend/web/src/features/modelManagerV2/subpanels/ModelPanel/ModelEdit.tsx

@@ -6,6 +6,7 @@ import {
   FormLabel,
   Heading,
   Input,
+  SimpleGrid,
   Text,
   Textarea,
 } from '@invoke-ai/ui-library';
@@ -66,25 +67,21 @@ export const ModelEdit = ({ form }: Props) => {
           <Heading as="h3" fontSize="md" mt="4">
             {t('modelManager.modelSettings')}
           </Heading>
-          <Flex gap={4}>
+          <SimpleGrid columns={2} gap={4}>
             <FormControl flexDir="column" alignItems="flex-start" gap={1}>
               <FormLabel>{t('modelManager.baseModel')}</FormLabel>
               <BaseModelSelect control={form.control} />
             </FormControl>
-          </Flex>
-          {data.type === 'main' && data.format === 'checkpoint' && (
-            <>
-              <Flex gap={4}>
+            <FormControl flexDir="column" alignItems="flex-start" gap={1}>
+              <FormLabel>{t('modelManager.variant')}</FormLabel>
+              <ModelVariantSelect control={form.control} />
+            </FormControl>
+            {data.type === 'main' && data.format === 'checkpoint' && (
+              <>
                 <FormControl flexDir="column" alignItems="flex-start" gap={1}>
                   <FormLabel>{t('modelManager.pathToConfig')}</FormLabel>
                   <Input {...form.register('config_path', stringFieldOptions)} />
                 </FormControl>
-                <FormControl flexDir="column" alignItems="flex-start" gap={1}>
-                  <FormLabel>{t('modelManager.variant')}</FormLabel>
-                  <ModelVariantSelect control={form.control} />
-                </FormControl>
-              </Flex>
-              <Flex gap={4}>
                 <FormControl flexDir="column" alignItems="flex-start" gap={1}>
                   <FormLabel>{t('modelManager.predictionType')}</FormLabel>
                   <PredictionTypeSelect control={form.control} />
@@ -93,9 +90,9 @@ export const ModelEdit = ({ form }: Props) => {
                   <FormLabel>{t('modelManager.upcastAttention')}</FormLabel>
                   <Checkbox {...form.register('upcast_attention')} />
                 </FormControl>
-              </Flex>
-            </>
-          )}
+              </>
+            )}
+          </SimpleGrid>
         </Flex>
       </form>
     </Flex>

invokeai/frontend/web/src/features/modelManagerV2/subpanels/ModelPanel/ModelView.tsx

@@ -1,4 +1,4 @@
-import { Box, Flex, Text } from '@invoke-ai/ui-library';
+import { Box, Flex, SimpleGrid, Text } from '@invoke-ai/ui-library';
 import { skipToken } from '@reduxjs/toolkit/query';
 import { useAppSelector } from 'app/store/storeHooks';
 import { ControlNetOrT2IAdapterDefaultSettings } from 'features/modelManagerV2/subpanels/ModelPanel/ControlNetOrT2IAdapterDefaultSettings/ControlNetOrT2IAdapterDefaultSettings';
@@ -24,57 +24,32 @@ export const ModelView = () => {
   return (
     <Flex flexDir="column" h="full" gap={4}>
       <Box layerStyle="second" borderRadius="base" p={4}>
-        <Flex flexDir="column" gap={4}>
-          <Flex gap={2}>
-            <ModelAttrView label={t('modelManager.baseModel')} value={data.base} />
-            <ModelAttrView label={t('modelManager.modelType')} value={data.type} />
-          </Flex>
-          <Flex gap={2}>
-            <ModelAttrView label={t('common.format')} value={data.format} />
-            <ModelAttrView label={t('modelManager.path')} value={data.path} />
-          </Flex>
-
+        <SimpleGrid columns={2} gap={4}>
+          <ModelAttrView label={t('modelManager.baseModel')} value={data.base} />
+          <ModelAttrView label={t('modelManager.modelType')} value={data.type} />
+          <ModelAttrView label={t('common.format')} value={data.format} />
+          <ModelAttrView label={t('modelManager.path')} value={data.path} />
+          {data.type === 'main' && <ModelAttrView label={t('modelManager.variant')} value={data.variant} />}
           {data.type === 'main' && data.format === 'diffusers' && data.repo_variant && (
-            <Flex gap={2}>
-              <ModelAttrView label={t('modelManager.repoVariant')} value={data.repo_variant} />
-            </Flex>
+            <ModelAttrView label={t('modelManager.repoVariant')} value={data.repo_variant} />
           )}
-
           {data.type === 'main' && data.format === 'checkpoint' && (
             <>
-              <Flex gap={2}>
-                <ModelAttrView label={t('modelManager.pathToConfig')} value={data.config_path} />
-                <ModelAttrView label={t('modelManager.variant')} value={data.variant} />
-              </Flex>
-              <Flex gap={2}>
-                <ModelAttrView label={t('modelManager.predictionType')} value={data.prediction_type} />
-                <ModelAttrView label={t('modelManager.upcastAttention')} value={`${data.upcast_attention}`} />
-              </Flex>
+              <ModelAttrView label={t('modelManager.pathToConfig')} value={data.config_path} />
+              <ModelAttrView label={t('modelManager.predictionType')} value={data.prediction_type} />
+              <ModelAttrView label={t('modelManager.upcastAttention')} value={`${data.upcast_attention}`} />
             </>
           )}
-
           {data.type === 'ip_adapter' && data.format === 'invokeai' && (
-            <Flex gap={2}>
-              <ModelAttrView label={t('modelManager.imageEncoderModelId')} value={data.image_encoder_model_id} />
-            </Flex>
+            <ModelAttrView label={t('modelManager.imageEncoderModelId')} value={data.image_encoder_model_id} />
           )}
-        </Flex>
+        </SimpleGrid>
+      </Box>
+      <Box layerStyle="second" borderRadius="base" p={4}>
+        {data.type === 'main' && data.base !== 'sdxl-refiner' && <MainModelDefaultSettings />}
+        {(data.type === 'controlnet' || data.type === 't2i_adapter') && <ControlNetOrT2IAdapterDefaultSettings />}
+        {(data.type === 'main' || data.type === 'lora') && <TriggerPhrases />}
       </Box>
-      {data.type === 'main' && data.base !== 'sdxl-refiner' && (
-        <Box layerStyle="second" borderRadius="base" p={4}>
-          <MainModelDefaultSettings />
-        </Box>
-      )}
-      {(data.type === 'controlnet' || data.type === 't2i_adapter') && (
-        <Box layerStyle="second" borderRadius="base" p={4}>
-          <ControlNetOrT2IAdapterDefaultSettings />
-        </Box>
-      )}
-      {(data.type === 'main' || data.type === 'lora') && (
-        <Box layerStyle="second" borderRadius="base" p={4}>
-          <TriggerPhrases />
-        </Box>
-      )}
     </Flex>
   );
 };

invokeai/frontend/web/src/features/modelManagerV2/subpanels/ModelPanel/TriggerPhrases.tsx

@@ -77,9 +77,17 @@ export const TriggerPhrases = () => {
     [updateModel, selectedModelKey, triggerPhrases]
   );
 
+  const onTriggerPhraseAddFormSubmit = useCallback(
+    (e: React.FormEvent<HTMLFormElement>) => {
+      e.preventDefault();
+      addTriggerPhrase();
+    },
+    [addTriggerPhrase]
+  );
+
   return (
     <Flex flexDir="column" w="full" gap="5">
-      <form>
+      <form onSubmit={onTriggerPhraseAddFormSubmit}>
         <FormControl w="full" isInvalid={Boolean(errors.length)} orientation="vertical">
           <FormLabel>{t('modelManager.triggerPhrases')}</FormLabel>
           <Flex flexDir="column" w="full">

invokeai/frontend/web/src/features/nodes/util/graph/addVAEToGraph.ts

@@ -9,6 +9,7 @@ import {
   CANVAS_TEXT_TO_IMAGE_GRAPH,
   IMAGE_TO_IMAGE_GRAPH,
   IMAGE_TO_LATENTS,
+  INPAINT_CREATE_MASK,
   INPAINT_IMAGE,
   LATENTS_TO_IMAGE,
   MAIN_MODEL_LOADER,
@@ -145,6 +146,16 @@ export const addVAEToGraph = async (
           field: 'vae',
         },
       },
+      {
+        source: {
+          node_id: isSeamlessEnabled ? SEAMLESS : isAutoVae ? modelLoaderNodeId : VAE_LOADER,
+          field: 'vae',
+        },
+        destination: {
+          node_id: INPAINT_CREATE_MASK,
+          field: 'vae',
+        },
+      },
 
       {
         source: {

invokeai/frontend/web/src/features/nodes/util/graph/buildCanvasInpaintGraph.ts

@@ -133,6 +133,8 @@ export const buildCanvasInpaintGraph = async (
         coherence_mode: canvasCoherenceMode,
         minimum_denoise: canvasCoherenceMinDenoise,
         edge_radius: canvasCoherenceEdgeSize,
+        tiled: false,
+        fp32: fp32,
       },
       [DENOISE_LATENTS]: {
         type: 'denoise_latents',
@@ -182,6 +184,16 @@ export const buildCanvasInpaintGraph = async (
           field: 'clip',
         },
       },
+      {
+        source: {
+          node_id: modelLoaderNodeId,
+          field: 'unet',
+        },
+        destination: {
+          node_id: INPAINT_CREATE_MASK,
+          field: 'unet',
+        },
+      },
       // Connect CLIP Skip to Conditioning
       {
         source: {
@@ -331,6 +343,16 @@ export const buildCanvasInpaintGraph = async (
           field: 'mask',
         },
       },
+      {
+        source: {
+          node_id: INPAINT_IMAGE_RESIZE_UP,
+          field: 'image',
+        },
+        destination: {
+          node_id: INPAINT_CREATE_MASK,
+          field: 'image',
+        },
+      },
       // Resize Down
       {
         source: {

invokeai/frontend/web/src/features/nodes/util/graph/buildCanvasOutpaintGraph.ts

@@ -157,6 +157,8 @@ export const buildCanvasOutpaintGraph = async (
         coherence_mode: canvasCoherenceMode,
         edge_radius: canvasCoherenceEdgeSize,
         minimum_denoise: canvasCoherenceMinDenoise,
+        tiled: false,
+        fp32: fp32,
       },
       [DENOISE_LATENTS]: {
         type: 'denoise_latents',
@@ -207,6 +209,16 @@ export const buildCanvasOutpaintGraph = async (
           field: 'clip',
         },
       },
+      {
+        source: {
+          node_id: modelLoaderNodeId,
+          field: 'unet',
+        },
+        destination: {
+          node_id: INPAINT_CREATE_MASK,
+          field: 'unet',
+        },
+      },
       // Connect CLIP Skip to Conditioning
       {
         source: {
@@ -453,6 +465,16 @@ export const buildCanvasOutpaintGraph = async (
           field: 'image',
         },
       },
+      {
+        source: {
+          node_id: INPAINT_IMAGE_RESIZE_UP,
+          field: 'image',
+        },
+        destination: {
+          node_id: INPAINT_CREATE_MASK,
+          field: 'image',
+        },
+      },
       // Resize Results Down
       {
         source: {

invokeai/frontend/web/src/features/nodes/util/graph/buildCanvasSDXLInpaintGraph.ts

@@ -135,6 +135,8 @@ export const buildCanvasSDXLInpaintGraph = async (
         coherence_mode: canvasCoherenceMode,
         minimum_denoise: refinerModel ? Math.max(0.2, canvasCoherenceMinDenoise) : canvasCoherenceMinDenoise,
         edge_radius: canvasCoherenceEdgeSize,
+        tiled: false,
+        fp32: fp32,
       },
       [SDXL_DENOISE_LATENTS]: {
         type: 'denoise_latents',
@@ -214,6 +216,16 @@ export const buildCanvasSDXLInpaintGraph = async (
           field: 'clip2',
         },
       },
+      {
+        source: {
+          node_id: modelLoaderNodeId,
+          field: 'unet',
+        },
+        destination: {
+          node_id: INPAINT_CREATE_MASK,
+          field: 'unet',
+        },
+      },
       // Connect Everything To Inpaint Node
       {
         source: {
@@ -342,6 +354,16 @@ export const buildCanvasSDXLInpaintGraph = async (
           field: 'mask',
         },
       },
+      {
+        source: {
+          node_id: INPAINT_IMAGE_RESIZE_UP,
+          field: 'image',
+        },
+        destination: {
+          node_id: INPAINT_CREATE_MASK,
+          field: 'image',
+        },
+      },
       // Resize Down
       {
         source: {

invokeai/frontend/web/src/features/nodes/util/graph/buildCanvasSDXLOutpaintGraph.ts

@@ -157,6 +157,8 @@ export const buildCanvasSDXLOutpaintGraph = async (
         coherence_mode: canvasCoherenceMode,
         edge_radius: canvasCoherenceEdgeSize,
         minimum_denoise: refinerModel ? Math.max(0.2, canvasCoherenceMinDenoise) : canvasCoherenceMinDenoise,
+        tiled: false,
+        fp32: fp32,
       },
       [SDXL_DENOISE_LATENTS]: {
         type: 'denoise_latents',
@@ -237,6 +239,16 @@ export const buildCanvasSDXLOutpaintGraph = async (
           field: 'clip2',
         },
       },
+      {
+        source: {
+          node_id: modelLoaderNodeId,
+          field: 'unet',
+        },
+        destination: {
+          node_id: INPAINT_CREATE_MASK,
+          field: 'unet',
+        },
+      },
       // Connect Infill Result To Inpaint Image
       {
         source: {
@@ -451,6 +463,16 @@ export const buildCanvasSDXLOutpaintGraph = async (
           field: 'image',
         },
       },
+      {
+        source: {
+          node_id: INPAINT_IMAGE_RESIZE_UP,
+          field: 'image',
+        },
+        destination: {
+          node_id: INPAINT_CREATE_MASK,
+          field: 'image',
+        },
+      },
       // Take combined mask and resize
       {
         source: {

invokeai/frontend/web/src/features/parameters/components/ImageSize/AspectRatioCanvasPreview.tsx

@@ -2,7 +2,7 @@ import { Flex } from '@invoke-ai/ui-library';
 import { StageComponent } from 'features/regionalPrompts/components/StageComponent';
 import { memo } from 'react';
 
-export const AspectRatioPreview = memo(() => {
+export const AspectRatioCanvasPreview = memo(() => {
   return (
     <Flex w="full" h="full" alignItems="center" justifyContent="center" position="relative">
       <StageComponent asPreview />
@@ -10,4 +10,4 @@ export const AspectRatioPreview = memo(() => {
   );
 });
 
-AspectRatioPreview.displayName = 'AspectRatioPreview';
+AspectRatioCanvasPreview.displayName = 'AspectRatioCanvasPreview';

invokeai/frontend/web/src/features/parameters/components/ImageSize/AspectRatioIconPreview.tsx

@@ -0,0 +1,75 @@
+import { useSize } from '@chakra-ui/react-use-size';
+import { Flex, Icon } from '@invoke-ai/ui-library';
+import { useImageSizeContext } from 'features/parameters/components/ImageSize/ImageSizeContext';
+import { AnimatePresence, motion } from 'framer-motion';
+import { memo, useMemo, useRef } from 'react';
+import { PiFrameCorners } from 'react-icons/pi';
+
+import {
+  BOX_SIZE_CSS_CALC,
+  ICON_CONTAINER_STYLES,
+  ICON_HIGH_CUTOFF,
+  ICON_LOW_CUTOFF,
+  MOTION_ICON_ANIMATE,
+  MOTION_ICON_EXIT,
+  MOTION_ICON_INITIAL,
+} from './constants';
+
+export const AspectRatioIconPreview = memo(() => {
+  const ctx = useImageSizeContext();
+  const containerRef = useRef<HTMLDivElement>(null);
+  const containerSize = useSize(containerRef);
+
+  const shouldShowIcon = useMemo(
+    () => ctx.aspectRatioState.value < ICON_HIGH_CUTOFF && ctx.aspectRatioState.value > ICON_LOW_CUTOFF,
+    [ctx.aspectRatioState.value]
+  );
+
+  const { width, height } = useMemo(() => {
+    if (!containerSize) {
+      return { width: 0, height: 0 };
+    }
+
+    let width = ctx.width;
+    let height = ctx.height;
+
+    if (ctx.width > ctx.height) {
+      width = containerSize.width;
+      height = width / ctx.aspectRatioState.value;
+    } else {
+      height = containerSize.height;
+      width = height * ctx.aspectRatioState.value;
+    }
+
+    return { width, height };
+  }, [containerSize, ctx.width, ctx.height, ctx.aspectRatioState.value]);
+
+  return (
+    <Flex w="full" h="full" alignItems="center" justifyContent="center" ref={containerRef}>
+      <Flex
+        bg="blackAlpha.400"
+        borderRadius="base"
+        width={`${width}px`}
+        height={`${height}px`}
+        alignItems="center"
+        justifyContent="center"
+      >
+        <AnimatePresence>
+          {shouldShowIcon && (
+            <Flex
+              as={motion.div}
+              initial={MOTION_ICON_INITIAL}
+              animate={MOTION_ICON_ANIMATE}
+              exit={MOTION_ICON_EXIT}
+              style={ICON_CONTAINER_STYLES}
+            >
+              <Icon as={PiFrameCorners} color="base.700" boxSize={BOX_SIZE_CSS_CALC} />
+            </Flex>
+          )}
+        </AnimatePresence>
+      </Flex>
+    </Flex>
+  );
+});
+
+AspectRatioIconPreview.displayName = 'AspectRatioIconPreview';

invokeai/frontend/web/src/features/parameters/components/ImageSize/ImageSize.tsx

@@ -1,6 +1,5 @@
 import type { FormLabelProps } from '@invoke-ai/ui-library';
 import { Flex, FormControlGroup } from '@invoke-ai/ui-library';
-import { AspectRatioPreview } from 'features/parameters/components/ImageSize/AspectRatioPreview';
 import { AspectRatioSelect } from 'features/parameters/components/ImageSize/AspectRatioSelect';
 import type { ImageSizeContextInnerValue } from 'features/parameters/components/ImageSize/ImageSizeContext';
 import { ImageSizeContext } from 'features/parameters/components/ImageSize/ImageSizeContext';
@@ -13,10 +12,11 @@ import { memo } from 'react';
 type ImageSizeProps = ImageSizeContextInnerValue & {
   widthComponent: ReactNode;
   heightComponent: ReactNode;
+  previewComponent: ReactNode;
 };
 
 export const ImageSize = memo((props: ImageSizeProps) => {
-  const { widthComponent, heightComponent, ...ctx } = props;
+  const { widthComponent, heightComponent, previewComponent, ...ctx } = props;
   return (
     <ImageSizeContext.Provider value={ctx}>
       <Flex gap={4} alignItems="center">
@@ -33,7 +33,7 @@ export const ImageSize = memo((props: ImageSizeProps) => {
           </FormControlGroup>
         </Flex>
         <Flex w="108px" h="108px" flexShrink={0} flexGrow={0}>
-          <AspectRatioPreview />
+          {previewComponent}
         </Flex>
       </Flex>
     </ImageSizeContext.Provider>

invokeai/frontend/web/src/features/parameters/components/ImageSize/constants.ts

@@ -1,7 +1,29 @@
 import type { ComboboxOption } from '@invoke-ai/ui-library';
 
 import type { AspectRatioID, AspectRatioState } from './types';
-
+// When the aspect ratio is between these two values, we show the icon (experimentally determined)
+export const ICON_LOW_CUTOFF = 0.23;
+export const ICON_HIGH_CUTOFF = 1 / ICON_LOW_CUTOFF;
+const ICON_SIZE_PX = 64;
+const ICON_PADDING_PX = 16;
+export const BOX_SIZE_CSS_CALC = `min(${ICON_SIZE_PX}px, calc(100% - ${ICON_PADDING_PX}px))`;
+export const MOTION_ICON_INITIAL = {
+  opacity: 0,
+};
+export const MOTION_ICON_ANIMATE = {
+  opacity: 1,
+  transition: { duration: 0.1 },
+};
+export const MOTION_ICON_EXIT = {
+  opacity: 0,
+  transition: { duration: 0.1 },
+};
+export const ICON_CONTAINER_STYLES = {
+  width: '100%',
+  height: '100%',
+  alignItems: 'center',
+  justifyContent: 'center',
+};
 export const ASPECT_RATIO_OPTIONS: ComboboxOption[] = [
   { label: 'Free' as const, value: 'Free' },
   { label: '16:9' as const, value: '16:9' },

invokeai/frontend/web/src/features/regionalPrompts/components/RPLayerListItem.tsx

@@ -75,7 +75,7 @@ export const RPLayerListItem = memo(({ layerId }: Props) => {
           <RPLayerSettingsPopover layerId={layerId} />
           <RPLayerMenu layerId={layerId} />
         </Flex>
-        <AddPromptButtons layerId={layerId} />
+        {!hasPositivePrompt && !hasNegativePrompt && !hasIPAdapters && <AddPromptButtons layerId={layerId} />}
         {hasPositivePrompt && <RPLayerPositivePrompt layerId={layerId} />}
         {hasNegativePrompt && <RPLayerNegativePrompt layerId={layerId} />}
         {hasIPAdapters && <RPLayerIPAdapterList layerId={layerId} />}

invokeai/frontend/web/src/features/regionalPrompts/components/StageComponent.tsx

@@ -6,6 +6,7 @@ import { useAppDispatch, useAppSelector } from 'app/store/storeHooks';
 import { useMouseEvents } from 'features/regionalPrompts/hooks/mouseEventHooks';
 import {
   $cursorPosition,
+  $isMouseOver,
   $lastMouseDownPos,
   $tool,
   isVectorMaskLayer,
@@ -14,7 +15,7 @@ import {
   layerTranslated,
   selectRegionalPromptsSlice,
 } from 'features/regionalPrompts/store/regionalPromptsSlice';
-import { renderers } from 'features/regionalPrompts/util/renderers';
+import { debouncedRenderers, renderers as normalRenderers } from 'features/regionalPrompts/util/renderers';
 import Konva from 'konva';
 import type { IRect } from 'konva/lib/types';
 import type { MutableRefObject } from 'react';
@@ -49,18 +50,10 @@ const useStageRenderer = (
   const { onMouseDown, onMouseUp, onMouseMove, onMouseEnter, onMouseLeave, onMouseWheel } = useMouseEvents();
   const cursorPosition = useStore($cursorPosition);
   const lastMouseDownPos = useStore($lastMouseDownPos);
+  const isMouseOver = useStore($isMouseOver);
   const selectedLayerIdColor = useAppSelector(selectSelectedLayerColor);
-
-  const renderLayers = useMemo(() => (asPreview ? renderers.layersDebounced : renderers.layers), [asPreview]);
-  const renderToolPreview = useMemo(
-    () => (asPreview ? renderers.toolPreviewDebounced : renderers.toolPreview),
-    [asPreview]
-  );
-  const renderBbox = useMemo(() => (asPreview ? renderers.bboxDebounced : renderers.bbox), [asPreview]);
-  const renderBackground = useMemo(
-    () => (asPreview ? renderers.backgroundDebounced : renderers.background),
-    [asPreview]
-  );
+  const layerIds = useMemo(() => state.layers.map((l) => l.id), [state.layers]);
+  const renderers = useMemo(() => (asPreview ? debouncedRenderers : normalRenderers), [asPreview]);
 
   const onLayerPosChanged = useCallback(
     (layerId: string, x: number, y: number) => {
@@ -147,17 +140,19 @@ const useStageRenderer = (
   }, [stageRef, width, height, wrapper]);
 
   useLayoutEffect(() => {
-    log.trace('Rendering brush preview');
+    log.trace('Rendering tool preview');
     if (asPreview) {
+      // Preview should not display tool
       return;
     }
-    renderToolPreview(
+    renderers.renderToolPreview(
       stageRef.current,
       tool,
       selectedLayerIdColor,
       state.globalMaskLayerOpacity,
       cursorPosition,
       lastMouseDownPos,
+      isMouseOver,
       state.brushSize
     );
   }, [
@@ -168,30 +163,38 @@ const useStageRenderer = (
     state.globalMaskLayerOpacity,
     cursorPosition,
     lastMouseDownPos,
+    isMouseOver,
     state.brushSize,
-    renderToolPreview,
+    renderers,
   ]);
 
   useLayoutEffect(() => {
     log.trace('Rendering layers');
-    renderLayers(stageRef.current, state.layers, state.globalMaskLayerOpacity, tool, onLayerPosChanged);
-  }, [stageRef, state.layers, state.globalMaskLayerOpacity, tool, onLayerPosChanged, renderLayers]);
+    renderers.renderLayers(stageRef.current, state.layers, state.globalMaskLayerOpacity, tool, onLayerPosChanged);
+  }, [stageRef, state.layers, state.globalMaskLayerOpacity, tool, onLayerPosChanged, renderers]);
 
   useLayoutEffect(() => {
     log.trace('Rendering bbox');
     if (asPreview) {
+      // Preview should not display bboxes
       return;
     }
-    renderBbox(stageRef.current, state.layers, state.selectedLayerId, tool, onBboxChanged, onBboxMouseDown);
-  }, [stageRef, asPreview, state.layers, state.selectedLayerId, tool, onBboxChanged, onBboxMouseDown, renderBbox]);
+    renderers.renderBbox(stageRef.current, state.layers, state.selectedLayerId, tool, onBboxChanged, onBboxMouseDown);
+  }, [stageRef, asPreview, state.layers, state.selectedLayerId, tool, onBboxChanged, onBboxMouseDown, renderers]);
 
   useLayoutEffect(() => {
     log.trace('Rendering background');
     if (asPreview) {
+      // The preview should not have a background
       return;
     }
-    renderBackground(stageRef.current, width, height);
-  }, [stageRef, asPreview, width, height, renderBackground]);
+    renderers.renderBackground(stageRef.current, width, height);
+  }, [stageRef, asPreview, width, height, renderers]);
+
+  useLayoutEffect(() => {
+    log.trace('Arranging layers');
+    renderers.arrangeLayers(stageRef.current, layerIds);
+  }, [stageRef, layerIds, renderers]);
 };
 
 type Props = {

invokeai/frontend/web/src/features/regionalPrompts/hooks/mouseEventHooks.ts

@@ -15,6 +15,7 @@ import {
 } from 'features/regionalPrompts/store/regionalPromptsSlice';
 import type Konva from 'konva';
 import type { KonvaEventObject } from 'konva/lib/Node';
+import type { Vector2d } from 'konva/lib/types';
 import { useCallback, useRef } from 'react';
 
 const getIsFocused = (stage: Konva.Stage) => {
@@ -23,21 +24,28 @@ const getIsFocused = (stage: Konva.Stage) => {
 
 export const getScaledFlooredCursorPosition = (stage: Konva.Stage) => {
   const pointerPosition = stage.getPointerPosition();
-
   const stageTransform = stage.getAbsoluteTransform().copy();
-
   if (!pointerPosition || !stageTransform) {
     return;
   }
-
   const scaledCursorPosition = stageTransform.invert().point(pointerPosition);
-
   return {
     x: Math.floor(scaledCursorPosition.x),
     y: Math.floor(scaledCursorPosition.y),
   };
 };
 
+const syncCursorPos = (stage: Konva.Stage): Vector2d | null => {
+  const pos = getScaledFlooredCursorPosition(stage);
+  if (!pos) {
+    return null;
+  }
+  $cursorPosition.set(pos);
+  return pos;
+};
+
+const BRUSH_SPACING = 20;
+
 export const useMouseEvents = () => {
   const dispatch = useAppDispatch();
   const selectedLayerId = useAppSelector((s) => s.regionalPrompts.present.selectedLayerId);
@@ -52,7 +60,7 @@ export const useMouseEvents = () => {
       if (!stage) {
         return;
       }
-      const pos = $cursorPosition.get();
+      const pos = syncCursorPos(stage);
       if (!pos) {
         return;
       }
@@ -61,12 +69,11 @@ export const useMouseEvents = () => {
       if (!selectedLayerId) {
         return;
       }
-      // const tool = getTool();
       if (tool === 'brush' || tool === 'eraser') {
         dispatch(
           maskLayerLineAdded({
             layerId: selectedLayerId,
-            points: [Math.floor(pos.x), Math.floor(pos.y), Math.floor(pos.x), Math.floor(pos.y)],
+            points: [pos.x, pos.y, pos.x, pos.y],
             tool,
           })
         );
@@ -109,33 +116,47 @@ export const useMouseEvents = () => {
       if (!stage) {
         return;
       }
-      const pos = getScaledFlooredCursorPosition(stage);
+      const pos = syncCursorPos(stage);
       if (!pos || !selectedLayerId) {
         return;
       }
-      $cursorPosition.set(pos);
       if (getIsFocused(stage) && $isMouseOver.get() && $isMouseDown.get() && (tool === 'brush' || tool === 'eraser')) {
         if (lastCursorPosRef.current) {
-          if (Math.hypot(lastCursorPosRef.current[0] - pos.x, lastCursorPosRef.current[1] - pos.y) < 20) {
+          // Dispatching redux events impacts perf substantially - using brush spacing keeps dispatches to a reasonable number
+          if (Math.hypot(lastCursorPosRef.current[0] - pos.x, lastCursorPosRef.current[1] - pos.y) < BRUSH_SPACING) {
             return;
           }
         }
-        lastCursorPosRef.current = [Math.floor(pos.x), Math.floor(pos.y)];
+        lastCursorPosRef.current = [pos.x, pos.y];
         dispatch(maskLayerPointsAdded({ layerId: selectedLayerId, point: lastCursorPosRef.current }));
       }
     },
     [dispatch, selectedLayerId, tool]
   );
 
-  const onMouseLeave = useCallback((e: KonvaEventObject<MouseEvent | TouchEvent>) => {
-    const stage = e.target.getStage();
-    if (!stage) {
-      return;
-    }
-    $isMouseOver.set(false);
-    $isMouseDown.set(false);
-    $cursorPosition.set(null);
-  }, []);
+  const onMouseLeave = useCallback(
+    (e: KonvaEventObject<MouseEvent | TouchEvent>) => {
+      const stage = e.target.getStage();
+      if (!stage) {
+        return;
+      }
+      const pos = syncCursorPos(stage);
+      if (
+        pos &&
+        selectedLayerId &&
+        getIsFocused(stage) &&
+        $isMouseOver.get() &&
+        $isMouseDown.get() &&
+        (tool === 'brush' || tool === 'eraser')
+      ) {
+        dispatch(maskLayerPointsAdded({ layerId: selectedLayerId, point: [pos.x, pos.y] }));
+      }
+      $isMouseOver.set(false);
+      $isMouseDown.set(false);
+      $cursorPosition.set(null);
+    },
+    [selectedLayerId, tool, dispatch]
+  );
 
   const onMouseEnter = useCallback(
     (e: KonvaEventObject<MouseEvent>) => {
@@ -144,7 +165,7 @@ export const useMouseEvents = () => {
         return;
       }
       $isMouseOver.set(true);
-      const pos = $cursorPosition.get();
+      const pos = syncCursorPos(stage);
       if (!pos) {
         return;
       }
@@ -162,7 +183,7 @@ export const useMouseEvents = () => {
           dispatch(
             maskLayerLineAdded({
               layerId: selectedLayerId,
-              points: [Math.floor(pos.x), Math.floor(pos.y), Math.floor(pos.x), Math.floor(pos.y)],
+              points: [pos.x, pos.y, pos.x, pos.y],
               tool,
             })
           );

invokeai/frontend/web/src/features/regionalPrompts/store/regionalPromptsSlice.ts

@@ -16,7 +16,7 @@ type DrawingTool = 'brush' | 'eraser';
 
 export type Tool = DrawingTool | 'move' | 'rect';
 
-type VectorMaskLine = {
+export type VectorMaskLine = {
   id: string;
   type: 'vector_mask_line';
   tool: DrawingTool;
@@ -24,7 +24,7 @@ type VectorMaskLine = {
   points: number[];
 };
 
-type VectorMaskRect = {
+export type VectorMaskRect = {
   id: string;
   type: 'vector_mask_rect';
   x: number;
@@ -109,7 +109,7 @@ export const regionalPromptsSlice = createSlice({
             y: 0,
             autoNegative: 'invert',
             needsPixelBbox: false,
-            positivePrompt: null,
+            positivePrompt: '',
             negativePrompt: null,
             ipAdapterIds: [],
           };

invokeai/frontend/web/src/features/regionalPrompts/util/getLayerBlobs.ts

@@ -20,7 +20,7 @@ export const getRegionalPromptLayerBlobs = async (
   const reduxLayers = state.regionalPrompts.present.layers;
   const container = document.createElement('div');
   const stage = new Konva.Stage({ container, width: state.generation.width, height: state.generation.height });
-  renderers.layers(stage, reduxLayers, 1, 'brush');
+  renderers.renderLayers(stage, reduxLayers, 1, 'brush');
 
   const konvaLayers = stage.find<Konva.Layer>(`.${VECTOR_MASK_LAYER_NAME}`);
   const blobs: Record<string, Blob> = {};

invokeai/frontend/web/src/features/regionalPrompts/util/renderers.ts

@@ -1,9 +1,14 @@
 import { getStore } from 'app/store/nanostores/store';
 import { rgbaColorToString, rgbColorToString } from 'features/canvas/util/colorToString';
 import { getScaledFlooredCursorPosition } from 'features/regionalPrompts/hooks/mouseEventHooks';
-import type { Layer, Tool, VectorMaskLayer } from 'features/regionalPrompts/store/regionalPromptsSlice';
+import type {
+  Layer,
+  Tool,
+  VectorMaskLayer,
+  VectorMaskLine,
+  VectorMaskRect,
+} from 'features/regionalPrompts/store/regionalPromptsSlice';
 import {
-  $isMouseOver,
   $tool,
   BACKGROUND_LAYER_ID,
   BACKGROUND_RECT_ID,
@@ -35,6 +40,7 @@ const BBOX_NOT_SELECTED_STROKE = 'rgba(255, 255, 255, 0.353)';
 const BBOX_NOT_SELECTED_MOUSEOVER_STROKE = 'rgba(255, 255, 255, 0.661)';
 const BRUSH_BORDER_INNER_COLOR = 'rgba(0,0,0,1)';
 const BRUSH_BORDER_OUTER_COLOR = 'rgba(255,255,255,0.8)';
+// This is invokeai/frontend/web/public/assets/images/transparent_bg.png as a dataURL
 const STAGE_BG_DATAURL =
   'data:image/png;base64,iVBORw0KGgoAAAANSUhEUgAAABQAAAAUCAIAAAAC64paAAAEsmlUWHRYTUw6Y29tLmFkb2JlLnhtcAAAAAAAPD94cGFja2V0IGJlZ2luPSLvu78iIGlkPSJXNU0wTXBDZWhpSHpyZVN6TlRjemtjOWQiPz4KPHg6eG1wbWV0YSB4bWxuczp4PSJhZG9iZTpuczptZXRhLyIgeDp4bXB0az0iWE1QIENvcmUgNS41LjAiPgogPHJkZjpSREYgeG1sbnM6cmRmPSJodHRwOi8vd3d3LnczLm9yZy8xOTk5LzAyLzIyLXJkZi1zeW50YXgtbnMjIj4KICA8cmRmOkRlc2NyaXB0aW9uIHJkZjphYm91dD0iIgogICAgeG1sbnM6ZXhpZj0iaHR0cDovL25zLmFkb2JlLmNvbS9leGlmLzEuMC8iCiAgICB4bWxuczp0aWZmPSJodHRwOi8vbnMuYWRvYmUuY29tL3RpZmYvMS4wLyIKICAgIHhtbG5zOnBob3Rvc2hvcD0iaHR0cDovL25zLmFkb2JlLmNvbS9waG90b3Nob3AvMS4wLyIKICAgIHhtbG5zOnhtcD0iaHR0cDovL25zLmFkb2JlLmNvbS94YXAvMS4wLyIKICAgIHhtbG5zOnhtcE1NPSJodHRwOi8vbnMuYWRvYmUuY29tL3hhcC8xLjAvbW0vIgogICAgeG1sbnM6c3RFdnQ9Imh0dHA6Ly9ucy5hZG9iZS5jb20veGFwLzEuMC9zVHlwZS9SZXNvdXJjZUV2ZW50IyIKICAgZXhpZjpQaXhlbFhEaW1lbnNpb249IjIwIgogICBleGlmOlBpeGVsWURpbWVuc2lvbj0iMjAiCiAgIGV4aWY6Q29sb3JTcGFjZT0iMSIKICAgdGlmZjpJbWFnZVdpZHRoPSIyMCIKICAgdGlmZjpJbWFnZUxlbmd0aD0iMjAiCiAgIHRpZmY6UmVzb2x1dGlvblVuaXQ9IjIiCiAgIHRpZmY6WFJlc29sdXRpb249IjMwMC8xIgogICB0aWZmOllSZXNvbHV0aW9uPSIzMDAvMSIKICAgcGhvdG9zaG9wOkNvbG9yTW9kZT0iMyIKICAgcGhvdG9zaG9wOklDQ1Byb2ZpbGU9InNSR0IgSUVDNjE5NjYtMi4xIgogICB4bXA6TW9kaWZ5RGF0ZT0iMjAyNC0wNC0yM1QwODoyMDo0NysxMDowMCIKICAgeG1wOk1ldGFkYXRhRGF0ZT0iMjAyNC0wNC0yM1QwODoyMDo0NysxMDowMCI+CiAgIDx4bXBNTTpIaXN0b3J5PgogICAgPHJkZjpTZXE+CiAgICAgPHJkZjpsaQogICAgICBzdEV2dDphY3Rpb249InByb2R1Y2VkIgogICAgICBzdEV2dDpzb2Z0d2FyZUFnZW50PSJBZmZpbml0eSBQaG90byAxLjEwLjgiCiAgICAgIHN0RXZ0OndoZW49IjIwMjQtMDQtMjNUMDg6MjA6NDcrMTA6MDAiLz4KICAgIDwvcmRmOlNlcT4KICAgPC94bXBNTTpIaXN0b3J5PgogIDwvcmRmOkRlc2NyaXB0aW9uPgogPC9yZGY6UkRGPgo8L3g6eG1wbWV0YT4KPD94cGFja2V0IGVuZD0iciI/Pn9pdVgAAAGBaUNDUHNSR0IgSUVDNjE5NjYtMi4xAAAokXWR3yuDURjHP5uJmKghFy6WxpVpqMWNMgm1tGbKr5vt3S+1d3t73y3JrXKrKHHj1wV/AbfKtVJESq53TdywXs9rakv2nJ7zfM73nOfpnOeAPZJRVMPhAzWb18NTAffC4pK7oYiDTjpw4YgqhjYeCgWpaR8P2Kx457Vq1T73rzXHE4YCtkbhMUXT88LTwsG1vGbxrnC7ko7Ghc+F+3W5oPC9pcfKXLQ4VeYvi/VIeALsbcLuVBXHqlhJ66qwvByPmikov/exXuJMZOfnJPaId2MQZooAbmaYZAI/g4zK7MfLEAOyoka+7yd/lpzkKjJrrKOzSoo0efpFLUj1hMSk6AkZGdat/v/tq5EcHipXdwag/sU033qhYQdK26b5eWyapROoe4arbCU/dwQj76JvVzTPIbRuwsV1RYvtweUWdD1pUT36I9WJ25NJeD2DlkVw3ULTcrlnv/ucPkJkQ77qBvYPoE/Ot658AxagZ8FoS/a7AAAACXBIWXMAAC4jAAAuIwF4pT92AAAAL0lEQVQ4jWM8ffo0A25gYmKCR5YJjxxBMKp5ZGhm/P//Px7pM2fO0MrmUc0jQzMAB2EIhZC3pUYAAAAASUVORK5CYII=';
 
@@ -51,6 +57,68 @@ const selectVectorMaskObjects = (node: Konva.Node) => {
   return node.name() === VECTOR_MASK_LAYER_LINE_NAME || node.name() === VECTOR_MASK_LAYER_RECT_NAME;
 };
 
+/**
+ * Creates the brush preview layer.
+ * @param stage The konva stage to render on.
+ * @returns The brush preview layer.
+ */
+const createToolPreviewLayer = (stage: Konva.Stage) => {
+  // Initialize the brush preview layer & add to the stage
+  const toolPreviewLayer = new Konva.Layer({ id: TOOL_PREVIEW_LAYER_ID, visible: false, listening: false });
+  stage.add(toolPreviewLayer);
+
+  // Add handlers to show/hide the brush preview layer
+  stage.on('mousemove', (e) => {
+    const tool = $tool.get();
+    e.target
+      .getStage()
+      ?.findOne<Konva.Layer>(`#${TOOL_PREVIEW_LAYER_ID}`)
+      ?.visible(tool === 'brush' || tool === 'eraser');
+  });
+  stage.on('mouseleave', (e) => {
+    e.target.getStage()?.findOne<Konva.Layer>(`#${TOOL_PREVIEW_LAYER_ID}`)?.visible(false);
+  });
+  stage.on('mouseenter', (e) => {
+    const tool = $tool.get();
+    e.target
+      .getStage()
+      ?.findOne<Konva.Layer>(`#${TOOL_PREVIEW_LAYER_ID}`)
+      ?.visible(tool === 'brush' || tool === 'eraser');
+  });
+
+  // Create the brush preview group & circles
+  const brushPreviewGroup = new Konva.Group({ id: TOOL_PREVIEW_BRUSH_GROUP_ID });
+  const brushPreviewFill = new Konva.Circle({
+    id: TOOL_PREVIEW_BRUSH_FILL_ID,
+    listening: false,
+    strokeEnabled: false,
+  });
+  brushPreviewGroup.add(brushPreviewFill);
+  const brushPreviewBorderInner = new Konva.Circle({
+    id: TOOL_PREVIEW_BRUSH_BORDER_INNER_ID,
+    listening: false,
+    stroke: BRUSH_BORDER_INNER_COLOR,
+    strokeWidth: 1,
+    strokeEnabled: true,
+  });
+  brushPreviewGroup.add(brushPreviewBorderInner);
+  const brushPreviewBorderOuter = new Konva.Circle({
+    id: TOOL_PREVIEW_BRUSH_BORDER_OUTER_ID,
+    listening: false,
+    stroke: BRUSH_BORDER_OUTER_COLOR,
+    strokeWidth: 1,
+    strokeEnabled: true,
+  });
+  brushPreviewGroup.add(brushPreviewBorderOuter);
+  toolPreviewLayer.add(brushPreviewGroup);
+
+  // Create the rect preview
+  const rectPreview = new Konva.Rect({ id: TOOL_PREVIEW_RECT_ID, listening: false, stroke: 'white', strokeWidth: 1 });
+  toolPreviewLayer.add(rectPreview);
+
+  return toolPreviewLayer;
+};
+
 /**
  * Renders the brush preview for the selected tool.
  * @param stage The konva stage to render on.
@@ -60,13 +128,14 @@ const selectVectorMaskObjects = (node: Konva.Node) => {
  * @param lastMouseDownPos The position of the last mouse down event - used for the rect tool.
  * @param brushSize The brush size.
  */
-const toolPreview = (
+const renderToolPreview = (
   stage: Konva.Stage,
   tool: Tool,
   color: RgbColor | null,
   globalMaskLayerOpacity: number,
   cursorPos: Vector2d | null,
   lastMouseDownPos: Vector2d | null,
+  isMouseOver: boolean,
   brushSize: number
 ) => {
   const layerCount = stage.find(`.${VECTOR_MASK_LAYER_NAME}`).length;
@@ -85,65 +154,9 @@ const toolPreview = (
     stage.container().style.cursor = 'none';
   }
 
-  let toolPreviewLayer = stage.findOne<Konva.Layer>(`#${TOOL_PREVIEW_LAYER_ID}`);
+  const toolPreviewLayer = stage.findOne<Konva.Layer>(`#${TOOL_PREVIEW_LAYER_ID}`) ?? createToolPreviewLayer(stage);
 
-  // Create the layer if it doesn't exist
-  if (!toolPreviewLayer) {
-    // Initialize the brush preview layer & add to the stage
-    toolPreviewLayer = new Konva.Layer({ id: TOOL_PREVIEW_LAYER_ID, visible: tool !== 'move', listening: false });
-    stage.add(toolPreviewLayer);
-
-    // Add handlers to show/hide the brush preview layer
-    stage.on('mousemove', (e) => {
-      const tool = $tool.get();
-      e.target
-        .getStage()
-        ?.findOne<Konva.Layer>(`#${TOOL_PREVIEW_LAYER_ID}`)
-        ?.visible(tool === 'brush' || tool === 'eraser');
-    });
-    stage.on('mouseleave', (e) => {
-      e.target.getStage()?.findOne<Konva.Layer>(`#${TOOL_PREVIEW_LAYER_ID}`)?.visible(false);
-    });
-    stage.on('mouseenter', (e) => {
-      const tool = $tool.get();
-      e.target
-        .getStage()
-        ?.findOne<Konva.Layer>(`#${TOOL_PREVIEW_LAYER_ID}`)
-        ?.visible(tool === 'brush' || tool === 'eraser');
-    });
-
-    // Create the brush preview group & circles
-    const brushPreviewGroup = new Konva.Group({ id: TOOL_PREVIEW_BRUSH_GROUP_ID });
-    const brushPreviewFill = new Konva.Circle({
-      id: TOOL_PREVIEW_BRUSH_FILL_ID,
-      listening: false,
-      strokeEnabled: false,
-    });
-    brushPreviewGroup.add(brushPreviewFill);
-    const brushPreviewBorderInner = new Konva.Circle({
-      id: TOOL_PREVIEW_BRUSH_BORDER_INNER_ID,
-      listening: false,
-      stroke: BRUSH_BORDER_INNER_COLOR,
-      strokeWidth: 1,
-      strokeEnabled: true,
-    });
-    brushPreviewGroup.add(brushPreviewBorderInner);
-    const brushPreviewBorderOuter = new Konva.Circle({
-      id: TOOL_PREVIEW_BRUSH_BORDER_OUTER_ID,
-      listening: false,
-      stroke: BRUSH_BORDER_OUTER_COLOR,
-      strokeWidth: 1,
-      strokeEnabled: true,
-    });
-    brushPreviewGroup.add(brushPreviewBorderOuter);
-    toolPreviewLayer.add(brushPreviewGroup);
-
-    // Create the rect preview
-    const rectPreview = new Konva.Rect({ id: TOOL_PREVIEW_RECT_ID, listening: false, stroke: 'white', strokeWidth: 1 });
-    toolPreviewLayer.add(rectPreview);
-  }
-
-  if (!$isMouseOver.get() || layerCount === 0) {
+  if (!isMouseOver || layerCount === 0) {
     // We can bail early if the mouse isn't over the stage or there are no layers
     toolPreviewLayer.visible(false);
     return;
@@ -200,85 +213,140 @@ const toolPreview = (
   }
 };
 
-const vectorMaskLayer = (
+/**
+ * Creates a vector mask layer.
+ * @param stage The konva stage to attach the layer to.
+ * @param reduxLayer The redux layer to create the konva layer from.
+ * @param onLayerPosChanged Callback for when the layer's position changes.
+ */
+const createVectorMaskLayer = (
   stage: Konva.Stage,
-  vmLayer: VectorMaskLayer,
-  vmLayerIndex: number,
+  reduxLayer: VectorMaskLayer,
+  onLayerPosChanged?: (layerId: string, x: number, y: number) => void
+) => {
+  // This layer hasn't been added to the konva state yet
+  const konvaLayer = new Konva.Layer({
+    id: reduxLayer.id,
+    name: VECTOR_MASK_LAYER_NAME,
+    draggable: true,
+    dragDistance: 0,
+  });
+
+  // Create a `dragmove` listener for this layer
+  if (onLayerPosChanged) {
+    konvaLayer.on('dragend', function (e) {
+      onLayerPosChanged(reduxLayer.id, Math.floor(e.target.x()), Math.floor(e.target.y()));
+    });
+  }
+
+  // The dragBoundFunc limits how far the layer can be dragged
+  konvaLayer.dragBoundFunc(function (pos) {
+    const cursorPos = getScaledFlooredCursorPosition(stage);
+    if (!cursorPos) {
+      return this.getAbsolutePosition();
+    }
+    // Prevent the user from dragging the layer out of the stage bounds.
+    if (
+      cursorPos.x < 0 ||
+      cursorPos.x > stage.width() / stage.scaleX() ||
+      cursorPos.y < 0 ||
+      cursorPos.y > stage.height() / stage.scaleY()
+    ) {
+      return this.getAbsolutePosition();
+    }
+    return pos;
+  });
+
+  // The object group holds all of the layer's objects (e.g. lines and rects)
+  const konvaObjectGroup = new Konva.Group({
+    id: getVectorMaskLayerObjectGroupId(reduxLayer.id, uuidv4()),
+    name: VECTOR_MASK_LAYER_OBJECT_GROUP_NAME,
+    listening: false,
+  });
+  konvaLayer.add(konvaObjectGroup);
+
+  stage.add(konvaLayer);
+
+  return konvaLayer;
+};
+
+/**
+ * Creates a konva line from a redux vector mask line.
+ * @param reduxObject The redux object to create the konva line from.
+ * @param konvaGroup The konva group to add the line to.
+ */
+const createVectorMaskLine = (reduxObject: VectorMaskLine, konvaGroup: Konva.Group): Konva.Line => {
+  const vectorMaskLine = new Konva.Line({
+    id: reduxObject.id,
+    key: reduxObject.id,
+    name: VECTOR_MASK_LAYER_LINE_NAME,
+    strokeWidth: reduxObject.strokeWidth,
+    tension: 0,
+    lineCap: 'round',
+    lineJoin: 'round',
+    shadowForStrokeEnabled: false,
+    globalCompositeOperation: reduxObject.tool === 'brush' ? 'source-over' : 'destination-out',
+    listening: false,
+  });
+  konvaGroup.add(vectorMaskLine);
+  return vectorMaskLine;
+};
+
+/**
+ * Creates a konva rect from a redux vector mask rect.
+ * @param reduxObject The redux object to create the konva rect from.
+ * @param konvaGroup The konva group to add the rect to.
+ */
+const createVectorMaskRect = (reduxObject: VectorMaskRect, konvaGroup: Konva.Group): Konva.Rect => {
+  const vectorMaskRect = new Konva.Rect({
+    id: reduxObject.id,
+    key: reduxObject.id,
+    name: VECTOR_MASK_LAYER_RECT_NAME,
+    x: reduxObject.x,
+    y: reduxObject.y,
+    width: reduxObject.width,
+    height: reduxObject.height,
+    listening: false,
+  });
+  konvaGroup.add(vectorMaskRect);
+  return vectorMaskRect;
+};
+
+/**
+ * Renders a vector mask layer.
+ * @param stage The konva stage to render on.
+ * @param reduxLayer The redux vector mask layer to render.
+ * @param reduxLayerIndex The index of the layer in the redux store.
+ * @param globalMaskLayerOpacity The opacity of the global mask layer.
+ * @param tool The current tool.
+ */
+const renderVectorMaskLayer = (
+  stage: Konva.Stage,
+  reduxLayer: VectorMaskLayer,
   globalMaskLayerOpacity: number,
   tool: Tool,
   onLayerPosChanged?: (layerId: string, x: number, y: number) => void
-) => {
-  let konvaLayer = stage.findOne<Konva.Layer>(`#${vmLayer.id}`);
-
-  if (!konvaLayer) {
-    // This layer hasn't been added to the konva state yet
-    konvaLayer = new Konva.Layer({
-      id: vmLayer.id,
-      name: VECTOR_MASK_LAYER_NAME,
-      draggable: true,
-      dragDistance: 0,
-    });
-
-    // Create a `dragmove` listener for this layer
-    if (onLayerPosChanged) {
-      konvaLayer.on('dragend', function (e) {
-        onLayerPosChanged(vmLayer.id, Math.floor(e.target.x()), Math.floor(e.target.y()));
-      });
-    }
-
-    // The dragBoundFunc limits how far the layer can be dragged
-    konvaLayer.dragBoundFunc(function (pos) {
-      const cursorPos = getScaledFlooredCursorPosition(stage);
-      if (!cursorPos) {
-        return this.getAbsolutePosition();
-      }
-      // Prevent the user from dragging the layer out of the stage bounds.
-      if (
-        cursorPos.x < 0 ||
-        cursorPos.x > stage.width() / stage.scaleX() ||
-        cursorPos.y < 0 ||
-        cursorPos.y > stage.height() / stage.scaleY()
-      ) {
-        return this.getAbsolutePosition();
-      }
-      return pos;
-    });
-
-    // The object group holds all of the layer's objects (e.g. lines and rects)
-    const konvaObjectGroup = new Konva.Group({
-      id: getVectorMaskLayerObjectGroupId(vmLayer.id, uuidv4()),
-      name: VECTOR_MASK_LAYER_OBJECT_GROUP_NAME,
-      listening: false,
-    });
-    konvaLayer.add(konvaObjectGroup);
-
-    stage.add(konvaLayer);
-
-    // When a layer is added, it ends up on top of the brush preview - we need to move the preview back to the top.
-    stage.findOne<Konva.Layer>(`#${TOOL_PREVIEW_LAYER_ID}`)?.moveToTop();
-  }
+): void => {
+  const konvaLayer =
+    stage.findOne<Konva.Layer>(`#${reduxLayer.id}`) ?? createVectorMaskLayer(stage, reduxLayer, onLayerPosChanged);
 
   // Update the layer's position and listening state
   konvaLayer.setAttrs({
     listening: tool === 'move', // The layer only listens when using the move tool - otherwise the stage is handling mouse events
-    x: Math.floor(vmLayer.x),
-    y: Math.floor(vmLayer.y),
-    // We have a konva layer for each redux layer, plus a brush preview layer, which should always be on top. We can
-    // therefore use the index of the redux layer as the zIndex for konva layers. If more layers are added to the
-    // stage, this may no longer be work.
-    zIndex: vmLayerIndex,
+    x: Math.floor(reduxLayer.x),
+    y: Math.floor(reduxLayer.y),
   });
 
   // Convert the color to a string, stripping the alpha - the object group will handle opacity.
-  const rgbColor = rgbColorToString(vmLayer.previewColor);
+  const rgbColor = rgbColorToString(reduxLayer.previewColor);
 
   const konvaObjectGroup = konvaLayer.findOne<Konva.Group>(`.${VECTOR_MASK_LAYER_OBJECT_GROUP_NAME}`);
-  assert(konvaObjectGroup, `Object group not found for layer ${vmLayer.id}`);
+  assert(konvaObjectGroup, `Object group not found for layer ${reduxLayer.id}`);
 
   // We use caching to handle "global" layer opacity, but caching is expensive and we should only do it when required.
   let groupNeedsCache = false;
 
-  const objectIds = vmLayer.objects.map(mapId);
+  const objectIds = reduxLayer.objects.map(mapId);
   for (const objectNode of konvaObjectGroup.find(selectVectorMaskObjects)) {
     if (!objectIds.includes(objectNode.id())) {
       objectNode.destroy();
@@ -286,26 +354,10 @@ const vectorMaskLayer = (
     }
   }
 
-  for (const reduxObject of vmLayer.objects) {
+  for (const reduxObject of reduxLayer.objects) {
     if (reduxObject.type === 'vector_mask_line') {
-      let vectorMaskLine = stage.findOne<Konva.Line>(`#${reduxObject.id}`);
-
-      // Create the line if it doesn't exist
-      if (!vectorMaskLine) {
-        vectorMaskLine = new Konva.Line({
-          id: reduxObject.id,
-          key: reduxObject.id,
-          name: VECTOR_MASK_LAYER_LINE_NAME,
-          strokeWidth: reduxObject.strokeWidth,
-          tension: 0,
-          lineCap: 'round',
-          lineJoin: 'round',
-          shadowForStrokeEnabled: false,
-          globalCompositeOperation: reduxObject.tool === 'brush' ? 'source-over' : 'destination-out',
-          listening: false,
-        });
-        konvaObjectGroup.add(vectorMaskLine);
-      }
+      const vectorMaskLine =
+        stage.findOne<Konva.Line>(`#${reduxObject.id}`) ?? createVectorMaskLine(reduxObject, konvaObjectGroup);
 
       // Only update the points if they have changed. The point values are never mutated, they are only added to the
       // array, so checking the length is sufficient to determine if we need to re-cache.
@@ -319,20 +371,9 @@ const vectorMaskLayer = (
         groupNeedsCache = true;
       }
     } else if (reduxObject.type === 'vector_mask_rect') {
-      let konvaObject = stage.findOne<Konva.Rect>(`#${reduxObject.id}`);
-      if (!konvaObject) {
-        konvaObject = new Konva.Rect({
-          id: reduxObject.id,
-          key: reduxObject.id,
-          name: VECTOR_MASK_LAYER_RECT_NAME,
-          x: reduxObject.x,
-          y: reduxObject.y,
-          width: reduxObject.width,
-          height: reduxObject.height,
-          listening: false,
-        });
-        konvaObjectGroup.add(konvaObject);
-      }
+      const konvaObject =
+        stage.findOne<Konva.Rect>(`#${reduxObject.id}`) ?? createVectorMaskRect(reduxObject, konvaObjectGroup);
+
       // Only update the color if it has changed.
       if (konvaObject.fill() !== rgbColor) {
         konvaObject.fill(rgbColor);
@@ -342,20 +383,16 @@ const vectorMaskLayer = (
   }
 
   // Only update layer visibility if it has changed.
-  if (konvaLayer.visible() !== vmLayer.isVisible) {
-    konvaLayer.visible(vmLayer.isVisible);
+  if (konvaLayer.visible() !== reduxLayer.isVisible) {
+    konvaLayer.visible(reduxLayer.isVisible);
     groupNeedsCache = true;
   }
 
-  if (konvaObjectGroup.children.length > 0) {
-    // If we have objects, we need to cache the group to apply the layer opacity...
-    if (groupNeedsCache) {
-      // ...but only if we've done something that needs the cache.
-      konvaObjectGroup.cache();
-    }
-  } else {
-    // No children - clear the cache to reset the previous pixel data
+  if (konvaObjectGroup.children.length === 0) {
+    // No objects - clear the cache to reset the previous pixel data
     konvaObjectGroup.clearCache();
+  } else if (groupNeedsCache) {
+    konvaObjectGroup.cache();
   }
 
   // Updating group opacity does not require re-caching
@@ -372,7 +409,7 @@ const vectorMaskLayer = (
  * @param onLayerPosChanged Callback for when the layer's position changes. This is optional to allow for offscreen rendering.
  * @returns
  */
-const layers = (
+const renderLayers = (
   stage: Konva.Stage,
   reduxLayers: Layer[],
   globalMaskLayerOpacity: number,
@@ -388,24 +425,57 @@ const layers = (
     }
   }
 
-  for (let layerIndex = 0; layerIndex < reduxLayers.length; layerIndex++) {
-    const reduxLayer = reduxLayers[layerIndex];
-    assert(reduxLayer, `Layer at index ${layerIndex} is undefined`);
+  for (const reduxLayer of reduxLayers) {
     if (isVectorMaskLayer(reduxLayer)) {
-      vectorMaskLayer(stage, reduxLayer, layerIndex, globalMaskLayerOpacity, tool, onLayerPosChanged);
+      renderVectorMaskLayer(stage, reduxLayer, globalMaskLayerOpacity, tool, onLayerPosChanged);
     }
   }
 };
 
 /**
- *
- * @param stage The konva stage to render on.
- * @param tool The current tool.
- * @param selectedLayerIdId The currently selected layer id.
- * @param onBboxChanged A callback to be called when the bounding box changes.
+ * Creates a bounding box rect for a layer.
+ * @param reduxLayer The redux layer to create the bounding box for.
+ * @param konvaLayer The konva layer to attach the bounding box to.
+ * @param onBboxMouseDown Callback for when the bounding box is clicked.
+ */
+const createBboxRect = (reduxLayer: Layer, konvaLayer: Konva.Layer, onBboxMouseDown: (layerId: string) => void) => {
+  const rect = new Konva.Rect({
+    id: getLayerBboxId(reduxLayer.id),
+    name: LAYER_BBOX_NAME,
+    strokeWidth: 1,
+  });
+  rect.on('mousedown', function () {
+    onBboxMouseDown(reduxLayer.id);
+  });
+  rect.on('mouseover', function (e) {
+    if (getIsSelected(e.target.getLayer()?.id())) {
+      this.stroke(BBOX_SELECTED_STROKE);
+    } else {
+      this.stroke(BBOX_NOT_SELECTED_MOUSEOVER_STROKE);
+    }
+  });
+  rect.on('mouseout', function (e) {
+    if (getIsSelected(e.target.getLayer()?.id())) {
+      this.stroke(BBOX_SELECTED_STROKE);
+    } else {
+      this.stroke(BBOX_NOT_SELECTED_STROKE);
+    }
+  });
+  konvaLayer.add(rect);
+  return rect;
+};
+
+/**
+ * Renders the bounding boxes for the layers.
+ * @param stage The konva stage to render on
+ * @param reduxLayers An array of all redux layers to draw bboxes for
+ * @param selectedLayerId The selected layer's id
+ * @param tool The current tool
+ * @param onBboxChanged Callback for when the bbox is changed
+ * @param onBboxMouseDown Callback for when the bbox is clicked
  * @returns
  */
-const bbox = (
+const renderBbox = (
   stage: Konva.Stage,
   reduxLayers: Layer[],
   selectedLayerId: string | null,
@@ -433,7 +503,6 @@ const bbox = (
     if (reduxLayer.bboxNeedsUpdate && reduxLayer.objects.length) {
       // We only need to use the pixel-perfect bounding box if the layer has eraser strokes
       bbox = reduxLayer.needsPixelBbox ? getLayerBboxPixels(konvaLayer) : getLayerBboxFast(konvaLayer);
-
       // Update the layer's bbox in the redux store
       onBboxChanged(reduxLayer.id, bbox);
     }
@@ -442,32 +511,8 @@ const bbox = (
       continue;
     }
 
-    let rect = konvaLayer.findOne<Konva.Rect>(`.${LAYER_BBOX_NAME}`);
-    if (!rect) {
-      rect = new Konva.Rect({
-        id: getLayerBboxId(reduxLayer.id),
-        name: LAYER_BBOX_NAME,
-        strokeWidth: 1,
-      });
-      rect.on('mousedown', function () {
-        onBboxMouseDown(reduxLayer.id);
-      });
-      rect.on('mouseover', function (e) {
-        if (getIsSelected(e.target.getLayer()?.id())) {
-          this.stroke(BBOX_SELECTED_STROKE);
-        } else {
-          this.stroke(BBOX_NOT_SELECTED_MOUSEOVER_STROKE);
-        }
-      });
-      rect.on('mouseout', function (e) {
-        if (getIsSelected(e.target.getLayer()?.id())) {
-          this.stroke(BBOX_SELECTED_STROKE);
-        } else {
-          this.stroke(BBOX_NOT_SELECTED_STROKE);
-        }
-      });
-      konvaLayer.add(rect);
-    }
+    const rect =
+      konvaLayer.findOne<Konva.Rect>(`.${LAYER_BBOX_NAME}`) ?? createBboxRect(reduxLayer, konvaLayer, onBboxMouseDown);
 
     rect.setAttrs({
       visible: true,
@@ -481,31 +526,41 @@ const bbox = (
   }
 };
 
-const background = (stage: Konva.Stage, width: number, height: number) => {
-  let layer = stage.findOne<Konva.Layer>(`#${BACKGROUND_LAYER_ID}`);
+/**
+ * Creates the background layer for the stage.
+ * @param stage The konva stage to render on
+ */
+const createBackgroundLayer = (stage: Konva.Stage): Konva.Layer => {
+  const layer = new Konva.Layer({
+    id: BACKGROUND_LAYER_ID,
+  });
+  const background = new Konva.Rect({
+    id: BACKGROUND_RECT_ID,
+    x: stage.x(),
+    y: 0,
+    width: stage.width() / stage.scaleX(),
+    height: stage.height() / stage.scaleY(),
+    listening: false,
+    opacity: 0.2,
+  });
+  layer.add(background);
+  stage.add(layer);
+  const image = new Image();
+  image.onload = () => {
+    background.fillPatternImage(image);
+  };
+  image.src = STAGE_BG_DATAURL;
+  return layer;
+};
 
-  if (!layer) {
-    layer = new Konva.Layer({
-      id: BACKGROUND_LAYER_ID,
-    });
-    const background = new Konva.Rect({
-      id: BACKGROUND_RECT_ID,
-      x: stage.x(),
-      y: 0,
-      width: stage.width() / stage.scaleX(),
-      height: stage.height() / stage.scaleY(),
-      listening: false,
-      opacity: 0.2,
-    });
-    layer.add(background);
-    stage.add(layer);
-    const image = new Image();
-    image.onload = () => {
-      background.fillPatternImage(image);
-    };
-    // This is invokeai/frontend/web/public/assets/images/transparent_bg.png as a dataURL
-    image.src = STAGE_BG_DATAURL;
-  }
+/**
+ * Renders the background layer for the stage.
+ * @param stage The konva stage to render on
+ * @param width The unscaled width of the canvas
+ * @param height The unscaled height of the canvas
+ */
+const renderBackground = (stage: Konva.Stage, width: number, height: number) => {
+  const layer = stage.findOne<Konva.Layer>(`#${BACKGROUND_LAYER_ID}`) ?? createBackgroundLayer(stage);
 
   const background = layer.findOne<Konva.Rect>(`#${BACKGROUND_RECT_ID}`);
   assert(background, 'Background rect not found');
@@ -528,15 +583,37 @@ const background = (stage: Konva.Stage, width: number, height: number) => {
   background.fillPatternOffset(stagePos);
 };
 
-const DEBOUNCE_MS = 300;
+/**
+ * Arranges all layers in the z-axis by updating their z-indices.
+ * @param stage The konva stage
+ * @param layerIds An array of redux layer ids, in their z-index order
+ */
+const arrangeLayers = (stage: Konva.Stage, layerIds: string[]): void => {
+  let nextZIndex = 0;
+  // Background is the first layer
+  stage.findOne<Konva.Layer>(`#${BACKGROUND_LAYER_ID}`)?.zIndex(nextZIndex++);
+  // Then arrange the redux layers in order
+  for (const layerId of layerIds) {
+    stage.findOne<Konva.Layer>(`#${layerId}`)?.zIndex(nextZIndex++);
+  }
+  // Finally, the tool preview layer is always on top
+  stage.findOne<Konva.Layer>(`#${TOOL_PREVIEW_LAYER_ID}`)?.zIndex(nextZIndex++);
+};
 
 export const renderers = {
-  toolPreview,
-  toolPreviewDebounced: debounce(toolPreview, DEBOUNCE_MS),
-  layers,
-  layersDebounced: debounce(layers, DEBOUNCE_MS),
-  bbox,
-  bboxDebounced: debounce(bbox, DEBOUNCE_MS),
-  background,
-  backgroundDebounced: debounce(background, DEBOUNCE_MS),
+  renderToolPreview,
+  renderLayers,
+  renderBbox,
+  renderBackground,
+  arrangeLayers,
+};
+
+const DEBOUNCE_MS = 300;
+
+export const debouncedRenderers = {
+  renderToolPreview: debounce(renderToolPreview, DEBOUNCE_MS),
+  renderLayers: debounce(renderLayers, DEBOUNCE_MS),
+  renderBbox: debounce(renderBbox, DEBOUNCE_MS),
+  renderBackground: debounce(renderBackground, DEBOUNCE_MS),
+  arrangeLayers: debounce(arrangeLayers, DEBOUNCE_MS),
 };

invokeai/frontend/web/src/features/settingsAccordions/components/ImageSettingsAccordion/ImageSizeCanvas.tsx

@@ -2,6 +2,7 @@ import { useAppDispatch, useAppSelector } from 'app/store/storeHooks';
 import { aspectRatioChanged, setBoundingBoxDimensions } from 'features/canvas/store/canvasSlice';
 import ParamBoundingBoxHeight from 'features/parameters/components/Canvas/BoundingBox/ParamBoundingBoxHeight';
 import ParamBoundingBoxWidth from 'features/parameters/components/Canvas/BoundingBox/ParamBoundingBoxWidth';
+import { AspectRatioIconPreview } from 'features/parameters/components/ImageSize/AspectRatioIconPreview';
 import { ImageSize } from 'features/parameters/components/ImageSize/ImageSize';
 import type { AspectRatioState } from 'features/parameters/components/ImageSize/types';
 import { selectOptimalDimension } from 'features/parameters/store/generationSlice';
@@ -41,6 +42,7 @@ export const ImageSizeCanvas = memo(() => {
       aspectRatioState={aspectRatioState}
       heightComponent={<ParamBoundingBoxHeight />}
       widthComponent={<ParamBoundingBoxWidth />}
+      previewComponent={<AspectRatioIconPreview />}
       onChangeAspectRatioState={onChangeAspectRatioState}
       onChangeWidth={onChangeWidth}
       onChangeHeight={onChangeHeight}

invokeai/frontend/web/src/features/settingsAccordions/components/ImageSettingsAccordion/ImageSizeLinear.tsx

@@ -1,13 +1,17 @@
 import { useAppDispatch, useAppSelector } from 'app/store/storeHooks';
 import { ParamHeight } from 'features/parameters/components/Core/ParamHeight';
 import { ParamWidth } from 'features/parameters/components/Core/ParamWidth';
+import { AspectRatioCanvasPreview } from 'features/parameters/components/ImageSize/AspectRatioCanvasPreview';
+import { AspectRatioIconPreview } from 'features/parameters/components/ImageSize/AspectRatioIconPreview';
 import { ImageSize } from 'features/parameters/components/ImageSize/ImageSize';
 import type { AspectRatioState } from 'features/parameters/components/ImageSize/types';
 import { aspectRatioChanged, heightChanged, widthChanged } from 'features/parameters/store/generationSlice';
+import { activeTabNameSelector } from 'features/ui/store/uiSelectors';
 import { memo, useCallback } from 'react';
 
 export const ImageSizeLinear = memo(() => {
   const dispatch = useAppDispatch();
+  const tab = useAppSelector(activeTabNameSelector);
   const width = useAppSelector((s) => s.generation.width);
   const height = useAppSelector((s) => s.generation.height);
   const aspectRatioState = useAppSelector((s) => s.generation.aspectRatio);
@@ -40,6 +44,7 @@ export const ImageSizeLinear = memo(() => {
       aspectRatioState={aspectRatioState}
       heightComponent={<ParamHeight />}
       widthComponent={<ParamWidth />}
+      previewComponent={tab === 'txt2img' ? <AspectRatioCanvasPreview /> : <AspectRatioIconPreview />}
       onChangeAspectRatioState={onChangeAspectRatioState}
       onChangeWidth={onChangeWidth}
       onChangeHeight={onChangeHeight}

invokeai/version/invokeai_version.py

@@ -1 +1 @@
-__version__ = "4.2.0a2"
+__version__ = "4.2.0a3"

scripts/invokeai-web.py

@@ -7,10 +7,12 @@ import os
 
 from invokeai.app.run_app import run_app
 
-logging.getLogger("xformers").addFilter(lambda record: "A matching Triton is not available" not in record.getMessage())
-
 
 def main():
+    logging.getLogger("xformers").addFilter(
+        lambda record: "A matching Triton is not available" not in record.getMessage()
+    )
+
     # Change working directory to the repo root
     os.chdir(os.path.abspath(os.path.join(os.path.dirname(__file__), "..")))
     run_app()

tests/app/util/test_controlnet_utils.py

@@ -3,6 +3,7 @@ import pytest
 from PIL import Image
 
 from invokeai.app.util.controlnet_utils import prepare_control_image
+from invokeai.backend.image_util.util import nms
 
 
 @pytest.mark.parametrize("num_channels", [1, 2, 3])
@@ -40,3 +41,10 @@ def test_prepare_control_image_num_channels_too_large(num_channels):
             device="cpu",
             do_classifier_free_guidance=False,
         )
+
+
+@pytest.mark.parametrize("threshold,sigma", [(None, 1.0), (1, None)])
+def test_nms_invalid_options(threshold: None | int, sigma: None | float):
+    """Test that an exception is raised in nms(...) if only one of the `threshold` or `sigma` parameters are provided."""
+    with pytest.raises(ValueError):
+        nms(np.zeros((256, 256, 3), dtype=np.uint8), threshold, sigma)

tests/test_model_probe.py

@@ -4,7 +4,7 @@ import pytest
 from torch import tensor
 
 from invokeai.backend.model_manager import BaseModelType, ModelRepoVariant
-from invokeai.backend.model_manager.config import InvalidModelConfigException
+from invokeai.backend.model_manager.config import InvalidModelConfigException, MainDiffusersConfig, ModelVariantType
 from invokeai.backend.model_manager.probe import (
     CkptType,
     ModelProbe,
@@ -78,3 +78,11 @@ def test_probe_handles_state_dict_with_integer_keys():
     }
     with pytest.raises(InvalidModelConfigException):
         ModelProbe.get_model_type_from_checkpoint(Path("embedding.pt"), state_dict_with_integer_keys)
+
+
+def test_probe_sd1_diffusers_inpainting(datadir: Path):
+    config = ModelProbe.probe(datadir / "sd-1/main/dreamshaper-8-inpainting")
+    assert isinstance(config, MainDiffusersConfig)
+    assert config.base is BaseModelType.StableDiffusion1
+    assert config.variant is ModelVariantType.Inpaint
+    assert config.repo_variant is ModelRepoVariant.FP16

tests/test_model_probe/sd-1/main/dreamshaper-8-inpainting/README

@@ -0,0 +1 @@
+This folder contains config files copied from [Lykon/dreamshaper-8-inpainting](https://huggingface.co/Lykon/dreamshaper-8-inpainting).

tests/test_model_probe/sd-1/main/dreamshaper-8-inpainting/model_index.json

@@ -0,0 +1,34 @@
+{
+  "_class_name": "StableDiffusionInpaintPipeline",
+  "_diffusers_version": "0.21.0.dev0",
+  "_name_or_path": "lykon-models/dreamshaper-8-inpainting",
+  "feature_extractor": [
+    "transformers",
+    "CLIPFeatureExtractor"
+  ],
+  "requires_safety_checker": true,
+  "safety_checker": [
+    "stable_diffusion",
+    "StableDiffusionSafetyChecker"
+  ],
+  "scheduler": [
+    "diffusers",
+    "DEISMultistepScheduler"
+  ],
+  "text_encoder": [
+    "transformers",
+    "CLIPTextModel"
+  ],
+  "tokenizer": [
+    "transformers",
+    "CLIPTokenizer"
+  ],
+  "unet": [
+    "diffusers",
+    "UNet2DConditionModel"
+  ],
+  "vae": [
+    "diffusers",
+    "AutoencoderKL"
+  ]
+}

tests/test_model_probe/sd-1/main/dreamshaper-8-inpainting/scheduler/scheduler_config.json

@@ -0,0 +1,23 @@
+{
+  "_class_name": "DEISMultistepScheduler",
+  "_diffusers_version": "0.21.0.dev0",
+  "algorithm_type": "deis",
+  "beta_end": 0.012,
+  "beta_schedule": "scaled_linear",
+  "beta_start": 0.00085,
+  "clip_sample": false,
+  "dynamic_thresholding_ratio": 0.995,
+  "lower_order_final": true,
+  "num_train_timesteps": 1000,
+  "prediction_type": "epsilon",
+  "sample_max_value": 1.0,
+  "set_alpha_to_one": false,
+  "skip_prk_steps": true,
+  "solver_order": 2,
+  "solver_type": "logrho",
+  "steps_offset": 1,
+  "thresholding": false,
+  "timestep_spacing": "leading",
+  "trained_betas": null,
+  "use_karras_sigmas": false
+}

tests/test_model_probe/sd-1/main/dreamshaper-8-inpainting/unet/config.json

@@ -0,0 +1,66 @@
+{
+  "_class_name": "UNet2DConditionModel",
+  "_diffusers_version": "0.21.0.dev0",
+  "_name_or_path": "/home/patrick/.cache/huggingface/hub/models--lykon-models--dreamshaper-8-inpainting/snapshots/15dcb9dec91a39ee498e3917c9ef6174b103862d/unet",
+  "act_fn": "silu",
+  "addition_embed_type": null,
+  "addition_embed_type_num_heads": 64,
+  "addition_time_embed_dim": null,
+  "attention_head_dim": 8,
+  "attention_type": "default",
+  "block_out_channels": [
+    320,
+    640,
+    1280,
+    1280
+  ],
+  "center_input_sample": false,
+  "class_embed_type": null,
+  "class_embeddings_concat": false,
+  "conv_in_kernel": 3,
+  "conv_out_kernel": 3,
+  "cross_attention_dim": 768,
+  "cross_attention_norm": null,
+  "down_block_types": [
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "CrossAttnDownBlock2D",
+    "DownBlock2D"
+  ],
+  "downsample_padding": 1,
+  "dual_cross_attention": false,
+  "encoder_hid_dim": null,
+  "encoder_hid_dim_type": null,
+  "flip_sin_to_cos": true,
+  "freq_shift": 0,
+  "in_channels": 9,
+  "layers_per_block": 2,
+  "mid_block_only_cross_attention": null,
+  "mid_block_scale_factor": 1,
+  "mid_block_type": "UNetMidBlock2DCrossAttn",
+  "norm_eps": 1e-05,
+  "norm_num_groups": 32,
+  "num_attention_heads": null,
+  "num_class_embeds": null,
+  "only_cross_attention": false,
+  "out_channels": 4,
+  "projection_class_embeddings_input_dim": null,
+  "resnet_out_scale_factor": 1.0,
+  "resnet_skip_time_act": false,
+  "resnet_time_scale_shift": "default",
+  "sample_size": 64,
+  "time_cond_proj_dim": null,
+  "time_embedding_act_fn": null,
+  "time_embedding_dim": null,
+  "time_embedding_type": "positional",
+  "timestep_post_act": null,
+  "transformer_layers_per_block": 1,
+  "up_block_types": [
+    "UpBlock2D",
+    "CrossAttnUpBlock2D",
+    "CrossAttnUpBlock2D",
+    "CrossAttnUpBlock2D"
+  ],
+  "upcast_attention": null,
+  "use_linear_projection": false
+}

tests/test_object_serializer_disk.py

@@ -99,6 +99,20 @@ def test_obj_serializer_ephemeral_writes_to_tempdir(tmp_path: Path):
     assert not Path(tmp_path, obj_1_name).exists()
 
 
+def test_obj_serializer_ephemeral_deletes_dangling_tempdirs_on_init(tmp_path: Path):
+    tempdir = tmp_path / "tmpdir"
+    tempdir.mkdir()
+    ObjectSerializerDisk[MockDataclass](tmp_path, ephemeral=True)
+    assert not tempdir.exists()
+
+
+def test_obj_serializer_does_not_delete_tempdirs_on_init(tmp_path: Path):
+    tempdir = tmp_path / "tmpdir"
+    tempdir.mkdir()
+    ObjectSerializerDisk[MockDataclass](tmp_path, ephemeral=False)
+    assert tempdir.exists()
+
+
 def test_obj_serializer_disk_different_types(tmp_path: Path):
     obj_serializer_1 = ObjectSerializerDisk[MockDataclass](tmp_path)
     obj_1 = MockDataclass(foo="bar")