128 Commits

Author	SHA1	Message	Date
Ryan Dick	bcd29c5d74	Remove all cases where we check the 'model.device'. This is no longer trustworthy now that partial loading is permitted.	2025-01-07 00:31:00 +00:00
Ryan Dick	d84adfd39f	Clean up FLUX control LoRA pre-processing logic.	2024-12-17 07:28:45 -05:00
Brandon Rising	70811d0bd0	Remove unexpected artifacts in output images	2024-12-17 07:28:45 -05:00
Brandon Rising	046d19446c	Rename Structural Lora to Control Lora	2024-12-17 07:28:45 -05:00
Brandon Rising	f53da60b84	Lots of updates centered around using the lora patcher rather than changing the modules in the transformer model	2024-12-17 07:28:45 -05:00
Brandon Rising	5a035dd19f	Support bnb quantized nf4 flux models, Use controlnet vae, only support 1 structural lora per transformer. various other refractors and bugfixes	2024-12-17 07:28:45 -05:00
Brandon Rising	f3b253987f	Initial setup for flux tools control loras	2024-12-17 07:28:45 -05:00
Ryan Dick	0bff4ace1b	Revert performance improvement, because it caused flux inference to fail on Mac: https://github.com/invoke-ai/InvokeAI/issues/7422	2024-12-03 15:18:58 +00:00
Ryan Dick	021552fd81	Avoid unnecessary dtype conversions with rope encodings.	2024-11-29 12:32:50 -05:00
Ryan Dick	be73dbba92	Use view() instead of rearrange() for better performance.	2024-11-29 12:32:50 -05:00
Ryan Dick	db9c0cad7c	Replace custom RMSNorm implementation with torch.nn.functional.rms_norm(...) for improved speed.	2024-11-29 12:32:50 -05:00
Ryan Dick	5d8dd6e26e	Fix FLUX regional negative prompts.	2024-11-28 18:49:29 +00:00
Ryan Dick	64364e7911	Short-circuit if there are no region masks in FLUX and don't apply attention masking.	2024-11-27 22:40:10 +00:00
Ryan Dick	6565cea039	Comment unused _prepare_unrestricted_attn_mask(...) for future reference.	2024-11-27 22:16:44 +00:00
Ryan Dick	3ebd8d6c07	Delete outdated TODO comment.	2024-11-27 22:13:25 +00:00
Ryan Dick	e970185161	Tweak flux regional prompting attention scheme based on latest experimentation.	2024-11-27 22:13:07 +00:00
Ryan Dick	b54463d294	Allow regional prompting background regions to attend to themselves and to the entire txt embedding.	2024-11-26 17:57:31 +00:00
Ryan Dick	faee79dc95	Distinguish between restricted and unrestricted attn masks in FLUX regional prompting.	2024-11-26 16:55:52 +00:00
Ryan Dick	e01f66b026	Apply regional attention masks in the single stream blocks in addition to the double stream blocks.	2024-11-25 22:40:08 +00:00
Ryan Dick	53abdde242	Update Flux RegionalPromptingExtension to prepare both a mask with restricted image self-attention and a mask with unrestricted image self attention.	2024-11-25 22:04:23 +00:00
Ryan Dick	94c088300f	Be smarter about selecting the global CLIP embedding for FLUX regional prompting.	2024-11-25 20:15:04 +00:00
Ryan Dick	3741a6f5e0	Fix device handling for regional masks and apply the attention mask in the FLUX double stream block.	2024-11-25 16:02:03 +00:00
Ryan Dick	2c23b8414c	Use a single global CLIP embedding for FLUX regional guidance.	2024-11-22 23:01:43 +00:00
Ryan Dick	20356c0746	Fixup the logic for preparing FLUX regional prompt attention masks.	2024-11-21 22:46:25 +00:00
Ryan Dick	bad1149504	WIP - add rough logic for preparing the FLUX regional prompting attention mask.	2024-11-20 22:29:36 +00:00
Ryan Dick	fda7aaa7ca	Pass RegionalPromptingExtension down to the CustomDoubleStreamBlockProcessor in FLUX.	2024-11-20 19:48:04 +00:00
Ryan Dick	85c616fa34	WIP - Pass prompt masks to FLUX model during denoising.	2024-11-20 18:51:43 +00:00
Ryan Dick	c6fc82f756	Infer the clip_extra_context_tokens param from the state dict for FLUX XLabs IP-Adapter V2 models.	2024-11-18 17:06:53 -08:00
David Burnett	9bd17ea02f	Get flux working with MPS on 2.4.1, with GGUF support	2024-10-23 10:20:42 +11:00
Ryan Dick	dde54740c5	Test out IP-Adapter with CFG.	2024-10-21 15:47:17 +00:00
Ryan Dick	fdccdd52d5	Fixes to get XLabsIpAdapterExtension running.	2024-10-21 15:43:00 +00:00
Ryan Dick	31ffd73423	Initial draft of integrating FLUX IP-Adapter inference support.	2024-10-21 15:42:56 +00:00
Ryan Dick	3fa1012879	Add IPAdapterDoubleBlocks wrapper to tidy FLUX ip-adapter handling.	2024-10-21 15:38:50 +00:00
Ryan Dick	c2a8fbd8d6	(minor) Move infer_xlabs_ip_adapter_params_from_state_dict(...) to state_dict_utils.py.	2024-10-21 15:38:50 +00:00
Ryan Dick	d6643d7263	Add model loading code for xlabs FLUX IP-Adapter (not tested).	2024-10-21 15:38:50 +00:00
Ryan Dick	f939dbdc33	Add is_state_dict_xlabs_ip_adapter() utility function.	2024-10-21 15:38:50 +00:00
Ryan Dick	24a0ca86f5	Add logic for loading an Xlabs IP-Adapter from a state dict.	2024-10-21 15:38:50 +00:00
Ryan Dick	95c30f6a8b	Add initial logic for inferring FLUX IP-Adapter params from a state_dict.	2024-10-21 15:38:50 +00:00
Ryan Dick	ac7441e606	Fixup typing/imports for IPDoubleStreamBlockProcessor.	2024-10-21 15:38:50 +00:00
Ryan Dick	9c9af312fe	Copy IPDoubleStreamBlockProcessor from 47495425db/src/flux/modules/layers.py (L221).	2024-10-21 15:38:50 +00:00
Ryan Dick	32c7cdd856	Add cfg_scale_start_step and cfg_scale_end_step to FLUX Denoise node.	2024-10-21 14:52:02 +00:00
Ryan Dick	6df4ee5fc8	Make negative_text_conditioning nullable on FLUX Denoise invocation.	2024-10-18 20:31:27 +00:00
Ryan Dick	371742d8f9	Add support for cfg_scale list on FLUX Denoise node.	2024-10-18 20:14:47 +00:00
Ryan Dick	73d4c4d56d	Naive implementation of CFG for FLUX.	2024-10-16 16:22:35 +00:00
Ryan Dick	4aace24f1f	Reduce peak memory utilization when preparing FLUX controlnet inputs.	2024-10-10 00:18:46 +00:00
Ryan Dick	3953e60a4f	Remove instantx_control_mode from FLUX ControlNet node.	2024-10-09 22:00:54 +00:00
Ryan Dick	8b1ef4b902	Fix bug with InstantX input image range.	2024-10-09 19:38:30 +00:00
Ryan Dick	859944f848	Fix support for InstantX non-union models (with no single blocks).	2024-10-09 18:51:53 +00:00
Ryan Dick	8bf8742984	(minor) Add comment about future memory optimization.	2024-10-09 16:16:04 +00:00
Ryan Dick	c78eeb1645	Shift the controlnet-type-specific logic into the specific ControlNet extensions and make the FLUX model controlnet-type-agnostic.	2024-10-09 16:12:09 +00:00