* Initial plan
* Warn user when credentials have expired in multiuser mode
Agent-Logs-Url: https://github.com/lstein/InvokeAI/sessions/f0947cda-b15c-475d-b7f4-2d553bdf2cd6
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Address code review: avoid multiple localStorage reads in base query
Agent-Logs-Url: https://github.com/lstein/InvokeAI/sessions/f0947cda-b15c-475d-b7f4-2d553bdf2cd6
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* bugfix(multiuser): ask user to log back in when authentication token expires
* feat: sliding window session expiry with token refresh
Backend:
- SlidingWindowTokenMiddleware refreshes JWT on each mutating request
(POST/PUT/PATCH/DELETE), returning a new token in X-Refreshed-Token
response header. GET requests don't refresh (they're often background
fetches that shouldn't reset the inactivity timer).
- CORS expose_headers updated to allow X-Refreshed-Token.
Frontend:
- dynamicBaseQuery picks up X-Refreshed-Token from responses and
updates localStorage so subsequent requests use the fresh expiry.
- 401 handler only triggers sessionExpiredLogout when a token was
actually sent (not for unauthenticated background requests).
- ProtectedRoute polls localStorage every 5s and listens for storage
events to detect token removal (e.g. manual deletion, other tabs).
Result: session expires after TOKEN_EXPIRATION_NORMAL (1 day) of
inactivity, not a fixed time after login. Any user-initiated action
resets the clock.
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore(backend): ruff
* fix: address review feedback on auth token handling
Bug fixes:
- ProtectedRoute: only treat 401 errors as session expiry, not
transient 500/network errors that should not force logout
- Token refresh: use explicit remember_me claim in JWT instead of
inferring from remaining lifetime, preventing silent downgrade of
7-day tokens to 1-day when <24h remains
- TokenData: add remember_me field, set during login
Tests (6 new):
- Mutating requests (POST/PUT/DELETE) return X-Refreshed-Token
- GET requests do not return X-Refreshed-Token
- Unauthenticated requests do not return X-Refreshed-Token
- Remember-me token refreshes to 7-day duration even near expiry
- Normal token refreshes to 1-day duration
- remember_me claim preserved through refresh cycle
Co-Authored-By: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
* chore(backend): ruff
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Co-authored-by: Claude Opus 4.6 (1M context) <noreply@anthropic.com>
Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com>
* Add comprehensive multi-user support specification and implementation plan
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Clarify Python tooling transition state
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Add executive summary for multi-user support specification
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Reorganize multiuser docs into subfolder and update with approved design decisions
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* docs: fix mkdocs syntax issues
* Fix Z-Image VAE operations not reserving working memory for OOM prevention (#8)
* Initial plan
* Fix Z-Image VAE encode/decode to request working memory
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Add test for Z-Image working memory estimation
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix unit test: only set config attribute for AutoencoderKL, not FluxAutoEncoder
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* style: fix ruff errors
* Fix test: use model_construct to bypass Pydantic validation for mock fields
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* chore(ruff): fix ruff errors
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* Phase 1: Add multi-user database schema and authentication foundation (#6)
* Initial plan
* Add Phase 1: Database schema, authentication utilities, and user service
- Added dependencies: passlib[bcrypt], python-jose[cryptography], email-validator
- Created migration_25 for multi-user support database schema
- users table with authentication fields
- user_sessions, user_invitations, shared_boards tables
- Added user_id columns to boards, images, workflows, session_queue, style_presets
- Created system user for backward compatibility
- Implemented authentication utilities
- Password hashing and validation (password_utils.py)
- JWT token generation and verification (token_service.py)
- Implemented user service
- Abstract base class (users_base.py)
- SQLite implementation (users_default.py)
- Full CRUD operations for users
- Authentication and admin management
- Added comprehensive tests for all components
- All code passes ruff linting and mypy type checking
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Refactor user service to use transaction pattern and improve security documentation
- Changed all database operations to use self._db.transaction() context manager
- Follows the same pattern as other services like board_records
- Improves thread safety and transaction management
- Added detailed security warnings for the hardcoded JWT secret key
- Documented that it must be moved to config before production
- Fixed whitespace linting issues
- All tests still pass with the refactored code
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix system user email to use valid domain
- Changed system user email from system@invokeai.local to system@system.invokeai
- .local domains are rejected by email-validator as special-use/reserved
- All end-to-end tests now pass successfully
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix test failures: password length, migration chain, and table existence checks
- Fixed bcrypt 72-byte password limit by truncating long passwords in hash_password and verify_password
- Fixed test_migration_25 to use callback directly instead of migrator to avoid fragmented chain error
- Updated system user email assertion in test from @invokeai.local to @system.invokeai
- Added table existence checks for boards, images, workflows, and session_queue before altering them
- This prevents "no such table" errors when migration runs in environments without all tables
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* (chore) fix ruff errors
* (chore) fix ruff errors (2)
* Fix test failures: use _conn instead of conn and improve password truncation
- Fixed test_user_service.py to use db._conn instead of db.conn (SqliteDatabase uses private _conn)
- Simplified password truncation logic in password_utils.py for better clarity
- Ensured UTF-8 decoding handles truncated bytes properly with errors="ignore"
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* chore(uv): updated uv lockfile
* Fix bcrypt password length errors by configuring passlib properly
- Added bcrypt__relax_truncate_checks=True to CryptContext to prevent errors on long passwords
- Removed min_length=8 constraint from pydantic models to allow service-level validation
- Service-level validation provides better error messages and more control
- Manual truncation code kept as safety net for passwords >72 bytes
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix bcrypt parameter name: use truncate_error instead of relax_truncate_checks
- Changed bcrypt__relax_truncate_checks=True to bcrypt__truncate_error=False
- The correct passlib parameter is truncate_error (not relax_truncate_checks)
- Setting it to False allows passwords >72 bytes without raising an error
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* fix(passwords) downgrade bcrypt to work with current passlib
* chore(uv): update locks file
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* Implement Phase 2: Authentication Service with JWT and FastAPI Integration (#11)
* Initial plan
* Add Phase 2 authentication service - auth dependencies, router, and service integration
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Address code review feedback - add token expiration constants and improve documentation
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix email validation to allow special-use domains like .local for testing
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix system user to not be admin - allows /auth/setup to work on fresh database
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* chore: typegen
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* Add Phase 3 integration tests and documentation for authentication middleware (#15)
* Initial plan
* Add Phase 3 integration tests for auth router endpoints
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Add Phase 3 completion documentation and testing guide
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* chore: ruff
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* chore: fix unresolved merged conflict markers
* Phase 4: Add multi-tenancy to boards and session queue services (#17)
* Initial plan
* Update boards service for multi-user support
- Add user_id parameter to board creation and filtering
- Update board_records base and SQLite implementation to filter by user
- Include shared boards and public boards in user queries
- Update boards router to require authentication
- Update invocation context to pass user_id from queue item
- Add user_id field to SessionQueueItem model
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Update session_queue service for multi-user support
- Add user_id parameter to enqueue_batch method
- Update ValueToInsertTuple to include user_id
- Update prepare_values_to_insert to accept and use user_id
- Update SQL INSERT statements to include user_id column
- Update session_queue router to require authentication
- Update retry_items_by_id to preserve user_id when retrying
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Add authentication imports to images, workflows, and style_presets routers
- Import CurrentUser dependency in all three routers
- Add CurrentUser parameter to upload_image endpoint
- Prepare for further authentication requirements in these services
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Add multiuser tests and Phase 4 documentation
- Create test_boards_multiuser.py with authentication tests
- Verify board operations require authentication
- Test board creation and listing with auth tokens
- Add comprehensive Phase 4 implementation summary
- Document all changes, data flows, and security considerations
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Add authentication to remaining board endpoints
- Add CurrentUser to get_board endpoint
- Add CurrentUser to update_board endpoint
- Add CurrentUser to delete_board endpoint
- Ensures all board operations require authentication
- Addresses code review feedback
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Feature(image boards): Implement per-user board isolation
- Complete verification report with all checks passed
- Document code quality, security, and testing results
- List all achievements and sign-off criteria
- Mark phase as READY FOR MERGE
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* chore: ruff
* chore: resolve conflicts in z_image_working_memory test
* chore: ruff
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* feat: Implement Phase 5 of multiuser plan - Frontend authentication (#19)
* Phase 5: Implement frontend authentication infrastructure
- Created auth slice with Redux state management for authentication
- Created auth API endpoints (login, logout, setup, me)
- Created LoginPage component for user authentication
- Created AdministratorSetup component for initial admin setup
- Created ProtectedRoute wrapper for route authentication checking
- Updated API configuration to include Authorization headers
- Installed and configured react-router-dom for routing
- Updated App component with authentication routes
- All TypeScript checks passing
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* chore(style): prettier, typegen and add convenience targets to makefile
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* feat: Implement Phase 6 frontend UI updates - UserMenu and admin restrictions
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
docs: Add comprehensive testing and verification documentation for Phase 6
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
docs: Add Phase 6 summary document
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* feat: Add user management script for testing multiuser features
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* feat: Implement read-only model manager access for non-admin users
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
feat: Add admin authorization to model management API endpoints
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
docs: Update specification and implementation plan for read-only model manager
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Phase 7: Comprehensive testing and security validation for multiuser authentication (#23)
* Initial plan
* Phase 7: Complete test suite with 88 comprehensive tests
- Add password utils tests (31 tests): hashing, verification, validation
- Add token service tests (20 tests): JWT creation, verification, security
- Add security tests (13 tests): SQL injection, XSS, auth bypass prevention
- Add data isolation tests (11 tests): multi-user data separation
- Add performance tests (13 tests): benchmarks and scalability
- Add comprehensive testing documentation
- Add phase 7 verification report
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* bugfix(backend): Fix issues with authentication token expiration handling
- Remove time.sleep from token uniqueness test (use different expiration instead)
- Increase token expiration test time from 1 microsecond to 10 milliseconds
- More reliable test timing to prevent flakiness
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Add Phase 7 summary documentation
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Fix test_performance.py missing logger fixture
Add missing logger fixture to test_performance.py that was causing test failures.
The fixture creates a Logger instance needed by the user_service fixture.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Add board isolation issue specification document
Document the three board isolation issues that need to be addressed:
1. Board list not updating when switching users
2. "Uncategorized" board shared among users
3. Admin cannot access all users' boards
Includes technical details, implementation plan, and acceptance criteria.
This document will be used to create a separate GitHub issue and PR.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Fix test failures in Phase 7 tests
- Fix board service API calls to use enum values (BoardRecordOrderBy, SQLiteDirection) instead of strings
- Fix board deletion test to use get_dto() instead of non-existent get() method
- Add exception handling to verify_password() for invalid hash formats
- Update SQL injection test to accept both 401 and 422 status codes (Pydantic validation)
All fixes ensure tests match actual API signatures and handle edge cases properly.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Fix token forgery test to properly decode and modify JWT payload
The test was attempting to modify the JWT payload by string replacement on the
base64-encoded data, which doesn't work since "false" doesn't appear literally
in the base64 encoding. Fixed to:
- Properly decode the base64 payload
- Parse the JSON
- Modify the is_admin field
- Re-encode the payload
- Create a forged token with the modified payload and original signature
- Verify it's rejected with 401 status
This properly tests that JWT signature verification prevents token forgery.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Fix test_verify_expired_token timing issue
Increased token expiration from 10ms to 100ms and wait time from 20ms to 200ms
to make the test more reliable. The previous timing was too tight and could
cause race conditions where the token hadn't actually expired yet when verified.
This ensures the test reliably validates that expired tokens are properly rejected.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Fix token verification to properly validate signatures and expiration
The verify_token() function was not explicitly enabling JWT verification options,
causing it to accept expired tokens and tokens with modified signatures/payloads.
Fixed by explicitly setting verification options:
- verify_signature: True (reject modified tokens)
- verify_exp: True (reject expired tokens)
- verify_nbf: True (verify "not before" claim)
- verify_iat: True (verify "issued at" claim)
- verify_aud: False (we don't use audience claims)
Also added catch-all exception handler for other validation errors.
This ensures all token security tests pass properly.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Revert to default python-jose verification behavior
Removed explicit options parameter from jwt.decode() call since python-jose
verifies signatures and expiration by default. The explicit options may have
been causing issues or conflicts with the library's default behavior.
Added comments clarifying that python-jose defaults are secure and handle
verification properly. Need user to test and provide debugging output if
issues persist.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Fix JWT expiration verification for python-jose 3.5.0
python-jose 3.5.0 has a bug where expiration verification doesn't work properly
by default. The jwt.decode() function is not rejecting expired tokens even when
they should be rejected.
Workaround implemented:
1. First, get unverified claims to extract the 'exp' timestamp
2. Manually check if current time >= exp time (token is expired)
3. Return None immediately if expired
4. Then verify signature with jwt.decode() for tokens that aren't expired
This ensures:
- Expired tokens are properly rejected
- Signature verification still happens for non-expired tokens
- Modified tokens are rejected due to signature mismatch
All three failing tests should now pass:
- test_verify_expired_token
- test_verify_token_with_modified_payload
- test_token_signature_verification
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix race condition in token verification - verify signature before expiration
Changed the order of verification in verify_token():
1. First verify signature with jwt.decode() - rejects modified/forged tokens
2. Then manually check expiration timestamp
Previous implementation checked expiration first using get_unverified_claims(),
which could cause a race condition where:
- Token with valid payload but INVALID signature would pass expiration check
- If expiration check happened to return None due to timing, signature was never verified
- Modified tokens could be accepted intermittently
New implementation ensures signature is ALWAYS verified first, preventing any
modified tokens from being accepted, while still working around the python-jose
3.5.0 expiration bug by manually checking expiration after signature verification.
This eliminates the non-deterministic test failures in test_verify_token_with_modified_payload.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* chore(app): ruff
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* Backend: Add admin board filtering and uncategorized board isolation
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix intermittent token service test failures caused by Base64 padding (#32)
* Initial plan
* Fix intermittent token service test failures due to Base64 padding
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Address code review: add constants for magic numbers in tests
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* chore(tests): ruff
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* Implement user isolation for session queue and socket events (WIP - debugging queue visibility) (#30)
* Add user isolation for queue events and field values filtering
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Add user column to queue list UI
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Add field values privacy indicator and implementation documentation
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Allow all users to see queue item status events while keeping invocation events private
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* chore(backend): ruff
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* Fix Queue tab not updating for other users in real-time (#34)
* Initial plan
* Add SessionQueueItemIdList invalidation to queue socket events
This ensures the queue item list updates in real-time for all users when
queue events occur (status changes, batch enqueued, queue cleared).
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Add SessionQueueItemIdList invalidation to queue_items_retried event
Ensures queue list updates when items are retried.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Improve queue_items_retried event and mutation invalidation
- Add individual item invalidation to queue_items_retried event handler
- Add SessionQueueStatus and BatchStatus tags to retryItemsById mutation
- Ensure consistency between event handler and mutation invalidation patterns
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Add privacy check for batch field values in Queue tab
Displays "Hidden for privacy" message for non-admin users viewing
queue items they don't own, instead of showing the actual field values.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* i18n(frontend): change wording of queue values suppressed message
* Add SessionQueueItemIdList cache invalidation to queue events
Ensures real-time queue updates for all users by invalidating the
SessionQueueItemIdList cache tag when queue events occur.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* Fix multiuser information leakage in Queue panel detail view (#38)
* Initial plan
* Implement multiuser queue information leakage fix
- Backend: Update sanitize_queue_item_for_user to clear session graph and workflow
- Frontend: Add permission check to disable detail view for unauthorized users
- Add test for sanitization logic
- Add translation key for permission denied message
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix prettier formatting for QueueItemComponent
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Address code review feedback
- Move Graph and GraphExecutionState imports to top of file
- Remove dependency on test_nodes in sanitization test
- Create minimal test invocation directly in test file
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Address additional code review feedback
- Create shallow copy to avoid mutating original queue_item
- Extract 'system' user_id to constant (SYSTEM_USER_ID)
- Add constant to both backend and frontend for consistency
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix pydantic validation error in test fixture
Add required timestamp fields (created_at, updated_at, started_at, completed_at) to SessionQueueItem in test fixture
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* fix(queue): Enforce user permissions for queue operations in multiuser mode (#36)
* Initial plan
* Add backend authorization checks for queue operations
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix linting issues in authorization changes
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Add frontend authorization checks for queue operations
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Add access denied messages for cancel and clear operations
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix access denied messages for all cancel/delete operations
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix merge conflict duplicates in QueueItemComponent
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* chore(frontend): typegen
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* fix(multiuser): Isolate client state per user to prevent data leakage (#40)
* Implement per-user client state storage to fix multiuser leakage
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix: Make authentication optional for client_state endpoints to support single-user mode
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Clear params state on logout/login to prevent user data leakage
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* feat(queue): show user/total pending jobs in multiuser mode badge (#43)
* Initial plan
* Add multiuser queue badge support - show X/Y format in multiuser mode
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Format openapi.json with Prettier
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Address code review feedback - optimize DB queries and improve code clarity
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* translationBot(ui): update translation files (#8767)
Updated by "Cleanup translation files" hook in Weblate.
Translate-URL: https://hosted.weblate.org/projects/invokeai/web-ui/
Translation: InvokeAI/Web UI
* Limit automated issue closure to bug issues only (#8776)
* Initial plan
* Add only-labels parameter to limit automated issue closure to bugs only
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* fix(multiuser): Isolate client state per user to prevent data leakage (#40)
* Implement per-user client state storage to fix multiuser leakage
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix: Make authentication optional for client_state endpoints to support single-user mode
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Clear params state on logout/login to prevent user data leakage
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Initial plan
* chore(backend) ruff & typegen
* Fix real-time badge updates by invalidating SessionQueueStatus on queue events
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Co-authored-by: Weblate (bot) <hosted@weblate.org>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* Convert session queue isolation logs from info to debug level
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Add JWT secret storage in database and app_settings service
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Add multiuser configuration option with default false
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Update token service tests to initialize JWT secret
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix app_settings_service to use proper database transaction pattern
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* chore(backend): typegen and ruff
* chore(docs): update docstrings
* Fix frontend to bypass authentication in single-user mode
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix auth tests to enable multiuser mode
Auth tests were failing because the login and setup endpoints now return 403 when multiuser mode is disabled (the default). Updated test fixtures to enable multiuser mode for all auth-related tests.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix model manager UI visibility in single-user mode
Model manager UI for adding, deleting and modifying models is now:
- Visible in single-user mode (multiuser: false, the default)
- Hidden in multiuser mode for non-admin users
- Visible in multiuser mode for admin users
Created useIsModelManagerEnabled hook that checks multiuser_enabled status
and returns true when multiuser is disabled OR when user is admin.
Updated all model manager components to use this hook instead of direct
is_admin checks.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* chore(backend): ruff
* chore(frontend): typegen
* Fix TypeScript lint errors
- Added multiuser_enabled field to SetupStatusResponse type in auth.ts
- Removed unused user variable reference in MainModelDefaultSettings.tsx
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix test_data_isolation to enable multiuser mode
Added fixture to enable multiuser mode for data isolation tests, similar to other auth tests.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Redirect login and setup pages to app in single-user mode
When multiuser mode is disabled, the LoginPage and AdministratorSetup components now redirect to /app instead of showing the login/setup forms. This prevents users from being stuck on the login page after browser refresh in single-user mode.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix test_auth.py to initialize JWT secret
Added setup_jwt_secret fixture to test_auth.py to initialize the JWT secret before running auth tests. This fixture was missing, causing token creation/verification to fail in auth router tests.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Prevent login form flash in single-user mode
Show loading spinner instead of login/setup forms when multiuser mode is disabled or when redirecting is about to happen. This prevents the unattractive flash of the login dialog when refreshing the page in single-user mode.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix board and queue operations in single-user mode
Changed boards, session_queue, and images routers to use CurrentUserOrDefault instead of CurrentUser. This allows these endpoints to work without authentication when multiuser mode is disabled (default), fixing the issue where users couldn't create boards or add jobs to the queue in single-user mode.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Add user management utilities and rename add_user.py
Created three user management scripts in the scripts/ directory:
- useradd.py (renamed from add_user.py) - add users with admin privileges
- userdel.py - delete users by email address with confirmation
- usermod.py - modify user details (name, password, admin status)
All scripts support both CLI and interactive modes for flexibility.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix ESLint errors in frontend code
- Fixed brace-style issue in App.tsx (else-if on same line)
- Removed unused useAppSelector imports from model manager components
- Fixed import sorting in ControlAdapterModelDefaultSettings.tsx
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Add userlist.py script for viewing database users
Created scripts/userlist.py to display all users in the database. Supports:
- Table format (default): Shows ID, email, display name, admin status, and active status
- JSON format (--json flag): Outputs user data as JSON for scripting/automation
Example usage:
python scripts/userlist.py # Table view
python scripts/userlist.py --json # JSON output
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix test_boards_multiuser.py test failures
Fixed test failures caused by ApiDependencies.invoker not being set properly:
- Added setup_jwt_secret fixture to initialize JWT secret for token generation
- Added enable_multiuser_for_tests fixture that sets ApiDependencies.invoker as a class attribute
- Updated tests to use enable_multiuser_for_tests fixture to ensure ApiDependencies is properly configured
- Removed MockApiDependencies class approach in favor of directly setting the class attribute
This fixes the AttributeError and ensures all tests have the proper setup.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* chore(backend): ruff
* Fix userlist.py SqliteDatabase initialization
Fixed AttributeError in userlist.py where SqliteDatabase was being passed the config object instead of config.db_path. The constructor expects a Path object (db_path) as the first argument, not the entire config object.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix test_boards_multiuser.py by adding app_settings service to mock
Added AppSettingsService initialization to the mock_services fixture in tests/conftest.py. The test was failing because setup_jwt_secret fixture expected mock_invoker.services.app_settings to exist, but it wasn't being initialized in the mock services.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* bugfix(scripts): fix crash in userlist.py script
* Fix test_boards_multiuser.py JWT secret initialization
Fixed setup_jwt_secret fixture to call set_jwt_secret() directly instead of trying to access non-existent app_settings service. Removed incorrect app_settings parameter from InvocationServices initialization in tests/conftest.py since app_settings is not an attribute of InvocationServices.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix CurrentUserOrDefault to require auth in multiuser mode
Changed get_current_user_or_default to raise HTTP 401 when multiuser mode is enabled and credentials are missing, invalid, or the user is inactive. This ensures that board/queue/image operations require authentication in multiuser mode while still working without authentication in single-user mode (default).
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* chore(front & backend): ruff and lint
* Add AdminUserOrDefault and fix model settings in single-user mode
Created AdminUserOrDefault dependency that allows admin operations to work without authentication in single-user mode while requiring admin privileges in multiuser mode. Updated model_manager router to use AdminUserOrDefault for update_model_record, update_model_image, and reidentify_model endpoints. This fixes the "Missing authentication credentials" error when saving model default settings in single-user mode.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix model manager operations in single-user mode
Changed all model manager endpoints from AdminUser to AdminUserOrDefault to allow model installation, deletion, conversion, and cache management operations to work without authentication in single-user mode. This fixes the issue where users couldn't add or delete models in single-user mode.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix syntax error in model_manager.py
Added Depends(AdminUserOrDefault) to all AdminUserOrDefault dependency parameters to fix Python syntax error where parameters without defaults were following parameters with defaults. Imported Depends from fastapi.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix FastAPI dependency injection syntax error
Removed type annotations from AdminUserOrDefault dependency parameters. FastAPI doesn't allow both Annotated type hints and = Depends() default values together. Changed from `_: AdminUserOrDefault = Depends(AdminUserOrDefault)` to `_ = Depends(AdminUserOrDefault)` throughout model_manager.py.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix delete_model endpoint parameter annotation
Changed delete_model endpoint to use Annotated[str, Path(...)] instead of str = Path(...) to match FastAPI's preferred syntax and fix the 422 Unprocessable Entity error when deleting models in single-user mode.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix parameter annotations for all AdminUserOrDefault endpoints
Changed all endpoints using AdminUserOrDefault from old syntax (key: str = Path(...)) to FastAPI's preferred Annotated syntax (key: Annotated[str, Path(...)]). This fixes 422 Unprocessable Entity errors when updating model settings and deleting models in single-user mode. Updated endpoints: delete_model_image, install_model, install_hugging_face_model, and convert_model.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Revert to correct AdminUserOrDefault usage pattern
Reverted model_manager.py to state before commit c47af8f and reapplied AdminUserOrDefault correctly. Changed from `_: AdminUser` to `current_admin: AdminUserOrDefault` using the same pattern as boards.py (`current_user: CurrentUserOrDefault`). This fixes all 422 errors in single-user mode while maintaining proper admin authentication in multiuser mode.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix parameter order for AdminUserOrDefault in model manager
Moved current_admin: AdminUserOrDefault parameter before all parameters with default values in model_manager.py endpoints. Python requires parameters without defaults to come before parameters with defaults. Fixed 8 endpoints: delete_model, bulk_delete_models, delete_model_image, install_model, install_hugging_face_model, cancel_model_install_job, convert_model, and do_hf_login.
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* chore(frontend): typegen
* chore(frontend): typegen again
* Docs(app): Comprehensive Documentation of Multiuser Features (#50)
* Implement Phase 8: Complete multiuser documentation (user, admin, and API guides)
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Update multiuser documentation for single-user mode and CLI scripts
- Document multiuser config option (true/false/absent)
- Explain single-user mode behavior (no login required)
- Document mode switching and legacy "system" user
- Update user management to reference CLI scripts (useradd, userdel, usermod, userlist)
- Note that web UI for user management is coming in future release
- Add adaptive API client example for both modes
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* docs(multiuser): bring user guide documentation up to date
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* docs(app): update multiuser documentation
* bugfix(app): fix misaligned database migration calls
* chore(tests): update migration test to accommodate resequencing of migrations
* fix(frontend): prevent caching of static pages
* chore(backend): ruff
* fix(backend): fix incorrect migration import
* Fix: Admin users can see image previews from other users' generations (#61)
* Initial plan
* Fix: strip image preview from InvocationProgressEvent sent to admin room
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* chore: ruff
* fix(backend): add migration_29 file
* chore(tests): fix migration_29 test
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* fix(queue): System user queue items show blank instead of `<hidden>` for non-admin users (#63)
* Initial plan
* fix(queue): System user queue items show blank instead of `<hidden>` for non-admin users
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* chore(backend): ruff
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* Hide "Use Cache" checkbox in node editor for non-admin users in multiuser mode (#65)
* Initial plan
* Hide use cache checkbox for non-admin users in multiuser mode
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix node loading hang when invoke URL ends with /app (#67)
* Initial plan
* Fix node loading hang when URL ends with /app
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Move user management scripts to installable module with CLI entry points (#69)
* Initial plan
* Add user management module with invoke-useradd/userdel/userlist/usermod entry points
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* chore(util): remove superceded user administration scripts
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>
* chore(backend): reorganized migrations, but something still broken
* Fix migration 28 crash when `client_state.data` column is absent (#70)
* Initial plan
* Fix migration 28 to handle missing data column in client_state table
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Consolidate multiuser DB migrations 27–29 into a single migration step (#71)
* Initial plan
* Consolidate migrations 27, 28, and 29 into a single migration step
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Add `--root` option to user management CLI utilities (#81)
* Initial plan
* Add --root option to user management CLI utilities
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix queue clear() endpoint to respect user_id for multi-tenancy (#75)
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Add tests for session queue clear() user_id scoping
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
chore(frontend): rebuild typegen
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
* fix: use AdminUserOrDefault for pause and resume queue endpoints (#77)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* fix: queue pause/resume buttons disabled in single-user mode (#83)
In single-user mode, currentUser is never populated (no auth), so
`currentUser?.is_admin ?? false` always returns false, disabling the buttons.
Follow the same pattern as useIsModelManagerEnabled: treat as admin
when multiuser mode is disabled, and check is_admin flag when enabled.
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* fix: enforce board ownership checks in multiuser mode (#84)
- get_board: verify current user owns the board (or is admin), return 403 otherwise
- update_board: verify ownership before updating, 404 if not found, 403 if unauthorized
- delete_board: verify ownership before deleting, 404 if not found, 403 if unauthorized
- list_all_board_image_names: add CurrentUserOrDefault auth and ownership check for non-'none' board IDs
test: add ownership enforcement tests for board endpoints in multiuser mode
- Auth requirement tests for get, update, delete, and list_image_names
- Cross-user 403 forbidden tests (non-owner cannot access/modify/delete)
- Admin bypass tests (admin can access/update/delete any user's board)
- Board listing isolation test (users only see their own boards)
- Refactored fixtures to use monkeypatch (consistent with other test files)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix: Clear auth state when switching from multiuser to single-user mode (#86)
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
* Fix race conditions in download queue and model install service (#98)
* Initial plan
* Fix race conditions in download queue and model install service
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
---------
Co-authored-by: copilot-swe-agent[bot] <198982749+Copilot@users.noreply.github.com>
Co-authored-by: lstein <111189+lstein@users.noreply.github.com>
Co-authored-by: Weblate (bot) <hosted@weblate.org>
Co-authored-by: Jonathan <34005131+JPPhoto@users.noreply.github.com>
When unsafe_disable_picklescan is enabled, instead of erroring on
detections or scan failures, a warning is logged.
A warning is also logged on app startup when this setting is enabled.
The setting is disabled by default and there is no change in behaviour
when disabled.
Adds full support for managing model-to-model relationships in the UI and backend.
Introduces RelatedModels subpanel for linking and unlinking models in model management.
- Adds REST API routes for adding, removing, and retrieving model relationships.
- New database migration: creates model_relationships table for bidirectional links.
- New service layer (model_relationships) for relationship management.
- Updated frontend: Related models float to top of LoRA/Main grouped model comboboxes for quick access.
- Added 'Show Only Related' toggle badge to MainModelPicker filter bar
**Amended commit to remove changes to ParamMainModelSelect.tsx and MainModelPicker.tsx to avoid conflict with upstream deletion/ rewrite**
Previously, custom node loading occurred _during module imports_. A consequence of this is that when a custom node import fails (e.g. its type clobbers an existing node), the app fails to start up.
In fact, any time we import basically anything from the app, we trigger custom node imports! Not good.
This logic is now in its own function, called as the API app starts up.
If a custom node load fails for any reason, it no longer prevents the app from starting up.
One other bonus we get from this is that we can now ensure custom nodes are loaded _after_ core nodes.
Any clobbering that may occur while loading custom nodes is now guaranteed to be a custom node clobbering a core node's type - and not the other way round.
Uvicorn's logging is rather verbose. This change adds a `log_level_network` config setting to independently control uvicorn's log outputs. The setting defaults to warning.
The change hides the helpful startup message that says the host and port we are running on.
For example: `Uvicorn running on http://0.0.0.0:9090 (Press CTRL+C to quit`
The ASGI lifespan handler is updated to log an equivalent message on startup, regardless of log level settings.
Besides being helpful, the launcher relies on a message like this to launch the app. So, previously, if the user set their log level to anything above info (e.g. warning or error), the launcher would fail to open the app. This change prevents that edge case.
This pops up every now and then and I could never figure it out. A user figured it out in #6936. The cause is appending a query string to the app URL.
For example:
```sh
http://127.0.0.1:9090/?__theme=dark
```
The query string breaking the static file serving, which prevents our translations from loading correctly. Instead of the JSON translations, FastAPI sends the index HTML page. The UI then errors when attempting to parse the translation JSON.
The query string ?__theme=dark is used by Gradio to force dark mode. I believe the users with this issue are doing the same thing the user in #6936 did (just change the port number on an existing bookmark) or their browser history/bookmark includes the query string.
Though this is technically a user-caused problem (we cannot prevent the user from using a malformed URL), we can work around it. When query string is used on the root path, we can redirect the browser to the root path without the query string.
This is done via very simple middleware.
Closes#6696Closes#6817Closes#6828Closes#6936Closes#6983
Around the time we (I) implemented pydantic events, I noticed a short pause between progress images every 4 or 5 steps when generating with SDXL. It didn't happen with SD1.5, but I did notice that with SD1.5, we'd get 4 or 5 progress events simultaneously. I'd expect one event every ~25ms, matching my it/s with SD1.5. Mysterious!
Digging in, I found an issue is related to our use of a synchronous queue for events. When the event queue is empty, we must call `asyncio.sleep` before checking again. We were sleeping for 100ms.
Said another way, every time we clear the event queue, we have to wait 100ms before another event can be dispatched, even if it is put on the queue immediately after we start waiting. In practice, this means our events get buffered into batches, dispatched once every 100ms.
This explains why I was getting batches of 4 or 5 SD1.5 progress events at once, but not the intermittent SDXL delay.
But this 100ms wait has another effect when the events are put on the queue in intervals that don't perfectly line up with the 100ms wait. This is most noticeable when the time between events is >100ms, and can add up to 100ms delay before the event is dispatched.
For example, say the queue is empty and we start a 100ms wait. Then, immediately after - like 0.01ms later - we push an event on to the queue. We still need to wait another 99.9ms before that event will be dispatched. That's the SDXL delay.
The easy fix is to reduce the sleep to something like 0.01 seconds, but this feels kinda dirty. Can't we just wait on the queue and dispatch every event immediately? Not with the normal synchronous queue - but we can with `asyncio.Queue`.
I switched the events queue to use `asyncio.Queue` (as seen in this commit), which lets us asynchronous wait on the queue in a loop.
Unfortunately, I ran into another issue - events now felt like their timing was inconsistent, but in a different way than with the 100ms sleep. The time between pushing events on the queue and dispatching them was not consistently ~0ms as I'd expect - it was highly variable from ~0ms up to ~100ms.
This is resolved by passing the asyncio loop directly into the events service and using its methods to create the task and interact with the queue. I don't fully understand why this resolved the issue, because either way we are interacting with the same event loop (as shown by `asyncio.get_running_loop()`). I suppose there's some scheduling magic happening.
For some reason, I started getting this indefinite hang when the app checks if port 9090 is available. After some fiddling around, I found that adding a timeout resolves the issue.
I confirmed that the util still works by starting the app on 9090, then starting a second instance. The second instance correctly saw 9090 in use and moved to 9091.
Some tech debt related to dynamic pydantic schemas for invocations became problematic. Including the invocations and results in the event schemas was breaking pydantic's handling of ref schemas. I don't really understand why - I think it's a pydantic bug in a remote edge case that we are hitting.
After many failed attempts I landed on this implementation, which is actually much tidier than what was in there before.
- Create pydantic-enabled types for `AnyInvocation` and `AnyInvocationOutput` and use these in place of the janky dynamic unions. Actually, they are kinda the same, but better encapsulated. Use these in `Graph`, `GraphExecutionState`, `InvocationEventBase` and `InvocationCompleteEvent`.
- Revise the custom openapi function to work with the new models.
- Split out the custom openapi function to a separate file. Add a `post_transform` callback so consumers can customize the output schema.
- Update makefile scripts.
We don't need to use the payload schema registry. All our events are dispatched as pydantic models, which are already validated on instantiation.
We do want to add all events to the OpenAPI schema, and we referred to the payload schema registry for this. To get all events, add a simple helper to EventBase. This is functionally identical to using the schema registry.
Our events handling and implementation has a couple pain points:
- Adding or removing data from event payloads requires changes wherever the events are dispatched from.
- We have no type safety for events and need to rely on string matching and dict access when interacting with events.
- Frontend types for socket events must be manually typed. This has caused several bugs.
`fastapi-events` has a neat feature where you can create a pydantic model as an event payload, give it an `__event_name__` attr, and then dispatch the model directly.
This allows us to eliminate a layer of indirection and some unpleasant complexity:
- Event handler callbacks get type hints for their event payloads, and can use `isinstance` on them if needed.
- Event payload construction is now the responsibility of the event itself (a pydantic model), not the service. Every event model has a `build` class method, encapsulating this logic. The build methods are provided as few args as possible. For example, `InvocationStartedEvent.build()` gets the invocation instance and queue item, and can choose the data it wants to include in the event payload.
- Frontend event types may be autogenerated from the OpenAPI schema. We use the payload registry feature of `fastapi-events` to collect all payload models into one place, making it trivial to keep our schema and frontend types in sync.
This commit moves the backend over to this improved event handling setup.
* introduce new abstraction layer for GPU devices
* add unit test for device abstraction
* fix ruff
* convert TorchDeviceSelect into a stateless class
* move logic to select context-specific execution device into context API
* add mock hardware environments to pytest
* remove dangling mocker fixture
* fix unit test for running on non-CUDA systems
* remove unimplemented get_execution_device() call
* remove autocast precision
* Multiple changes:
1. Remove TorchDeviceSelect.get_execution_device(), as well as calls to
context.models.get_execution_device().
2. Rename TorchDeviceSelect to TorchDevice
3. Added back the legacy public API defined in `invocation_api`, including
choose_precision().
4. Added a config file migration script to accommodate removal of precision=autocast.
* add deprecation warnings to choose_torch_device() and choose_precision()
* fix test crash
* remove app_config argument from choose_torch_device() and choose_torch_dtype()
---------
Co-authored-by: Lincoln Stein <lstein@gmail.com>
HF login, legacy yaml confs, and default init file are all handled during app setup.
All directories are created as they are needed by the app.
No need to check for a valid root dir - we will make it if it doesn't exist.
We have two problems with how argparse is being utilized:
- We parse CLI args as the `api_app.py` file is read. This causes a problem pytest, which has an incompatible set of CLI args. Some tests import the FastAPI app, which triggers the config to parse CLI args, which receives the pytest args and fails.
- We've repeatedly had problems when something that uses the config is imported before the CLI args are parsed. When this happens, the root dir may not be set correctly, so we attempt to operate on incorrect paths.
To resolve these issues, we need to lift CLI arg parsing outside of the application code, but still let the application access the CLI args. We can create a external app entrypoint to do this.
- `InvokeAIArgs` is a simple helper class that parses CLI args and stores the result.
- `run_app()` is the new entrypoint. It first parses CLI args, then runs `invoke_api` to start the app.
The `invokeai-web` project script and `invokeai-web.py` dev script now call `run_app()` instead of `invoke_api()`.
The first time `get_config()` is called to get the singleton config object, it retrieves the args from `InvokeAIArgs`, sets the root dir if provided, then merges settings in from `invokeai.yaml`.
CLI arg parsing is now safely insulated from application code, but still accessible. And we don't need to worry about import order having an impact on anything, because by the time the app is running, we have already parsed CLI args. Whew!
