gogurtenjoyer 382d85ee23 Fix memory issues when installing models on Windows (#8652) ...

* Wrap GGUF loader for context managed close()

Wrap gguf.GGUFReader and then use a context manager to load memory-mapped GGUF files, so that they will automatically close properly when no longer needed. Should prevent the 'file in use in another process' errors on Windows.

* Additional check for cached state_dict

Additional check for cached state_dict as path is now optional - should solve model manager 'missing' this and the resultant memory errors.

* Appease ruff

* Further ruff appeasement

* ruff

* loaders.py fix for linux

No longer attempting to delete internal object.

* loaders.py - one more _mmap ref removed

---------

Co-authored-by: Lincoln Stein <lincoln.stein@gmail.com>

2025-11-16 09:25:52 -05:00

..

gguf

Fix memory issues when installing models on Windows (#8652)

2025-11-16 09:25:52 -05:00

scripts

refactor: model manager v3 (#8607)

2025-10-15 10:18:53 +11:00

__init__.py

Move requantize.py to the quatnization/ dir.

2024-08-26 20:17:50 -04:00

bnb_llm_int8.py

Simplify the state management in InvokeLinear8bitLt and add unit tests. This is in preparation for wrapping it to support streaming of weights from cpu to gpu.

2024-12-24 14:32:11 +00:00

bnb_nf4.py

Install sub directories with folders correctly, ensure consistent dtype of tensors in flux pipeline and vae

2024-08-26 20:17:50 -04:00