Optimizing SD Load model

[AshD]

Currently when Loading a SD model, it loads all the sub models.

Was wondering what is the best way to share VAE Encoder, Decoder, Tokenizer and TextEncoder sub-models between different SD Models, so that they are not all loaded again when SD Model is switched. In most cases only the Unet model is different.

Thanks,
Ash

[saddam213]

I was literally just looking into that now :)

I was looking at supporting TAESD, or Tiny AutoEncoder which would require some sort of sharing ability, or way or overriding
https://huggingface.co/deinferno/taesd-onnx

OnnxStack.Core supports any arrangement of models in the ModelSet class, so might see if I can restructure this in a way thats a bit more versatile

We also could support it just in OnnxStack.StableDiffusion using the ModelTemplates perhaps, for a less intrusive change

[saddam213]

Had a bit of a think about this, The best approach to this would be to handle it all in OnnxStack.StableDiffusion

The ModelSet structure in OnnxStack.Core is made in a very specific way to encapsulate a sets of models and manage their lifetime and disposal, I think if ModelSet starts sharing sessions we will end up in messy situation with unmanaged resources

However we can simply make "sub module" ModelSet objects, Just a model set that is the 2 VAE models, or the Tokenizer and TextEncoder for example

Then in OnnxStack.StableDiffusion we can create a new service to wrap the calls to IOnnxModelService and IPromptService in the Diffusers sending inference to the desired ModelSet

`ModelSet' sub-module would be lazy loaded, so first model that needs it will load it, if next model loaded needs it it will stay loaded, otherwise unloaded

How to link a ModelSet to a sub-module ModelSet is something I have not looked at, but should be no real drama

Will open a PR for this soon, With initial goal of shared Tokenizer model

[NeusZimmer]

The approach i did in python that works fairly good: wrapping the pipelines and its submodels, I defined those submodels as singletons to avoid a possible memory mess calling them from multiple sources and saving loading/unloading permanently, for VAEs & text encoder is saving some time and also disk space as you may share same submodels between all your models.

[AshD]

For my use case (Windows App with One model at a time), it would be good enough to check if the sub model is loaded (same file) and not load it again.

Other people's use cases (Server hosted with multiple models concurrently) may be more complex, requiring more fine tuned control of loading sub models.

Regards,
Ash