Making image_model scene data removes the hardcoded-engine assumption. The illustrated _loaded_pipelines dictionary introduces a separate memory risk, though: every newly selected checkpoint remains resident, so an episode using several overrides can recreate the original OOM even when each model fits on its own.
I would give the cache an explicit residency budget and define when an inactive pipeline can be evicted. An active generation needs a lease so its pipeline cannot disappear halfway through a scene. A sequence alternating three checkpoints would be a useful regression case, recording resident memory after each switch rather than testing only a fresh process with one model.
Thanks Ahmet, that's a fair catch. The snippet in the post is deliberately simplified: it fixes the hardcoded-model problem but, as you say, _loaded_pipelines only ever grows, so an episode with several overrides can bring the same memory pressure back. An explicit residency budget with eviction of inactive pipelines is the natural next step, and the lease idea matters because a pipeline shouldn't be unloaded halfway through a scene. The three-checkpoint alternating test, recording resident memory after each switch, is a great regression case too. I'll fold that into a follow-up. Appreciate the detailed review.
That distinction between the illustrative cache and the follow-up residency policy makes sense. One extra measurement for the alternating-checkpoint test is peak memory during each switch, rather than only the resident total after the switch.
A policy can satisfy its steady-state budget and still fail while loading the next pipeline before evicting the old one. Recording the load and eviction order alongside that peak would show whether the budget needs to reserve headroom for transitions, especially when an active scene prevents immediate eviction.
Ahmet Özel
AI Engineer. Computer Vision, RAG and LLM agents.
Making image_model scene data removes the hardcoded-engine assumption. The illustrated _loaded_pipelines dictionary introduces a separate memory risk, though: every newly selected checkpoint remains resident, so an episode using several overrides can recreate the original OOM even when each model fits on its own.
I would give the cache an explicit residency budget and define when an inactive pipeline can be evicted. An active generation needs a lease so its pipeline cannot disappear halfway through a scene. A sequence alternating three checkpoints would be a useful regression case, recording resident memory after each switch rather than testing only a fresh process with one model.