[Bug]: RAM Memory leak issue - RAM consumption keeps increasing

Is there an existing issue for this?

• I have searched the existing issues and checked the recent builds/commits of both this extension and the webui

What happened?

When trying to use the extension without any model cache being switched on, after 10 tries, the SSH connection to my EC2 instance fails. This always happens during build_controlnet_model function, because the output : Loading model {model} gets printed but the loading state_dict does not.

Steps to reproduce the problem

1. Use AWS EC2 instance
2. I am running bash webui.sh with which the entire program starts and set model cache off, apply settings and restart without any args
3. Generate images using img2img roughly 10 times
4. Somewhere inbetween the SSH connection gets disconneted exactly after stating loading model [model name] - this is not limited to one model, happens across all models.

What should have happened?

The SSH connection should not have gotten closed. There seems to be some error here. It happens suddenly as well without any proper reproducible number of steps. But it does fail. It doesn't fail for normal automatic1111, and is a controlnet web-ui problem

Commit where the problem happens

webui: [22bcc7be] controlnet: 2270f364e167b9531daf9a8bd1d62cb2dbfa4d00

What browsers do you use to access the UI ?

Google Chrome

Command Line Arguments

No

Console logs

Usually it is supposed to be:

Loading model: control_v11p_sd15_inpaint [ebff9138]
Loaded state_dict from [/home/ec2-user/stable-diffusion-webui/models/ControlNet/control_v11p_sd15_inpaint.pth]
Loading config: /home/ec2-user/stable-diffusion-webui/extensions/sd-webui-controlnet/models/control_v11p_sd15_inpaint.yaml
ControlNet model control_v11p_sd15_inpaint [ebff9138] loaded.

But while it fails it stops at:

Loading model: control_v11p_sd15_inpaint [ebff9138] 

The rest of the log isn't visible and SSH gets disconnected.

Additional information

Happens with model cache as well. But after many more tries