Multiple Inference Pipeline cause GPU memory leak

[erwin132]

Hi!
My name is Erwin, I am on team of AI, and we currently experimenting using inference pipeline. We trying to make multiple AI CCTV Camera stream with inference pipeline.

My basic system,

1. FastAPI for the restAPI (this is the "gate" for adding camera stream and view the annotated stream)
2. Inference Pipeline

• we wanted to use inference pipeline because the capability of multiple thread and clean from RAM leak (previously we tried using plain thread module from python, it didn't go well, because of RAM leak or leftover, even we already remove the camera)
• We initialized new Inference Pipeline instance / object, each time we initiate camera or AI module
  for example
  camera1
  has 3 AI module
• object detection
• face recognition
• license plate recognition

---

3 Inference Pipeline intialized

Then,
If we want to remove the camera, we delete that instance / object of Inference Pipeline

Problem,

1. After deletion the VRAM GPU is decrease, but... there is a hiccup.

for example,
if I initiated camera1 (initial VRAM = 100MB), VRAM will increase let's say (VRAM = 200 MB), and then I stopped the camera1, the VRAM is decrease, but there is little left over (VRAM = 120MB). There is 20MB stuck in out VRAM.

2. I also already done a bit of analysis, I don't know exactly how to describe, maybe you can help

So in here I tried to benchmark the VRAM GPU
My scenario in this benchmark is
I'm trying to turn on one camera, then turn it off; turn on two cameras, then turn off two cameras, and so on. My observations:

• Graph for one camera: there’s an upward trend, which is bad.
• Graph for two cameras: the trend is stable.
• Graph for three cameras: it’s stable at around 4400 - 4500 MB, with a slightly downward trend.
• Graph for four cameras: stable at around 4600 - 4700 MB.

Stable means that if a new camera is added when it's in the stable range, the increase in RAM isn’t too high, compared to the first and second times a camera was added.

My system,

PyPI Package:
Inference-GPU: 0.23.0

Software & Hardware
OS: Ubuntu 22.04.5 LTS x86_64
GPU: Tesla P4 8GB (Nvidia driver: 535.183.01 )
CPU : Intel Xeon Silver 4216 (12) @ 2.095GHz

[PawelPeczek-Roboflow]

hi there - sorry for late response, just realised this question was asked.

few questions:

• what models do you run through the pipeline? Roboflow ones or your custom ones?
  • if Roboflow models are in use, the left-over in memory may be model manager which has capability to keep up to N models in memory - this can be limited by MAX_ACTIVE_MODELS env variable
• what is the methodology of termination - do you proceed with the InferencePipeline.join() awaiting for the graceful shutdown?
• does the trend continue up to OOM, or allocation of memory backs to normal when we are about to OOM, but we need new allocation
• is the memory enlisted as belonging to specific process in nvidia-smi?

[PawelPeczek-Roboflow]

and also - the setup seems to be fitting what we prepared for video-processing inside inference server: https://inference.roboflow.com/workflows/video_processing/overview/ this is not yet fully stable, but may fit your needs