[FEATURE] Profile Memory for Index Build Service

Is your feature request related to a problem?

The Index Build Service orchestrates the index build process in three steps

1. Building a GPU Index
2. Converting a CPU index to GPU Index
3. Writing a CPU Index to disc. The artifacts from the previous phase are eagerly deleted during this process. We would like to profile the GPU and CPU memory consumption over the course of this process

What solution would you like?

Specifically GPU and CPU memory consumption before and after every phase of execution must be observed to verify the right amount of memory was assigned and successfully cleaned up after every phase or an error condition.

What alternatives have you considered?

For measuring RAM consumed by CPU, we used Memory Profiler For measuring NVIDIA GPU memory we used nvidia_smi from p3nvml

Do you have any additional context?

Faiss Index Build Process

1. The service allocates GPU memory when a faiss GpuIndex is instantiated
2. It then allocates CPU RAM memory when the GpuIndex.copyTo method is called to generate a CPU compatible index. The GPU Index is deleted after this.
3. The CPU Index is persisted to storage and deleted from main memory.