Mesh/material memory leak

Maybe related to https://github.com/ignitionrobotics/ign-rendering/issues/39

After investigating this issue more, it is more suited to be inside https://github.com/ignitionrobotics/ign-rendering as it occurs for both GUI and camera sensors. Please move the issue there if possible.

Environment

• OS Version: Ubuntu 20.04
• Ignition - built from source
  • Both Dome and Edifice experience this issue
• Ogre Version:
  • Both ogre and ogre2 (2.1) experience this issue

Description

• Expected behavior: Various models can be added and removed arbitrary number of times without running out of memory.
• Actual behavior: Mesh/material (texture) is not freed after a model is removed. It affects both GUI and camera sensors. For GUI, reinserting the same model repeatedly introduces some extra memory usage, but at much smaller magnitude (negligible in the grand scheme of things).

I should note that this behaviour might be advantageous for headless simulation if an environment repeatedly utilises a limited number of models, as reinsertion of a model is much faster if its resources already loaded to memory. Duality of bug/feature is real with this one. Therefore, having an option to allow both behaviours might be preferable, i.e option A - unload all resources after removing model, option B - keep the resources (with some policy that makes sure the system does not run out of memory/exceed some threshold).

Steps to reproduce

1. ign gazebo -s
2. ign gazebo -g
3. Open RAM usage monitor of your choice. Optionally, monitor VRAM usage as well.
4. Run the following example script ign_gui_memory_leak_reproducibility_script.bash (gist)
   • This script iteratively inserts and removes models with mesh geometry and material texture using service calls
   • Note: Script downloads models from Fuel (total of ~650MB)
   • Note: About ~4GB RAM will be leaked with the utilised 50 models

#!/usr/bin/env bash

## Get list of first <n_models> models from <owner> (using Fuel)
owner="googleresearch"
n_models=50 # 50 corresponds to ~650MB disk usage and ~4GB RAM (leaked) usage, use less if your system does not allow it
echo "Getting list of '$n_models' models owned by '$owner'. This might take few minutes during the first execution... Please be patient :)"
model_names=$(ign fuel list -o $owner -t model -r | head -n $n_models | sed -n -e 's/^.*models\///p')

## Download models if needed (done separately because "/world/default/create" service might timeout)
for model_name in $model_names; do
    if [[ ! -d "$HOME/.ignition/fuel/fuel.ignitionrobotics.org/$owner/models/$model_name" ]]; then
        model_uri="https://fuel.ignitionrobotics.org/1.0/$owner/models/$model_name"
        echo "Downloading model '$model_uri'"
        ign fuel download -t model -u "$model_uri" &
    fi
done
for job in $(jobs -p); do
    wait $job
    echo "Model downloaded"
done

world_name="default"
# world_name="camera_sensor"
for model_name in $model_names; do
    ## Spawn model
    echo "Spawning model $model_name"
    model_uri="https://fuel.ignitionrobotics.org/1.0/$owner/models/$model_name"
    ign service -s "/world/$world_name/create" --timeout 5000 \
        --reqtype ignition.msgs.EntityFactory \
        --reptype ignition.msgs.Boolean \
        -r 'sdf_filename: "'$model_uri'" name: "'$model_name'"' 1>/dev/null
    sleep 0.5
    ## Remove model
    echo "Removing model $model_name"
    ign service -s "/world/$world_name/remove" --timeout 5000 \
        --reqtype ignition.msgs.Entity \
        --reptype ignition.msgs.Boolean \
        -r 'type: 2 name: "'$model_name'"' 1>/dev/null
    echo ""
done

Service calls are used here to easy reproducibility. This issue occurs also when using C++ API directly (I originally experienced the issue while using gym-ignition).

Output

Below is a video of performing the steps above. Notice also that the aligned bounding boxes of objects remain visible if object was removed while selected (and they cannot be removed). Speculation: This might be the small negligible amount of memory that accumulates on model reinsertion for GUI (or part of it).

https://user-images.githubusercontent.com/22929099/113486605-00994180-94b4-11eb-8945-597deb17d6c7.mp4

I tried to investigate the issue with Heaptrack, however, only a fraction of the leaked memory gets logged (as far as I can see). Peak resident memory (RSS) matches the total RAM usage (4.8GB), but I was not able to figure out what the largest contributor is. The mesh/texture data is not logged. I am not sure if it's caused by having Ruby in the loop or because rendering engine is loaded as plugin?

Log file: heaptrack.ruby.3172402.gz

Summary:

Consumption (each spike is insertion of a new model):

Overlapping collision geometries

While making the reproducibility example, I also noticed that if I resume the simulation after all the model insertions/deletions, the server freezes completely and outputs a bunch of ODE collision-related messages. I have seen these before when two or more models have their collision geometry largely overlapping. Therefore, it seems the collision geometry is not immediately removed from server when the simulation is paused. Is this a design choice or a bug?

ODE Message 2: Trimesh-trimesh contach hash table bucket overflow - close contacts might not be culled in AddContactToNode() [collision_trimesh_trimesh.cpp:224]