NVIDIA GPU

Available GPU

In order to see available GPU

nvidia-smi

Process related to GPU

Sometimes nvidia-smi show that a GPU is available, while it is inaccessible. It is most of the time due to GPU process not killed properly. You can list the processes related to the GPU's by typing

lsof /dev/nvidia*

Then you can kill the process bonded to the GPU.

kill ProcessPIDNumber

PS: Sometimes you have to repeat this command to ensure that the process has been completely terminated.

Limit the number of GPU used

If the server have 4 GPUs [0,1,2,3]

CUDA_VISIBLE_DEVICES=GPU_number python myprogram.py

ex

CUDA_VISIBLE_DEVICES=1 python myprogram.py

Depending on the program, if you don't specify it, it could use all the GPU which is usually not what we want. The other option is that if CUDA_VISIBLE_DEVICES isn't specified GPU 0 will be used by default and that GPU might be busy while others are free.