通常情况下，停止进程显存会释放
但是如果在不正常情况关闭进程，可能不会释放，这个时候就会出现这样的情况：
Mon Oct 19 16:00:00 2020
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 410.104 Driver Version: 410.104 CUDA Version: 10.0 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage | GPU-Util Compute M. |
|===============================+======================+======================|
| 0 Tesla P100-PCIE... Off | 00000000:00:0D.0 Off | 0 |
| N/A 38C P0 35W / 250W | 16239MiB / 16280MiB | 0% Default |
+-------------------------------+----------------------+----------------------+
+-----------------------------------------------------------------------------+
| Processes: GPU Memory |
| GPU PID Type Process name Usage |
|=============================================================================|
+-----------------------------------------------------------------------------+
解决方式，当然是干掉正常使用显存的进程
想要释放进程，当然需要找到进程
fuser -v /dev/nvidia*
USER PID ACCESS COMMAND
/dev/nvidia0: root 26031 F...m python
root 26035 F...m python
root 26041 F...m python
root 26050 F...m python
root 32512 F...m ZMQbg/1
/dev/nvidiactl: root 26031 F...m python
root 26035 F...m python
root 26041 F...m python
root 26050 F...m python
root 32512 F.... ZMQbg/1
/dev/nvidia-uvm: root 26031 F.... python
root 26035 F.... python
root 26041 F.... python
root 26050 F.... python
root 32512 F.... ZMQbg/1
然后使用kill -9 26031杀死进程，进程 释放资源，需要一次对上面查询到的进程进行依次kill
不出意外就正常了：
--------------------------------------------------------------------------分割线----------------------------------------------
如果想一行命令全部搞定，直接使用：
fuser -v /dev/nvidia* |awk '{for(i=1;i<=NF;i++)print "kill -9 " $i;}' | sh