TENSorflow多个会话与多个GPU

Question

我有一个带有2个GPU的工作站,我正在尝试同时运行多个tensorflow作业,所以我可以同时训练多个模型,等等.

例如,我试图通过在script1.py中使用python API将会话分成不同的资源:

with tf.device("/gpu:0"):
    # do stuff

在script2.py中:

with tf.device("/gpu:1"):
    # do stuff

在script3.py中

with tf.device("/cpu:0"):
    # do stuff

如果我自己运行每个脚本,我可以看到它正在使用指定的设备.(此外,这些模型非常适合单个GPU,即使两者都可用,也不会使用另一个.)

但是,如果一个脚本正在运行并且我尝试运行另一个脚本,我总是会收到此错误:

I tensorflow/core/common_runtime/local_device.cc:40] Local device intra op parallelism threads: 8
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:909] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:103] Found device 0 with properties: 
name: GeForce GTX 980
major: 5 minor: 2 memoryClockRate (GHz) 1.2155
pciBusID 0000:01:00.0
Total memory: 4.00GiB
Free memory: 187.65MiB
I tensorflow/stream_executor/cuda/cuda_gpu_executor.cc:909] successful NUMA node read from SysFS had negative value (-1), but there must be at least one NUMA node, so returning NUMA node zero
I tensorflow/core/common_runtime/gpu/gpu_init.cc:103] Found device 1 with properties: 
name: GeForce GTX 980
major: 5 minor: 2 memoryClockRate (GHz) 1.2155
pciBusID 0000:04:00.0
Total memory: 4.00GiB
Free memory: 221.64MiB
I tensorflow/core/common_runtime/gpu/gpu_init.cc:127] DMA: 0 1 
I tensorflow/core/common_runtime/gpu/gpu_init.cc:137] 0:   Y Y 
I tensorflow/core/common_runtime/gpu/gpu_init.cc:137] 1:   Y Y 
I tensorflow/core/common_runtime/gpu/gpu_device.cc:702] Creating    TensorFlow device (/gpu:0) -> (device: 0, name: GeForce GTX 980, pci bus id: 0000:01:00.0)
I tensorflow/core/common_runtime/gpu/gpu_device.cc:702] Creating TensorFlow device (/gpu:1) -> (device: 1, name: GeForce GTX 980, pci bus id: 0000:04:00.0)
I tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:42] Allocating 187.40MiB bytes.
E tensorflow/stream_executor/cuda/cuda_driver.cc:932] failed to allocate 187.40M (196505600 bytes) from device: CUDA_ERROR_OUT_OF_MEMORY
F tensorflow/core/common_runtime/gpu/gpu_bfc_allocator.cc:47] Check failed: gpu_mem != nullptr  Could not allocate GPU device memory for device 0. Tried to allocate 187.40MiB
Aborted (core dumped)

似乎每个张量流程都试图在加载时抓住机器上的所有GPU,即使并非所有设备都将用于运行模型.

我看到有一个选项可以限制每个进程使用的GPU数量

tf.GPUOptions(per_process_gpu_memory_fraction=0.5)

...我还没有尝试过,但这似乎会让两个进程尝试共享每个GPU的50%而不是在单独的GPU上运行每个进程...

有谁知道如何配置tensorflow只使用一个GPU并让另一个可用于另一个张量流程？

Answer 1

TensorFlow将尝试使用对其可见的所有GPU设备(相当于一部分内存).如果要在不同的GPU上运行不同的会话,则应执行以下操作.

1. 在不同的Python进程中运行每个会话.

2. 使用CUDA_VISIBLE_DEVICES环境变量的不同值启动每个进程.例如,如果您的脚本被调用my_script.py并且您有4个GPU,则可以运行以下命令:

   $ CUDA_VISIBLE_DEVICES=0 python my_script.py  # Uses GPU 0.
   $ CUDA_VISIBLE_DEVICES=1 python my_script.py  # Uses GPU 1.
   $ CUDA_VISIBLE_DEVICES=2,3 python my_script.py  # Uses GPUs 2 and 3.
   

   请注意,TensorFlow中的GPU设备仍将从零开始编号(即"/gpu:0"等),但它们将对应于您可见的设备CUDA_VISIBLE_DEVICES.