那些遇到过的问题

在调用train_on_batch,fit等时,Keras使用了太多的GPU内存

Mar*_*tin 35 memory theano keras tensorflow

我一直在搞乱Keras,到目前为止还喜欢它.在使用相当深的网络时,我遇到了一个大问题:当调用model.train_on_batch或model.fit等时,Keras分配的GPU内存比模型本身需要的多得多.这不是因为尝试训练一些非常大的图像,而是网络模型本身似乎需要大量的GPU内存.我创建了这个玩具示例来展示我的意思.这基本上是在发生什么:

我首先创建一个相当深的网络,并使用model.summary()来获取网络所需的参数总数(在本例中为206538153,相当于大约826 MB).然后我使用nvidia-smi来查看Keras分配了多少GPU内存,我可以看到它非常有意义(849 MB).

然后我编译网络,并确认这不会增加GPU内存使用量.正如我们在这种情况下所看到的,此时我有近1 GB的VRAM可用.

然后我尝试将一个简单的16x16图像和1x1基础事实提供给网络,然后一切都爆炸了,因为Keras开始再次分配大量内存,这对我来说无缘无故.关于训练网络的东西似乎需要比仅拥有模型更多的内存,这对我来说没有意义.在其他框架中,我已经在这个GPU上训练了更深层次的网络,所以这让我觉得我使用的是Keras错误(或者我的设置或者Keras中存在错误,但当然很难确定).

这是代码:

from scipy import misc
import numpy as np
from keras.models import Sequential
from keras.layers import Dense, Activation, Convolution2D, MaxPooling2D, Reshape, Flatten, ZeroPadding2D, Dropout
import os

model = Sequential()

model.add(Convolution2D(256, 3, 3, border_mode='same', input_shape=(16,16,1)))
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2)))
model.add(Convolution2D(512, 3, 3, border_mode='same'))
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2)))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(Convolution2D(1024, 3, 3, border_mode='same'))
model.add(MaxPooling2D(pool_size=(2,2), strides=(2,2)))
model.add(Convolution2D(256, 3, 3, border_mode='same'))
model.add(Convolution2D(32, 3, 3, border_mode='same'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Flatten())
model.add(Dense(4))
model.add(Dense(1))

model.summary()

os.system("nvidia-smi")
raw_input("Press Enter to continue...")    

model.compile(optimizer='sgd',
              loss='mse', 
              metrics=['accuracy'])

os.system("nvidia-smi")              
raw_input("Compiled model. Press Enter to continue...")

n_batches = 1
batch_size = 1
for ibatch in range(n_batches):
    x = np.random.rand(batch_size, 16,16,1)
    y = np.random.rand(batch_size, 1)

    os.system("nvidia-smi")
    raw_input("About to train one iteration. Press Enter to continue...")

    model.train_on_batch(x, y)         
    print("Trained one iteration")
Run Code Online (Sandbox Code Playgroud)

这为我提供了以下输出:

Using Theano backend.
Using gpu device 0: GeForce GTX 960 (CNMeM is disabled, cuDNN 5103)
/usr/local/lib/python2.7/dist-packages/theano/sandbox/cuda/__init__.py:600: UserWarning: Your cuDNN version is more recent than the one Theano officially supports. If you see any problems, try updating Theano or downgrading cuDNN to version 5.
  warnings.warn(warn)
____________________________________________________________________________________________________
Layer (type)                     Output Shape          Param #     Connected to                     
====================================================================================================
convolution2d_1 (Convolution2D)  (None, 16, 16, 256)   2560        convolution2d_input_1[0][0]      
____________________________________________________________________________________________________
maxpooling2d_1 (MaxPooling2D)    (None, 8, 8, 256)     0           convolution2d_1[0][0]            
____________________________________________________________________________________________________
convolution2d_2 (Convolution2D)  (None, 8, 8, 512)     1180160     maxpooling2d_1[0][0]             
____________________________________________________________________________________________________
maxpooling2d_2 (MaxPooling2D)    (None, 4, 4, 512)     0           convolution2d_2[0][0]            
____________________________________________________________________________________________________
convolution2d_3 (Convolution2D)  (None, 4, 4, 1024)    4719616     maxpooling2d_2[0][0]             
____________________________________________________________________________________________________
convolution2d_4 (Convolution2D)  (None, 4, 4, 1024)    9438208     convolution2d_3[0][0]            
____________________________________________________________________________________________________
convolution2d_5 (Convolution2D)  (None, 4, 4, 1024)    9438208     convolution2d_4[0][0]            
____________________________________________________________________________________________________
convolution2d_6 (Convolution2D)  (None, 4, 4, 1024)    9438208     convolution2d_5[0][0]            
____________________________________________________________________________________________________
convolution2d_7 (Convolution2D)  (None, 4, 4, 1024)    9438208     convolution2d_6[0][0]            
____________________________________________________________________________________________________
convolution2d_8 (Convolution2D)  (None, 4, 4, 1024)    9438208     convolution2d_7[0][0]            
____________________________________________________________________________________________________
convolution2d_9 (Convolution2D)  (None, 4, 4, 1024)    9438208     convolution2d_8[0][0]            
____________________________________________________________________________________________________
convolution2d_10 (Convolution2D) (None, 4, 4, 1024)    9438208     convolution2d_9[0][0]            
____________________________________________________________________________________________________
convolution2d_11 (Convolution2D) (None, 4, 4, 1024)    9438208     convolution2d_10[0][0]           
____________________________________________________________________________________________________
convolution2d_12 (Convolution2D) (None, 4, 4, 1024)    9438208     convolution2d_11[0][0]           
____________________________________________________________________________________________________
convolution2d_13 (Convolution2D) (None, 4, 4, 1024)    9438208     convolution2d_12[0][0]           
____________________________________________________________________________________________________
convolution2d_14 (Convolution2D) (None, 4, 4, 1024)    9438208     convolution2d_13[0][0]           
____________________________________________________________________________________________________
convolution2d_15 (Convolution2D) (None, 4, 4, 1024)    9438208     convolution2d_14[0][0]           
____________________________________________________________________________________________________
convolution2d_16 (Convolution2D) (None, 4, 4, 1024)    9438208     convolution2d_15[0][0]           
____________________________________________________________________________________________________
convolution2d_17 (Convolution2D) (None, 4, 4, 1024)    9438208     convolution2d_16[0][0]           
____________________________________________________________________________________________________
convolution2d_18 (Convolution2D) (None, 4, 4, 1024)    9438208     convolution2d_17[0][0]           
____________________________________________________________________________________________________
convolution2d_19 (Convolution2D) (None, 4, 4, 1024)    9438208     convolution2d_18[0][0]           
____________________________________________________________________________________________________
convolution2d_20 (Convolution2D) (None, 4, 4, 1024)    9438208     convolution2d_19[0][0]           
____________________________________________________________________________________________________
convolution2d_21 (Convolution2D) (None, 4, 4, 1024)    9438208     convolution2d_20[0][0]           
____________________________________________________________________________________________________
convolution2d_22 (Convolution2D) (None, 4, 4, 1024)    9438208     convolution2d_21[0][0]           
____________________________________________________________________________________________________
convolution2d_23 (Convolution2D) (None, 4, 4, 1024)    9438208     convolution2d_22[0][0]           
____________________________________________________________________________________________________
convolution2d_24 (Convolution2D) (None, 4, 4, 1024)    9438208     convolution2d_23[0][0]           
____________________________________________________________________________________________________
maxpooling2d_3 (MaxPooling2D)    (None, 2, 2, 1024)    0           convolution2d_24[0][0]           
____________________________________________________________________________________________________
convolution2d_25 (Convolution2D) (None, 2, 2, 256)     2359552     maxpooling2d_3[0][0]             
____________________________________________________________________________________________________
convolution2d_26 (Convolution2D) (None, 2, 2, 32)      73760       convolution2d_25[0][0]           
____________________________________________________________________________________________________
maxpooling2d_4 (MaxPooling2D)    (None, 1, 1, 32)      0           convolution2d_26[0][0]           
____________________________________________________________________________________________________
flatten_1 (Flatten)              (None, 32)            0           maxpooling2d_4[0][0]             
____________________________________________________________________________________________________
dense_1 (Dense)                  (None, 4)             132         flatten_1[0][0]                  
____________________________________________________________________________________________________
dense_2 (Dense)                  (None, 1)             5           dense_1[0][0]                    
====================================================================================================
Total params: 206538153
____________________________________________________________________________________________________
None
Thu Oct  6 09:05:42 2016       
+------------------------------------------------------+                       
| NVIDIA-SMI 352.63     Driver Version: 352.63         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 960     Off  | 0000:01:00.0      On |                  N/A |
| 30%   37C    P2    28W / 120W |   1082MiB /  2044MiB |      9%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      1796    G   /usr/bin/X                                     155MiB |
|    0      2597    G   compiz                                          65MiB |
|    0      5966    C   python                                         849MiB |
+-----------------------------------------------------------------------------+
Press Enter to continue...
Thu Oct  6 09:05:44 2016       
+------------------------------------------------------+                       
| NVIDIA-SMI 352.63     Driver Version: 352.63         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 960     Off  | 0000:01:00.0      On |                  N/A |
| 30%   38C    P2    28W / 120W |   1082MiB /  2044MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      1796    G   /usr/bin/X                                     155MiB |
|    0      2597    G   compiz                                          65MiB |
|    0      5966    C   python                                         849MiB |
+-----------------------------------------------------------------------------+
Compiled model. Press Enter to continue...
Thu Oct  6 09:05:44 2016       
+------------------------------------------------------+                       
| NVIDIA-SMI 352.63     Driver Version: 352.63         |                       
|-------------------------------+----------------------+----------------------+
| GPU  Name        Persistence-M| Bus-Id        Disp.A | Volatile Uncorr. ECC |
| Fan  Temp  Perf  Pwr:Usage/Cap|         Memory-Usage | GPU-Util  Compute M. |
|===============================+======================+======================|
|   0  GeForce GTX 960     Off  | 0000:01:00.0      On |                  N/A |
| 30%   38C    P2    28W / 120W |   1082MiB /  2044MiB |      0%      Default |
+-------------------------------+----------------------+----------------------+

+-----------------------------------------------------------------------------+
| Processes:                                                       GPU Memory |
|  GPU       PID  Type  Process name                               Usage      |
|=============================================================================|
|    0      1796    G   /usr/bin/X                                     155MiB |
|    0      2597    G   compiz                                          65MiB |
|    0      5966    C   python                                         849MiB |
+-----------------------------------------------------------------------------+
About to train one iteration. Press Enter to continue...
Error allocating 37748736 bytes of device memory (out of memory). Driver report 34205696 bytes free and 2144010240 bytes total 
Traceback (most recent call last):
  File "memtest.py", line 65, in <module>
    model.train_on_batch(x, y)         
  File "/usr/local/lib/python2.7/dist-packages/keras/models.py", line 712, in train_on_batch
    class_weight=class_weight)
  File "/usr/local/lib/python2.7/dist-packages/keras/engine/training.py", line 1221, in train_on_batch
    outputs = self.train_function(ins)
  File "/usr/local/lib/python2.7/dist-packages/keras/backend/theano_backend.py", line 717, in __call__
    return self.function(*inputs)
  File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 871, in __call__
    storage_map=getattr(self.fn, 'storage_map', None))
  File "/usr/local/lib/python2.7/dist-packages/theano/gof/link.py", line 314, in raise_with_op
    reraise(exc_type, exc_value, exc_trace)
  File "/usr/local/lib/python2.7/dist-packages/theano/compile/function_module.py", line 859, in __call__
    outputs = self.fn()
MemoryError: Error allocating 37748736 bytes of device memory (out of memory).
Apply node that caused the error: GpuContiguous(GpuDimShuffle{3,2,0,1}.0)
Toposort index: 338
Inputs types: [CudaNdarrayType(float32, 4D)]
Inputs shapes: [(1024, 1024, 3, 3)]
Inputs strides: [(1, 1024, 3145728, 1048576)]
Inputs values: ['not shown']
Outputs clients: [[GpuDnnConv{algo='small', inplace=True}(GpuContiguous.0, GpuContiguous.0, GpuAllocEmpty.0, GpuDnnConvDesc{border_mode='half', subsample=(1, 1), conv_mode='conv', precision='float32'}.0, Constant{1.0}, Constant{0.0}), GpuDnnConvGradI{algo='none', inplace=True}(GpuContiguous.0, GpuContiguous.0, GpuAllocEmpty.0, GpuDnnConvDesc{border_mode='half', subsample=(1, 1), conv_mode='conv', precision='float32'}.0, Constant{1.0}, Constant{0.0})]]

HINT: Re-running with most Theano optimization disabled could give you a back-trace of when this node was created. This can be done with by setting the Theano flag 'optimizer=fast_compile'. If that does not work, Theano optimizations can be disabled with 'optimizer=None'.
HINT: Use the Theano flag 'exception_verbosity=high' for a debugprint and storage map footprint of this apply node.
Run Code Online (Sandbox Code Playgroud)

有几点需要注意:

  • 我试过了Theano和TensorFlow后端.两者都有相同的问题,并在同一行中耗尽内存.在TensorFlow中,似乎Keras预先分配了大量内存(大约1.5 GB),因此nvidia-smi无法帮助我们跟踪那里发生的事情,但我得到了相同的内存异常.再次,这指向(我的使用)Keras的错误(尽管很难确定这些事情,它可能是我的设置的东西).
  • 我尝试在Theano中使用CNMEM,其行为类似于TensorFlow:它预先分配了大量内存(大约1.5 GB)但在同一个地方崩溃.
  • 关于CudNN版本有一些警告.我尝试使用CUDA而不是CudNN运行Theano后端,但我得到了相同的错误,因此这不是问题的根源.
  • 如果您想在自己的GPU上进行测试,您可能希望根据您需要测试多少GPU内存来使网络更深/更浅.
  • 我的配置如下:Ubuntu 14.04,GeForce GTX 960,CUDA 7.5.18,CudNN 5.1.3,Python 2.7,Keras 1.1.0(通过pip安装)
  • 我已经尝试更改模型的编译以使用不同的优化器和损失,但这似乎没有改变任何东西.
  • 我已经尝试将train_on_batch函数更改为使用fit,但它有同样的问题.
  • 我在StackOverflow上看到了一个类似的问题 - 为什么这个Keras模型需要超过6GB的内存？ - 但据我所知,我的配置中没有这些问题.我从来没有安装过多个版本的CUDA,而且我已经多次检查了我的PATH,LD_LIBRARY_PATH和CUDA_ROOT变量,而不是我可以计算的次数.
  • Julius建议激活参数本身占用GPU内存.如果这是真的,有人可以更清楚地解释一下吗？我已经尝试将我的卷积层的激活函数更改为明确硬编码的函数,据我所知,没有可学习的参数,这并没有改变任何东西.而且,这些参数似乎不太可能占用与网络本身其余部分一样多的内存.
  • 经过全面测试,我可以训练的最大网络是大约453 MB的参数,在我的~2 GB GPU内存中.这是正常的吗？
  • 在对我的GPU中适合的一些较小的CNN测试Keras后,我可以看到GPU RAM的使用非常突然.如果我运行一个大约100 MB参数的网络,99%的时间在训练期间它将使用少于200 MB的GPU RAM.但每隔一段时间,内存使用量就会达到1.3 GB左右.假设这些尖峰导致我的问题似乎是安全的.我从来没有在其他框架中看到过这些峰值,但它们可能有充分的理由存在吗？如果有人知道是什么原因造成的,如果有办法避免它们,请加入!

Jul*_*.M. 18

忘记激活和渐变也采用vram,而不仅仅是参数,增加内存使用量是一个非常常见的错误.backprob计算本身使得训练阶段几乎是神经网络的前向/推断使用的VRAM的两倍.

因此,在创建网络的开始时,仅分配参数.但是,当训练开始时,激活(每个小批量的时间)被分配,以及反向计算,增加了内存使用.

  • 对于1 GB的gpu,你的网络非常庞大.删除一堆图层 (5认同)
  • @Martin我不认为keras根本没有进行任何内存分配.它只是Tensorflow或Theano的包装器(内部可能会将任务转移到某些nvidia驱动程序). (4认同)
  • 你看到现在使用的非常深的网络是在12 GB vram gpus(大多数是最近的tesla k80)上训练的,不是在1 GB卡上. (2认同)
  • 一些事情:Julius,它是一个2 GB的GPU.我希望能够使用~1 GB的网络,因为我之前已经将它与其他框架一起使用了.这不是关于如何正确设计CNN的问题.这是一个玩具示例,用于显示Keras如何分配VRAM的问题. (2认同)
  • 你的网络比你想象的要大.它真的很大! (2认同)

ind*_*you 13

Theano和Tensorflow都增加了创建的符号图,尽管两者都不同.

要分析内存消耗的发生方式,您可以从较小的模型开始,然后将其增长以查看内存中的相应增长.同样,您可以增长batch_size以查看内存中的相应增长.

以下是batch_size基于初始代码增加的代码段:

from scipy import misc
import numpy as np
from keras.models import Sequential
from keras.layers import Dense, Activation, Convolution2D, MaxPooling2D, Reshape, Flatten, ZeroPadding2D, Dropout
import os
import matplotlib.pyplot as plt


def gpu_memory():
    out = os.popen("nvidia-smi").read()
    ret = '0MiB'
    for item in out.split("\n"):
        if str(os.getpid()) in item and 'python' in item:
            ret = item.strip().split(' ')[-2]
    return float(ret[:-3])

gpu_mem = []
gpu_mem.append(gpu_memory())

model = Sequential()
model.add(Convolution2D(100, 3, 3, border_mode='same', input_shape=(16,16,1)))
model.add(Convolution2D(256, 3, 3, border_mode='same'))
model.add(Convolution2D(32, 3, 3, border_mode='same'))
model.add(MaxPooling2D(pool_size=(2,2)))
model.add(Flatten())
model.add(Dense(4))
model.add(Dense(1))

model.summary()
gpu_mem.append(gpu_memory())

model.compile(optimizer='sgd',
              loss='mse', 
              metrics=['accuracy'])
gpu_mem.append(gpu_memory())


batches = []
n_batches = 20
batch_size = 1
for ibatch in range(n_batches):
    batch_size = (ibatch+1)*10
    batches.append(batch_size)
    x = np.random.rand(batch_size, 16,16,1)
    y = np.random.rand(batch_size, 1)
    print y.shape

    model.train_on_batch(x, y)         
    print("Trained one iteration")

    gpu_mem.append(gpu_memory())

fig = plt.figure()
plt.plot([-100, -50, 0]+batches, gpu_mem)
plt.show()
Run Code Online (Sandbox Code Playgroud)

此外,对于速度Tensorflow占用了所有可用的GPU内存.要停止这一点,你需要添加config.gpu_options.allow_growth = True在get_session()

# keras/backend/tensorflow_backend.py
def get_session():
    global _SESSION
    if tf.get_default_session() is not None:
        session = tf.get_default_session()
    else:
        if _SESSION is None:
            if not os.environ.get('OMP_NUM_THREADS'):
                config = tf.ConfigProto(allow_soft_placement=True,
                    )
            else:
                nb_thread = int(os.environ.get('OMP_NUM_THREADS'))
                config = tf.ConfigProto(intra_op_parallelism_threads=nb_thread,
                                        allow_soft_placement=True)
            config.gpu_options.allow_growth = True
            _SESSION = tf.Session(config=config)
        session = _SESSION
    if not _MANUAL_VAR_INIT:
        _initialize_variables()
    return session
Run Code Online (Sandbox Code Playgroud)

现在,如果你运行prev片段,你会得到如下情节:

Theano: Theano Tensorflow: Tenserflow

Theano:model.compile()无论需要什么记忆,在训练开始时,它几乎加倍.这是因为Theano增加了符号图以进行反向传播,并且每个张量需要相应的张量来实现梯度的向后流动.内存需求似乎没有增长,batch_size这对我来说是意料之外的,因为占位符大小应该增加以适应来自CPU-> GPU的数据流入.

Tensorflow:即使在Keras model.compile()没有呼叫get_session()直到实际呼叫的时间之后,也没有分配GPU内存_initialize_variables().Tensorflow似乎为了速度而在内存中占用内存,因此内存不会线性增长batch_size.

说完所有Tensorflow似乎都是内存饥渴,但对于大图来说它非常快......另一方面Theano非常高效,但在训练开始时需要花费大量时间来初始化图形.之后它也很快.

归档时间：

查看次数：

15292 次

最近记录：

7 年，6 月 前
为什么这款Keras型号需要超过6GB的内存？ 8
更多相关链接
相关归档
使用什么来获得Spark内存使用情况的图形视图(使用YARN)？ 9
如何从 keras 中的双向 LSTM 中提取状态 c 和 h 7
为什么远远超过CPU缓存大小时内存访问时间会增加 7
多 GPU 训练比 Tensorflow 上的单 GPU 慢 7
在张量流中创建三角矩阵 6
某些阵列大小的性能减损 5
将编译标志传递给 bazel (TensorFlow) 5
经过重新训练的keras模型评估在循环中调用时会泄漏内存 5
摆脱基于文件的通信 3
Keras 中的两个并行但不同的数据集作为多个输入？ 2
难疑归档
如何将空目录添加到Git存储库？ 4039
Dockerfile中的`COPY`和`ADD`命令有什么区别？ 1991
如何将Git托管项目中的所有本地更改还原到以前的状态？ 1830
如何在Vim中进行不区分大小写的搜索 1579
如何在Python中追加文件？ 1446
int32的最大值是多少？ 1383
插入...值(SELECT ... FROM ...) 1340
虚拟成员在构造函数中调用 1270
CSS calc()函数中的Sass变量 1182
如何随机化(shuffle)一个JavaScript数组？ 1138