我在一个共享计算资源的环境中工作,也就是说,我们有一些服务器机器配备了几个Nvidia Titan X GPU.
对于小到中等大小的型号,12GB的Titan X通常足以让2-3人在同一GPU上同时进行训练.如果模型足够小以至于单个模型没有充分利用Titan X的所有计算单元,那么与在另一个训练过程之后运行一个训练过程相比,这实际上可以导致加速.即使在并发访问GPU确实减慢了单个培训时间的情况下,仍然可以灵活地让多个用户同时在GPU上运行.
TensorFlow的问题在于,默认情况下,它在启动时会在GPU上分配全部可用内存.即使对于一个小的2层神经网络,我也看到12 GB的Titan X已用完.
有没有办法让TensorFlow只分配4GB的GPU内存,如果有人知道这个数量对于给定的模型来说足够了？
当我跑步时,sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})我得到了InternalError: Blas SGEMM launch failed.这是完整的错误和堆栈跟踪:
InternalErrorTraceback (most recent call last)
<ipython-input-9-a3261a02bdce> in <module>()
1 batch_xs, batch_ys = mnist.train.next_batch(100)
----> 2 sess.run(train_step, feed_dict={x: batch_xs, y_: batch_ys})
/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.pyc in run(self, fetches, feed_dict, options, run_metadata)
338 try:
339 result = self._run(None, fetches, feed_dict, options_ptr,
--> 340 run_metadata_ptr)
341 if run_metadata:
342 proto_data = tf_session.TF_GetBuffer(run_metadata_ptr)
/usr/local/lib/python2.7/dist-packages/tensorflow/python/client/session.pyc in _run(self, handle, fetches, feed_dict, options, run_metadata)
562 try:
563 results = self._do_run(handle, target_list, unique_fetches,
--> 564 feed_dict_string, options, run_metadata) …我安装了tensorflow-gpu来在我的GPU上运行我的tensorflow代码.但我不能让它运行.它继续给出上述错误.以下是我的示例代码,后跟错误堆栈跟踪:
import tensorflow as tf
import numpy as np
def check(W,X):
return tf.matmul(W,X)
def main():
W = tf.Variable(tf.truncated_normal([2,3], stddev=0.01))
X = tf.placeholder(tf.float32, [3,2])
check_handle = check(W,X)
with tf.Session() as sess:
tf.initialize_all_variables().run()
num = sess.run(check_handle, feed_dict =
{X:np.reshape(np.arange(6), (3,2))})
print(num)
if __name__ == '__main__':
main()
我的GPU是相当不错的GeForce GTX 1080 Ti,拥有11 GB的vram,并且没有其他任何重要的运行(只是chrome),你可以在nvidia-smi中看到:
Fri Aug 4 16:34:49 2017
+-----------------------------------------------------------------------------+
| NVIDIA-SMI 381.22 Driver Version: 381.22 |
|-------------------------------+----------------------+----------------------+
| GPU Name Persistence-M| Bus-Id Disp.A | Volatile Uncorr. ECC |
| Fan Temp Perf Pwr:Usage/Cap| Memory-Usage …