在Google Colaboratory上,GPU对Pytorch的执行速度比CPU慢

Question

GPU在大约16秒内训练这个网络.CPU在大约13秒内完成.(我没有评论/评论适当的行来进行测试).谁能看到我的代码或pytorch安装有什么问题？(我已经检查过GPU是否可用,并且GPU上有足够的可用内存.

from os import path
from wheel.pep425tags import get_abbr_impl, get_impl_ver, get_abi_tag
platform = '{}{}-{}'.format(get_abbr_impl(), get_impl_ver(), get_abi_tag())

accelerator = 'cu80' if path.exists('/opt/bin/nvidia-smi') else 'cpu'
print(accelerator)
!pip install -q http://download.pytorch.org/whl/{accelerator}/torch-0.4.0-{platform}-linux_x86_64.whl torchvision
print("done")

#########################

import torch
from datetime import datetime

startTime = datetime.now()

dtype = torch.float
device = torch.device("cpu") # Comment this to run on GPU
# device = torch.device("cuda:0") # Uncomment this to run on GPU

# N is batch size; D_in is input dimension;
# H is hidden dimension; D_out is output dimension.
N, D_in, H, D_out = 64, 1024, 128, 8

# Create random Tensors to hold input and outputs.
x = torch.randn(N, D_in, device=device, dtype=dtype)
t = torch.randn(N, D_out, device=device, dtype=dtype)

# Create random Tensors for weights.
w1 = torch.randn(D_in, H, device=device, dtype=dtype, requires_grad=True)
w2 = torch.randn(H, D_out, device=device, dtype=dtype, requires_grad=True)
w3 = torch.randn(D_out, D_out, device=device, dtype=dtype, requires_grad=True)

learning_rate = 1e-9
for i in range(10000):
    y_pred = x.mm(w1).clamp(min=0).mm(w2).clamp(min=0).mm(w3)

    loss = (y_pred - t).pow(2).sum()

    if i % 1000 == 0:
        print(i, loss.item())

    loss.backward()

    # Manually update weights using gradient descent
    with torch.no_grad():
        w1 -= learning_rate * w1.grad
        w2 -= learning_rate * w2.grad

        # Manually zero the gradients after updating weights
        w1.grad.zero_()
        w2.grad.zero_()

print(datetime.now() - startTime)

Answer 1

我看到你正在计划你不应该计时的东西(dtype,device,...的定义).这里有趣的是创建输入,输出和权重张量.

startTime = datetime.now()
# Create random Tensors to hold input and outputs.
x = torch.randn(N, D_in, device=device, dtype=dtype)
t = torch.randn(N, D_out, device=device, dtype=dtype)
torch.cuda.synchronize()
print(datetime.now()-startTime)

# Create random Tensors for weights.
startTime = datetime.now()
w1 = torch.randn(D_in, H, device=device, dtype=dtype, requires_grad=True)
w2 = torch.randn(H, D_out, device=device, dtype=dtype, requires_grad=True)
w3 = torch.randn(D_out, D_out, device=device, dtype=dtype, requires_grad=True)
torch.cuda.synchronize()
print(datetime.now()-startTime)

和训练循环

startTime = datetime.now()

for i in range(10000):
    y_pred = x.mm(w1).clamp(min=0).mm(w2).clamp(min=0).mm(w3)

    loss = (y_pred - t).pow(2).sum()

    if i % 1000 == 0:
        print(i, loss.item())

    loss.backward()

    # Manually update weights using gradient descent
    with torch.no_grad():
        w1 -= learning_rate * w1.grad
        w2 -= learning_rate * w2.grad

        # Manually zero the gradients after updating weights
        w1.grad.zero_()
        w2.grad.zero_()
torch.cuda.synchronize()
print(datetime.now() - startTime)

为什么GPU速度较慢

我使用GTX1080和非常好的CPU在我的机器上运行它,因此绝对时间较低,但解释应该仍然有效.如果您打开Jupyter笔记本并在CPU上运行它:

0:00:00.001786 time to create input/output tensors
0:00:00.003359 time to create weight tensors
0:00:04.030797 time to run training loop

现在你将设备设置为cuda,我们将其称为"冷启动"(以前没有任何内容在此笔记本中的GPU上运行)

0:00:03.180510 time to create input/output tensors
0:00:00.000642 time to create weight tensors
0:00:03.534751 time to run training loop

您可以看到运行训练循环的时间减少了一小部分,但是有3秒的开销,因为您需要将张量从CPU移动到GPU RAM.

如果再次运行它而不关闭Jupyter笔记本:

0:00:00.000421 time to create input/output tensors
0:00:00.000733 time to create weight tensors
0:00:03.501581 time to run training loop

开销消失了,因为Pytorch使用缓存内存分配器来加快速度.

您可以注意到,您在训练循环中获得的加速非常小,这是因为您正在进行的操作是在相当小的张量上.在处理小型架构和数据时,我总是进行快速测试,看看我是否真的通过在GPU上运行来获得任何东西.例如,如果我设置N, D_in, H, D_out = 64, 5000, 5000, 8,训练循环在GTX1080上运行3.5秒,在CPU上运行85秒.