Tensorflow CUDA - CUPTI error: CUPTI could not be loaded or symbol could not be found

Question

I use the Tensorflow v 1.14.0. I work on Windows 10. And here is how relevant environment variables look in the PATH:

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\bin
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\libnvvp
C:\Program Files (x86)\NVIDIA Corporation\PhysX\Common
C:\Users\sinthes\AppData\Local\Programs\Python\Python37
C:\Users\sinthes\AppData\Local\Programs\Python\Python37\Scripts
C:\Program Files\NVIDIA Corporation\NVIDIA NvDLISR
C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\cuda\bin

Maybe also worth to mention, just in case it might be relevant.. I use Sublime Text 3 for development and I do not use Anaconda. I find it a bit cumbersome to make updates on tensorflow in the conda environment so I just use Sublime Text right now. (I was using Anaconda (Spyder) previously but I uninstalled it from my computer.)

Things seem to work fine except with some occasional strange warnings. But one consistent warning I get is the following whenever I run the fit function.

E tensorflow/core/platform/default/device_tracer.cc:68] CUPTI error: CUPTI could not be loaded or symbol could not be found.

And here is how I call the fit function:

history = model.fit(x=train_x,
                    y=train_y,
                    batch_size=BATCH_SIZE,
                    epochs=110,
                    verbose=2,
                    callbacks=[tensorboard, checkpoint, reduce_lr_on_plateau],
                    validation_data=(dev_x, dev_y),
                    shuffle=True,
                    class_weight=class_weight,
                    steps_per_epoch=None,
                    validation_steps=None)

I just wonder why I see the CUPTI Error message during the run time? It is only printed out once. Is that something that I need to fix or is it something that can be ignored? This message does not tell anything concrete to me to be able to take any action.

Answer 1

在pathWindows 中添加这个：

C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.0\extras\CUPTI\libx64

Answer 2

NVIDIA® CUDA 分析工具接口 (CUPTI) 是一个动态库，可用于创建针对 CUDA 应用程序的分析和跟踪工具。

CPUTI 似乎已由 Tensorflow 开发人员添加以允许分析。如果您不介意异常或调整您的环境路径，您可以简单地忽略该错误，以便在执行期间可以找到动态链接库 (DLL)。

在您的 CUDA 安装目录中，有一个extras\CUPTI\lib64包含cupti64_101.dll正在尝试加载的目录。将该目录添加到您的路径应该可以解决问题，例如，

SET PATH=C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v10.1\extras\CUPTI\lib64;%PATH%

注意，如果接下来出现INSUFFICIENT_PRIVILEGES 错误，请尝试以管理员身份运行您的程序。

Answer 3

这个答案是为Ubuntu-16.04.

当我升级到Tensorflow-1.14withPython2.7和时，我遇到了这个问题Python3.6。我不得不添加/usr/local/cuda/extras/CUPTI/lib64到LD_LIBRARY_PATH同export LD_LIBRARY_PATH=/usr/local/cuda/extras/CUPTI/lib64:$LD_LIBRARY_PATH和注销和登录。source ~/.bashrc没有帮助。请注意，我的cuda文件夹指向cuda-10.0.

Answer 4

遇到了同样的问题。这就是为我解决的问题，以防其他人遇到类似的问题解决这个问题。

我收到的错误：

function cupti_interface_->Subscribe( &subscriber_, (CUpti_CallbackFunc)ApiCallback, this)failed with error CUPTI could not be loaded or symbol could not be found.

• Windows 服务器 2019
• 张量流2.5
• Cuda 11.2（设置CUDA_PATH环境变量并添加到PATH环境变量中）
• 库德恩8.1.0

我已经C:\Program Files\NVIDIA GPU Computing Toolkit\CUDA\v11.2\extras\CUPTI\lib64在PATH环境变量中设置了，但仍然收到错误。

在 cmd 提示符下运行，在目录where /r c:\ cupti*.dll中找到了 dll c:\Program Files\NVIDIA Corporation\Nsight Systems 2020.4.3\target-windows-x64\。只需将此目录添加到PATH环境变量即可修复该错误。

Answer 5

这是解决“我的”问题的方法：

我刚刚将我的替换tensorflow v 1.14为tensorflow v 1.13.1. 并且不再有CUPTI 错误消息。甚至其他一些奇怪的警告/问题也消失了。所有问题显然都应该有具体的原因，但不幸的是，Tensorflow（很多时候）没有提供可理解的错误/警告消息，从而提供有助于解决问题的好/公平的想法。我最终花了几个小时（甚至几天）来解决这些奇怪的问题，这大大降低了我的生产力。

对我来说，一个普遍的学习（可能与这里分享相关）是我不应该急于将我的张量流安装升级到最新版本。最新的几乎从来都不稳定，每当我尝试时，我最终都会花费大量时间来解决由张量流引起的问题。糟糕的文档和错误消息使得使用起来非常困难。

如果有人有更好的答案，非常欢迎他/她分享他/她对我在这个问题中分享的问题的见解。