如何在自定义路径中保存/加载张量流集线器模块？

Question

该tensorflow_hub库的维护者取得了它的每一个方便用户下载和使用预先训练tensorflow模块,如:

import tensorflow_hub as hub

embed = hub.Module("https://tfhub.dev/google/universal-sentence-encoder/1")

但从sys.stderr它看起来就像是将模块本地保存到临时目录,即

INFO:tensorflow:使用/ var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules来缓存模块.INFO:tensorflow:使用Embeddings_en/sharded_0从检查点b'/ var/folders/j6/xczfl75n3sbfwpg4190gpb44vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables'初始化变量模块/ Embeddings_en/sharded_0:0 INFO:tensorflow:初始化变量模块/ Embeddings_en /来自检查点b'/ var/folders/j6/xczfl75n3sbfwpg4190gpb44vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables'的sharded_1:0 with Embeddings_en/sharded_1 INFO:tensorflow:从检查点b'/ var /初始化变量模块/ Embeddings_en/sharded_10:0文件夹/ J6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3 /变量/变量与Embeddings_en/sharded_10 INFO:tensorflow:初始化变量模块/ Embeddings_en/sharded_11:从检查点b 0,'/变种/文件夹/ J6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables'with Embeddings_en/sharded_11 INFO:tensorflow:Initialize varia ble module/Embeddings_en/sharded_12:0来自检查点b'/ var/folders/j6/xczfl75n3sbfwpg4190gpb4vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables'with Embeddings_en/sharded_12 INFO:tensorflow:从检查点初始化变量模块/ Embeddings_en/sharded_13:0 b '/变种/文件夹/ J6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3 /变量/变量' 与Embeddings_en/sharded_13 INFO:tensorflow:初始化变量模块/ Embeddings_en/sharded_14:从检查点b 0,'/变种/文件夹/ J6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3 /变量/变量与Embeddings_en/sharded_14 INFO:tensorflow:初始化变量模块/ Embeddings_en/sharded_15:从检查点b 0,'/变种/文件夹/ J6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3 /变量/变量'with Embeddings_en/sharded_15 INFO:tensorflow:初始化变量模块/ Embeddings_en/sha 来自检查点b'/ var/folders/j6/xczfl75n3sbfwpg4190gpb44vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables'的rded_16:0 with Embeddings_en/sharded_16 INFO:tensorflow:从检查点b'/ var /初始化变量模块/ Embeddings_en/sharded_2:0文件夹/ J6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3 /变量/变量与Embeddings_en/sharded_2 INFO:tensorflow:初始化变量模块/ Embeddings_en/sharded_3:从检查点b 0,'/变种/文件夹/ J6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3 /变量/变量与Embeddings_en/sharded_3 INFO:tensorflow:从检查点b 0,:初始化变量模块/ Embeddings_en/sharded_4 '/变种/文件夹/ J6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3 /变量/变量' 与Embeddings_en/sharded_4 INFO:tensorflow:从检查点b'/ var/fo初始化变量模块/ Embeddings_en/sharded_5:0 lders/J6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3 /变量/变量与Embeddings_en/sharded_5 INFO:tensorflow:初始化变量模块/ Embeddings_en/sharded_6:从检查点B 0,'/变种/文件夹/ J6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3 /变量/变量与Embeddings_en/sharded_6 INFO:tensorflow:从检查点b 0,:初始化变量模块/ Embeddings_en/sharded_7 '/变种/文件夹/ J6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3 /变量/变量' 与Embeddings_en/sharded_7 INFO:tensorflow:使用Embeddings_en/sharded_8从检查点b'/ var/folders/j6/xczfl75n3sbfwpg4190gpb44vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables'初始化变量模块/ Embeddings_en/sharded_8:0 INFO:tensorflow:初始化变量模块/ Embeddings_en /来自检查点b'/ var/folders/j6/xczfl75n3sbfwpg4190gpb104vn的sharded_9:0 带有Embeddings_en/sharded_9的lxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables'信息:tensorflow:初始化变量模块/ Encoder_en/DNN/ResidualHidden_​​0 /权重:0来自检查点b'/ var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3 /变量/变量与Encoder_en/DNN/ResidualHidden_​​0 /权重INFO:tensorflow:初始化变量模块/ Encoder_en/DNN/ResidualHidden_​​1 /权重:0从检查点b'的/ var /文件夹/ J6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3 /变量/变量'与Encoder_en/DNN/ResidualHidden_​​1 /权重INFO:tensorflow:初始化变量模块/ Encoder_en/DNN/ResidualHidden_​​2 /权重:0来自检查点b'/ var/folders/j6/xczfl75n3sbfwpg4190gpb44vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables /变量'与Encoder_en/DNN/ResidualHidden_​​2 /权重INFO:tensorflow:初始化变量模块/ Encoder_en/DNN/ResidualHidd en_3/projection:0来自检查点b'/ var/folders/j6/xczfl75n3sbfwpg4190gpb44vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables'with Encoder_en/DNN/ResidualHidden_​​3/projection INFO:tensorflow:Initialize variable module/Encoder_en/DNN/ResidualHidden_​​3 /权重:0从检查点b '的/ var /文件夹/ J6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3 /变量/变量' 与Encoder_en/DNN/ResidualHidden_​​3 /权重INFO:tensorflow:初始化变量模块/ SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_0 /偏压: 0从检查点b '的/ var /文件夹/ J6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3 /变量/变量' 与SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_0 /偏压INFO:tensorflow:初始化变量模块/ SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_0 /权重:0从checkpoint b'/ var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e 8838bf21a2d3 /变量/变量与SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_0 /权重INFO:tensorflow:初始化变量模块/ SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_1 /偏压:从检查点B 0,'/变种/文件夹/ J6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3 /变量/变量与SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_1 /偏压INFO:tensorflow:初始化变量模块/ SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_1 /权重:0从检查点b'的/ var /文件夹/ J6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3 /变量/变量与SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_1 /权重INFO:tensorflow:初始化变量模块/ SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_2 /偏压:从检查点b 0,'/变种/文件夹/ J6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3 /变量/变量的使用SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_2 /偏压INFO:tensorflow:初始化变量模块/ SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_2 /权重:从检查点B 0, '/变种/文件夹/ J6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3 /变量/变量' 与SHARED_RANK_ANSWER/response_encoder_0/tanh_layer_2 /权重信息:tensorflow:初始化变量模块/ SNLI /分类器/ LinearLayer/bias:0来自检查点b'/ var/folders/j6/xczfl75n3sbfwpg4190gpb44vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables'with SNLI/Classifier/LinearLayer/bias信息:tensorflow:初始化变量模块/ SNLI /分类器/ LinearLayer /权重:0来自检查点b'/ var/folders/j6/xczfl75n3sbfwpg4190gpb4vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables'与SNLI /分类器/ LinearLayer /权重信息: tensorflow:初始化变量模块/ SNLI /分类器/ tanh_layer_0/bias:0来自检查点b'/ var/folders/j6/xczfl75n3sbfwpg4190gpb44vnlxt/T/具有SNLI /分类器/ tanh_layer_0/bias的tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3/variables/variables':INFO:tensorflow:初始化变量模块/ SNLI /分类器/ tanh_layer_0 /权重:0来自检查点b'/ var/folders/j6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3 /变量/变量与SNLI /分类/ tanh_layer_0 /权重INFO:tensorflow:初始化变量模块/ global_step:从检查点b 0, '/变种/文件夹/ J6/xczfl75n3sbfwpg4190gpb104vnlxt/T/tfhub_modules/c6f5954ffa065cdb2f2e604e740e8838bf21a2d3 /变量/变量' 与global_step

机器重启后,模块被删除,hub.Module('...')再次运行代码将重新下载模块.

是否可以将模块保存到自定义目录,然后从自定义目录加载？

如果可能,如何在自定义路径中保存/加载张量流集线器模块？

Answer 1

您可以从url +'下载模型需求？tf-hub-format = compressed'

我尝试下载elmo,它工作

url = https://tfhub.dev/google/elmo/2 +'？tf-hub-format = compressed'

例如:https://tfhub.dev/google/elmo/2？tf-hub-format = compression

该模型将作为tarfile下载到您的机器上.

一旦你解压缩文件,它将有tfhub_module.pb

Answer 2

• 获取 URL 并将其更改为：

https：//开头tfhub.dev /谷歌/万能句编码器/ 1

到：

https：//开头storage.googleapis.com/tfhub-modules /谷歌/万能句编码器/ 1。tar.gz

• 使用 Curl 或浏览器下载。
• 提取到首选位置（例如/home/admin/embed/）
• 将您的代码更改为：

.

import tensorflow_hub as hub 
embed = hub.load('/home/admin/embed/')

Answer 3

1）找到您的模型：例如https://tfhub.dev/google/imagenet/inception_v1/feature_vector/1

\n\n

2）获取真实下载路径：

\n\n

\n

将 tfhub.dev 替换为 storage.googleapis.com/tfhub-modules 并附加\n .tar.gz 作为后缀。

\n

\n\n

3）准备缓存：

\n\n

\n

在某些平台上，TF hub 会记录缓存目录，但有些平台不会记录\xe2\x80\x99t。在代码中指定缓存位置更加可靠。只需在调用 tfhub 之前将以下代码放入您的文件中即可。

\n

\n\n

os.environ["TFHUB_CACHE_DIR"] = \'/tmp/tfhub\'\n

\n\n

参考：如何在没有互联网连接的情况下本地运行 TF hub

\n

Answer 4

确实可以指定/使用自定义目录:) 有关说明，请参阅https://www.tensorflow.org/hub/basics上名为“缓存模块”的部分