将Tensorflow图转换为使用Estimator,使用`sampled_softmax_loss`或`nce_loss`在损失函数中获取'TypeError:数据类型不被理解'

Question

我试图将Tensorflow的官方基本word2vec实现转换为使用tf.Estimator.问题是当使用Tensorflow Estimators时,损失函数(sampled_softmax_loss或nce_loss)会出错.它在原始实现中完美地运行.

这是Tensorflow的官方基本word2vec实现:

https://github.com/tensorflow/tensorflow/blob/master/tensorflow/examples/tutorials/word2vec/word2vec_basic.py

以下是我实施此代码的Google Colab笔记本,该代码正常运行.

https://colab.research.google.com/drive/1nTX77dRBHmXx6PEF5pmYpkIVxj_TqT5I

这是Google Colab笔记本,我在其中更改了代码,因此它使用Tensorflow Estimator,它不起作用.

https://colab.research.google.com/drive/1IVDqGwMx6BK5-Bgrw190jqHU6tt3ZR3e

为方便起见,这里是我定义的Estimator版本的精确代码 model_fn

batch_size = 128
embedding_size = 128  # Dimension of the embedding vector.
skip_window = 1  # How many words to consider left and right.
num_skips = 2  # How many times to reuse an input to generate a label.
num_sampled = 64  # Number of negative examples to sample.

def my_model( features, labels, mode, params):

    with tf.name_scope('inputs'):
        train_inputs = features
        train_labels = labels

    with tf.name_scope('embeddings'):
        embeddings = tf.Variable(
          tf.random_uniform([vocabulary_size, embedding_size], -1.0, 1.0))
        embed = tf.nn.embedding_lookup(embeddings, train_inputs)

    with tf.name_scope('weights'):
        nce_weights = tf.Variable(
          tf.truncated_normal(
              [vocabulary_size, embedding_size],
              stddev=1.0 / math.sqrt(embedding_size)))
    with tf.name_scope('biases'):
        nce_biases = tf.Variable(tf.zeros([vocabulary_size]))

    with tf.name_scope('loss'):
        loss = tf.reduce_mean(
            tf.nn.nce_loss(
                weights=nce_weights,
                biases=nce_biases,
                labels=train_labels,
                inputs=embed,
                num_sampled=num_sampled,
                num_classes=vocabulary_size))

    tf.summary.scalar('loss', loss)

    if mode == "train":
        with tf.name_scope('optimizer'):
            optimizer = tf.train.GradientDescentOptimizer(1.0).minimize(loss)

        return tf.estimator.EstimatorSpec(mode, loss=loss, train_op=optimizer)

这里是我称之为估算和培训的地方

word2vecEstimator = tf.estimator.Estimator(
        model_fn=my_model,
        params={
            'batch_size': 16,
            'embedding_size': 10,
            'num_inputs': 3,
            'num_sampled': 128,
            'batch_size': 16
        })

word2vecEstimator.train(
    input_fn=generate_batch,
    steps=10)

这是我在调用Estimator培训时得到的错误消息:

INFO:tensorflow:Calling model_fn.
---------------------------------------------------------------------------
TypeError                                 Traceback (most recent call last)
<ipython-input-22-955f44867ee5> in <module>()
      1 word2vecEstimator.train(
      2     input_fn=generate_batch,
----> 3     steps=10)

/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py in train(self, input_fn, hooks, steps, max_steps, saving_listeners)
    352 
    353       saving_listeners = _check_listeners_type(saving_listeners)
--> 354       loss = self._train_model(input_fn, hooks, saving_listeners)
    355       logging.info('Loss for final step: %s.', loss)
    356       return self

/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py in _train_model(self, input_fn, hooks, saving_listeners)
   1205       return self._train_model_distributed(input_fn, hooks, saving_listeners)
   1206     else:
-> 1207       return self._train_model_default(input_fn, hooks, saving_listeners)
   1208 
   1209   def _train_model_default(self, input_fn, hooks, saving_listeners):

/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py in _train_model_default(self, input_fn, hooks, saving_listeners)
   1235       worker_hooks.extend(input_hooks)
   1236       estimator_spec = self._call_model_fn(
-> 1237           features, labels, model_fn_lib.ModeKeys.TRAIN, self.config)
   1238       global_step_tensor = training_util.get_global_step(g)
   1239       return self._train_with_estimator_spec(estimator_spec, worker_hooks,

/usr/local/lib/python3.6/dist-packages/tensorflow/python/estimator/estimator.py in _call_model_fn(self, features, labels, mode, config)
   1193 
   1194     logging.info('Calling model_fn.')
-> 1195     model_fn_results = self._model_fn(features=features, **kwargs)
   1196     logging.info('Done calling model_fn.')
   1197 

<ipython-input-20-9d389437162a> in my_model(features, labels, mode, params)
     33                 inputs=embed,
     34                 num_sampled=num_sampled,
---> 35                 num_classes=vocabulary_size))
     36 
     37     # Add the loss value as a scalar to summary.

/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/nn_impl.py in nce_loss(weights, biases, labels, inputs, num_sampled, num_classes, num_true, sampled_values, remove_accidental_hits, partition_strategy, name)
   1246       remove_accidental_hits=remove_accidental_hits,
   1247       partition_strategy=partition_strategy,
-> 1248       name=name)
   1249   sampled_losses = sigmoid_cross_entropy_with_logits(
   1250       labels=labels, logits=logits, name="sampled_losses")

/usr/local/lib/python3.6/dist-packages/tensorflow/python/ops/nn_impl.py in _compute_sampled_logits(weights, biases, labels, inputs, num_sampled, num_classes, num_true, sampled_values, subtract_log_q, remove_accidental_hits, partition_strategy, name, seed)
   1029   with ops.name_scope(name, "compute_sampled_logits",
   1030                       weights + [biases, inputs, labels]):
-> 1031     if labels.dtype != dtypes.int64:
   1032       labels = math_ops.cast(labels, dtypes.int64)
   1033     labels_flat = array_ops.reshape(labels, [-1])

TypeError: data type not understood

编辑:根据请求,这是input_fn的典型输出

print(generate_batch(batch_size=8, num_skips=2, skip_window=1))

(array([3081, 3081,   12,   12,    6,    6,  195,  195], dtype=int32), array([[5234],
       [  12],
       [   6],
       [3081],
       [  12],
       [ 195],
       [   6],
       [   2]], dtype=int32))

Answer 1

找到了答案

错误清楚地表明您的标签类型无效。

您试图传递 numpy 数组而不是张量。有时，Tensorflow 会在后台执行从 ndarray 到 Tensor 的隐式转换（这就是为什么您的代码在 Estimator 之外工作的原因），但在这种情况下却不会。

。

不，官方暗示。从占位符提供数据。占位符始终是一个张量，因此它不依赖于隐式事物。

但是，如果您直接使用 numpy 数组作为输入调用损失函数（注意：在图构建阶段调用，因此参数内容会嵌入到图中），它可能会起作用（但是，我没有检查它）。

这段代码：

nce_loss(labels=[1,2,3]) 在图构建过程中仅被调用一次。标签将作为常量静态嵌入到图中，并且可能可以是任何张量兼容的类型（列表、ndarray 等）

此代码：```Python def model(label_input): nce_loss(labels=label_input)

estimator(model_fun=model).train() ``` 无法静态嵌入标签变量，因为它的内容在图构建期间未定义。因此，如果您输入除张量之外的任何内容，它都会抛出错误。

从

https://www.reddit.com/r/MachineLearning/comments/a39pef/r_tensorflow_estimators_managing_simplicity_vs/

所以我使用了labels=tf.dtypes.cast( train_labels, tf.int64)并且有效