TensorFlow 如何通过 array_ops.cc 生成 gen_array_ops.py？

Question

TensorFlow 自动生成代码。我很好奇 TF 是如何生成gen_array_ops.py的array_ops.cc？

生成的python文件在 python3.6/site-packages/tensorflow/python/ops/gen_array_ops.py

"""Python wrappers around TensorFlow ops.

This file is MACHINE GENERATED! Do not edit.
Original C++ source file: array_ops.cc
"""
...
...

Answer 1

Python 代码生成是在构建时通过 Bazel 完成的。您可以在 中找到相关定义tensorflow/tensorflow.bzl，我将在此处仅发布标题：

# Generates a Python library target wrapping the ops registered in "deps".
#
# Args:
#   name: used as the name of the generated target and as a name component of
#     the intermediate files.
#   out: name of the python file created by this rule. If None, then
#     "ops/gen_{name}.py" is used.
#   hidden: Optional list of ops names to make private in the Python module.
#     It is invalid to specify both "hidden" and "op_whitelist".
#   visibility: passed to py_library.
#   deps: list of dependencies for the intermediate tool used to generate the
#     python target. NOTE these `deps` are not applied to the final python
#     library target itself.
#   require_shape_functions: leave this as False.
#   hidden_file: optional file that contains a list of op names to make private
#     in the generated Python module. Each op name should be on a line by
#     itself. Lines that start with characters that are invalid op name
#     starting characters are treated as comments and ignored.
#   generated_target_name: name of the generated target (overrides the
#     "name" arg)
#   op_whitelist: if not empty, only op names in this list will be wrapped. It
#     is invalid to specify both "hidden" and "op_whitelist".
#   cc_linkopts: Optional linkopts to be added to tf_cc_binary that contains the
#     specified ops.

def tf_gen_op_wrapper_py(
        name,
        out = None,
        hidden = None,
        visibility = None,
        deps = [],
        require_shape_functions = False,
        hidden_file = None,
        generated_target_name = None,
        op_whitelist = [],
        cc_linkopts = [],
        api_def_srcs = []):
    # ...

这被称为间接调用tf_gen_op_wrapper_private_py，您可以通过它在tensorflow/python/build_defs.bzl. 对于 的情况array_ops，您将在tensorflow/python/BUILD以下位置找到它：

tf_gen_op_wrapper_private_py(
    name = "array_ops_gen",
    visibility = [
        "//learning/brain/python/ops:__pkg__",
        "//tensorflow/compiler/tests:__pkg__",
        "//tensorflow/contrib/quantization:__pkg__",
        "//tensorflow/python/kernel_tests:__pkg__",
    ],
)

这条规则有什么作用？它调用您可以在其中找到源的程序tensorflow/python/framework/python_op_gen_main.cc（即主要入口点，它使用其他相邻的源文件）。本质上，它是一个程序，它遍历通过REGISTER_OP宏（在 中定义tensorflow/core/framework/op.h）注册的操作并相应地生成 Python 代码。我现在无法详细说明，但如果您想了解详细信息，您应该能够浏览代码。