如何从boost线程调用Python？

Question

我有一个Python应用程序调用C++ boost python库,它都可以工作.但是,我有一个回调C++到Python的场景,其中来自boost线程的C++调用python,我在C++端获得访问冲突.如果我使用python线程执行完全相同的回调,它可以完美地工作.因此我怀疑我不能简单地使用boost线程从C++回调Python,但需要做一些额外的工作吗？

Answer 1

最可能的罪魁祸首是,在调用Python代码时,线程不会持有全局解释器锁(GIL),从而导致未定义的行为.验证所有进行直接或间接Python调用的路径,在调用Python代码之前获取GIL.

GIL是CPython解释器周围的互斥体.此互斥锁可防止对Python对象执行并行操作.因此,在任何时间点,允许最多一个线程(已获取GIL的线程)对Python对象执行操作.当存在多个线程时,调用Python代码而不保存GIL会导致未定义的行为.

C或C++线程有时在Python文档中称为外来线程.Python解释器无法控制外来线程.因此,外来线程负责管理GIL以允许与Python线程并发或并行执行.必须仔细考虑:

• 堆栈展开,因为Boost.Python可能抛出异常.
• 间接调用Python,例如copy-constructors或destructors

一种解决方案是使用了解GIL管理的自定义类型来包装Python回调.

使用RAII风格的类来管理GIL提供了一种优雅的异常安全解决方案.例如,使用以下with_gil类,在with_gil创建对象时,调用线程获取GIL.当with_gil对象被破坏时,它恢复GIL状态.

/// @brief Guard that will acquire the GIL upon construction, and
///        restore its state upon destruction.
class with_gil
{
public:
  with_gil()  { state_ = PyGILState_Ensure(); }
  ~with_gil() { PyGILState_Release(state_);   }

  with_gil(const with_gil&)            = delete;
  with_gil& operator=(const with_gil&) = delete;
private:
  PyGILState_STATE state_;
};

它的用法:

{
  with_gil gil;                      // Acquire GIL.
  // perform Python calls, may throw
}                                    // Restore GIL.

通过能够管理GIL with_gil,下一步是创建一个正确管理GIL的仿函数.以下py_callable类将包装boost::python::object并获取调用Python代码的所有路径的GIL:

/// @brief Helper type that will manage the GIL for a python callback.
///
/// @detail GIL management:
///           * Acquire the GIL when copying the `boost::python` object
///           * The newly constructed `python::object` will be managed
///             by a `shared_ptr`.  Thus, it may be copied without owning
///             the GIL.  However, a custom deleter will acquire the
///             GIL during deletion
///           * When `py_callable` is invoked (operator()), it will acquire
///             the GIL then delegate to the managed `python::object`
class py_callable
{
public:

  /// @brief Constructor that assumes the caller has the GIL locked.
  py_callable(const boost::python::object& object)
  {
    with_gil gil;
    object_.reset(
      // GIL locked, so it is safe to copy.
      new boost::python::object{object},
      // Use a custom deleter to hold GIL when the object is deleted.
      [](boost::python::object* object)
      {
        with_gil gil;
        delete object;
      });
  }

  // Use default copy-constructor and assignment-operator.
  py_callable(const py_callable&) = default;
  py_callable& operator=(const py_callable&) = default;

  template <typename ...Args>
  void operator()(Args... args)
  {
    // Lock the GIL as the python object is going to be invoked.
    with_gil gil;
    (*object_)(std::forward<Args>(args)...);
  }

private:
  std::shared_ptr<boost::python::object> object_;
};

通过管理boost::python::object自由空间,人们可以自由地复制,shared_ptr而无需持有GIL.这允许我们安全地使用默认生成的复制构造函数,赋值运算符,析构函数等.

一个人会使用py_callable如下:

// thread 1
boost::python::object object = ...; // GIL must be held.
py_callable callback(object);       // GIL no longer required.
work_queue.post(callback);

// thread 2
auto callback = work_queue.pop();   // GIL not required.
// Invoke the callback.  If callback is `py_callable`, then it will
// acquire the GIL, invoke the wrapped `object`, then release the GIL.
callback(...);   

这是一个完整的示例,演示了如何使用Python扩展调用Python对象作为来自C++线程的回调:

#include <memory>  // std::shared_ptr
#include <thread>  // std::this_thread, std::thread
#include <utility> // std::forward
#include <boost/python.hpp>

/// @brief Guard that will acquire the GIL upon construction, and
///        restore its state upon destruction.
class with_gil
{
public:
  with_gil()  { state_ = PyGILState_Ensure(); }
  ~with_gil() { PyGILState_Release(state_);   }

  with_gil(const with_gil&)            = delete;
  with_gil& operator=(const with_gil&) = delete;
private:
  PyGILState_STATE state_;
};

/// @brief Helper type that will manage the GIL for a python callback.
///
/// @detail GIL management:
///           * Acquire the GIL when copying the `boost::python` object
///           * The newly constructed `python::object` will be managed
///             by a `shared_ptr`.  Thus, it may be copied without owning
///             the GIL.  However, a custom deleter will acquire the
///             GIL during deletion
///           * When `py_callable` is invoked (operator()), it will acquire
///             the GIL then delegate to the managed `python::object`
class py_callable
{
public:

  /// @brief Constructor that assumes the caller has the GIL locked.
  py_callable(const boost::python::object& object)
  {
    with_gil gil;
    object_.reset(
      // GIL locked, so it is safe to copy.
      new boost::python::object{object},
      // Use a custom deleter to hold GIL when the object is deleted.
      [](boost::python::object* object)
      {
        with_gil gil;
        delete object;
      });
  }

  // Use default copy-constructor and assignment-operator.
  py_callable(const py_callable&) = default;
  py_callable& operator=(const py_callable&) = default;

  template <typename ...Args>
  void operator()(Args... args)
  {
    // Lock the GIL as the python object is going to be invoked.
    with_gil gil;
    (*object_)(std::forward<Args>(args)...);
  }

private:
  std::shared_ptr<boost::python::object> object_;
};

BOOST_PYTHON_MODULE(example)
{
  // Force the GIL to be created and initialized.  The current caller will
  // own the GIL.
  PyEval_InitThreads();

  namespace python = boost::python;
  python::def("call_later",
    +[](int delay, python::object object) {
      // Create a thread that will invoke the callback.
      std::thread thread(+[](int delay, py_callable callback) {
        std::this_thread::sleep_for(std::chrono::seconds(delay));
        callback("spam");
      }, delay, py_callable{object});
      // Detach from the thread, allowing caller to return.
      thread.detach();
  });
}

互动用法:

>>> import time
>>> import example
>>> def shout(message):
...     print message.upper()
...
>>> example.call_later(1, shout)
>>> print "sleeping"; time.sleep(3); print "done sleeping"
sleeping
SPAM
done sleeping