如何在 google Pub/Sub 中快速发布多条消息？

Question

如何快速发布多条消息到pubsub？没有多处理和多线程，因为代码已经在线程中

下面的代码每秒发布 40 条消息

publisher = pubsub.PublisherClient(
    credentials=credentials,
    batch_settings=types.BatchSettings(
         max_messages=1000,  # default is 100
        max_bytes=1 * 1000 * 1000,  # 1 MiB
        max_latency=0.1,  # default is 10 ms
    )
)

topic_name = 'projects/{project_id}/topics/{topic}'.format(
    project_id=PROJECT_ID,
    topic=TOPIC_PUBSUB,
)

for data in results:
    bytes_json_data = str.encode(json.dumps(data))
    future = publisher.publish(topic_name, bytes_json_data)
    future.result()

Answer 1

您不应一次发布一条消息然后等待future，而应该一次发布所有消息，然后等待futures最后发布。它看起来像：

from concurrent import futures
...
publish_futures = []
for data in results:
    bytes_json_data = str.encode(json.dumps(data))
    future = publisher.publish(topic_name, bytes_json_data)
    publish_futures.append(future)
...
futures.wait(publish_futures, return_when=futures.ALL_COMPLETED)

文档中有一个带有示例代码的详细示例。