如何使用Boto3在AWS EMR集群中等待步骤完成

Question

给定步骤ID，我想等待该AWS EMR步骤完成。我该如何实现？有内置功能吗？

在撰写本文时，EMR的Boto3 Waiters允许等待集群运行和集群终止事件：

EMR服务员

Answer 1

现在有一个可用于步骤完成事件的服务员。它是在最新的boto3版本中添加的。

http://boto3.readthedocs.io/en/latest/reference/services/emr.html#EMR.Waiter.StepComplete

示例代码：

import boto3

client = boto3.client("emr")
waiter = client.get_waiter("step_complete")
waiter.wait(
    ClusterId='the-cluster-id',
    StepId='the-step-id',
    WaiterConfig={
        "Delay": 30,
        "MaxAttempts": 10
    }
)

Answer 2

我想出了以下代码（如果设置max_attempts为 0 或更少，那么它将简单地等待，直到没有正在运行/挂起的步骤）：

def wait_for_steps_completion(emr_client, emr_cluster_id, max_attempts=0):
    sleep_seconds = 30
    num_attempts = 0

    while True:
        response = emr_client.list_steps(
            ClusterId=emr_cluster_id,
            StepStates=['PENDING', 'CANCEL_PENDING', 'RUNNING']
        )
        num_attempts += 1
        active_aws_emr_steps = response['Steps']

        if active_aws_emr_steps:
            if 0 < max_attempts <= num_attempts:
                raise Exception(
                    'Max attempts exceeded while waiting for AWS EMR steps completion. Last response:\n'
                    + json.dumps(response, indent=3, default=str)
                )
            time.sleep(sleep_seconds)
        else:
            return