Amazon Kinesis：在同步Kinesis分片和租约时捕获到异常

Question

我正在尝试让Snowplow在AWS上工作。当我尝试在实例上运行流丰富的服务时，出现此异常：

[main] INFO com.amazonaws.services.kinesis.clientlibrary.lib.worker.Worker - Syncing Kinesis shard info
[main] ERROR com.amazonaws.services.kinesis.clientlibrary.lib.worker.ShardSyncTask - Caught exception while sync'ing Kinesis shards and leases
[cw-metrics-publisher] WARN com.amazonaws.services.kinesis.metrics.impl.CWPublisherRunnable - Could not publish 4 datums to CloudWatch

我不认为错误归因于Cloud Watch：

同步Kinesis分片和租约时发现异常

Answer 1

正如上面的评论中提到的，当您缺乏Kinesis Client Library (KCL)所需的 AWS 资源的权限时，就会出现此错误所需的 AWS 资源的权限时，就会出现此错误。这可以是 DynamoDB、CloudWatch 或 Kinesis。对于 Snowplow 的 Stream Enrich 组件，您需要以下权限：

\n\n

\n
• 输入运动流的读取权限（收集器好）
\n
• 输出运动流的写权限（丰富好的和丰富坏的）
\n
• 列出对 kinesis 流的权限
\n
• 对 DynamoDB 状态表的读/写/创建权限（表名称是流丰富中的 \xe2\x80\x9cappName\xe2\x80\x9d 值）application.conf）
\n
• 将MetricData 放入Cloudwatch
\n

\n\n

满足这些需求的 IAM 策略模板化版本如下：

\n\n

{\n  "Version": "2012-10-17",\n  "Statement": [\n    {\n      "Effect": "Allow",\n      "Action": [\n        "kinesis:DescribeStream",\n        "kinesis:GetShardIterator",\n        "kinesis:GetRecords",\n        "kinesis:ListShards"\n      ],\n      "Resource": [\n        "${collector_stream_out_good}"\n      ]\n    },\n    {\n      "Effect": "Allow",\n      "Action": [\n          "kinesis:ListStreams"\n      ],\n      "Resource": "*"\n    },\n    {\n      "Effect": "Allow",\n      "Action": [\n        "kinesis:DescribeStream",\n        "kinesis:PutRecord",\n        "kinesis:PutRecords"\n      ],\n      "Resource": [\n        "${enricher_stream_out_good}",\n        "${enricher_stream_out_bad}"\n      ]\n    },\n    {\n      "Effect": "Allow",\n      "Action": [\n        "dynamodb:CreateTable",\n        "dynamodb:DescribeTable",\n        "dynamodb:Scan",\n        "dynamodb:GetItem",\n        "dynamodb:PutItem",\n        "dynamodb:UpdateItem",\n        "dynamodb:DeleteItem"\n      ],\n      "Resource": [\n        "${enricher_state_table}"\n      ]\n    },\n    {\n      "Effect": "Allow",\n      "Action": [\n        "cloudwatch:PutMetricData"\n      ],\n      "Resource": "*"\n    }\n  ]\n}\n

\n\n

我写了一篇博文，其中介绍了 Stream Enrich 和其他 Snowplow 组件所需的 IAM 权限，因为有关确切的文档所需权限的文档很少/不存在。

\n\n

希望有帮助！

\n