AWS step-function mapState iterate over large payloads

Question

I have a state-machine consisting of a first pre-process task that generates an array as output, which is used by a subsequent map state to loop over. The output array of the first task has gotten too big and the state-machine throws the error States.DataLimitExceeded: The state/task 'arn:aws:lambda:XYZ' returned a result with a size exceeding the maximum number of characters service limit.

Here is an example of the state-machine yaml:

stateMachines:
  myStateMachine:
    name: "myStateMachine"
    definition:
      StartAt: preProcess
      States:
        preProcess:
          Type: Task
          Resource:
            Fn::GetAtt: [preProcessLambda, Arn]
          Next: mapState
          ResultPath: "$.preProcessOutput"
        mapState:
          Type: Map
          ItemsPath: "$.preProcessOutput.data"
          MaxConcurrency: 100
          Iterator:
            StartAt: doMap
            States:
              doMap:
                Type: Task
                Resource:
                  Fn::GetAtt: [doMapLambda, Arn]
                End: true
          Next: ### next steps, not relevant

A possible solution I came up with would be that state preProcess saves its output in an S3-bucket and state mapState reads directly from it. Is this possible? At the moment the output of preProcess is

ResultPath: "$.preProcessOutput"

and mapState takes the array

ItemsPath: "$.preProcessOutput.data" as input.

How would I need to adapt the yaml that the map state reads directly from S3?

Answer 1

我目前也在工作中解决类似的问题。由于步骤函数存储其整个状态，因此随着 json 映射所有值而不断增长，您很快就会遇到问题。

解决这个问题的唯一真正方法是使用阶跃函数的层次结构。也就是说，您的阶跃函数上的阶跃函数。所以你有了：

parent -> [batch1, batch2, batch...N]

然后每个批次都有许多单个作业：

batch1 -> [j1,j2,j3...jBATCHSIZE]

我有一个非常简单的步骤函数，我发现 at~4k大约是我开始达到状态限制之前可以拥有的最大批量大小。

这不是一个很好的解决方案，嘿它有效。