对动态缩放flink作业有任何想法吗？

Question

If there is a kafka topic with 10 partitions and we'd like to use flink to consume the topic. We want the system to allocate slots dynamically according to workload, which means if the workload is low, the flink job can use less slots(with less parallelism), and if the workload is high it can run with higher parallelism. Is there a good way to achieve this? It seems that the parallelism can be changed with stopping the job first. if so, does the pause period affect real-time feature of the application? Any other ideas to change the parallelism? Thank you very much.

Answer 1

有一个REST api调用，用于修改正在运行的作业的并行性，但是当前重新分配状态的唯一方法是创建一个保存点并从其重新启动，因此这就是重新缩放的工作原理（至少目前如此）。

如果您的应用程序正在使用事件时间处理，则结果应不受重启的影响，但当然会因停机而延迟。

更新：以前有一个CLI命令可以进行重新缩放，但是在Flink 1.9.0中暂时禁用了该命令。参见FLINK-12312。