如何在故障转移模式下配置Flume 1.x(flume-ng)？

Question

有关在CDH3中配置故障转移模式的Flume(0,9x)节点的大量信息.

但CDH4中Flume(1.x)配置的配置格式完全不同.如何在故障转移模式下配置Flume 1.x(flume-ng)？

Answer 1

在flume-ng中,您可以定义一组所谓的"接收器"(事件使用者),它们都连接到一个通道,并为该组指定"故障转移"策略,因此如果其中一个接收器发生故障,事件将被重定向到另一个.

假设我们有两个接收器 - main_sink和backup_sink,并且都配置为使用来自一个通道的事件并将事件传递到某个目标.我们将main_sink的优先级设置为大于backup_sink的优先级,因此只要有效,Flume就会将事件从channel转发到main_sink.但是,如果main_sink失败,Flume会将其降级到失败的汇池,在那里它将被分配一个冷却期.同时,来自频道的事件将被转发到backup_sink.

完整示例可能如下所示:

# channels
agent.channels = mem_channel
agent.channels.mem_channel.type = memory

# sources
agent.sources = event_source
agent.sources.event_source.type = avro
agent.sources.event_source.bind = 127.0.0.1
agent.sources.event_source.port = 10000
agent.sources.event_source.channels = mem_channel

# sinks
agent.sinks = main_sink backup_sink

agent.sinks.main_sink.type = avro
agent.sinks.main_sink.hostname = 127.0.0.1
agent.sinks.main_sink.port = 10001
agent.sinks.main_sink.channel = mem_channel

agent.sinks.backup_sink.type = avro
agent.sinks.backup_sink.hostname = 127.0.0.1
agent.sinks.backup_sink.port = 10002
agent.sinks.backup_sink.channel = mem_channel

# sink groups    
agent.sinkgroups = failover_group
agent.sinkgroups.failover_group.sinks = main_sink backup_sink
agent.sinkgroups.failover_group.processor.type = failover
agent.sinkgroups.failover_group.processor.priority.main_sink = 10
agent.sinkgroups.failover_group.processor.priority.backup_sink = 5

您可以在Flume用户指南中找到有关该主题的更多详细信息.