kafka connect 和 kafka 主机要求

Question

我正在使用 Couchbase 水槽连接器。CB 和 kafka 位于不同 AWS 区域的 2 个不同 EC2 实例中。我正在关注这些文档：

1. https://docs.couchbase.com/kafka-connector/current/quickstart.html
2. https://kafka.apache.org/documentation/#connect_configuring

基于这些，我认为 connect 必须在也安装了 kafka 的主机上运行。我的连接是否可以在远程主机上运行，​​以便我从远程 kafka 读取并将消息接收到远程 CB 存储桶中？有专门针对此的文档吗？

另外，我收到以下错误：

        at org.apache.kafka.connect.util.ConvertingFutureCallback.result(ConvertingFutureCallback.java:79)
    at org.apache.kafka.connect.util.ConvertingFutureCallback.get(ConvertingFutureCallback.java:66)
    at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:118)
Caused by: org.apache.kafka.connect.errors.ConnectException: Failed to find any class that implements Connector and which name matches com.couchbase.connect.kafka.CouchbaseSinkConnector, available connectors are: PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSinkConnector, name='org.apache.kafka.connect.file.FileStreamSinkConnector', version='2.3.0', encodedVersion=2.3.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.file.FileStreamSourceConnector, name='org.apache.kafka.connect.file.FileStreamSourceConnector', version='2.3.0', encodedVersion=2.3.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockConnector, name='org.apache.kafka.connect.tools.MockConnector', version='2.3.0', encodedVersion=2.3.0, type=connector, typeName='connector', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSinkConnector, name='org.apache.kafka.connect.tools.MockSinkConnector', version='2.3.0', encodedVersion=2.3.0, type=sink, typeName='sink', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.MockSourceConnector, name='org.apache.kafka.connect.tools.MockSourceConnector', version='2.3.0', encodedVersion=2.3.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.SchemaSourceConnector, name='org.apache.kafka.connect.tools.SchemaSourceConnector', version='2.3.0', encodedVersion=2.3.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSinkConnector, name='org.apache.kafka.connect.tools.VerifiableSinkConnector', version='2.3.0', encodedVersion=2.3.0, type=source, typeName='source', location='classpath'}, PluginDesc{klass=class org.apache.kafka.connect.tools.VerifiableSourceConnector, name='org.apache.kafka.connect.tools.VerifiableSourceConnector', version='2.3.0', encodedVersion=2.3.0, type=source, typeName='source', location='classpath'}
    at org.apache.kafka.connect.runtime.isolation.Plugins.connectorClass(Plugins.java:180)
    at org.apache.kafka.connect.runtime.isolation.Plugins.newConnector(Plugins.java:152)
    at org.apache.kafka.connect.runtime.AbstractHerder.getConnector(AbstractHerder.java:519)
    at org.apache.kafka.connect.runtime.AbstractHerder.validateConnectorConfig(AbstractHerder.java:289)
    at org.apache.kafka.connect.runtime.standalone.StandaloneHerder.putConnectorConfig(StandaloneHerder.java:192)
    at org.apache.kafka.connect.cli.ConnectStandalone.main(ConnectStandalone.java:115)

Answer 1

我认为 connect 必须在也安装了 kafka 的主机上运行。

有 Kafka库，是的，但没有实际运行的代理，不是。

换句话说，您无法将 Kafka Connect 作为单独的服务下载，因此当您获取它时它已经附带了 Kafka 库。

我的连接是否可以在远程主机上运行，​​以便我从远程 kafka 读取并将消息接收到远程 CB 存储桶中？

这是推荐的部署。类似于您不会在经纪人上运行普通生产者和消费者的方式。Kafka Connect 应该受到同样的对待。

引导服务器和 couchbase 连接字符串将是它们各自的远程地址

文档有点在这里

• https://kafka.apache.org/documentation/#connect_running
• https://docs.confluence.io/current/connect/userguide.html#deployment-considerations

另外，我收到以下错误：Failed to find any class that implements Connector and which name matches com.couchbase.connect.kafka.CouchbaseSinkConnector

如果您查看 的列表available connectors are:，您会发现您的未列出。这意味着您没有plugin.path使用库正确加载连接器属性。

• https://docs.confluence.io/current/connect/managing/install.html
• https://docs.confluence.io/current/connect/managing/community.html

正如 Couchbase 文档所述：

修改plugin.path属性以包含其父目录kafka-connect-couchbase-<version>.jar

/connector-plugins您还可以在使用端点发布配置之前验证连接器是否已加载