我尝试使用以下配置启动 JDBC 接收器连接器:
{
"name": "crm_data-sink_hh",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSinkConnector",
"tasks.max": 6,
"topics": "crm_account,crm_competitor,crm_event,crm_event_participation",
"connection.url": "jdbc:postgresql://db_host/hh?prepareThreshold=0",
"connection.user": "db_user",
"connection.password": "${file:db_hh_kafka_connect_pass}",
"dialect.name": "PostgreSqlDatabaseDialect",
"insert.mode": "upsert",
"pk.mode": "record_value",
"pk.fields": "guid",
"errors.tolerance": "all",
"errors.log.enable":true,
"errors.log.include.messages":true,
"errors.deadletterqueue.topic.name":"crm_data_deadletterqueue",
"errors.deadletterqueue.context.headers.enable":true
}
}
Run Code Online (Sandbox Code Playgroud)
但是当连接器处于运行状态时没有任务正在运行:
curl -X GET http://kafka-connect:10900/connectors/crm_data-sink_hh/status
{"name":"crm_data-sink_hh","connector":{"state":"RUNNING","worker_id":"172.16.24.14:10900"},"tasks":[],"type":"sink"}
Run Code Online (Sandbox Code Playgroud)
我多次遇到这个问题,但我很困惑,因为它是随机发生的。我的问题和这个问题非常相似。我将不胜感激任何帮助!
更新。11/04/2019(不幸的是,现在我只有INFO级别的日志)
最后,经过几次尝试,我通过更新现有连接器的配置来启动连接器来运行任务crm_data-sink_db_hh:
$ curl -X GET http://docker61:10900/connectors/crm_data-sink_db_hh/status
{"name":"crm_data-sink_db_hh","connector":{"state":"RUNNING","worker_id":"192.168.1.198:10900"},"tasks":[],"type":"sink"}
$ curl -X GET http://docker61:10900/connectors/crm_data-sink_db_hh/status
{"name":"crm_data-sink_db_hh","connector":{"state":"RUNNING","worker_id":"192.168.1.198:10900"},"tasks":[],"type":"sink"}
$ curl -X PUT -d @new_config.json http://docker21:10900/connectors/crm_data-sink_db_hh/config -H 'Content-Type: application/json'
$ curl -X GET http://docker61:10900/connectors/crm_data-sink_db_hh/status
{"name":"crm_data-sink_db_hh","connector":{"state":"UNASSIGNED","worker_id":"192.168.1.198:10900"},"tasks":[],"type":"sink"}
$ curl …Run Code Online (Sandbox Code Playgroud) 我似乎经常根据查询从 JdbcConnectionSource 创建一个 Kafka Connect 连接器,并且连接器创建成功,状态为“RUNNING”,但没有创建任何任务。查看我的容器的控制台日志,我没有看到任何错误的迹象,我可以说:没有错误,没有警告,没有解释任务失败的原因。我可以让其他连接器工作,但有时不能。
当连接器无法创建 RUNNING 任务时,如何获得更多信息以进行故障排除?
我将在下面发布我的连接器配置示例。
我正在使用 Kafka Connect 5.4.1-ccs。
连接器配置(它是 JDBC 背后的 Oracle 数据库):
{
"name": "FiscalYear",
"config": {
"connector.class": "io.confluent.connect.jdbc.JdbcSourceConnector",
"tasks.max": 1,
"connection.url": "jdbc:oracle:thin:@(DESCRIPTION=(ADDRESS=(PROTOCOL=TCP)(HOST=myhost.example.com)(PORT=1521))(LOAD_BALANCE=OFF)(FAILOVER=OFF)(CONNECT_DATA=(SERVER=DEDICATED)(SERVICE_NAME=MY_DB_PRI)(UR=A)))",
"connection.user":"myuser",
"connection.password":"mypass",
"mode": "timestamp",
"timestamp.column.name": "MAINT_TS",
"topic.prefix": "MyTeam.MyTopicName",
"poll.interval.ms": 5000,
"value.converter" : "org.apache.kafka.connect.json.JsonConverter",
"value.converter.schemas.enable": "false",
"numeric.mapping": "best_fit",
"_comment": "The query is wrapped in `select * from ()` so that JdbcSourceConnector can automatically append a WHERE clause.",
"query": "SELECT * FROM (SELECT fy_nbr, min(fy_strt_dt) fy_strt_dt, max(fy_end_dt) fy_end_dt FROM myuser.fsc_dt fd WHERE …Run Code Online (Sandbox Code Playgroud)