小编jap*_*izo的帖子

使用表白名单选项更新 Debezium MySQL 连接器

我正在使用 Debezium (0.7.5) MySQL 连接器,如果我想使用选项更新此配置,我试图了解什么是最佳方法table.whitelist

假设我创建了一个连接器,如下所示:

curl -i -X POST -H "Accept:application/json" -H  "Content-Type:application/json" http://debezium-host/connectors/ -d '
{
  "name": "MyConnector",
  "config": {
      "connector.class": "io.debezium.connector.mysql.MySqlConnector",
      "connect.timeout.ms": "60000",
      "tasks.max": "1",
      "database.hostname": "myhost",
      "database.port": "3306",
      "database.user": "***",
      "database.password": "***",
      "database.server.id": "3227197",
      "database.server.name": "MyServer",
      "database.whitelist": "myDb",
      "table.whitelist": "myDb.table1,myDb.table2",
      "database.history.kafka.bootstrap.servers": "kb0:9092,kb1:9092,kb2:9092",
      "database.history.kafka.topic": "MyConnectorHistoryTopic",
      "max.batch.size": "1024",
      "snapshot.mode": "initial",
      "decimal.handling.mode": "double"
    }
}'
Run Code Online (Sandbox Code Playgroud)

一段时间后(2 周),我需要向myDb.table3table.whitelist选项添加一个新表 ( ) (该表是旧表,它是在连接器之前创建的)

我尝试的是:

  • 暂停连接器。
  • 删除了历史主题(也许这就是问题所在?)。
  • 通过 API 更新配置端点更新了配置。
  • 恢复连接器。

通过 API 更新命令:

curl -i -X …
Run Code Online (Sandbox Code Playgroud)

mysql apache-kafka apache-kafka-connect debezium

5
推荐指数
1
解决办法
5050
查看次数

Debezium 无法对大表大小进行快照

我想我可能错过了一些配置,但我们正在尝试使用 Debezium 对一个包含大约 800 万条记录的表中的所有行进行快照,一段时间后它会停止。

连接器配置是:

{
   "connector.class":"io.debezium.connector.mysql.MySqlConnector",
   "database.user":"MyUser",
   "database.server.id":"12345",
   "tasks.max":"1",
   "database.history.kafka.bootstrap.servers":"MyKafka:9092",
   "database.history.kafka.topic":"MyConnectorHistory",
   "database.server.name":"MyDbName",
   "database.port":"3306",
   "table.whitelist":"BigTable",
   "decimal.handling.mode":"double",
   "database.hostname":"***",
   "database.password":"***",
   "name":"MyConnector",
   "database.whitelist":"MyDb",
   "snapshot.mode":"initial_only",
   "connect.timeout.ms":"60000"
}

Run Code Online (Sandbox Code Playgroud)

连接器开始扫描行:

  April 24th 2019, 13:06:52.573 2019-04-24 16:06:52,569 INFO   MySQL|MyDbName|snapshot  Step 9: - 2040000 of 8609643 rows scanned from table 'MyDb.BigTable' after 00:59:29.129   [io.debezium.connector.mysql.SnapshotReader]
  ... other prints
  April 24th 2019, 12:17:28.448 2019-04-24 15:17:28,447 INFO   MySQL|MyDbName|snapshot  Step 9: - 50000 of 8609643 rows scanned from table 'MyDb.BigTable' after 00:10:05.008   [io.debezium.connector.mysql.SnapshotReader]
    April 24th 2019, 12:07:43.183   2019-04-24 15:07:43,183 INFO   MySQL|MyDbName|snapshot …
Run Code Online (Sandbox Code Playgroud)

apache-kafka apache-kafka-connect debezium

1
推荐指数
1
解决办法
2661
查看次数