通过AWS上的ELB在Kubernetes上公开单独的Kafka经纪人

Question

我运行在AWS上一KOPS Kubernetes集群,努力使kubernetes -卡夫卡例如工作与弹性负载均衡.以下是两个经纪人的外部服务部分:

kind: Service
apiVersion: v1
metadata:
  name: outside-0
  namespace: kafka
spec:
  selector:
    app: kafka
    kafka-broker-id: "0"
  ports:
  - protocol: TCP
    targetPort: 9094
    port: 32400
    nodePort: 32400
  type: NodePort
---
kind: Service
apiVersion: v1
metadata:
  name: outside-1
  namespace: kafka
spec:
  selector:
    app: kafka
    kafka-broker-id: "1"
  ports:
  - protocol: TCP
    targetPort: 9094
    port: 32401
    nodePort: 32401
  type: NodePort

这是我尝试通过ELB公开这些代理(实际的FQDN替换为my.copmany.com).

apiVersion: v1
kind: Service
metadata:
  name: kafka-0
  annotations:
    dns.alpha.kubernetes.io/external: kafka-0.kafka.my.company.com
spec:
  type: LoadBalancer
  ports:
  - port: 32400
    name: outside
    targetPort: 32400
  selector:
    app: outside-0
---
apiVersion: v1
kind: Service
metadata:
  name: kafka-1
  annotations:
    dns.alpha.kubernetes.io/external: kafka-1.kafka.my.company.com
spec:
  type: LoadBalancer
  ports:
  - port: 32401
    name: outside
    targetPort: 32401
  selector:
    app: outside-1

综观AWS ELB控制台显示了可用于每个卡夫卡ELB经纪人并产生0〜3的情况下kafka-1.kafka.my.company.com:9094使用卡夫卡命令行客户机超时.如何outside-0通过kafka-0LoadBalancer服务公开NodePort 服务？还是有其他方法被考虑？

Answer 1

Kakfa非常关注需要直接访问作为主题领导者的服务器的客户.为实现这一目标,我做了以下工作:

1)设置configmap,根据pod的序数值动态覆盖advertised.listeners的值

POD_ID=${HOSTNAME##*-}

kafka-server-start.sh server.properties \
      --override advertised.listeners=INSIDE://`hostname -f`:9092,OUTSIDE://kafka-${POD_ID}.kafka.my.company.com:9094 \
      --override broker.id=${POD_ID} \
      --override listeners=INSIDE://:9092,OUTSIDE://:9094

2)为每个Kafka pod创建一个LoadBalancer服务.更改选择器以匹配您的kafka-pod-id.

    apiVersion: v1
    kind: Service
    metadata:
      name: kafka-0
      annotations:
        dns.alpha.kubernetes.io/external: kafka-0.kafka.my.company.com
    spec:
      externalTrafficPolicy: Local
      type: LoadBalancer
      ports:
      - port: 9094
        name: outside
        targetPort: 9094
      selector:
        app: kafka
        kafka-pod-id: "0"
---     
apiVersion: v1
        kind: Service
        metadata:
          name: kafka-1
          annotations:
            dns.alpha.kubernetes.io/external: kafka-1.kafka.my.company.com
        spec:
          externalTrafficPolicy: Local
          type: LoadBalancer
          ports:
          - port: 9094
            name: outside
            targetPort: 9094
          selector:
            app: kafka
            kafka-pod-id: "1"

Answer 2

根据文档（Kuebrnetes 服务类型）：

\n\n

\n

LoadBalancer：使用云提供商\xe2\x80\x99s\n 负载均衡器向外部公开服务。外部负载均衡器将路由到的 NodePort 和 ClusterIP 服务是自动创建的。

\n

\n\n

您不应该为 NodePort 和 LoadBalancer 类型定义单独的服务，而只能LoadBalancer指定nodePort指定的服务（请测试并尝试添加/删除一些选项，因为我没有可以测试它的环境）：

\n\n

apiVersion: v1\nkind: Service\nmetadata:\n  name: kafka-0\nspec:\n  type: LoadBalancer\n  ports:\n  - port: 32400\n    name: kafka\n    nodePort: 32400\n  selector:\n    app: kafka\n    kafka-broker-id: "0"\n---\napiVersion: v1\nkind: Service\nmetadata:\n  name: kafka-1\nspec:\n  type: LoadBalancer\n  ports:\n  - port: 32401\n    name: kafka\n    nodePort: 32401\n  selector:\n    app: kafka\n    kafka-broker-id: "1"\n

\n\n

\n
• Kubernetes 集群应配置 AWS API 访问。kubelet、kube-apiserver、kube-controller-manager/cloud-controller-manager有云配置选项。
\n

\n