当Kafka从Python脚本开始时,kafka-server-stop.sh无法正常工作

Question

在远程节点上部署一些Apache Kafka实例后,我观察到了kafka-server-stop.sh作为Kafka存档一部分的脚本问题.

默认情况下,它包含:

#!/bin/sh
# Licensed to the Apache Software Foundation (ASF) under one or more
# contributor license agreements.  See the NOTICE file distributed with
# this work for additional information regarding copyright ownership.
# The ASF licenses this file to You under the Apache License, Version 2.0
# (the "License"); you may not use this file except in compliance with
# the License.  You may obtain a copy of the License at
# 
#    http://www.apache.org/licenses/LICENSE-2.0
# 
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.
ps ax | grep -i 'kafka\.Kafka' | grep java | grep -v grep | awk '{print $1}' | xargs kill -SIGTERM

如果我执行apache kafka而不是后台进程,这个脚本效果很好,例如:

/var/lib/kafka/bin/kafka-server-start.sh /var/lib/kafka/config/server.properties

当我将它作为后台进程执行时它也可以工作:

/var/lib/kafka/bin/kafka-server-start.sh /var/lib/kafka/config/server.properties &

但是在我的远程节点上,我使用这个python脚本执行它(使用Ansible):

#!/usr/bin/env python
import argparse
import os
import subprocess

KAFKA_PATH = "/var/lib/kafka/"

def execute_command_pipe_output(command_to_call):
  return subprocess.Popen(command_to_call, stdout=subprocess.PIPE, stderr=subprocess.STDOUT)

def execute_command_no_output(command_to_call):
  with open(os.devnull, "w") as null_file:
    return subprocess.Popen(command_to_call, stdout=null_file, stderr=subprocess.STDOUT)  

def start_kafka(args):
  command_to_call = ["nohup"]
  command_to_call += [KAFKA_PATH + "bin/zookeeper-server-start.sh"]
  command_to_call += [KAFKA_PATH + "config/zookeeper.properties"]

  proc = execute_command_no_output(command_to_call)

  command_to_call = ["nohup"]
  command_to_call += [KAFKA_PATH + "bin/kafka-server-start.sh"]
  command_to_call += [KAFKA_PATH + "config/server.properties"]

  proc = execute_command_no_output(command_to_call)

def stop_kafka(args):
  command_to_call = [KAFKA_PATH + "bin/kafka-server-stop.sh"]

  proc = execute_command_pipe_output(command_to_call)
  for line in iter(proc.stdout.readline, b''):
    print line,

  command_to_call = [KAFKA_PATH + "bin/zookeeper-server-stop.sh"]

  proc = execute_command_pipe_output(command_to_call)
  for line in iter(proc.stdout.readline, b''):
    print line,


if __name__ == "__main__":
  parser = argparse.ArgumentParser(description="Starting Zookeeper and Kafka instances")
  parser.add_argument('action', choices=['start', 'stop'], help="action to take")

  args = parser.parse_args()

  if args.action == 'start':
    start_kafka(args)
  elif args.action == 'stop':
    stop_kafka(args)
  else:
    parser.print_help()

执行后

manage-kafka.py start
manage-kafka.py stop

Zookeeper关闭(应该是),但Kafka仍在运行.

什么更有趣,当我调用(手动)

nohup /var/lib/kafka/bin/kafka-server-stop.sh

要么

nohup /var/lib/kafka/bin/kafka-server-stop.sh &

kafka-server-stop.sh正确关闭Kafka实例.我怀疑这个问题可能是由某些Linux/Python引起的.

Answer 1

在找出解决问题的粗鲁方法之前，我面对了很多问题。因此，发生的事情是Kafka突然关闭，但该端口仍在使用中。

请按照以下步骤操作：

1. 找到在该端口上运行的进程的进程ID ： lsof -t -i :YOUR_PORT_NUMBER。##这是为Mac
2. 杀死那个过程 kill -9 process_id

Answer 2

Kafka需要在动物园管理员关闭之前完成关闭过程。

因此，启动动物园管理员，然后经纪人将重试关闭过程。

我有一个类似的案例。问题是我的配置没有等待kafka代理关闭。

希望这对某人有帮助。我花了一段时间才弄清楚...

Answer 3

我的猜测：kafka-server-stop.sh 使用 shell 管道。所以波本需要shell=True争论。

请参阅https://docs.python.org/2/library/subprocess.html#subprocess.Popen