Hive/HBase集成 - Zookeeper会话立即关闭

Question

我们有一个使用Cloudera Manager配置CDH3u2的8节点集群.我们有一个专用的主节点,它运行我们唯一的zookeeper实例.当我配置hive来运行从主节点执行的本地hadoop时,我从HBase中检索数据没有问题.当我通过配置单元运行分布式map/reduce时,当从节点连接到zookeeper时,我收到以下错误.

HBase能够连接到ZooKeeper,但连接立即关闭.这可能表示服务器连接太多(默认为30).

我们尝试将最大连接设置得更高(我们甚至尝试删除限制).这是一个用户很少的开发集群,我知道问题不在于连接太多(我可以使用./zkCli从slave节点连接到zookeeper).

服务器端日志表明会话已被客户端终止.

客户端hadoop日志说:'引起:org.apache.zookeeper.KeeperException $ ConnectionLossException:KeeperErrorCode = /Lbase的ConnectionLoss

知道为什么我无法通过Hive Map/Reduce来维护与zookeeper的连接吗？

hbase和zookeeper的配置是:

# Autogenerated by Cloudera SCM on Wed Dec 28 08:42:23 CST 2011
tickTime=2000
initLimit=10
syncLimit=5
dataDir=/var/zookeeper
clientPort=2181
maxClientCnxns=1000
minSessionTimeout=4000
maxSessionTimeout=40000

HBase Site-XML是:

<property>
  <name>hbase.rootdir</name>
  <value>hdfs://alnnimb01:8020/hbase</value>
  <description>The directory shared by region servers. Should be fully-qualified to include the filesystem to use. E.g: hdfs://NAMENODE_SERVER:PORT/HBASE_ROOTDIR</description>
</property>

<property>
  <name>hbase.master.port</name>
  <value>60000</value>
  <description>The port master should bind to.</description>
</property>

<property>
  <name>hbase.cluster.distributed</name>
  <value>true</value>
  <description>The mode the cluster will be in. Possible values are false: standalone and pseudo-distributed setups with managed Zookeeper true: fully-distributed with unmanaged Zookeeper Quorum (see hbase-env.sh)</description>
</property>


<property>
  <name>hbase.master.info.port</name>
  <value>60010</value>
  <description>The port for the hbase master web UI Set to -1 if you do not want the info server to run.</description>
</property>
















<property>
  <name>zookeeper.znode.parent</name>
  <value>/hbase</value>
  <description>Root ZNode for HBase in ZooKeeper. All of HBase's ZooKeeper files that are configured with a relative path will go under this node. By default, all of HBase's ZooKeeper file path are configured with a relative path, so they will all go under this directory unless changed.</description>
</property>

<property>
  <name>zookeeper.znode.rootserver</name>
  <value>root-region-server</value>
  <description>Path to ZNode holding root region location. This is written by the master and read by clients and region servers. If a relative path is given, the parent folder will be ${zookeeper.znode.parent}. By default, this means the root location is stored at /hbase/root-region-server.</description>
</property>


  <property>
    <name>hbase.zookeeper.property.clientPort</name>
    <value>2181</value>
    <description>The ZooKeeper client port to which HBase clients will     connect</description>
  </property>

  <property>
<name>hbase.zookeeper.quorum</name>
<value>alnnimb01.aln.experian.com</value>
<description>Comma separated list of servers in the ZooKeeper Quorum. For example, "host1.mydomain.com,host2.mydomain.com,host3.mydomain.com".</description>

Answer 1

事实证明，Hive 提交的 Map/Reduce 正在尝试连接到“localhost”的 Zookeeper，无论配置文件中的 Zookeeper.quorom 是如何设置的。我更改了 /etc/hosts，使别名“localhost”指向我的主节点的 IP，并保持与 Zookeeper 的连接。仍在寻找更好的解决方案，但这目前可行。