Hadoop IOException登录失败

Question

我对Hadoop很新.但是,我已经能够在我的服务器上以群集模式成功设置带有Java 7的hadoop 2.7.3.一切都很好.

但是,当我尝试切换到Java 8并启动dfs时,出现错误:

Exception in thread "main" java.io.IOException: failure to login
    at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:824)
    at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634)
    at org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:315)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at org.apache.hadoop.hdfs.tools.GetConf.main(GetConf.java:332)
Caused by: javax.security.auth.login.LoginException: java.lang.NullPointerException: invalid null input: name
    at com.sun.security.auth.UnixPrincipal.<init>(UnixPrincipal.java:71)
    at com.sun.security.auth.module.UnixLoginModule.login(UnixLoginModule.java:133)
    at sun.reflect.NativeMethodAccessorImpl.invoke0(Native Method)
    at sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)
    at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)
    at java.lang.reflect.Method.invoke(Method.java:498)
    at javax.security.auth.login.LoginContext.invoke(LoginContext.java:755)
    at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
    at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
    at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:799)
    at org.apache.hadoop.security.UserGroupInformation.getLoginUser(UserGroupInformation.java:761)
    at org.apache.hadoop.security.UserGroupInformation.getCurrentUser(UserGroupInformation.java:634)
    at org.apache.hadoop.hdfs.tools.GetConf.run(GetConf.java:315)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:70)
    at org.apache.hadoop.util.ToolRunner.run(ToolRunner.java:84)
    at org.apache.hadoop.hdfs.tools.GetConf.main(GetConf.java:332)

    at javax.security.auth.login.LoginContext.invoke(LoginContext.java:856)
    at javax.security.auth.login.LoginContext.access$000(LoginContext.java:195)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:682)
    at javax.security.auth.login.LoginContext$4.run(LoginContext.java:680)
    at java.security.AccessController.doPrivileged(Native Method)
    at javax.security.auth.login.LoginContext.invokePriv(LoginContext.java:680)
    at javax.security.auth.login.LoginContext.login(LoginContext.java:587)
    at org.apache.hadoop.security.UserGroupInformation.loginUserFromSubject(UserGroupInformation.java:799)
    ... 6 more

似乎hadoop碰巧无法使用我的工具用户名登录.我尝试从其他来源查找并遵循他们的指导,例如添加环境变量HADOOP_USER_NAME,删除和重置整个datanode和namenode等等.但是,在这种情况下,它们都不适用于我.

关于这个问题的奇怪之处在于,当我尝试切换回Java 7. Hadoop像以前一样适合我.但是,我不认为应该存在问题,因为Java版本之间存在差异,因为Hadoop基于其文档与它们兼容.此外,已在群集中设置受信任的ssh.

下面是我的.bashrc配置fyi:

export JAVA_HOME="/scratch/dsat_server/jdk1.8.0_121"
export HADOOP_PREFIX="/scratch/dsat_server/hadoop-2.7.3"
export HADOOP_COMMON_LIB_NATIVE_DIR="$HADOOP_PREFIX/lib/native"
export HADOOP_CONF_DIR="$HADOOP_PREFIX/conf"
export HADOOP_OPTS="-Djava.net.preferIPv4Stack=truei-Djava.library.path=$HADOOP_PREFIX/lib"

Answer 1

我从使用Java 8的docker容器运行hbase客户端时遇到了同样的问题.这显然是由类com.sun.security.auth.module.UnixLoginModule引起的,该类使用本机调用来获取unix用户名.在我的例子中,它没有映射到docker中,并且该类抛出NullPointerException.这不是hadoop本身的错误.

为了指示hadoop绕过OS用户名的查找,我能够在所有初始化之前添加以下代码行:

UserGroupInformation.setLoginUser(UserGroupInformation.createRemoteUser("hduser"));

在您的情况下,您正在运行服务器,因此您注入代码的选项是有限的.相反,有两种选择:

1. 请尝试使用IBM JDK
2. 尝试调试worker上的OS用户设置($ whoami).如果它显示"无法找到用户ID XXXX的名称",请检查/ etc/passwd设置