从UDF查找Hbase Tbl(Beeline,Hbase,Delegation Tokens)

Question

我需要编写自定义UDF来从Hbase表进行数据查找.

注意:我已经用HIVE进行了单元测试.它似乎工作.

但是当我使用相同的UDF Beeline时,它失败了.默认情况下,Cloudera限制模拟并仅允许hive用户在Beeline中运行查询.在作业启动时,YarnChild正在设置以下委托令牌.

我想添加令牌(Kind:HBASE_AUTH_TOKEN)来处理Hbase.

Kind: mapreduce.job
Kind: HDFS_DELEGATION_TOKEN 
Kind: kms-dt

我研究并发现了HbaseStorageHandler如何使用委托令牌(即HBASE_AUTH_TOKEN)用于Hbase.所以我使用了相同的功能集,但它也没有用.

来自HbasestorageHandler的函数(获取令牌到Job):

private void addHBaseDelegationToken(Configuration conf, JobConf jconf) throws IOException {
        if (User.isHBaseSecurityEnabled(conf)) {
            try {
                logger.info("isHbaseSecurityEnabled :True ");
                User e = User.getCurrent();
                logger.info("isHbaseSecurityEnabled :User ==> " + e.toString());
                Token authToken = getAuthToken(conf, e);
                logger.info("isHbaseSecurityEnabled :AuthToken==> "+authToken.toString());
                Job job = new Job(conf);
                if(authToken == null) {
                    UserGroupInformation ugi = UserGroupInformation.getLoginUser();
                    ugi.setAuthenticationMethod(UserGroupInformation.AuthenticationMethod.KERBEROS);
                    e.obtainAuthTokenForJob(jconf);
  } else {
                    logger.info("authToken is not null"+authToken.toString());
                    job.getCredentials().addToken(authToken.getService(), authToken);
                }

                logger.info("obtained Token /....");
            } catch (InterruptedException var5) {
                throw new IOException("Error while obtaining hbase delegation token", var5);
            }
        }

    }


private static Token<AuthenticationTokenIdentifier> getAuthToken(Configuration conf, User user) throws IOException, InterruptedException {
        ZooKeeperWatcher zkw = new ZooKeeperWatcher(conf, "mr-init-credentials", (Abortable) null);

        Token var4;
        try {
            String e = ZKClusterId.readClusterIdZNode(zkw);
            logger.info("====== clusterID : " + e);
            var4 = (new AuthenticationTokenSelector()).selectToken(new Text(e), user.getUGI().getTokens());
             if (var4 == null) {
                logger.info("var4 is null===========================");
            } else {
                logger.info("====== Hbase Token : " + var4.toString());
            }
        } catch (KeeperException var8) {
            throw new IOException(var8);
        } catch (NullPointerException np) {
            return null;
        } finally {
            zkw.close();
        }
        return var4;
 }

在UDF的configure()中调用addHBaseDelegationToken()之后.我得到以下异常.我不知道如何让hvie用户与Hbase交谈,因为hive.keytab由Cloudera及其安全处理.

任何输入可能会有所帮助.谢谢 !

异常StackTrace:

2018-10-11 04:48:07,625 WARN [main] org.apache.hadoop.security.UserGroupInformation:PriviledgedActionException as:hive(auth:SIMPLE)原因:javax.security.sasl.SaslException:GSS启动失败[引起GSSException :没有提供有效凭据(机制级别:找不到任何Kerberos tgt)] 2018-10-11 04:48:07,627 WARN [main] org.apache.hadoop.hbase.ipc.RpcClientImpl:连接到服务器时遇到异常:javax.security.sasl.SaslException:GSS启动失败[由GSSException引起:未提供有效凭据(机制级别:无法找到任何Kerberos tgt)] 2018-10-11 04:48:07,628 FATAL [main] org.apache .hadoop.hbase.ipc.RpcClientImpl:SASL身份验证失败.最可能的原因是凭据丢失或无效.考虑'kinit'.javax.security.sasl.SaslException:GSS启动失败[由GSSException引起:没有提供有效凭据(机制级别:无法找到任何Kerberos tgt)]在com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java) :211)在org.apache.hadoop.hbase.security.HBaseSaslRpcClient.saslConnect(HBaseSaslRpcClient.java:181)在org.apache.hadoop.hbase.ipc.RpcClientImpl $ Connection.setupSaslConnection(RpcClientImpl.java:618)的组织.位于org.apache的org.apache.hadoop.hbase.ipc.RpcClientImpl $ Connection $ 2.run(RpcClientImpl.java:744)的apache.hadoop.hbase.ipc.RpcClientImpl $ Connection.access $ 700(RpcClientImpl.java:163). hadoop.hbase.ipc.RpcClientImpl $ Connection $ 2.run(RpcClientImpl.java:741)位于org的javax.security.auth.Subject.doAs(Subject.java:422)的java.security.AccessController.doPrivileged(Native Method) .apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)at org.apache.hadoop.hbase.ipc.RpcClientImpl $ Connection.setupIOstreams(RpcClientImpl.j)AVA:741)在org.apache.hadoop.hbase.ipc.RpcClientImpl $ Connection.writeRequest(RpcClientImpl.java:907)在org.apache.hadoop.hbase.ipc.RpcClientImpl $ Connection.tracedWriteRequest(RpcClientImpl.java:874)在org.apache.hadoop.hbase.ipc.RpcClientImpl.call(RpcClientImpl.java:1246)在org.apache.hadoop.hbase.ipc.AbstractRpcClient.callBlockingMethod(AbstractRpcClient.java:227)在org.apache.hadoop.hbase .ipc.AbstractRpcClient $ BlockingRpcChannelImplementation.callBlockingMethod(AbstractRpcClient.java:336)在org.apache.hadoop.hbase.protobuf.generated.ClientProtos $ ClientService $ BlockingStub.execService(ClientProtos.java:34118)在org.apache.hadoop.hbase .protobuf.ProtobufUtil.execService(ProtobufUtil.java:1633)在org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel $ 1.call(RegionCoprocessorRpcChannel.java:104)在org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel $ 1.call (RegionCoprocessorRpcChannel.java:94)org.apache.hadoop.hbase.client.RpcRetryingCaller.callWithRetries(RpcRetryingCalle)r.java:136)在org.apache.hadoop.hbase.ipc.RegionCoprocessorRpcChannel.callExecService(RegionCoprocessorRpcChannel.java:107)在org.apache.hadoop.hbase.ipc.CoprocessorRpcChannel.callBlockingMethod(CoprocessorRpcChannel.java:73)在组织.apache.hadoop.hbase.protobuf.generated.AuthenticationProtos $ $的AuthenticationService BlockingStub.getAuthenticationToken(AuthenticationProtos.java:4512)在org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:86)在组织.apache.hadoop.hbase.security.token.TokenUtil $ 1.run(TokenUtil.java:111)位于java.security的org.apache.hadoop.hbase.security.token.TokenUtil $ 1.run(TokenUtil.java:108)位于org.apache.hadoop的org.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)的javax.security.auth.Subject.doAs(Subject.java:422)上的.AccessController.doPrivileged(Native Method) .hbase.security.User $ SecureHadoopUser.runAs(User.java:340)位于com.ba的org.apache.hadoop.hbase.security.token.TokenUtil.obtainToken(TokenUtil.java:108)rclaycardus.hadoop.utils.udfs.HbaseTblLookupUDF.configure(HbaseTblLookupUDF.java:131)位于org.apache.hadoop.hive的org.apache.hadoop.hive.ql.exec.MapredContext.setup(MapredContext.java:120). ql.exec.ExprNodeGenericFuncEvaluator.initialize(ExprNodeGenericFuncEvaluator.java:143)在org.apache.hadoop.hive.ql.exec.Operator.initEvaluators(Operator.java:954)在org.apache.hadoop.hive.ql.exec. Operator.initEvaluatorsAndReturnStruct(Operator.java:980)org.apache.hadoop.hive.ql.exec.SelectOperator.initializeOp(SelectOperator.java:63)at org.apache.hadoop.hive.ql.exec.Operator.initialize( Operator.java:385)org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:469)org.apache.hadoop.hive.ql.exec.Operator.initializeChildren(Operator.java: 425)org.apache.hadoop.hive.ql.exec.TableScanOperator.initializeOp(TableScanOperator.java:196)org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)at org .apache.hadoop.hive.ql.exec.MapOperator.initializeOp(MapOperator.java :431)org.apache.hadoop.hive.ql.exec.Operator.initialize(Operator.java:385)org.apache.hadoop.hive.ql.exec.mr.ExecMapper.configure(ExecMapper.java:126) )在sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法)在sun.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)在sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)在java.lang.reflect中.在org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)的org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)org中的Method.invoke(Method.java:498) .apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)在org.apache.hadoop.mapred.MapRunner.configure(MapRunner.java:38)在sun.reflect.NativeMethodAccessorImpl.invoke0(本机方法)在阳光下.reflect.NativeMethodAccessorImpl.invoke(NativeMethodAccessorImpl.java:62)at sun.reflect.DelegatingMethodAccessorImpl.invoke(DelegatingMethodAccessorImpl.java:43)at java.lang.reflect.Me thod.invoke(Method.java:498)org.apache.hadoop.util.ReflectionUtils.setJobConf(ReflectionUtils.java:106)org.apache.hadoop.util.ReflectionUtils.setConf(ReflectionUtils.java:75)at org位于org.apache.hadoop.mapred.MapTask.run的org.apache.hadoop.mapred.MapTask.runOldMapper(MapTask.java:455)中的.apache.hadoop.util.ReflectionUtils.newInstance(ReflectionUtils.java:133)(MapTask .java:343)atg.apache.hadoop.mapred.YarnChild $ 2.run(YarnChild.java:164)at java.security.AccessController.doPrivileged(Native Method)at javax.security.auth.Subject.doAs(Subject. java:422)atg.apache.hadoop.security.UserGroupInformation.doAs(UserGroupInformation.java:1920)org.apache.hadoop.mapred.YarnChild.main(YarnChild.java:158)引起:GSSException:无有效凭据提供(机制级别:无法找到任何的Kerberos TGT)在sun.security.jgss.krb5.Krb5InitCredential.getInstance(Krb5InitCredential.java:147)在sun.security.jgss.krb5.Krb5MechFactory.getCredentialElement(Krb5MechFactory.java:122)在sun.security.jgss.krb5.Krb5MechFactory.getMechanismContext(Krb5MechFactory.java:187)在sun.security.jgss.GSSManagerImpl.getMechanismContext(GSSManagerImpl.java:224)在sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java :212)at sun.security.jgss.GSSContextImpl.initSecContext(GSSContextImpl.java:179)at com.sun.security.sasl.gsskerb.GssKrb5Client.evaluateChallenge(GssKrb5Client.java:192)... 66 more

已尝试以下选项:

https://github.com/apache/oozie/blob/master/core/src/main/java/org/apache/oozie/action/hadoop/HbaseCredentials.java

https://github.com/ibm-research-ireland/sparkoscope/blob/master/yarn/src/main/scala/org/apache/spark/deploy/yarn/security/HBaseCredentialProvider.scala